From Parameters to Behaviors: Unsupervised Compression of the Policy Space
Authors: Davide Tenedini
Riccardo Zamboni
Mirco Mutti
Marcello Restelli
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
unsupervised reinforcement learning
unsupervised representation learning
Primary area: reinforcement learning

Abstract: Despite its recent successes, Deep Reinforcement Learning (DRL) is notoriously sample-inefficient. We argue that this inefficiency stems from the standard practice of optimizing policies directly in the high-dimensional and highly redundant parameter space $\\Theta$. This challenge is greatly compounded in multi-task settings. In this work, we develop a novel, unsupervised approach that compresses the policy parameter space $\\Theta$ into a low-dimensional latent space $\\mathcal Z$. We train a generative model $g:\\mathcal Z\\to\\Theta$ by optimizing a behavioral reconstruction loss, which ensures that the latent space is organized by functional similarity rather than proximity in parameterization. We conjecture that the inherent dimensionality of this manifold is a function of the environment's complexity, rather than the size of the policy network. We validate our approach in continuous control domains, showing that the parameterization of standard policy networks can be compressed up to five orders of magnitude while retaining most of its expressivity. As a byproduct, we show that the learned manifold enables task-specific adaptation via Policy Gradient operating in the latent space $\\mathcal{Z}$.

Published as a conference paper at ICLR 2026
FROM PARAMETERS TO BEHAVIORS: UNSUPERVISED COMPRESSION OF THE POLICY SPACE
Davide Tenedini
Riccardo Zamboni & Mirco Mutti Marcello Restelli
Politecnico di Milano
Technion
(davide.tteedimng tenedini, riccardo. zamboni, res.eelll..ettlllleee........... mirco. mrrco.m@techhion.ace
ABSTRACT
Despite its recent successes, Deep Reinforcement Learning (DRL) is notoriously sample-ineffiiient. We argue that this inefficiency stems from the standard practice of optimizing policies directly in the high-dimensional and highly redundant pa- rameter space θ This challenge is greatly compounded in multi-task settings. In this work, we develop a novel, unsupervised approach that compresses the policy parameter space & into a low-dimensional latent space Z. We train a generative model g 2 - θ by optimizing a behavioral reconstruction loss, which ensures that the latent space organized by functional similarity rather than proximity parameterization. We conjecture that the inherent dimensionality of this manifold is a function of the environment's complexity, rather than the size the policy network. We validate our approach in continuous control domains, showing that the parameterization orders of magnitude while retaining most of its expressivity. As byproduct, we show that the learned manifold enables task-specific adaptation via Policy Gradient operating in the latent space 2.
standard policy networks can be compressed up to five
INTRODUCTION
High-dimensional parameterization of policies via deep neural networks has been a key driver of recent successes in Deep Reinforcement Learning (among others, Andrychowicz et al., 2020; Smith et al., 2022; Bakhtin et al., 2022; Wurman et al., 2022: Degrave et al., 2022). A major drawback of this approach, however, a significant increase in sample complexity, which further compounded when the agent is called to solve multiple and potentially unknown tasks, typically requiring learning tabula rasa (Agarwal et al., 2022). This inefficiency often stems from a fundamental redundancy in the parameter space, where a large set of distinct weight configurations maps to much smaller set effective behaviors. Various approaches tried to solve this limitation as a byproduct. such explicitly learning diverse behaviors (Eysenbach et al., 2018; Zahavy et al., 2023; De Paola et al., 2025; Zamboni et al., 2025), or enforcing small policy networks in asymmetric actor-critic architectures (Degrave al., 2022; Mastikhina et al., 2025).
In this paper, we address this limitation directly through the lens of the Manifold Hypothesis (Cayton al., 2005), widely accepted tenet of Machine Learning, and we hypothesize that it holds in Reinforcement Learning as well, namely that:
The manifold of realizable behaviors is intrinsically low-dimensional and largely independent of the network's 's parameler count.
In view of this hypothesis, propose paradigm shift from learning in the parameter space to learning in the (latent) behavior space itself. To do so, the agent first needs to learn a latent repre- sentation of the possible behaviors, which, according to the aforementioned hypothesis, should be dow-diimnsl and policy network invariant. Then, it needs to find a way to leverage this repre- sentation to solve different tasks efficiently. The proposed solution is novel two-stage framework

---

Published as a conference paper at ICLR 2026
directly inspired by the Unsupervised RL formalism (Laskin et al., 2021). allowing for the explicir exploitation of this latent structure. In first pre pre-training phase, we learn a latent representation of the behavior manifold by leveraging generative model in a fully unsupervised fashion, that is, without including any information related to specific task, i.e., reward. In this way, we can learn a latent structure that models the intrinsic nature of the environment dynamics, rather than its coupling with task, and preserve the end-to-end differentiability that makes gradient-based optimization effective. In a second fine-tuning phase, we leverage the pre-trained representation to fine-tune policies on specific tasks known posteriori, avoiding the need to learn from scratch. In particular, the fine-tuning phase involves performing gradient steps in the latent space, thereby optimizing latent behaviors directly. This approach enables the agent to explore the inherently low-dimensional behavior space rather than the high-dimensional parameter space.
In this paper, we address the following:
Research Questions:
(Q1) Is it possible to learn a oow-dimensional latent representation of high-dimensional policy parameter space in an unsupervised fashion?
(Q2) What are the properties of such a latent representation, if exists? Is its intrinsic dimension function of the behavioral complexity rather than the size of the parameter
space?
(Q3) How can we fine-tune on specific tasks leveraging the low-dimensional space? Does this offer any advantages?
Content Outline and Contributions. First, in Section 3, we formulate the problem of learning latent representation of behaviors in an unsupervised fashion and then leveraging it to solve specific tasks. Then, in Section 4, we characterize our proposed solution to this problem, namely, addressing it in a two-stage pipeline. Finally, in Section 5, we perform experiments extensively to address the Research Questions. We demonstrate that the proposed pipeline is indeed able to learn low- dimensional latent representations (Q1), which are more influenced by the environment than by the size of the compressed policies (Q2). Finally, we demonstrate that learning over this reduced space can make simple algorithms competitive against complex state-of-the-arr DRI. algorithms (Q3).
PRELIMINARIES
Notation. In the following, we denote a set with a calligraphic letter A and its size as [A]. For a finite set, ((A) denotes the probability simplex. For continuous spaces, denote the set of Borel probability measures as P(A). When the context allows, we use A(A) to refer generally to the space of valid distributions over A. For two distributions PI; E (AA), we define a general measure of divergence between distributions with D(p || P2).
Interaction Protocol. As base model for interaction, we consider a (finite-horizon) Controlled Markoy Process (CMP). A CMP is defined as the tuple . = (S,A, P, u, T'), where S is the state space and A is the action space. At the start of an episode, the initial state so of M is drawn from an initial state distribution μ E A(S). Upon observing so, the agent takes action au E A, and the system transitions to $1 P(- Su, ag) according to the transition model P : S x + (SS). The process is repeated until T' is reached and sT is generated, with T < oo being the horizon
of an episode. The agent selects actions according to decision policy क : S - A(AA such that r(a s) denotes the conditional probability of taking action upon observing state Deploying policy т over M leads to the generation of trajectories T, defined as a sequence of state-action pairs ;= (so. ap, $1, a1,................. ... ST). Furthermore, a policy induces a state distribution di A(S) over the state space of the CMP M defined as (s) T EI. Pr(st s). also induces a state- action distribution d E ASS x A), defined as d= a) = T(a 8) (s), which we will denote as the behavior of the policy. In the following, we will consider deterministic policies no S + A represented by neural networks parameterized by a set of weights 0 E e, where e C R is the policy parameter space, with P being the total number of parameters. We define the Policy Space Ie as the collection of policies that can be represented by e. For brevity of notation, we denote a policy To as its set of parameters O and the policy space Ih as the parameter space O that induces it.
P
2

---

Published as a conference paper at ICLR 2026
(Unsupervised) RL. In RL, agent learns how to solve (downstream) tasks, encoded by different reward signals.To this end, we define a Markov Decision Process (MDP, Puterman, 2014) MR := M U R as a coupling of CMP M and a reward function R: : S x A - R. which the agent observes after every state transition. In the Unsupervised Reinforcement Learning (URI., Laskin et al., 2021) framework, the reward signal is not always available the agent from the beginning. It often belongs to (potentially infinite) family of tasks R. also unknown to the agent. URL is then composed of two phases: (1) an unsupervised pre-training phase involves the agent interacting with CMP to acquire purpose knowledge without receiving any reward signal, which is distilled into a pre-traie trained model M: (2) the supervised fine-tuning phase begins once a reward function R E is revealed. At this point, the CMP becomes a standard MDP M. and the agent leverages the pre-trained model M to find a set of policy parameters that maximizes the expected return for the given task, namely as:
0
arg max J (0, M)) arg max B(o,a))d: M [R (s. a)].
(1)
gge
9
Policy Optimization. Policy Optimization (PO, Deisenroth et al., 2013), which involves optimizing the policy parameters directly, has shown surprisingly good results. This is especially true for deep neural policies, where first-order methods have been extensively employed. A popular approach to PO is Policy Gradient (PG, Peters & Schaal, 2008), which updates the parameters by simple gradient ascent 0' + + (O). Among others, Policy Gradient with Parameter-based Exploration (PGPE, Sehnke et al., 2008) is PG algorithm that handles exploration in the parameter space by sampling the policy parameters 9 from a hyper-policy Vs- parameterized by 6. PGPE optimizes trajectory- -hased version of the objective defined in Eq. defined
JR M) Ey-ppt 0~,0~ MM [R(T))
(2)
where R(()) =
R(T)== ET R(st, 0.) is the return of trajectory, and p(T 0) - μ($a) IT P(S
St, ax) (0 $t) is the probability density of a trajectory. In PGPE, the parameter vector & is usually updated via gradient ascent using Monte Carlo estimator of the gradient computed over N E N trajectories:
N Σ
JR = N
V Loguo (O))
(3)
Generative Models. Generative models have achieved remarkable success in density estimation for multi-modal data, drawing significant interest from the RL community. Among others, Autoen- coders (AE, Hinton & Salakhutdinov, 2006) are type of artificial neural network used to learn efficient data encoding in an unsupervised manner. The aim is first to learn encoded representations of data and then generate the input data (as closely as possible) from the learned encoded represen- tations. More specifically, their goal is to map a data space X' C R" to a latent space Z C RR with & < n. AEs are composed of an encoder, a function fE X + Z, parameterized by vector which maps a data sample x E X' to a latent code z E 2, and a decoder, a function 9ç- : Z + X, parameterized by vector Ç, which reconstructs the data sample & E X' from the latent code z in such a way that 9c ~ fe An AE is typically trained by minimizing the reconstruction error (aE(œ)= = d(x, (gc fe (z)), where d is a metric that measures the distance of samples in the data space. These architectures are particularly compelling light of the Manifold Hypothesis (Cayton et al., 2005): AEs learn the underlying structure by compressing the data into compact latent space that represents the manifold and then reconstructing the original data from it, as illustrated in Fig. 1. Unfortunately, AEs are far from being bullet- proof. In cases where no plausible embedding exists, even networks (fg- 9c) which come close to perfectly reconstructing the manifold A will incur numerical instability (Cornish et al., 2020). In some other cases, is possible to resolve these topological issues by increasing the latent dimension k. For instance, a dimensionality of
Autoencoder
DATA SPACE
LATENT SPACE
encoder
x=q(f(x))
g g=if-1
M
decodde
k 2d* + 1 is enough to topologically embed any manifold of dimension d' in Rà (Theorem
v3, Hurewicz & Wallman, 2015).
Figure 1: Autoencoder Spaces and Data Manifold.
For instance, Gaussian hyper-policies will be parameterized by their mean and standard deviation, i.c. (1,, T).
3

---

Published as a conference paper at ICLR 2026
Unsupervised
Supervised
Latent Behavior Compression
Latent Behavior Optimization
Policy Dataset
Generation
Figure 2: Pipeline of Unsupervised Compression of the Policy Space.
PROBLEM FORMULATION
By looking closely to Eq. I, one should notice that to solve an RL task, the agent just needs to focus on visiting the states and actions that matter for the task. Yet, this simple intuition hides few traps. First of all, different policy parameters 0 E θ might induce nearly identical distributions over actions. Yet, even different distributions over actions could lead comparable state-action distributions due to the complex structure of the environment. Finally, in almost all problems of interest, there may be multiple and potentially unknown tasks that the agent could be called upon to solve, and would be risky to deem any state-action distribution irrelevant without additional information on the task structure.
In this work, we aim to address these issues by focusing on behaviors rather than parameters, under the lens of the Manifold Hypothesis: we want to learn a latent manifold of realizable behaviors, and we do this by compressing parameters inducing similar behaviors to the same latent representation. For policy parameters space θ C R", we define Z S R as a k-dimensional latent space, with k < and we look for function Z - & that maps latent vector E Z, which also refer to as latent code, to a corresponding policy parameter vector 0 (()). As a result, any policy could be written Te - g(*) Zz
We refer to this problem as Latent Behavior Compression, which formally defined as finding the generative function 9* 2 -> O, such that:
VO θ, 3z E Z: g" arg min D (dea || d=
(4)
This task is essentially unsupervised, as any notion of specific task is absent. Indeed, it is somewhat similar to the Policy Space Compression framework (Mutti et al., 2022), in the latter, the authors aim to reduce the cardinality of the policy space, rather than its dimensionality. Moreover, the constraints defining valid compression are stricter than ours, resulting in optimization problem that is known to be NP-hard.
Once such low-dimensional space and generative function available, solving for different tasks will require searching over a simpler space than the original one. We call this process Latent Behavior Optimization. In other words, the standard PO problem of Eq. which requires finding an optimal policy parameter vector 0* E θ, will be reformulated as the problem of finding an optimal latent code that, via the generative function g. yields 0", For given task with reward R R. the policy optimization problem is now defined as:
z' arg max JR = arg max [ (0=00900 (z)) ZEZ
2E2
Contrary to the Latent Behavior Compression task, this task is essentially supervised, it is well- defined as soon as the agent is provided with a reward. the following, we will show how the URL framework indeed provide essential tools in addressing the problems.
4

---

Published as a conference paper at ICLR 2026
4 METHOD: UNSUPERVISED COMPRESSION OF THE POLICY SPACE
To address the sample inefficiency inherent in sighddimensional policy parameter spaces, we propose a paradigm shift from directly optimizing in the parameter space to learning within a compact, low- dimensional policy manifold that captures the true diversity of behaviors. This is achieved through a two-phase framework: completely unsupervised, task-agnostic pre-training phase to discover the manifold, followed by supervised, task-specific fine-tuning phase. As illustrated in Fig. 2, this framework is composed of three steps: (1) generating behaviorally diverse dataset of policies, (2) learning the latent policy manifold via generative model, and (3) performing fine-tuning by optimizing over this learned latent space.
Policy Dataset Generation. Many manifold reconstruction algorithms depend on efficiently covering the manifold with samples (Bernstein et al., 2000; Cheng et al., 2005; Fefferman et al., 2016). Thus, the first stage of our framework involves generating dataset De of policies intended to cover the manifold of behaviorally diverse policies.
A naïve option is to randomly sample N policies, De {0: J by drawing their parameters from a uniform distribution. Unfortunately, it is well-understood that such naïve sampling from the parameter space is unlikely to produce uniform coverage of the behavior space, as it tends to favor functionally similar, often non-expinn non-exploring, policies.
To address this bias, explicit measure of behavioral diversity is needed. Looking at Eq. 4, one notices that optimizing such a measure directly requires estimating the divergence between two state-action distributions dig d Unfortunately, this would not only be computationally intensive but also require sampling from the environment for potentially vast set of policies. To avoid this, we will take into account an upper bound to this quantity in the case finite-horizon tasks (Prop. E.1, Metelli et al., 2018), namely D(m Ter). ). In practice, we substitute this measure with the L2 distance of two policies in the action space, evaluated on a finite subset of the state space, or formally:
M
D(To ) =
Σ (To (. (1) ) Te (. ss))
Based on this proxy, we apply Novelty Search algorithm (Lehman & Stanley, 2011) by computing novelty score, p(Te). for each policy based its average divergence from its k, nearest neighbors: p(TTe) k. - D(To m.,)-
Then, a high score indicates a behaviorally unique policy. Using this metric, we form the final dataset De by selecting only the top percentile of policies with the highest novelty scores, ensuring a dataset of behaviorally diverse policies.
Latent Behavior Compression. In the second stage, we learn the low-dimensional manifold from the filtered policy dataset De. Potentially, any generative model would do the work. Still, here we are interested in learning latent low- now-dimensional representations while preserving the end-to-end differentiability that makes gradient-based optimization effective. For these reasons, we employ symmetric autoencoder architecture with encoder Se θ - Z and a decoder dC 2 + O. While a standard autoencoder minimizes a parameter reconstruction error, our goal is to compress policy behavior. We therefore introduce novel Behavioral Reconstruction Loss, which trains the autoencoder to minimize the expected behavioral divergence between the original policy and its reconstruction:
CB(E. C) = Ee-Ds (TT
[()c))))))))
This objective frees the decoder from reproducing the exact parameter values, allowing it to discover any parameterization that generates the desired behavior. As a result, the latent space Z becomes organized purely by functional similarity, effectively capturing the policy manifold. In practice, we use an empirical estimator of the behavioral reconstruction loss based on the notion of divergence in the action space. For this purpose, we train our autoencoders to minimize the Mean Squared Error between action vectors relative to subset of the state space sampled at each gradient step, resulting in the estimator Ln(E, () - NM Σ Σ5= =| (5z) (o.)) where N is the number of policies, and M is the number of sampled states.
5

---

Published as a conference paper at ICLR 2026
Latent Behavior Optimization. In the final stage, we leverage the learned latent manifold for rapid, task-specific fine-tuning. With the decoder parameters ( frozen, 9ç. becomes a deterministic and differentiable function that generates policies from latent codes. This structure allows to adapt wide range of PG methods to operate in the latent space. By applying the chain rule, the standard policy gradient can be back-propagated through the frozen decoder to update the latent code
V (z) V VoJ"
(5)
where Vo) is the conventional policy gradient and V - (z) is the Jacobian of the decoder. This provides general recipe for adapting popular PG algorithms is particularly advantageous for purameter-exploring PG methods, like PGPE, which notoriously struggle with high-dimensional parameter spaces. By operating on the low-dimensional latent space. these algorithms regain their effectiveness while still controlling the expressive power of the original large network.?
framework. This approach
Remarks. In this section, we proposed three specific instantiations for each phase. Yet. we emphasize that the proposed pipeline represents the most relevant contribution per se, independently of how is realized, i.e., how the policies are collected, which divergence measure is used, which generative model or PO algorithm over the latent space is employed.
5 EXPERIMENTS
We now investigate through extensive empirical corroboration how the proposed method addresses the research questions. In order to do so, we will mainly focus on the unsupervised pre-training phase of the proposed pipeline seen Fig. 2, in which a latent representation is built out of general datasets of policies not designed to address any specific task explicitly, and we report the empirical results in Subsec. 5.1. Finally, we make sure that such a latent space can indeed be leveraged in later supervised fine-tuning phases soon task is provided, and report the results in Subsec. 5.2. A detailed description of the environments and experimental settings can be found in Appendix B.
Experimental Domains. The experiments are performed to illustrate essential features of Latent Behavior Compression, and for this reason, the domains are selected for being challenging while keeping high interpretability. The first is the Mountain Car Continuous (MC, Moore, 1990) environment. To evaluate the quality and characteristics of the latent space, we define four downstream tasks: standard and left have the goal state on the right and left hill, respectively; speed and height incentivize the car to keep a high speed and vertical coordinate, respectively, without terminating the episode. We also consider three environments from the MuJoCo suite (Todorov et al., 2012; Tassa et al., 2018). For Reacher (RC), we define four downstream tasks: speed, which incentivizes the fingertip to move with high linear velocity; clockwise and c-clockwise reward the agent for each step the fingertip is rotating clockwise and counterclockwise, respectively; and radial, which promotes the retraction and extension of the arm. For Hopper (HP), we define four downstream tasks: forward, backward, and standstill reward the agent for positive, negative, or dlose-to-zers velocity along the r-axis respectively; jump rewards the agent for achieving certain position along the z-axis. Finally, for HalfCheetah (HC), we define four downstream tasks: forward and backward are defined as for HP: frontflis ip and backflip reward the agent each time it performs frontflip and backflip, respectively.
Experimental Regimes. The experiments are performed over set of different parameters. In MC, we took into account three Policy Sizes (Small, Medium, and Large) with roughly 10 10ss and 10a parameters respectively, three Policy Dataset Sizes (10k, 50k, and 100k generated policies, with a 10% novelty- based cut-down), and three Latent Space Sizes (ID, 2D, and 3D), In RC, we focused on specific configuration with Medium policies, Policy Dataset size of 100k, but five possible latent space sizes (ID, 2D, 3D, 5D, and 8D). In HP and HC, we focused on Policy Datasets of 10k, using Medium policies, with three latent space sizes (5D, 8D, and 16D))²
2 Additionally, running PGPE over the latent space does not actually require computing the Jacobian of the decoder V=9c- (z), as explained Appendix B. Notably, the AE archtecctree and training papar-parameters are left the same for every experiment, regardless of configuration or environment.

---

Published as a conference paper at ICLR 2026
(a) height, Small, ID, MC
(b) standard, Medium, ID, MC
(c) height, Large, ID, MC
(d) speed, Medium, ID, RC
(e) standard, Small,
(f) standard, Medium,
(g) standard, Large, 2D, MC
(h) speed, Medium,
2D, MC
MC
(i) speed, Small, 3D, MC
() standard, Medium, 3D. MC
(k) speed, Large, 3D. MC
speed, Medium, 3D, RC
Figure 3: Landscape of the Latent Behavior Manifold. Lighter and darker colors indicate higher and lower returns of the decoded policy. The plots shown here represent subset of the full results reported in Appendix B. We consider a specific seed with different tasks (height, standard, speed), policy size (Small, Medium, Large), and encoding dimension (ID, 2D, 3D), for both MC (first three columns, datasets of 50k policies) and RC (last column, datasets of 100k policies).
5.1 UNSUPERVISED PRE-TRAINING
First, we address the first two research questions, that is:
(Q1) Is possible to learn low-dimensional latent representation of high-dimensional policy parameter space in an unsupervised fashion?
(Q2) What are the properties of such a latent representation, it exists? Is its intrinsic dimension function of the behavioral complexity rather than the size of the parameter space?
To do so, we discretize the latent space into a subset (2: and perform evaluations of the decoded policies { IN This allows for a rough estimate of the quality of the policies compressed latent manifold. Further details on how such discretization was performed are in Appendix B.
Landscape of the Latent Behavior Manifold. We visually inspected the latent spaces trained under different conditions and environments, and we report handful in Fig. 3. Interestingly, it is apparent that the latent spaces, regardless of the choice of encoding dimension (top-to-bottom) or policy size (left-to-right), encode some behaviorally diverse policies with high performance. For instance, in Fig. 3e.3f, and 3g, 2D latent space can encode policies of all three sizes, but the landscape grows more complex with the larger policy sizes. We speculate that this is due the increased range of behaviors expressed by larger policies and the hardness of high-compression regimes. Indeed, the same trend is present for different tasks, as in Fig. 3i,3k. On the other hand, by changing the encoding dimension as in Fig. 3b,3f,3j. it is clear how certain behavioral areas at high performance are able to grow larger, creating a better optimization landscape. Unfortunately yet, the compression is only as good as the dataset used to learn the latent space: when behavior is scarcely represented in the dataset, as is the case for the task height in Fig. 3a,3c, it is unlikely that the learned representation will encode it in large areas, encode it at all. As for RC, the compression architecture struggles to compress the policies at higher compression regimes (Fig. 3d,3h), as the environment more challenging and presents wider range of behaviors. On the other hand, large of good quality compression are present for larger dimensions of the encoding (Fig. 31), confirming the expected theoretical behavior (Hurewicz & Wallman, 2015).
7

---

Published as a conference paper at ICLR 2026
Table 1: Quality of Latent Behavior Compression in MC. We report the performance recovery for three tasks. We report mean and standard deviation computed over 3 seeds. Bold indicates best across latent dimensions for each experiment setting; shading indicates best per task.
Config
Standard
Speed
Height
Datasel
Policy
ID
2D
3D
ID
2D
3D
ID
2D
3D
Small
10k.
0.51
0.66
0.74
0.15
0.15
0.27
0.164
0.16g
0.27a.on
50k
0.
0.93
0.94
10
0.42 15
0. 0.44
0.11 11
0.45a. 14
0.17+.2
10UK
0.50+.00
0.72+.21
0.72+.21
0,15+.01
0.40+.37
0.32+.11
0.29+.04
0.40+.28
0.42+.10
Medium
10k
0.
1.01
1.024.00 .0
0.25
0. 0.
0.
0.364 22
0.71 20
0.7
50k
01
1.01
1.02
0.14
0.851.05
0.98 1.9
0.15
0.451.03
0.471.01
0.51 y. .00
4.02
1.02
0.14g.03 03
0.60 24
0.97
0.22 04
0.44 01
0. 0.53 10
Large
10K
1. 1.02 00
1.01
1.01g 00
0.79
0.872..0 06
1.04:.02
0.78 14
0.84
0. 0.874 0.6
50k
1. 1.02 00
1.01
1.01 00
0.68 06
0.97
1.00
0.47 03
0.55
0.57 13
100k
1.01 +.00
1.01
1.014.00
0.73
0.924.06
0.99
0.39 07
0.54+.04
0. 0.744.27
Quality of Latent Behavior Compression. We also compared the policies encoded in the latent space with the ones in the training dataset. They were compared by examining the performance recovery, that is, the ratio between the performances of policies decoded from the latent space and those in the dataset. The values for MC are reported in Table 1, First, it is clear that increasing policy size frequently leads to better performance recovery,
the number of latent dimensions resulting in higher performance as well. Interestingly, some configurations appear to recover higher performances than the ones in the training dataset. This may be due to the generalization abilities of the AE, but it may also be influenced by variance in the policy evaluation process. On the contrary, note that 1D latent spaces trained on Small policies fail to learn any meaningful encoding of the behaviors, collapsing to uniform representation. We attribute this phenomenon to the instability of the learning process when the latent dimensions are not sufficient. Interestingly, this issue is almost always fixed by increasing the number of latent dimensions and does not arise with large policies, which show excellent performance recovery. Finally, our analysis does not indicate that the dataset's dimension has any meaningful influence on performance recovery.
Generalization. We further investigated the generalization capabilities of Latent Behavior Com- pression in more complex environments. Focusing on larger latent spaces where grid visualization infeasible, we estimate performance recovery via random sampling of the latent space instead. Table 2 reports the results for HC and HP. These experiments corroborate the observations in MC and provide statistically significant evidence that the latent space generalizes beyond the training set. More specifically, increasing the latent dimension consistently improves performance recovery in HC, while such trend is likely obscured in HP by the high variance in the sampling process. Finally, we observe that generalization varies by task complexity, as indicated by the lower recovery rates for difficult tasks like front flip and backflip in HC.
With these experiments, we provided a positive answer to (Q1): the proposed unsu- pervised pipeline is indeed capable of encoding behaviorally meaningful policies in a wide range of configuration and in multiple environments, ultimately leading to a compression of up to five orders of magnitude," As for (Q2), we found that while larger policies produce richer behavioral manifolds, even a one miedimensional latent space is often sufficient to capture wide range of behaviors, supporting the hypothesis that the intrinsic dimensionality of the policy behavior manifold is dictated by the environment complexity rather than by the cardinality of the parameterization. Additionally, we study the scalability and generalization capabilities of the latent space. Finally, we extracted some evidence for the existence of a critical intrinsic dimension in the behavioral manifold, but how to leverage this evidence to learn the best latent representation possible is out of the scope of our work.
Takeaways.
*Values related to the left task have been omitted as they do not present major differences from the standard task. Instead, they are reported in Table 3 of Appendix B. "More precisely, a 121801:1 compression rate peak.
8

---

Published as a conference paper at ICLR 2026
Table 2: Quality of Latent Behavior Compression in HP and HC. We report the performance recovery for four tasks by the mean performance recovery and 95% confidence interval over 10 seeds. Bold indicates best per task.
Latent Dimensions
Environment
Task
5D
8D
16D
Mean
95% CI
Mean
95 % CI
Mean
95% CI
forward
1.33
11003 1.69)
1.59
11.21, 1.99
1.48
1.21, 1.78
backward
2.66
0.95, 5.96
1.29
1111,,.47
1.20
1.07,1.33 1.33
Hopper
standstill
1.49
1.06, 2.08
1.51
[1.02,
1.57
1.07, 2.32
jump
3.83
00.64, 9.32]
1.54
2.14]
2.42
3.35,,3.75
forward
1.63
[1.02, 2.31
1.80
11.00,2.17
1.84
1.31,2.39
backward
1.27
1.04, 1.55
1.52
[1.19, 1.85
1.72
1.37, 2.06
HalfCheetah
frontflip
0.54
00.29, 0.86
0.74
[0.99, 0.99
1.20
0.60, 1.63
backflip
0.55
00.31, 0.86]
0.75
[0.51, 0.99
1.23
0.881, 1.66)
5.2 SUPERVISED FINE-TUNING
Finally, we address the last research question, namely:
(Q3) How can we fine-tune against specific tasks, leveraging the low-dimensional space? Does this offer any advantages?
To achieve this, we compared the effect supervised fine-tuning on the latent space using simple variant of PGPE, which we refer to as Latent PGPE, with various baselines, including PPO (Schulman al., 2017), DDPG (Lillicrap al., 2016), TD3 (Fujimoto al., 2018), and SAC (Haarnoja et al., 2018). We also tested PGPE (Sehnke et al., 2008) in the high- gigh-dimensional parameter space, referred to as Parameter PGPE, as a sanity check. Our implementation of both Parameter PGPE and Latent PGPE can be found in Appendix B. All the algorithms were run in the best-performing configuration for the policy sizes. Interestingly, DRL. baselines always struggle to optimize small policies and typically perform better with larger ones. On the contrary, Parameter PGPE benefits from having reduced set of parameters to control; however, generally suffers from high sample complexity. This evidence is reported in Appendix B (Fig. 7,8,9).
From these comparisons (reported in Fig. and Fig. for MC and RC, respectively). we were able to extract three main findings. First of all, the convergence rate and performance of Latent PGPE are positively correlated with the number of dimensions of the latent space, which confirms that larger latent spaces are indeed better shaped and with easier optimization landscape, hinted in Subsec. 5. as well. Secondly, Latent PGPE converges faster than all baselines in 7 out of 8 tasks, even though does not always converge to the optimum. Finally, Latent PGPE is able achieve comparable. if not better, performances to most of the baselines, even for complex tasks like speed Fig. 4c, 5a. However, we observe that fails solve the height task in Fig. 4d, due to the scarce representation of the high-performance policies in the unsupervised policy dataset.
Takeaways. These results provide positive answer to (Q3): Leveraging the learned oow-dimensional representation of the behavioral manifold, the agent does not only achieve faster convergence, but
better performance than late-of-the-art DRI. algorithms in challenging sparse tasks. Unfortunately, the trade-off is that the fine-tuning performance is tied to the quality of the learned representation.
6 CONCLUSIONS
We proposed novel, unsupervised framework to address the sample inefficiency of Deep Rein- forcement Learning by shifting the focus from parameter space to behavior space. Our approach successfully learns a compact latent manifold of policies, organized by behavioral similarity, using generative model with an unsupervised behavioral reconstruction loss. Empirically, we showed that this approach compress policy parameterizations by several orders of magnitude while preserving their functional expressivity. This compressed representation also allows for more efficient fine-tuning for downstream tasks via cradient-based optimization in the diveninnniiona latent space.

---

Published as a conference paper at ICLR 2026
Latent PACC
Labunt PAPE
Labent PGPE
Seat Datannt
D32G
SAC
TDS
Darama 2GZE
100
-23
Timesteps
Timesteps (b) left
(a) standard
R,tum
6.00
3.50
1,00
1.0
375
0.23
2.0
Timesieps
(c) speed
(d) height
Figure 4: Performance comparison in MC for different tasks. We report the average and 95% confidence interval over 10 runs.
Lxtert TOPE
Extert PGPE
Tackes INPPS
Labeet PNGPE
Latwot FIRE
Doet Bataret
RMC
Parmmoter PGPS
8.0
60
13
8.0
32
0.6
Timeelnge
Timestape
aa) speed
(b) clockwise
14
16
16
Timesteps
Timesteps
(c) c-clockwise
(d) radial
Figure 5: Performance comparison in RC for different tasks. We report the average and 95% confidence interval over 10 runs. For clarity, the worst-performing baselines are omitted. full study reported in Appendix B.
Future Directions. Our framework intentionally modular, and we view it as a blueprint for a new class of more efficient DRL agents. We believe this approach can inspire significant future research into its core components, including the use alternative behavioral divergences, more advanced generative architectures for compression, and the adaptation of different algorithms for latent behavior optimization. Furthermore, many RL approaches that condition value functions (Faccio et al., 2021) leeaa-esrrers (Rakelly et al., 2019) directly on policy parameters could greatly benefit from our compression, as it provides a compact and semantically meaningful representation to replace raw parameter vectors.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work presents fundamental research in reinforcement learning theory and algorithms. We have carefully reviewed the Code of Ethics and confirm that our research raises no ethical concerns.
REPRODUCIBILITY STATEMENT
All experiments were run 50 cores of an Intel(R) Xeon(R) Gold 64118H CPU, 1 TB of RAM, and one NVIDIA HI00 GPU. The total wall-clock time to re-run all experiments is approximately 50 hours. Our proposed approach detailed in Section with full implementation details and hyperparameters included in Appendix B. The source code is available in the supplementary material and at this link.
LLM DISCLOSURE
We used L.MSs in a limited capacity as writing assistance tool. Specifically, L.MSs were employed to help refine the clarity and readability of selected paragraphs throughout the paper. All content has been reviewed and verified by the authors, who take full responsibility for the accuracy and originality of all statements in this paper.
ACKNOWLEDGMENTS
We thank Andrea Fraschini for his assistance in conducting the baseline experiments and Keenan Crane for the extremely clear visualization of autoencoder spaces used in Fig. 1 and that can be found at this link. This work was supported by the Italian Ministry of University and Research (MUR) under the National Recovery and Resilience Plan (NRRP), and by the European Union (EU) under the NextGenerationEUU project (CUP D43C24001670008)
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Reincarnating reinforcement learning: Reusing prior computation accelerate progress. Advances in Neural Information Processing Systems, 18-289555-2971, 2022.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20) 2020.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, (2(27))22118881222 2024.
Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sandra Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378:1067-1074, 2022.
Mira Bernstein, Vin De Silva, John C Langford, and Joshua B Tenenbaum. Graph approximations to geodesics on embedded manifolds. Technical report, Department of Psychology, Stanford University, 2000.
Lawrence Cayton et al. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17):1 2005.
11

---

Published as a conference paper at ICLR 2026
Oscar Chang, Robert Kwiatkowski, Siyuan Chen, and Hod Lipson. Agent embeddings: A latent representation for vole-balancing networks. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 665, 2019.
Siu-Wing Cheng, Tamal K Dey, and Edgar A Ramos. Manifold reconstruction from point samples. In ACM-SIAM Symposium on Discrete Algorithms, volume 5, pp. 1018-1027, 2005.
Rob Comishh Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International Conference Machine Learning, pp. 1133-2143, 2020.
Vincenzo De Paola, Riccardo Zamboni, Mirco Mutti, and Marcello Restelli. Enhancing diversity in parallel agents: A maximum state entropy exploration story. In International Conference Machine Learning, 2025.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):4144419 2022.
Mare Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends@ in Robotics, 2(1-2):1-142, 2013.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018.
Francesco Faccio, Louis Kirsch, and Jürgen Schmidhuber. Parameter-based value functions. In International Conference on Learning Representations, 2021.
Francesco Faccio, Vincent Herrmann, Aditya Ramesh, Louis Kirsch, and Jürgen Schmidhuber. Goal- conditioned generators of deep policies. In AAAI Conference Artificial Intelligence, volume 37, pp. 7503-7511, 2023.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-004, 2016.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596, 2018
Karol Gregor, Danilo Rezende, and Daan Wierstra. Variational intrinsic control. Workshop
International Conference Learning Representations, 2017.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, pp. 1861-1870, 2018.
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farles Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. International Conference on Learning Representations, 2019.
Shashank Hegde, Sumeet Batra, KR Zentner, and Gaurav Sukhatme. Generating behaviorally diverse policies with latent diffusion models. Advances in Neural Information Processing Systems, 36: 7541-7554, 2023.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 31 3133558)):504-500 2006.
Witold Hurewicz and Henry Wallman. Dimension theory, volume 4. Princeton University Press, 2015.
Sham M Kakade. A natural policy gradient. Advances in Neural Information Processing Systems, 14, 2001.
Vera Kürková and Paul C. Kainen. Functionally equivalent feedforward neural networks. Neural Computation, 6(3):543-555)) 1994.
12

---

Published as a conference paper at ICLR 2026
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks, 2021.
Joel Lehman and Kenneth Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Conference Genetic and Evolutionary Compulation, pp. 211-218. 2011.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Olya Mastikhina, Dhruv Sreenivas, and Pablo Samuel Castro. Optimistic critics can empower small actors. In Reinforcement Learning Conference, 2025.
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. Advances in Neural Information Processing Systems, 31, 2018.
Atsushi Miyamae, Yuichi Nagata, Isao Ono, and Shigenobu Kobayashi. Natural policy gradient methods with varameter-basee exploration for control tasks. Advances Neural Information Processing Systems, 23, 2010.
Alessandro Montenegro, Marco Mussi, Alberto Maria Metelli, and Matteo Papini. Learning optimal deterministic policies with stochastic policy gradients. In International Conference on Machine Learning, pp. 36160-3621 2024.
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in Neural Information Processing Systems, 27, 2014.
Andrew William Moore. Efficient memory-based learning for robot control. Technical report, University of Cambridge, 1990.
Mirco Mutti, Stefano Del Col, and Marcello Restelli. Reward-free policy space compression for reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 3187-3203, 2022.
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint 2209 2022.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein. On the expressive power of deep neural networks. In International Conference on Machine Learning, volume 70, pp. 2847-2854, 2017.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy metu-reiifforcmment learning via probabilistic context variables. In International Conference on Machine Learning, pp. 5331-5340, 2019.
Nemanja Rakicevic, Antoine Cully, and Petar Kormushev. Policy manifold search: Exploring the manifold hypothesis for diversity-based neuroevolution. In Genetic and Evolutionary Computation Conference, pp. 901-909, 2021.
Thomas Rückstiess, Frank Sehnke, Tom Schaul, Daan Wierstra. Yi Sun, and Jürgen Schmidhuber. Exploring parameter space in reinforeement learning. Paladyn, 1(1):14-24, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
13

---

Published as a conference paper at ICLR 2026
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems, 34:16481-16493, 2021.
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper-repreesentatons as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems, 0777966-7790,, 2022.
Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In International Conference on Machine Learning, pp. 43947-43966, 2024.
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmid- huber. Policy gradients with parameter-based exploration for control. In International Conference Artificial Neural Networks, pp. 387-396, 2008.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208 07860, 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Frez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite. arXiv preprint arXiv:1801.0000 00060000.000..... 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee -ased control. In International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Nihat Engin Toklu, Pawel Liskowski, and Rupesh Kumar Srivastava. ClipUp: A simple and powerful optimizer for distribution- based policy evolution. In International Conference on Parallel Problem Solvingfrom Nature, pp. 515-527, 2020.
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 602(7896): 223- 223-228,72 2022.
Tom Zahavy, Yannick Schroecker, Feryal Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with DOMiNO: Diversity optimization maintaining near optimality. In International Conference on Learning Representations, 2023.
Riccardo Zamboni, Mirco Mutti, and Marcello Restelli. Towards principled unsupervised multi-agen reinforcement learning. Advances Neural Information Processing Systems, 38, 2025.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Simplification of the Policy Space, A variety of works (among others, Gregor et al., 2017; Eysenbach et al., 2018; Achiam et al., 2018; Hansen et al., 2019) have proposed methods to simplify the policy space. Yet, those policies should be generally intended as mere initializations for supervised fine- tuning, which falls back to operating in the original policy space once the downstream task is revealed. To the best of our knowledge, the only other work defining a formal criterion to operate a compression of the policy space is Mutti al. (2022). Yet, this paper seeks way to reduce the cardinality of the policy space, rather than its dimensionality. Moreover, the constraints defining a valid compression stricter than ours, resulting in optimization problem that is known to be NP-hard. Finally, their work does not provide a way to perform supervised fine-tuning in a scalable way.
Weight Space Learning The goal of our work, namely, to learn latent representation of neural network parameters, is shared by the field of Weight Space Learning (WSI.). In RI., recent works have investigated task-specific generative models, conditioning them on goals (Faccio et al., 2023) or performance checkpoints (Peebles et al., 2022), unlike our fully unsupervised approach. Our approach is more in line with the works on hxpper-rerrexentatiions (Schürholt et al., 2021; 2022; 2024) of networks trained for supervised tasks. Hyper-representttions are task-agnostic low-dimensional embeddings of neural networks learned from zoo of trained models. Beyond its novel application to RL, our approach differs from these works in two ways. First, our "zoo" does not require trained experts. Second, and more critically, WSL methods must often contend with the vast number of parameter-space symmetries (e.g., neuron permutations and scaling (Kárková & Kainen, 1994)) using complex architectures or workarounds. Our behavioral reconstruction loss fundamentally sidesteps this challenge. By compressing policies based on their function rather than their specific parameters, our autoencoder naturally assimilates these redundant parameterizations. This concept of functional compression is similar to "policy fingerprinting" (Faccio et al., 2023), which also gauges behavior independent of weights.
Policy Manifold and Quality Diversity. The idea of employing generative models to learn compressed representation of the policy space has received some recent attention outside of the Weight Space Learning field. Rakicevic al. (2021) hypothesized that there might be a low-dimensional manifold embedded in the policy parameter space, even if they did not characterize it formally. Chang et al. (2019) trains a Variational AE reconstruct the weights of pre-trained expert policies to learn expert-agent embeddings and analyze the latent structure of the solution space. A similar architecture has also been applied in the field of Quality Diversity, either to improve the sample search Rakicevic et al. (2021) or to distill a large policy archive into a compact generative model (Hegde et al., 2023). Notably, all the methods above employ VAE architectures with purmmeter-recoosttuciiii loss, which allows only moderate compression ratios of up to 19: (Hegde et al., 2023). In comparison, this paper introduces a fully unsupervised pipeline that focuses on compressing a behavioral loss, intending to provide compact space for latent policy optimization as well while retaining greater compression abilities.
efficiency of diversity-
Policy Optimization. First-order methods have been extensively employed to address PO (Peters & Schaal, 2008; Lillicrap et al., 2016) as well as natural gradients (Kakade, 2001) and trust-region methods (Schulman et al., 2017). Yet in this work, we built on the long tradition of PGPE Algo- rithms (Sehnke et al., 2008; Rückstiess et al., 2010; Miyamae et al., 2010; Montenegro et al., 2024), as their hard scalability to large parameter spaces is notoriously a blocking factor. Finally, we notice that Rakicevic et al. (2021) indeed proposed method to optimize the diversity of the policies by taking into account the Jacobian of the decoder in a VAE architecture.
B EXPERIMENTAL DETAILS
Environments. We evaluate our methods on four control environments from the Gymnasium library: Mountain Car Continuous, Reacher, Hopper, and HalfCheetah. The first is a classic control environment, which consists of a car placed stochastically in the middle of a sinusoidal valley, with the goal state on top of the right hill. The state is defined by two continuous variables: the position of the car along the z-axis p 1.2, 0.6], and the velocity of the car [-0.07, 0.07]. The only possible action is to apply an acceleration a E [-1, 1] to the car. The standard task is defined as R 10. until the goal is reached and reward of Raundard, 100 is obtained, and
15

---

Published as a conference paper at ICLR 2026
the episode ends. If the goal is not reached, the episode ends after 999 steps. We introduce three additional tasks: left, which is the same as the standard task, but with the goal moved to the top of the left hill (p < -1.1;; hei which gives reward of Breight. = h2 at each time step for which h 2 0.2, with h = sin(3p) * 0.45 + 0.55 being the height of the car; speed, which gives a reward of Ropeed. x² at each time step. In the left. task, the episode ends when the car reaches the left goal, while in the height and speed tasks, the episode ends when the car reaches the right goal.
The second environment, Reacher, is classic continuous control task consisting of a two-jointed robot arm, moving in a 2D space, with an end-effector called fingertip. The state is originally 10-dimensional, but we remove the coordinates of the target and the vector between the fingertip and the target. We end up with a 6-dimensional state composed of: cos(g1), cos(g2), sin(g1). and sin sin().). the cosines and sines of the two joint angles, and w1 and w2, their angular velocities. For the purpose of normalization, we consider the state bounded between the vectors [-1, 11 -1, -1, -5, -5] and 1,,1, 5]. The agent controls the arm by applying a distinct torque to each hinge, making the action space A = 1J3. We disregard the standard task and instead define four new behavioral tasks that have the same reward shape Rusk = 1 if the condition is met, or 0 otherwise. In the speed task, the condition is that the linear velocity of the tip is greater than 6. In the clockwise and c-clockwise se tasks, the condition is that the tangential velocity of the fingertip is greater than -1 I, or 1, respectively. Finally, in the radial task, the condition is that the radial velocity of the tip is greater than 3. The episodes terminate after 50 steps.
The third environment, Hopper, is classic continuous control task that models 1-legged robot with 3 joints. The state space is 1 Il-dimensional, capturing the robot's z-position, joint angles, and corresponding velocities. For the purpose of normalization, we consider the state bounded between the vectors [0.7, -0.2, 227, 2277, -0.8, -5.0, -5.0, -5.0, -5.0, -5.0, 5.0.-5.0p and [1.5, 0.2, 0.0, 0.0, 0.8, 5.0,5.0, 5.0, 5.0, 5.0]. The agent controls the arm by applying a distinct torque to each hinge, making the action space A = [-1, 1] Analogously to Reacher, we disregard the standard task and instead define four new behavioral tasks that have the same reward shape Runk = 1 if the condition is met, or O otherwise. In the forward, backward, and standstill tasks, the condition is that the r-axis velocity is greater than 1, lower than -I, or between -0.05 and 0.05. In the jump task, the 2 position has to be greater than 1.3. The episodes terminate after 1000 steps.
The fourth environment, HalfCheetah, is classic continuous control task that models 2D, two-legged robot with 6 actuated joints. The state space is 18-dimensional, including the robot's body position, rotational angles, and joint velocities. For the purpose of normal- ization, we consider the state bounded between the vectors [0, - -, 1 -0.52, -0.785. - 0.4, -1, 11.2 -05. 55.0, -5.0, -5.0, -5.0, -5.0, 550, -5.0, -5.0, -5.0) and [1.5, n, 1.05, 0.785, 0.785, 0.7, 0.7,0.87.0.3 5.0, 5.0, 5.0, 5.0, 5.0, 5.0]. The agent controls the arm by applying dis- tinct torque to each hinge, making the action space A = [-1, 116 The tasks are almost the same as for Hopper. In the forward and backward, the condition is that the z-axis velocity is greater than 2, or lower than -2. In the frontflip and backflig the task condition is that the cumulative rotation along the y-axis increases or decreases by full rotation from the initial angle. The episodes terminate after 1000 steps.
Policies. To model the policies, we use fully-connected, feed-forward, deterministic MLPs. Our choice of focusing on deterministic policies is dictated by the use of PGPE as an optimization algorithm in the last stage; however, our pipeline is designed to be general. As such, we believe there is no apparent limitation to the use of stochastic policies instead, The input layer has neurons and is preceded by normalization layer that standardizes the state features have zero mean and unit variance. The hidden linear layers are followed by elu nonlinearities. The last layer has |A] neurons, followed by tanh activation to squash the action into the valid range. We test three different shapes of policies in Mountain Car: Small policies composed of a single 4-neuron hidden layer; Medium policies composed of two 32-neuron hidden layers; Large policies composed of a 400-ncuron hidden layer followed by a 300-neuron hidden layer. The number of parameters of the policies increases roughly by two orders of magnitude at each interval l''mmll = 17, I Mediium = 1, 185, PLange = 121, 801). For Hopper and HalfCheetah, we use the same shape of Medium policies. While in Reacher we use Medium policies composed of two 64-neuron hidden layers, with PRencher 4738.
Policy Divergence. To compute the divergence between policies, we instead estimate the distance of the deterministic actions over a subset of states. We consider the state spaces bounded as previously
16

---

Published as a conference paper at ICLR 2026
described, and extract roughly M- 3000 states. In Mountain Car, find them by discretizing the two dimensions and creating a grid, while in MuloCo environments, we simply sample them uniformly from the bounded state space. In the k, -NN phase, we use kn = 15, and compute the distance between two policies as:
D( || TTx)
Σ [To(S))) - To (s:)
While in the manifold learning phase, compute it as:
M D( || )TOr = (TT( - Tor (8:)
Autoencoder. We use a simple, fully-connected. feed-forward, deterministic MLP to model the autoencoder. The shape of the autoencoder is the same for all the experiments. The input and output layers have size P. with the input layer being preceded by a standardization layer, and the output layer not being activated; the encoder has 25-neuron hidden layer followed by 10-neuron hidden layer; the decoder has the mirrored shape of the encoder. The first layer of the encoder and the first layer of the decoder are followed by e elu nonlinearities. The autoencoder is trained for 50 epochs using the Adam optimizer with an initial learning rate of 0.0001 and batch size of 64. We employ a learning rate scheduler that halves the learning rate after 15 epochs of non-improvement, evaluated on a 20% random hold-out set. The empirical loss used to train the autoencoder is defined as:
N
M To (8)) To. (8))
LB = N
2
==
where N is the number of policies the training dataset, M' = 1000 the size of the subset of the state set that sample at each gradient step, and Té, is the reconstructed policy. In Mountain Car, we set the latent dimension of the autoencoders to k = 2, 3, while in Reacher, we use i = 1, 5, 8. In Hopper and HalfCheetah, = 5, 8, 16.
When we evaluate a latent space, we first compute the interquartile range for each dimension based on the spread of the training codes. Then, we discretize each dimension by a variable number of points depending on the dimension of the latent code: 100 points for ID, 50 points for 2D, and 17 points for 3D. For more than three latent dimensions, we sample 10000 policies from the same bounded latent space random. The decision is based solely on computational feasibility and serves the purpose of having a rough conservative estimate of the range encoded behaviors.
Performance Recovery. When comparing the policies found in the latent space with the ones belonging to the original dataset, we compute a behavior recovery metric in the following way. First, we average the dataset lower and upper bounds for all tasks across three seeds with the same configuration, lbp, ubp. Then we do the same for the discretized set of policies reconstructed from the latent space, 1bg, why. Finally, for each task, we compute the performance recovery as ubp-ibo In Table 3, we provide the analysis for the reward left, which was omitted in Table 1.
ub, =bbp
(Latent) PGPE. As byproduct of the low low-dimensionality of the latent space, this framework is well suited to parameter-exploring PG methods. Algorithms like PGPE struggle with a nigh-dimensional set of parameters, such those of standard DRL network with hundreds of thousands of parameters. Yet, they can instead operate on the now-dimensional set of latent parameters while maintaining the expressivity of the original parameter space. As a bonus, the extension of PGPE the latent space does not require computing the Jacobian of the decoder as in Eq. 5, as it can be seen as deterministic addition to the black-box process that evaluates the parameters produced by the Gaussian hyperpolicy Ve. where & = (H, or) is the vector of means and standard deviations parameterizing the Gaussian distributions over the latent parameters. In fact, the objective defined in Eq. 2 can be rewritten under the latent PG formulation as (z) = Ex (=)) [R(r)], with the only change being that the probability density of the trajectories is given by the policy induced by the latent parameters as
p(r] x)) = (((s)) I P( 8z at) (=) | st.. Finally, the gradient estimator at Eq. is left unchanged, but for the change in parameter space from O: to Z;.
17

---

Published as a conference paper at ICLR 2026
Table 3: Quality of Latent Behavior Compression in MC. We report the performance recovery for the left task. We report mean and standard deviation computed over 3 seeds.
Coulig.
Left
Policy
Large Medium Small
10k
0.73g.16
0.66
0.98
50k
0.61=.10
0.98+ 08
0.99 02
100k
0.73
0.80+.21
0.944.00
10k
0.95_.os
1.014.01
1.01+.01
50k
0.78
1.011,
1.01 00
100k
0.82
1.01 .00
1.01
1.012.00
1.00g o1
1.011 .00
50k
1.01=.00
1.01
1.00
100k
1.01+.00
1.01+.00
1.00+.00
We base our implementation of PGPE on ask-and-tell implementation with symmetric sam- pling (Toklu et al., 2020). We modify it to allow for numpy parallelization, reward normalization, center learning rate scheduling, learning log instead of Ø, and natural gradient computation. The center is optimized with Adam, with momentum 0.2. The log-standard deviation instead is learned through simple gradient ascent with fixed learning rate. For Mountain Car, perform 75 seeded runs on 75 different autoencoders with the same hyperparameters: center learning rate 0.05, population size 4, initial standard deviation 0.6, standard deviation learning rate 0.1, and 50 generations. In Reacher, the learning rate has a linear annealing down to 20% of the initial value, and we use the same hyperparameters for all runs, which are the following: center learning rate 0.1, population size 10, initial standard deviation 0.3, standard deviation learning rate 0.1, and 200 generations. In both cases, each sample of the population is evaluated on a single episode.
Given the episodic nature of PGPE, each generation can take an arbitrary number of environment steps to evaluate the samples of the population. Since the center is changed only once the end of a generation, evaluate the learning curve at different sample checkpoints, contrary to the standard StableBasclines3 approach. This causes different runs to have different evaluation checkpoints (æ-axis of the learning curve) and a different number of total samples used. To visualize an aggregate learning curve across multiple runs, we take two measures: we interpolate the various curves along the r-axis; extend the final evaluation of each along the x-axis that all runs have identical lengths, with the assumption that the algorithm has already converged by the end of the training. To ensure the stability of the final evaluation, we evaluate the center solution on 100 episodes (instead of 10 episodes used during training).
Reproducibility, In MC, we perform two main experiments. First, we study different configurations by creating 27 different datasets. We seed all the steps of the pipeline with seeds O through 26. In order, we use seeds 0-8 for Small policies, 9-17 for Medium policies, and 18-26 for Large policies. In each batch, the first three seeds are used for datasets of 10k policies, the next three for datasets of 50k, and the last three for datasets of 100k policies. The second experiment focuses on datasets of 10k Medium policies, and it is run with seeds starting from 100. In Reacher, we focus datasets of 100k Medium policies with seeds starting from O. In Hopper and HalfCheetah, we focus on datasets of 10k Medium policies with seeds starting from 0.
ADDITIONAL EXPERIMENTS
Baseline Hyperparameters. Here we provide the hyperparameters used to train the baselines for each environment. Where not specified, we use the StableBaselines default parameters. In MC, we used higher standard deviations for the stochastic processes used by TD3 and DDPG for exploration: 0.75 and 0.65, respectively. For DDPG, we also used a smaller replay buffer size of 50000. For SAC, we used soft update coefficient (tau) of 0.01, train frequency of 32, entropy coefficient of 0. 32 gradient steps per rollout, replay buffer size of 50000, and we used generalized State Dependent Exploration (gSDE). For PPO, used learning rate of 0.0001, 32 steps per rollout, batch size of 256, 4 epochs optimization of the surrogate loss, lambda value of 0.9 for the Generalized Advantage Estimator (GAE), clip range of 0. entropy coefficient of 0. value function coefficient
18

---

Published as a conference paper at ICLR 2026
100
Mianded
Lan
speed
100
301gat
Probabiitt Mass
10-
2011
10.4
10-4
10-2
10-
10-
100
50
100
100
50
0
50
100
0.0
0.5
1.0
3.5
2.0
200
4.00
600
800
Return
Return
Return
Return
Figure 6: Reward distribution comparison datasets of 10k Small (blue) and Large (orange) policies in MC. The y-axis uses a logarithmic scale. We report the average and 95% CI over 5 seeds.
DDPG
PPO
SAC
TD3
PGPE
Small
Medium
Large
Yeulees
Teanacc
st standard
(b) left
(c) speed
(d) height
Figure 7: Baseline Ablation study in MC. We report the average and 95% confidence interval over 10 runs.
of 0.19, max gradient norm of and used gSDE. In RC, kept the same hyperparameters used for MC, changing only the standard deviation of DDPG to 0.5.
Baseline Ablation Study. We report complete baseline studies for both MC and RC. MC, we study how the baselines operate with different-sized policies. We report our results in Figure 7. We observe that almost all algorithms struggle with optimizing small policies. In Figure 8, we report separate study for PGPE, in order to offer a cleaner visualization, given the major difference in sample complexity. We can observe the opposite behavior, namely, PGPE is often more sample-efficient and better-performing when using smaller policies. Finally, in Figure 9, we report the complete study of baselines for the RC environment.
Policy Representation Range. We analyze the behavioral distributions of randomly sampled Small and Large policies in the MC environment (Figure 8). For simpler tasks (standard, left, and speed), both architectures cover similar reward range, though their probability distributions differ. However, in the more complex height ght task, Large policies exhibit significantly broader support, reaching higher maximum rewards than their Small counterparts. While this observation relies random weight sampling and is limited to a single environment, it aligns with theoretical findings on the relationship between parameter count and network expressivity (Montúfar al., 2014; Raghu et al., 2017; Bahri et al.. 2024), suggesting that larger networks are naturally capable of representing richer diversity of behaviors.
Latent Behavior Manifolds. To complement the results in the main paper, we provide an extensive set of visualizations of the learned latent behavior manifolds. These plots illustrate how the latent representations organize policies across different tasks, policy sizes, and encoding dimensions. They cover both environments studied in this work MMounaai Car (MC) and Reacher (RC)- and show how the manifold structure emerges consistently across settings. The visualizations serve two purposes: (i) to confirm that the latent space captures meaningful behavioral structure qualitatively, and (ii) to demonstrate the consistency of this organization across seeds and settings. For clarity in the main text, we only reported a subset of representative plots; here, to enable a more thorough inspection and reproducibility,
visualize one seed per configuration in MC and all seeds for RC. Each figure shows the latent spaces for 1, 2, and 3 dimensions for all tasks. The tasks are ordered from left to right follows: standard, left, speed, height for MC, and speed, clockwise, c-clockwise, and radial for RC.
19

---

Published as a conference paper at ICLR 2026
Small
Medium
Large
(a) standard
(b) left
(c) speed
(d) height
Figure 8: PGPE Ablation study in MC. We report the average and 95% confidence interval over 10
runs.
DDPS
PPD
SAC
TDX
PGPS
(a) speed
(b) clockwise
(c) c-clockwise
(d) radial
Figure 9: Baseline study in RC. We report the average and 95% confidence interval over 10 runs.
Figure 10: MC Small, 10k Seed 0
20

---

Published as a conference paper at ICLR 2026
Figure 11: MC Small, 50k Seed 3
Figure 12: MC Small, 100k- Seed 6
21

---

Published as a conference paper at ICLR 2026
Figure 13: MC Medium, 10k Seed 9
Figure 14: MC Medium, 50k Seed 12
22

---

Published as a conference paper at ICLR 2026
Figure 15: MC- Medium, 100k Seed 15
Figure 16: MC Large, 10k Seed 18
23

---

Published as a conference paper at ICLR 2026
Figure 17: MC Large, 50k Seed 21
Figure 18: MC- Large, 100k Seed 24
24

---

Published as a conference paper at ICLR 2026
Figure 19: RC Seed
Figure 20: RC Seed
25

---

Published as a conference paper at ICLR 2026
Figure 21: RC-
Seed 2
Figure 22: RC Seed
26

---

Published as a conference paper at ICLR 2026
Figure 23: RC Seed
Figure 24: RC- Seed
27

---

Published as a conference paper at ICLR 2026
Figure 25: RC
Seed 6
Figure 26: RC Seed 7
28

---

Published as a conference paper at ICLR 2026
Figure 27: RC Seed 8
XIX
Figure 28: RC Seed 9
29


Reviews:
[{"review_id": "zR1dlsx7bf", "reviewer": "Reviewer_RZJt", "summary": "This paper proposes an unsupervised method to address the sample inefficiency of deep reinforcement learning. The approach learns a compact latent representation of the policy parameter space by optimizing *behavioral* similarity, allowing policies to be optimized in a low-dimensional space. Experiments on classic continuous control benchmarks demonstrate that this representation not only accelerates convergence and allows significant policy compression but also produces semantically meaningful latent representations.", "questions": "* In line 047, the term *behavior* is introduced as \"a latent representation of the possible behaviors.\" Since its precise meaning is not clear at this point and is formally defined later in line 099, it would be helpful to emphasize it with italics to signal its special usage.\n\n* In the experiments, it would be useful to specify the architectures used for the policies. I am particularly interested in how the depth and width vary across different policy sizes (small, medium, large), as prior work has shown that network depth can significantly affect the expressivity of deep RL policies.\n\n* I could not identify the algorithm used for training the policies. If it is not described in the main text, this information should be added for completeness.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper studies a core problem in deep reinforcement learning: instead of optimizing the expected return over the entire space of states and actions, can we focus on visiting sttes and actions that matter for the task in hand?\n\n* The presentation and writing quality of the paper are strong. The problem is well motivated, and the research questions are explicitly stated, making it easy to understand the focus of the study. I particularly enjoyed the background and problem formulation sections—they are cohesive, detailed, and accessible, making the paper enjoyable and understandable even for readers who may not be very familiar with the field, like myself.\n\n* Experiments are also descent quality and help answer the main questions authors presented early in the paper.", "weaknesses": "* The quality of Figure 1 is quite low; the grey text is difficult to read, and zooming in causes significant pixelation, which detracts from the overall presentation of the paper.\n\n* Figures 4 and 5 should be larger, as the curves are currently hard to distinguish. Additionally, the color choices make it difficult to differentiate the plots—for example, in Figure 4, DDPG, PPO, SAC, and TD3 all use similar green/blue colors with solid lines. Using more distinct colors and line styles would greatly improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XzAnAzS0ql", "reviewer": "Reviewer_Rt5R", "summary": "The paper proposes a method of optimizing a policy not directly in parameter space, but in a lower-dimensional learned latent space. This space is learned by an autoencoder trained on a dataset of policy parameters. The training set for the autoencoder is generated using policies discovered in an 'unsupervised' (i.e., reward-free) setting via novelty search, with the L2 distance between actions on some subset of the state space serving as the distance measure. Given a reward function, a concrete policy can then be optimized directly in the learned latent space using policy gradient algorithms. The paper shows that in mountain-car-continuous and reacher environments, this latent optimization often learns faster than methods like DDPG, PPO, or SAC, which learn in parameter space.", "questions": "- What are the best-performing policies for the various tasks in the training datasets? A direct comparison (with confidence intervals) between the policies found by Latent PGPE and the best policies already in the training set is needed to assess the true value of the latent optimization procedure.\n- Lines 359-360: \"We speculate that this is due to the increased range of behaviors expressed by larger policies\". Can the authors provide evidence (e.g., a quantitative measure of behavioral diversity) for an increased range of behaviors for larger policies in the training datasets?\n- Lines 361-362: \"...certain behavioral areas at high performance are able to grow larger, creating more nuanced decoded policies\". Do larger areas in the latent space really correspond to more _behaviorally_ nuanced policies, as opposed to e.g., simply multiple different parameters encoding the same behavior (a known redundancy)? Can this be tested?\n- Line 464: \"bitter lesson\" is obviously a loaded term in the context of RL. However, I can't see how the obvious dependency of the performance on the quality of the learned latent space relates to Sutton's essay, or why it is a \"bitter lesson\" in general. This seems like a straightforward and expected trade-off.\n- How would the subset of the state space (used to measure policy diversity for the training set) be chosen if the state space is too large or not well-defined enough for uniform sampling?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed framework has clear intuitive benefits and is well-motivated by the manifold hypothesis.\n- The experiments are clearly described and thoroughly documented.\n- In the tested environments, some advantages of the method can be seen (mainly faster convergence of the policy's performance, although not always).", "weaknesses": "- The method is empirically tested in only two very simple continuous control settings: Mountain Car Continuous and Reacher. It is difficult to judge the promise of the method without evaluation in much more challenging and open environments.\n- The method's core contribution (latent optimization) is significantly undermined if, in the best-case scenario, it simply recovers high-performing policies that were _already present_ in the training dataset generated by novelty search. The main benefit of the learned latent representation would be that it _generalizes_, i.e., that it allows for the efficient discovery of policies that are substantially different (in capability or performance) from those in the training set. It is not clearly shown in the paper that this happens in practice. While the paper mentions (lines 375-394) that some configurations recover performances _higher_ than in the training set, it is unclear if this result is statistically significant or within the bounds of evaluation noise. This point is critical to evaluating the method's true contribution.\n- The paper does not mention the significant challenges and problems arising when learning latent representations of neural network parameters (e.g., scalability to large networks, or weight space symmetries). The literature on weight space learning is ignored by the paper, even though it addresses exactly this topic and might lead to significant improvements. See for example \"Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights\" (Schürholt et al., 2022), \"Goal-conditioned generators of deep policies\" (Faccio et al., 2023), or \"Learning useful representations of recurrent neural network weight matrices\" (Herrmann et al., 2024). Going beyond the toy domains used in the paper would likely require insights from these works.\n\nWhile I find the proposed method interesting, I vote for a weak reject. I believe that a more thorough evaluation is necessary, specifically one that demonstrates statistically significant generalization beyond the training set policies and tests the method in more challenging environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5k7lj0X65I", "reviewer": "Reviewer_uJZc", "summary": "The paper tackles the issue of sample inefficiency in deep reinforcement learning caused by the high-dimensional parametrization of policies. The authors propose a two-stage approach: first, they compress the policy parameter space into a low-dimensional behavioral latent space using autoencoders; then, they leverage this learned latent representation to efficiently adapt and learn new tasks.", "questions": "1. Could you elaborate on the advantages and disadvantages of reducing cardinality versus dimensionality?\n\n2. Could you provide more details on how the policy network is updated without computing the Jacobian of the decoder? In Appendix B, what does $\\phi$ represent in the expression $z \\sim v_\\phi$?\n\n3. Could you offer some intuition regarding the trade-off among the size of the original policy network, the dimension of the latent space, and the resulting policy performance? For instance, does a larger policy network generally lead to better performance, but also require a larger encoded latent space?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a clear motivation, grounded in the Manifold Hypothesis, for learning and operating within a low-dimensional behavioral manifold.\n2. The work addresses the sample inefficiency of DRL. At the pre-training stage, it uses unsupervised RL, which does not require one to solve actual RL tasks\n3. The experiments show that the policy networks can be compressed up to five orders of magnitude\n4. The experiments show that fine-tuning the latent space allows faster convergence and low sample complexity", "weaknesses": "My main concern is that although the work is well motivated and supported by good intuition, the empirical evidence provided in the experiments is not sufficiently strong to support some of the authors’ claims. In particular:\n\n1. Complexity of the latent space: The authors evaluated latent space complexity through visualizations of the return of the encoded policies. However, the interpretation of these figures, especially for the 3D latent space, is somewhat ad hoc and lacks quantitative rigor. It would strengthen the paper to include concrete metrics that quantify latent space complexity rather than relying solely on visual inspection.\n\n2. Performance recovery metric: To assess performance recovery, the authors compare the upper and lower performance bounds of policies found in the latent space versus those from the original datasets. In my view, this comparison is insufficient to convincingly demonstrate recovery, as it only reflects the spread of the distribution and neglects the distribution shape or central tendency. A more informative metric (such as divergence measures) could provide a clearer picture of how well the latent policies capture the original performance characteristics.\n\n3. Convergence behavior: While Figures 4 and 5 show that Latent PGPE converges faster than other baselines, it does not consistently reach the optimal policy—approximately half of the time (e.g., Figures 4a, 4c, 4d, 5b, and 5c), convergence is suboptimal. In 5b, Parameter PGPE is significantly better than latent PGPE. This suggests that the chosen latent space may not be sufficiently expressive to fully recover the original policy space. This observation also reinforces my earlier point that spread-based metrics may fail to adequately capture true performance recovery.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)