Periodic Skill Discovery
Authors: Jonghae Park
Daesol Cho
Jusuk Lee
Dongseok Shim
Inkyu Jang
H. Jin Kim
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
unsupervised skill discovery
periodicity
Primary area: reinforcement_learning
TL;DR: We propose Periodic Skill Discovery (PSD), an unsupervised framework that discovers diverse periodic behaviors.

Abstract: Unsupervised skill discovery in reinforcement learning (RL) aims to learn diverse behaviors without relying on external rewards. However, current methods often overlook the periodic nature of learned skills, focusing instead on increasing the mutual dependency between states and skills or maximizing the distance traveled in latent space. Considering that many robotic tasks—particularly those involving locomotion—require periodic behaviors across varying timescales, the ability to discover diverse periodic skills is essential. Motivated by this, we propose Periodic Skill Discovery (PSD), a framework that discovers periodic behaviors in an unsupervised manner. The key idea of PSD is to train an encoder that maps states to a circular latent space, thereby naturally encoding periodicity in the latent representation. By capturing temporal distance, PSD can effectively learn skills with diverse periods in complex robotic tasks, even with pixel-based observations. We further show that these learned skills achieve high performance on downstream tasks such as hurdling. Moreover, integrating PSD with an existing skill discovery method offers more diverse behaviors, thus broadening the agent’s repertoire.
Our code and demos are available at https://jonghaepark.github.io/psd

Periodic Skill Discovery
Jonghae Park!
Daesol Cho
Jusuk Lee' Dongseok Shim' Inkyu Jang' H. Jin Kim!
'Seoul National University Georgia Institute of Technology bdfire12340amu ac.kr
Abstract
Unsupervised skill discovery in reinforcement learning (RL) aims to learn diverse behaviors without relying externl rewards. However, current methods often overlook the periodic nature of learned skills, focusing instead on increasing the mutual dependence between states and skills or maximizing the distance traveled in latent space. Considering that many robotic masks-pattcuulart those involving Docomotion-requirre periodic behaviors across varying timescales, the ability to discover diverse periodic skills is essential. Motivated by this, we propose Periodic Skill Discovery (PSD), a framework that discovers periodic behaviors in an unsupervised manner. The key idea of PSD is to train an encoder that maps states to a circular latent space, thereby naturally encoding periodicity in the latent representation. By capturing temporal distance, PSD can effectively learn skills with diverse periods in complex robotic tasks, even with pixel-based observations. We further show that these learned skills achieve high performance on downstream tasks such as hurdling. Moreover, integrating PSD with an existing skill discovery method offers more diverse behaviors, thus broadening the agent's 's repertoire. Our code and demos available https: // bettps:/jjongee jonghaepan..a github. io/pad
Introduction
fundamental observation in nature is that nearly all forms of locomotion are inherently periodic. Rhythmic gaits of quadrupeds, the oscillatory motions of fish, and even human walking patterns share a distinct periodic structure, which can be flexibly modulated across multiple timescales [39, 28, 31]. This inherent periodicity not only enables energy-efficieed movement [39, 82] but also provides adaptability under varying conditions [28, 31, 82]. Motivated by this understanding, robotics research has leveraged periodic priors to effectively control complex behaviors in various challenging environments [77, 87, 78, 52, 86, 83, 47].
In contrast, unsupervised skill discovery methods [25, 84, 34, 13, 89, 20, 59, 50, 96] despite their success learning diverse behaviors without external reward-hawe rarely addressed the role of periodicity. They primarily focus maximizing the mutual information (MI) between skills and states [25, 84, 13, 89, 20, 50] or maximizing state deviation based on given metric [65-67, 75], both of which encourage state diversity, thereby biasing the learned skills toward discovering where to go. However, none of these methods address how to behave, which requires modeling the periodic structure of behaviors behevilos-especiially across multiple timescales.
To address this gap, we propose a novel unsupervised skill discovery objective for learning periodic behaviors, which we call Periodic Skill Discovery (PSD). The main idea of PSD is to train an encoder that maps the state space to a circular latent space, where moving along this cireular structure naturally implies repetition-a fundamental property of periodicity. This geometric connection between circular embeddings and periodic behaviors makes our approach both intuitive and effective for capturing periodicity. Specifically, the latent space of PSD is designed to encode temporal distance, so that moving along a larger circle corresponds to a longer period, directly linking latent geometry to actual period (Figure 1).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Latent space of PSD
Figure 1: Visualization of the circular latent space for Walker2D and HalfCheetah. The core idea of PSD is to map the state space into circular latent space. where temporal distance is encoded geometrically. The figure visualizes an actual policy learned by PSD, where following larger circular paths (blue- + magenta) corresponds to longer-period behaviors.
While the circular representation is being updated, PSD jointly trains an RL policy using a single-step intrinsic reward defined in this latent space. By encouraging the policy move along the circular path in latent space, the RI. agent can achieve periodic skills varying lengths using only single-step reward signals.
Through experiments on various robotic continuous-contre tasks, we empirically demonstrate that PSD can discover diverse periodic skills across multiple timescales. These learned skills are also shown to be effective in solving complex downstream tasks that require multi-timescale prediction (e.g.. hurdling). Furthermore, since PSD encodes temporal distance in manner that is invariant the underlying state representation [94, 67, 68], it can also discover periodic skills even in pixel- pixel-based robotic environments. Moreover, PSD be effectively combined with the existing skill discovery method, METRA [67], thereby broadening the scope of learned behaviors. We empirically find that this combination leads more diverse and structured skill repertoires than either method alone.
To sum up, our contributions can be summarized as follows:
We introduce PSD, novel skill discovery objective that learns periodic behaviors across multiple timescales by mapping states to circular latent space, enabling the agent to exhibit temporally structured behaviors with controllable periodicity.
The discovered skills are predictive over multiple horizons, enabling agents to solve complex downstream tasks (e.g., hurdling) more effectively.
By encoding temporal distance rather than relying on specific state representations, PSD) can discover various periodic behaviors even in pixel-based environments.
PSD can be combined with the existing skill discovery method, METRA [67], expanding the range of learnable behaviors.
Related Work
Learning Periodic Motion Recent research has proposed various approaches to learning periodic motion in robotics. In the domain of legged robots, conventional methods often rely on carefully designed foot contact schedules [4, 8, 7] or central pattern generators (CPGs) [77, 87, 78] to manage gait patterns. In RL-based approaches, hand-crafied reward functions [52, 86, 83, or constraints [47] are widely used to encourage specific gait behaviors. These reward functions often incorporate phase variables [8, 52, 86] inform the current gait phase, or leverage predefined foot trajectories [83, 98, 58, 2] to establish joint targets via inverse kinematics. While effective in guiding legged robots to achieve desired walking patterns, these approaches present significant limitations in terms of generalizability and scalability. Designing such reward functions requires extensive manual tuning and domain-specific knowledge, making challenging to expand these methods to a wide range of robotic platforms or high-dimensional observations.

---

Another line of research in learning structured periodic motion focuses on representing motion data using frequency-domate features [57. 11, 92, 5, 97, 88]. In particular, PAE [88] leverages Fourier transforms to encode motion data into a latent phase space, capturing nonlinear local periodicities across different body segments and enabling structured motion representations. Building upon this, FLD [56] introduced an RL stage to PAE, proposing robust policy learning framework that generates periodic behaviors over long-term horizons. Despite its contributions, FLD relies on offline data to pre-train the autoencoder, limiting its applicability to the given data distribution. Furthermore, it requires manually engineered reward functions for individual body segments, which hinders its scalability to high- sigh-dimensional inputs such as pixel- based observations.
Mutual Information- based Skill Discovery
widely adopted approach to unsupervised skill discovery is to learn skills that maximize the mutual information (MI) between states S and skill Z, namely (S; 2). By maximizing (S; 2). each distinct skill variable x corresponds to distinguishable states 8, which encourages skill policy to visit diverse set of states. For example, DIAYN [25]
maximizes a variational lower bound of [(S; Z) through the following objective:
I(S; 2) = - (2))))) + (2) = Ez,r [(z 8)] [=llog((3)
(1)
1
2 E2,r (log go (= })] + (constant) = Ea,r 2.
|| (=))))1
+ +cconstant),
(2)
where (a(2 |s) is a skill discriminator that infers the skill from given state s. The agent is rewarded whenever it visits state where the discriminator can predict the skill with high confidence.
However, MI-based methods tend to discover skills that are easy to distinguish, rather than skills with diverse temporal patterns. The objective can be fully optimized simply by making the visited states maximally separated for each skill (i.e., minimizing HZ S)). often leading to simple or static behaviors as there is no additional motivation for exploration [89, 50, 66, 67]. Moreover, Gaussian N ((((s), o² oT)) the MI objective can be viewed
when (= s) is parameterized
goal-reaching objective in the latent space as shown in Eq. (2) [20, 653. Consequently, MI-based skill discovery methods do not consider periodic nature of behaviors, leaving temporal aspects of skills underexplored.
Distance-Maximizing Skill Discovery As alternative to MI-based approaches, distance- maximizing methods have been proposed [65-67, 75]. Formally, they maximize the following objective:
JOSD == B(r,)) ((or)) ((s)))
2] s.t. [1o(z)- -
s d(r,g) Vz,yE E D,
(3)
where D is the replay buffer, and ø S + Z is trainable function that maps states into latent representations. Here, the metric d enforces an upper bound on latent transitions so that differences in the latent space do not exceed the distance measured by d. Under this constraint, the RL agent learns to maximize [(((+++) (sxt)l in certain directions z, thereby discovering diverse skills that traverse the largest distances in latent space. Specifically, different choices of the metric dusuch as Euclidean [65], controllabilit-awwal distance [66], temporal distance [67], and language- distance encouraee different types of behavioral diversity.
However, a key limitation of maxiamiiim approaches that they discover skills which maximally deviate under their own metrics, yielding only "hard-to-ace toacheve- behaviors. For instance, METRA [67] employs a temporal distance as its metric and thus strongly prefers fast-moving skills to maximize temporal state deviations. This suggests that these distance-maximizin maximizing approaches provide no incentive to adjust the temporal patterns of the learned skills, making difficult to capture multi-timescale periodic behaviors.
Ad Advantages of PSD Prior approaches in robotics often require extensive domain -specfc knowledge or offline data to learn periodic motion, while unsupervised skill discovery methods fail to capture the periodic structure of behaviors. To overcome these limitations, our proposed method, PSD, constructs a circular latent space that captures multi-timescall periodicity in an unsupervised manner. Moreover, by encoding temporal distance in the latent space, PSD becomes invariant to the underlying state representation and scales to high-dimensional observations. Overall, PSD offers generalizable and scalable framework for capturing multi-timescale periodicity, enabling RI. agents to autonomously achieve periods of diverse lengths.
3

---

3 Periodic Skill Discovery
In this section, deseribe objective designed learn circular latent representations that capture periodicity, Leveraging this latent structure, we define intrinsic reward functions to train a skill policy that discovers periodic behaviors.
3.1 Preliminaries
For unsupervised skill discovery, we consider Markov decision process (MDP) M (8, A, in the absence of external reward. Here, S is the state space, A is the action space, and P S x x ASS) denotes the transition function.
this setup, we define a positive integer L as the period variable, which conditions the policy π va s, L.) to produce behaviors with period 2T. Formally, we refer to T |s L.) as periodic skill policy, which satisfies
8t- S122L where {st*k 22 ~ Pr.. E 1..
Here, PRL denotes the distribution over state trajectories induced by the policy 5(a |s, L). At the beginning of each training episode, the period variable L is sampled from prior distribution pLL), which we assume to be uniform over a bounded set of positive integers L E [Lmin Lmax]. Once sampled, L remains fixed throughout the episode. We then roll out the periodic skill policy π (a L) using the chosen L to collect a skill trajectory.
3.2 Circular Latent Representation to Capture Periodicity
To capture periodicity in unsupervised manner, train an encoder é S x N R that maps a state and a period variable L to a latent circle of diameter L. For simplicity, we denote (1) o(s, so that (.)(.) highlights the dependence L. Formally, PSD) maximizes the following constrained objective:
((0L())L) JPSD == =B ( - - - k (101(s(+L) + (L)))))
(4)
s.t. (101()++) - (1()2222 $ L,
(5)
(001())11 (x))ll S L sin(T/2L)
A (L, St, 8841,
8141, St42) E D,
(6)
where D is the replay buffer and l: > 0 a constant.
To construct circular latent representation, JESD encourages the encoder Φr, to map 8; and s to opposite points of the latent circle of diameter L. This is achieved by maximizing | (51+2)) Φz (st) ll while the first constraint ensures that this distance does not exceed L. The second constraint enforces equal angular spacing between consecutive states, where each adjacent pair is separated by angle of L/L, resulting in regular arrangement along the circle. Specifically, the term L sin( / / corresponds to the side length of regular 2L-gon inscribed in a circle of diameter L. As a result, the encoded states are positioned the vertices of the polygon, evenly distributed along the circular latent space (Figure 2), which facilitates the design of a single-step intrinsic reward described in Section 3.3.
(x
Lsin(i/2L))
(2(5,)
Figure 2: Latent space of PSD.
Illustration of the circular structure induced by optimizing JESD-
Additionally, to prevent arbitrary translations in the latent space, we include the term -k 1 (5++L)+ & (st)ll in Eq. (4). This ensures that the midpoint of opposite points is placed at the origin, aligning circles of different diameters to share the same center and form concentric circles for each L.
By optimizing JesD, the latent representation is structured to capture temporal distances. States that are L steps apart are mapped to opposite points on the latent circle, and after 2L steps, the latent trajectory returns to its initial point, completing full loop. We formally prove in Appendix A that optimizing JeSD induces a regular 2I. in latent space, where the encoded states satisfy Φ (sz) (5(+22) and dr (S++L) lies opposite to (s;) on circle of diameter L.

---

Algorithm Periodic Skill Discovery (PSD)
1: Initialize: policy π, encoder Φ sampling bound Lmin replay buffer DD Lagrange multiplier
2: for each training epoch do
3: Update Lmin- Lmax if AdaptiveSanpling is enabled
4: for each episode the epoch do
5: Sample L ~ p(L, where L E mmin Lmax]
6: Execute (a 8, L) for the entire episode, and store transitions (I., St, at, 84+1) in
7:
end for
8: Update (s) by maximizing JPSD.o using samples from D
9:
Compute intrinsic reward FPsD
10: Update π (a 8, L) with resD using SAC
11:
end for
Tractable Implementation we use dual gradient descent [9, 24] with Lagrange multipliers \1,2 2 0 as follows:
To implement constrained objective JPSD in tractable manner,
J E [ (n())) -k (11on((stt)) (n))21
A1 min(c, L - ((0L(((4)) - (2((s))12)
A2 min(c, [ 11(b)())+11) (1()))))))
where 0 small relaxation constant introduced to improve training stability [94, 67]. The tuple (L, $t-1, S1+L) is sampled from the replay buffer, which stores trajectories collected by the skill policy (aa s, L).
3.3 Single-Step Transition Reward for Periodic Behavior
While circular representation is being learned, the RL agent is jointly trained with an intrinsic reward that encourages periodic behavior. Since the circular latent space is designed to capture periodicity, rewarding the policy for moving along this circular space naturally promotes the learning of periodic behaviors. To this end, first quantify how much single-step latent transition deviates from the optimal length:
A = (101()+1) (s:)ll2 Linn((/2L))
Here, I. sin (x/2L) is the optimal single-step length from Eq. (6). We then define the intrinsic reward FPSD as follows:
FPSD Se+1, L) exp exp
where K > 0 is a positive constant. Maximizing "PSD penalizes deviation from the optimal single- step distance the latent space, thereby encouraging the policy TTa L) to follow the circular path and complete a full cycle of period 2L, where (s.) (St+2r)) By leveraging the latent representation PSD, the RL agent can discover diverse skills with varying periods using only a single-step reward design, without requiring entire rollouts specialized objectives for each period.
3.4 Adaptive Sampling Method
To enable agent to discover maximally diverse range of periods without any prior knowledge of its inherent period ranges, we introduce adaptive sampling method that dynamically adjusts the sampling range [ mmax] during training. The idea to evaluate the performance of the policy conditioned the boundary of the current sampling range. The performance is measured by the average cumulative sum of TPSD as follows:
R2 = (L( Σ TesD (150( Li] for L E {Lmin Linxx h
where p(r [.) denotes the distribution over state trajectories induced by the policy T r(a|s, s, L). Notably, since the TPSD is defined as exp(-xA²) (0, 1]. R2 upper bounded by T. which corresponds to the maximum episode length. We use this upper bound set threshold for how accurately the
5

---

Ant
HalfChoctah
Humanoid
Hopper
Walker2D
PSD(Ours)
METRA
DIAYN
DADS
Figure 3: Comparison of skill trajectories in the frequency domain. We apply Fourier transform skill trajectories, where each skill is uniformly sampled from the skill prior of each method. The resulting spectrum illustrates the frequency (æ-axis) and amplitude (y-axis), representing the temporal patterns of each skill. The accompanying bar chart visualizes the four most dominant frequencies-rankel by amplitude-and highlights the range of discovered periods.
policy x(a 8, L) follows the desired circular path in the latent space. The bounds updated as follows: Liaa
( Lmmax + N if R.aa aT
Lmmin N if R aT Liin + N if R BT
Liin
Limx N if R. BT
Here, a and B are threshold coefficients, where a > B> O, and a positive integer that determines the step size for adjusting the bounds. Since FrsD quantifies the deviation between the optimal and actual latent transitions, a large R, indicates that the current skill policy has the capability to achieve the currently given period ranges and is ready expand its skills. In such cases, the corresponding bound is extended. Conversely, if Rg. is too small, the current bound is rejected and the previous value is restored. This mechanism enables the agent discover dynamically feasible period bounds, thereby broadening the range of achievable periods. Details of the full algorithm and hyperparameters provided Appendix C.
3.5 Algorithm Summary
To summarize, we train the encoder to construct the circular latent representation, and jointly optimize the policy T(a L) with the single-step intrinsic reward resD using SAC [33]. The full procedure is described in Algorithm 1, and additional implementation details are provided in Appendix C.
4 Experiments
The main goal of our experiments is to demonstrate that PSD can discover diverse periodic skills across multiple timescales by learning circular latent representation. We also evaluate whether the discovered skills are useful for solving downstream tasks. In addition, we examine the scalability of PSD to high-dimensional observations such pixel inputs. Finally, explore the potential of combining PSD with existing unsupervised skill discovery methods to enhance the agent's behavioral diversity.

---

=110
16
State value
200
300
20
33
40
State value
200 400 800
800
1000
1200
1400
timestep
Figure 5: Trajectories of the skill policy and corresponding latent representation. The figure shows the joint trajectories of Ant (top) and Walker2D (bottom) and 2D PCA projection of their latent encodings learned by PSD. Within a single episode, we switch the period variable L at fixed time intervals. The resulting behavior of the skill policy exhibits period of 2L timesteps.
1-steg emoor = 0.09%
L-slep embr- 0,09%
1-step emor 0.04%
L-step ettor 0.03%
40c
400
300
300
optimal value empiniaal average
200
200
200
1.35
99
(a) L= 10
(b) L 20
Figure 6: Histogram of the representation learned by PSD in the Ant environment. The average values 1-step distance (st) - Φ (s and L-step distance [ - (1(++)) converge to their optimal values, indicating that the constraints of the objective JPSD effectively satisfied.
Experimental Setup We evaluate PSD on five robotic locomotion tasks in the MuJoCo envi- ronment [10, 91], both in state and pixel do- main: Ant, HalfChectah, Humanoid, Hopper, and Walker2D (Figures 4 and 7).
Ant
HalfChestah
Humunoid
Hopper
Walker2D
Figure
MuJoCo locomotion environments.
Baselines We compare PSD with the four state-of-the-an unsupervised skill discovery methods: DIAYN [25] is mutual information-bast method that discovers skills by training skill discrimina- tor (=(=|s) to infer the skill from given state. DADS [84], similar to DIAYN, trains skill dynamics go(s' (8,2) 2) increase the mutual dependence between state transitions and the skill, enabling the agent to learn diverse state transitions conditioned on the skill variable 2. CSD [66] and METRA [67] maximizingiiiiigg skill discovery methods. These methods discover skills by maximizing the latent distance traveled in a specific direction of the skill vector z. Specifically, CSD uses controllability-awal distance metric, and METRA uses temporal distance metric.
fall into the category of
Question 1. Can PSD discover diverse periodic skills across multiple timescales?
We first check whether PSD can learn a circular latent space constructed by the encoder d, and whether the skill policy [a s, L) actually produces behaviors with period of 2L across different values of L. Figure shows the trajectories of representative states along with a 2D PCA projection of the corresponding latent trajectories for the Ant and Walker2D environments. For varying values of L, PSD successfully constructs circular latent space whose diameter is proportional to the period variable L, and learns behaviors with the desired period of 2L. example, in the Ant environment with T. 20, we can observe that the resulting behavior completes approximately five full cycles of period 27. (= 40) over 200 timesteps. These results suggest that, by leveraging a circular-shaped latent space, PSD can learn policy that produces behaviors with controllable periodicity.

---

6
(a) Pixel-based observation
(b) Frequency spectrum of skill trajectory
Figure 7: Frequeney spectrum of skills in pixel-based environments. We visualize the pixel- based observations used as input to PSD, along with the resulting frequency spectrum of skill trajectories obtained via Fourier transform. The accompanying bar chart highlights the top-3 frequency components ranked by amplitude.
Table I: Comparison of downstream task performance. We evaluate PSD against existing skill discovery methods. High-level policies are trained using PPO with the skill policies kept frozen, All reported values are average returns over 10 seeds.
Downstream task
DIAYN
DADS
CSD
METRA
PSD (Ours)
HalfCheetahh-hurde
0.6:
0,940.
0.8=06
1,9408
3,8=2.0
Walker2D-hurdle
2.6+0.5
1.9+0.3
4.1=13
3.1+05
5,4
HalfCheetah-frice -rrictionn
13.2.13.
12.4
12.5.38
30.1113.1
13.4
Walker2D-friction 4.6.112
1.611.1
5.3
5.21116
8.7
To quantitatively evaluate whether the learned circular representation satisfies the objective JPSD. we sample Ik transitions from the replay buffer D and assess whether the 1-step constraint in Eq. (6) and the T.et constraint Eq. (5) are approximately satisfied. Figure plots histograms of the 1-step distance ( &u (s++1)] and the L-step distance | (sx) Φ (s)+)))l in the circular latent space, computed over sampled transitions. As shown in the figure, both distances converge closely to their theoretical optima, sin /2L) and L, with a small relative error. This strong alignment between empirical measurements and analytical predictions indicates that the encoder á successfully enforces the geometrie regularity of the circular latent space during training. The full experimental results of Figures and are provided in Appendix D.
Next, we compare PSD with prior skill discovery methods-DIAYS TRA-that aim to learn diverse behaviors via policies of the form xa s, 2), conditioned on different skill variables z. For each method, we uniformly sample 16 skills from its skill prior and roll them out in the environment collect corresponding skill trajectories. For comparison, each trajectory is normalized per dimension using statistics computed from random-action rollouts. The normalized trajectories are then projected to a one-dimensional subspace using Principal Component Analysis (PCA). Finally, we apply Fourier transform to each projected trajectory to analyze its frequency components and extract the four highest frequencies by amplitude, which are visualized as bar charts.
DADS, CSD, and ME-
As shown Figure 3, PSD consistently discovers skills that exhibit wide range of frequencies due to its explicit modeling of circular periodicity. In contrast, distance maximmiin approaches like METRA and CSD tend concentrate on narrow frequency bands and often produce inconsistent, indistinguishable behaviors in Hopper and Walker2D, limiting the diversity of the discovered temporal patterns and frequencies. Also, MI-based methods often produce either static or partially random behaviors, as they do not incorporate the temporal aspects of skills.
Question 2. Are the discovered skills useful for solving downstream tasks?
To evaluate the utility of the discovered skills, conduct downstream experiments by training a high- level policy (L s) (or r² (z s) for baseline methods). For each method, the skill policy is kept frozen, and the high-level policy is trained using PPO [80] to select skills that maximize task-specific rewards. We design downstream environments featuring two types of bhallenges-hurdles and varying ground friction. In the hurdle task, the agent should select skills to jump over the irregularly placed hurdles, which requires adaptive coordination between multi-timeseht task, the agent should select skills to robustly walk across terrain whose surface friction coefficients are randomly assigned. (sce Appendix C for details).
skills. Similarly, in the friction

---

6-
4
2
-10
-10
0
10
2
20
0
20
1.5
2.5
3
(a) Ant
(b) Walker2D
Figure 8: Visualization of traveled distance and frequency spectrum of skills learned via the combination METRA and PSD. Colors indicate skills conditioned on the same value of the period variable L. Videos are available at project page.
Since our method is not explicitly optimized for exploration in the state space, we add an external velocity- reward Text to encourage forward progress. For fair comparison, the same external reward is linearly combined with the intrinsic rewards of all baseline methods. Table shows that PSD outperforms the baselines on most tasks, demonstrating that PSD provides skills that are both adaptable and robust.
Question 3. Iy PSD scalable to high-dimenssonall observations such as pixel-based input?
Since PSD encodes periodicity by capturing temporal distances between states, its latent space is invariant the specific state representation. To validate this, conducted experiments in pixel-based Ant and HalfCheetah environments, as depicted in Figure 7, and found that PSD successfully learns periodic behaviors even from raw pixel inputs. These results demonstrate that PSD generalizes ro- bustly to visual domains without any modification to its objective or reward formulation, highlighting its scalability high-dimensional inputs. Additional analyses are provided in Appendix D.
Question Can PSD become fully unsupervised by combining with other unsupervised methods?
Since FeSD is designed to provide additional variations in the learned behaviors, it could be combined with any type of reward, even with an unsupervised one. To validate it, we combine PSD with METRA [67], enabling fully unsupervised extension. As discussed Section 2, METRA optimizes the following objective:
max Ep(r.e)
Σ (mm 2m (m))) s.t. (10m(s)) (m)()) V(s,s') ) E Sudj, (9)
<
where Φm denotes the encoder used in METRA. METRA discovers exploratory skills that maximally deviate along directions in latent space, while constraining the latent distance between adjacent states to 1, thus capturing temporal distance. PSD naturally aligns with this formulation, as both methods capture temporal aspects of skills: METRA adjusts the temporal direction of skills (i.e., the skill variable 2), whereas PSD modulates their temporal length (i.e., the period variable L). By jointly training both encoders and using the sum of their rewards, we can obtain skill policy that enables independent control over both variables (z and L), as follows:
Tal I.) < arg max E P
Σ 1=
(8m1)))) (m))) exp(=xA(D)3)
(10)
PMETRA
FPSD
In Figure 8, we visualize the traveled XY-coordinates (or X-coordinates) alongside the frequency
spectrum of the corresponding skill trajectories. By adjusting the variables z and L of the policy Ta s, L), the agent can modulate both the movement direction and the period of skills in fully unsupervised manner, yielding a more diverse behavioral repertoire. This result suggests that the temporal property of PSD is orthogonal to the exploratory objectives of METRA, making it a complementary component for constructing fully unsupervised policies. (see Appendix for implementation details)

---

Conclusion
We introduce Periodic Skill Discovery (PSD), framework for unsupervised skill discovery that captures the periodic nature of behaviors by embedding states into circular latent space. By optimiz- ing a constrained objective that encodes temporal distance, PSD enables agents learn skills with controllable periods. Our experiments demonstrate that PSD discovers diverse and temporally struc- tured skills across various MuJoCo environments, and scales to raw pixel observations. Furthermore, combining PSD with METRA leads to richer behaviors by jointly modulating temporal direction and period. Overall, PSD provides scalable and principled framework for discovering temporally structured behaviors in reinforcement learning.
Limitations and Future Work. While our experiments primarily focus on locomotion tasks, due to their suitability for showcasing multi-timescale behaviors, the PSD framework is applicable to any domain that exhibits periodic structure. However, PSD may underperform
settings with large persistent external disturbances (e.g., constant interference from another agent) where periodic behavior becomes infeasible. An interesting future direction is to extend PSD to non non-perionn non-periodic tasks, such robotic manipulation, by generalizing the latent geometry beyond circular structures. Moreover, directly integrating frequency-dome analysis, such as Fourier representations, into the training process could further improve PSD in capturing temporal patterns.
Acknowledgments
We would like to thank Kanggyu Park for his invaluable support, and the anonymous reviewers for their insightful comments. This work was supported by Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFC-IT2402-17.
10

---

References
[1] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 10299, 2018.
[2] Philip Arm, Mayank Mittal, Hendrik Kolvenbach, and Marco Hutter. Pedipulate: Enabling manipulation skills using quadruped robot's leg. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 5717-5723, IEEE, 2024.
[3] Junik Bae, Kwanyoung Park, and Youngwoon Lee. TLDR: Unsupervised goal-conditioned rl via temporal distance- aware representations. arXiv preprint (0XXix:2407.084666 2024.
[4] Victor Barasuol, Jonas Buchli, Claudio Semini, Marco Frigerio, Edson R De Pieri, and Dar- win G Caldwell. A reactive controller framework for quadrupedal locomotion on challenging terrain. In 2013 IEEE International Conference on Robotics and Automation, pages IEEE, 2013.
[5] Philippe Beaudoin, Pierre Poulin, and Michiel van de Panne. Adapting wavelet compression human motion capture clips. In Proceedings of Graphics Interface 2007, pages 313-318, 2007.
[6] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[7] C Dario Bellicoso, Fabian Jenelten, Christian Gehring, and Marco Hutter. Dynamic locomotion through online nonlinear motion optimization for quadrupedal robots. IEEE Robotics and Automation Letters, (1)::2261-2263 2018.
[8] Gerardo Bledt, Patrick M Wensing, Sam Ingersoll, and Sanghae Kim. Contact model fusion for event-based locomotion unstructured terrains. 2018 IEEE International Conference Robotics and Automation (ICRA), pages 4399-4406. IEEE, 2018
[9] Stephen Boyd. Convex optimization. Cambridge UP, 2004.
[10] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. arXiv preprint
2016.
[11] Armin Bruderlin and Lance Williams. Motion signal processing. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 97-104, 1995.
[12] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2018.
[13] Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, Discover and Learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[14] Rafael Cathomen, Mayank Mittal, Marin Vlastelica, and Marco Hutter. Divide, Discover, Deploy: Factorized skill learning with symmetry and style priors. In Conference on Robot Learning, pages 750-768, PMLR, 2025.
[15] Felix Chalumeau, Raphael Boige, Bryan Lim, Valentin Macé, Maxime Allard, Arthur Flajolet, Antoine Cully, and Thomas Pierrot. Neuroevolution is a competitive alternative to reinforce- ment learning for skill discovery. In The Eleventh International Conference Learning Representations, 2022.
[16] Konstantinos Chatzilygeroudis, Antoine Cully, Vassilis Vassiliades, and Jean Baptiste Mouret. Quality-diversin optimization: a novel branch of stochastic optimization. In Black box optimization, machine learning, and no-free lunch theorems, pages 109-135. Springer, 2021.
[17] Daesol Cho, Jigang Kim, and H Jin Kim. Unsupervised reinforcement learning for transferable manipulation skill discovery. IEEE Robotics and Automation Letters. 7(3):7455-7462 2022.

---

[18] Daesol Cho, Seungjae Lee, and H Jin Kim. Diversify & Conquer: Outcome-direeted curriculum r1 via out- -fittiibbbiitni disagreement. Advances in Neural Information Processing Systems, 36:53593-5366 2023.
[19] Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directed reinforcement learning by uncer- tainty & temporal aware curriculum goal generation. In The Eleventh International Conference on Learning Representations, 2023.
[20] Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Varia- tional empowerment as representation learning for goal-based reinforcement learning. arXiv preprint arXiv:2106.01400) 2021.
[21] Antoine Cully. Autonomous skill discovery with quality- and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 81-89, 2019.
[22] Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245-259, 2017.
[23] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in neural information processing systems, 32, 2019.
[24] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Advances in Neural Information Processing Systems, 34:27813-27825, 2021.
[25] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representations, 2018
[26] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-Learning: Learning to achieve goals via recursive classification. In International Conference on Learning Represen- tations, 2020.
[27] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Process- ing Systems, 35:35603-35620, 2022.
[28] Claire T Farley and C Richard Taylor. A mechanical trigger for the trot-gallop transition in horses. Science, 253(5017):3063300 1991
[29] Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Ried- miller. Self-supervised learning of image embedding for continuous control. arXiv preprint arXiv:1901.00943. 2019.
[30] Justin Fu, John Co-Reyes, and Sergey Levine. EX2: Exploration with exemplar models for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[31] Michael C Granatosky, Caleb M Bryce, Jandy Hanna, Aidan Fitzsimons, Myra F Laird, Kelsey Stilson, Christine E Wall, and Callum F Ross. Inter-stride variability triggers gait transitions mammals and birds. Proceedings of the Royal Sociery B, 285(1893):2018)
2018.
[32] Luca Grillotti and Antoine Cully, Unsupervised behavior discovery with quality-diversity optimization. IEEE Transactions on Evolutionary Computation, 26(6):1539-1552 2022.
[33] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870 PMLR, 2018.
[34] Steven Hansen, Will Dabney, Andre Barreto. Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. 2019.
[35] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. International Conference on Learning Representations, 2019.
12

---

[36] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pages 2681-2691. PMLR, 2019.
[37] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages (850-6681, 2020.
[38] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[39] Donald F Hoyt and C Richard Taylor. Gait and the energetics of locomotion in horses. Nature, 292(5820):239-244 1981.
[40] Edward S Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. In The Eleventh International Conference on Learning Representations, 2023.
[41] Jiaheng Hu, Zizhao Wang, Peter Stone, and Roberto Martín-Martín Disentangled unsuper- vised skill discovery for efficient hicrarchical reinforcement learning. Advances in Neural Information Processing Systems, 37:76529-76552, 2024.
[42] Zheyuan Jiang. Jingyue Gao, and Jianyu Chen. Unsupervised skill discovery via recurrent skill training. Advances in Neural Information Processing Systems, 35:39034-39046, 2022.
[43] Plerad-Alexandr Kamienny, Jean Tarbouriech, Sylvain Lamprier, Alessandro Lazaric, and Ludovic Denoyer. Direct then Diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. arXiv preprint arXiv:2110.144: 2021.
[44] Hyunseung Kim, Byung Kun Lee, Hojoon Lee, Dongyoon Hwang, Sejik Park, Kyushik Min, and Jaegul Choo. Learning to discover skills through guidance. Advances in Neural Information Processing Systems, 06:28226-28254, 2023.
[45] Hyunseung Kim, Byungkun Lee, Hojoon Lee, Dongyoon Hwang, Donghu Kim, and Jaegul Choo. Do's and Don't ts: Learning desirable skills with instruction videos. Advances in Neural Information Processing Systems, 37:47741-47708 2024.
[46] Jackyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. In International Conference on Machine Learning, pages 5572-5582. PMLR, 2021.
[47] Yunho Kim, Hyunsik Oh, Jeonghyun Lee, Jinhyeok Choi, Gwanghyeon Ji, Moonkyu Jung, Donghoon Youm, and Jemin Hwangbo. Not only rewards but also constraints: Applications on legged robot locomotion. IEEE Transactions on Robotics, 2024.
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[49] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: representations for reinforcement learning. In International conference on machine learning. pages 5639-56500 PMLR, 2020.
Contrastive unsupervised
[50] Michael Laskin, Hao Liu, Xue Bin Peng. Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CIC: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161, 2022.
[51] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1 1(4):541-551, 1989.
[52] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xea 2020.
13

---

[53] Sang-Hyun Lee and Seung- Woo Seo. Unsupervised skill discovery for learning shared structures across changing environments. International Conference on Machine Learning. pages 19185-19199. PMLR, 2023.
[54] Seungjae Lee, Daesol Cho, Jonghae Park, and H Jin Kim. CQM: Curriculum reinforeement learning with quantized world model, Advances in Neural Information Processing Systems, 36:78824-78845, 2023.
[55] Joel Lehman and Kenneth O Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary compulation, pages 211 1-121, 2011.
[56] Chenhao Li, Flijah Stanger-Jones, Steve Heim, and Sang bae Kim. FL.D: Fourier latent dynamics for structured motion representation and learning. In The Twelfth International Conference on Learning Representations, 2024.
[57] Zicheng Liu, Steven J Gortler, and Michael F Cohen. Hierarchical spacetime control. In Proceedings of the st annual conference on Computer graphics and interactive techniques, pages 35 42, 1994.
[58] Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895-10904, 2023.
[59] Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreog- rapher: Learning and adapting skills in imagination. In The Eleventh International Conference Learning Representations, 2022.
[60] Russell Mendonea, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 24:24,,9-24399, 2021.
[61] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04900 2015.
[62] Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learn- ing temporal distances: Contrastive successor features can provide metric structure for decision-making. In International Conference on Machine Learning, pages 37076-37096.
PMLR, 2024.
[63] Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 866-875, 2021.
[64] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. based exploration with neural density models. International conference on machine learning, pages 2721-2730. PMLR, 2017.
[65] Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Represen- tations, 2022.
[66] Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability-aware unsuper- vised skill discovery. In International Conference on Machine Learning, pages 27225-27245. PMLR, 2023.
[67] Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable unsupervised rl with metric- aware abstraction. In The Twelfth International Conference on Learning Representations, 2023.
[68] Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert repre- sentations. International Conference on Machine Learning, pages 39737-39761. PMLR, 2024.
14

---

[69] Xue Bin Peng, Yunrong Guo. Lina Halper, Sergey Levine, and Sanja Fidler. ASE: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 1-14 2022.
[70] Thomas Pierrot, Valentin Macé, Felix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Perrin-Gilbert. Diversity policy gradient for sample efficient
qualiiiie quality-diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1075-1083, 2022.
[71] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International conference machine learning, pages 7750-7761. PMLR, 2020.
[72] Vitchyr Pong. Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference mod- els: Model-free deep rl for model-based control. In International Conference on Learning Representations, 2018.
[73] Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-Fit: State-covering self-supervised reinforcement learning. In International Conference on Machine Learning, pages 7783-7792. PMLR, 2020.
[74] Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
[75] Seungeun Rho, Laura Smith, Tianyu Li, Sergey Levine, Xue Bin Peng, and Sehoon Ha. Language guided skill discovery. In The Thirteenth International Conference on Learning Representations, 2024.
[76] Seungeun Rho, Kartik Garg, Morgan Byrd, and Sehoon Ha. Unsupervised skill discovery as exploration for learning agile locomotion, In Conference on Robot Learning, pages 2678-2694. PMLR, 2025.
[77] Ludovic Righetti and Auke Jan Ijspeert. Pattern generators with sensory feedback for the control of quadruped locomotion. In 2008 IEEE International Conference Robotics and Automation, pages 819-824. IEEE, 2008
[78] Felix Ruppert and Alexander Badri-Spröwitz. Learning plastic matching of robot dynamics in closed-loop central pattern generators. Nature Machine Intelligence, 4(7):652 660, 2022.
[79] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative reinforcement learning. arXiv preprint arXiv:1703.03864 2017.
[80] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: /707.007777 2017.
[81] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference on machine learning, pages 8583-8592. PMLR, 2020.
[82] Milad Shafice, Guillaume Bellegarda, and Auke Ijspeert. Viability leads to the emergence of gait transitions in learning agile quadrupedal locomotion on challenging terrains. Nature Communications, 15(1):3073, 2024.
[83] Yecheng Shao, Yongbin Jin, Xianwei Liu, Weiyan He, Hongtao Wang, and Wei Yang. Learning free gait transition for quadruped robots via phase-guided controller. IEEE Roboties and Automation Letters, 1230) 1237, 2021.
[84] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representa- tions, 2019.
[85] Pranav Shyam, Wojeiech Jaškowski, and Faustino Gomez. Model-based active exploration. In International conference machine learning, pages 5779-5788, PMLR, 2019.
15

---

[86] Jonah Siekmann, Yesh Godse, Alan Fern, and Jonathan Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 7309-7315. IEEE, 2021.
[87] Alexander Spröwitz, Alexandre Tuleu, Massimo Vespignani, Mostafa Ajallooeian, Emilie Badri, and Auke Jan ljspeert. Towards dynamic trot gait locomotion: Design, control, and experiments with cheetab-cub, a compliant quadruped robot. The International Journal of Robotics Research, (2(8):932-950) 2013.
[88] Sebastian Starke, Ian Mason, and Taku Komura. DeepPhase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
[89] DJ Strouse, Kate Baumli, David Warde-Farley, Volodymyr Mnih, and Steven Stenberg Hansen. Learning more skills through optimistic exploration. In International Conference on Learning Representations, 2021.
[90] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # EXPLORATION: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[91] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model- control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[92] Munetoshi Unuma, Ken Anjyo, and Ryozo Takeuchi. Fourier principles for emotion- human figure animation. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 91-96, 1995.
[93] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017.
[94] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pages 3641 1-36430. PMLR, 2023.
[95] Zizhao Wang, Jiaheng Hu, Caleb Chuck, Stephen Chen, Roberto Martin-Martín, Amy Zhang, Scott Niekum, and Peter Stone. SkiLD: Unsupervised skill discovery guided by factor interactions. Advances in Neural Information Processing Systems, 37:77748-77776, 2024.
[96] Rushuai Yang. Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang. Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. In International Conference on Machine Learning, pages 39183-311 39204. PMLR, 2023.
[97] M Ersin Yumer and Niloy Mitra. Spectral style transfer for human motion between indepen- dent actions. ACM Transactions on Graphics (TOG), 35(4): 2016.
[98] John Z Zhang, Shuo Yang, Gengshan Yang, Arun L Bishop, Swaminathan Gurumurthy, Deva Ramanan, and Zachary Manchester. SLoMo: A general system for legged robot motion imitation from casual videos. IEEE Robotics and Automation Letters, 8(11):7554-716) 2023.
[99] Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. In The Twelfth International Conference on Learning Representations, 2023.
[100] Chongyi Zheng, Jens Tuyls, Joanne Peng, and Benjamin Eysenbach. Can MISL fly? analysis and ingredients for mutual information skill learning. In The Thirteenth International Conference on Learning Representations, 2024.
16

---

A Theoretical Results
Theorem 1. Given a positive integer Or, an optimal solution to Jrsp and only if it forms regular gon of diameter L centered at the origin.
Proof. We prove the claim by showing both directions of the equivalence.
(=) Suppose Φr, forms regular 2L-gon of diameter L centered at the origin. Then, for all (1,81, 5t-1, 114 E D, we have ( 2.) - or (8))l22 = L and Φr. = - or. (8.). which implies (1++()))t) + (xe)21 0. Thus, the objective becomes JESD E k. 0] = L. Since Eq. (5) requires (8++L - Φr. (*))]l2 & L,, any feasible solution must satisfy JpSD S T,, and no higher value can be attained. Under this condition, the given regular 2L-gon satisfies both constraints
in Eq. (5) and Eq. (6). Hence, ØL is optimal.
(+) Suppose ΦL is optimal and achieves JPSD L. Then (+++L) (2(s))l2 L and [lo ($++L) + (s)]l2 = O, implying Φ (5(++)) = - (st) and (L(s)) lies on hypersphere of radius L/2 centered (8++1) (s, S L sin (x/2L), which implies that the maximum angular distance between adjacent points L/L. Under this condition, reaching the antipodal point dr. (8++ starting from @b. (8+) is only possible the points are equally spaced along the cireumference of a great circle the hypersphere, with angular distance of exactly n/ between adjacent points.
the origin for all (1, $141, S(+L) D. From Eq. (6), we have
(st)
Hence, or forms a regular 2L-gon of diameter L centered at the origin.
B Extended Related Work
PSD primarily falls into the category of unsupervised skill discovery methods [1, 25, 84, 34, 13, 89, 20, 43, 46, 59, 50, 17, 42, 65, 69, 66, 53, 96, 44, 67, 45, 3, 75, 100, 41, 95, 14, 76], which aim to learn a diverse set of skills without external rewards, and the resulting skills can be effectively adapted to downstream tasks leveraged for high-level planning. In this regard, these methods also share conceptual similarities with Quality-Diversity (QD) algorithms [55, 61, 74, 22, 79, 21, 16, 63, 70, 32, 15], evolutionary optimization framework that iteratively explores and refines diverse behavioral patterns without external task rewards, using behavior descriptors as implicit objectives to guide exploration. Both frameworks aim to discover a broad repertoire of distinct and high- sigh-performin behaviors by optimizing for diversity rather than maximizing single task lask-specific reward.
Discovering these diverse and useful repertoires inherently requires agent to explore the environ- ment broadly and encourage skills to visit a wide range of states. This emphasis on broad exploration shows strong connection exploration methods [38, 6, 64, 90, 30, 36, 85, 12, 73, 81, 71, 60, 40, 19, 54, 18] that explicitly aim to maximize state coverage through intrinsic rewards. From this perspective, the PSD framework can also be viewed exploring the environment in the frequency domain (see Figure 3), as it learns diverse periodic behaviors across multiple timescales in an unsupervised manner through adaptive sampling method.
Moreover, since PSD learns
latent representation where distances between states capture their temporal relationships. it is closely related to prior works [72, 29, 23, 35, 26, 27, 94, 67, 68, 62, 99] that aim to encode temporal structure in RL representations.
17

---

C Experimental Details
C.1
Environments
MuJoCo locomotion environments We adopt MuJoCo environments including Ant, HalfCheetah, Humanoid, Hopper, and Walker2D [10, 91] to evaluate our method and baselines. Episode lengths are set to 200 timesteps for Ant and HalfCheetah, and 400 timesteps for Humanoid, Hopper, and Walker2D. For state-based observations, we follow the default Gym setting [10]. which includes proprioceptive information in the observation space. However, since both CSD and METRA rely on global position information to construct their latent representations, we include the global position in the observation when applying these methods, following the setups described in their original papers. For the pixel-based experiments of PSD, we use 90 x 90 x 3 RGB images captured from tracking camera (view shown Figure 7) as input to both the RL agent and the encoder Φ, without incorporating any additional proprioceptive information.
Downstream tasks environments In the HalfCheetah- hurddle and Walker2D-hurdle environments, the high-level policy receives a reward whenever the agent successfully jumps over a hurdle. For HalfCheetah-huree the hurdle positions are [2.5, 4.0, 7.0, 10.0, 15.0, 22.0, 30.0]. with a height of 0.26, which is higher than the setting used in METRA [67]. For Walker2D-hurdie positions are [1.2, 2.7, 5.8, 7.0, 9.2, 11.0, 12.8, 14.2), with a height of 0. both environments, the hurdles are unevenly spaced, requiring multi-timessale coordination for successful locomotion. We provide the distance to the nearest hurdle as part of the task-specific input Suck to the high-level policy. The episode lengths are set to 300 timesteps for SelfChhhcaah-hhree and 600 timesteps for Walker2D-hurdle
the hurdle
In the SorCCheetah-frictiin and Dalker2D-fricttin environments, the agent is rewarded for main- taining forward velocity, which encourages robust locomotion while avoiding falls under changing friction conditions. For implementation simplicity, we do not modify the ground friction directly. Instead, we sequentially change the friction parameters of the agent's s feet in the XML file every 100 timesteps, cycling through the values [0.5, 1.5, 2.0], given that the default friction parameter in MuloCo 1.0. Additionally, the current friction coefficient is provided as task-specific input 8isk to the high-level policy, enabling it to adapt to the changing frictions. The episode lengths are set to 500 for both HalfCheetah-friction and Walker2D-friction.
C.2 Implementation Details
We implement PSD on top of the publicly available PyTorch SAC implementation' For fair compari- son, we implement all baseline methods within the same codebase as PSD ensure consistency in training procedures and infrastructure. To train the high-level policy for downstream tasks, we use PPO implemented in public PyTorch repository". All experiments are conducted on an NVIDIA A6000 GPU, and training for each task typically completes within 24 hours.
Training PSD For training PSD, we uniformly sample four discrete values of the period variable from the range [mmin Lmax], including both bounds, to ensure coverage of the range while maintaining training efficiency. As shown Appendix D.2, we found that this sampling strategy is sufficient, as the model is able to generalize to intermediate L values via interpolation.
Additionally, since T, is scalar value, directly feeding it into the encoder ő, the policy and the Q-function may limit the representational capacity of these networks. To address this, we apply sinu- soidal positional embeddings commonly used in transformers [93] and diffusion models [37] to project into a higher-dimensional space. As an example, rather than using L directly in the policy in the form of <[a 8, L). we use (a 5, Embed(L)), where (inbed()) denotes the embedded representation of L as follows:
[snn[ wi) if mod 2 = 0, cos ( w;)
Embeed(L) = [eo, G1, ep-1). where e;
if mod 2 = 1,
18

---

where the frequency term w; is defined as wi = 10000 2////// We apply this sinusoidal embedding to the period variable L whenever it is used as input the network, enabling the model to better distinguish and generalize across different temporal scales. In addition, we found that using fixed values for Å and X2 works well in practice, when optimizing FrsD via dual gradient descent method. The full set of hyperparameters is summarized in Table 2.
Training baseline methods For baseline methods, we closely followed the implementation details described in their original papers. For METRA, we use 2-dimensional continuous skill vector E IR2 for Ant and Humanoid, and 16-dimensional discrete skills for other environments. CSD, 16-dimensional discrete skill vector is used for all environments. For both METRA and CSD, continuous skills are sampled from a standard Gaussian distribution and normalized to have unit norm, and discrete skills are designed to be zero-centered one-hot vectors. For DADS, we use 2- dimensional continuous skills sampled from the uniform range [-1, 1]a for Ant and Humanoid, and 16-dimensional
16-dimensional one-hot vectors for the remaining environments. In DIAYN, one-hot vectors across all environments. For training the low-level policy for downstream tasks, we use 16-dimensional discrete skills for all baseline methods.
Adaptive sampling method For the adaptive sampling method, we evaluate the periodic skill policy conditioned on the current boundary periods every Ik episodes. To measure performance, roll out episodes and compute the average cumulative sum of TesD. defined as Rg Bp(r(r) Ep(r E rysp]. For the threshold coefficients a and 3, we found that setting a 0.9 and 3 = 0.4 works well in practice. To avoid abrupt narrowing the radius range in the early stages of training, each bound is allowed to shrink only after it has first been expanded, i.e., after Rg > 0.97 has been satisfied at least once. The full algorithm is described in Algorithm 2, and a complete list of hyperparameters provided in Table 2.
Training PSD with pixel-based observations For experiments using pixel-based observations, we use CNN-based encoder [51] to process visual inputs. To capture temporal continuity, we concate- nate consecutive frames as input. We also apply random cropping as form of data augmentation, following CURL. [49]. We found that action repeat was not necessary to achieve stable training in our setup. A complete list of hyperparameters is provided in Table 3.
Task-specint reward Since PSD is designed enrich the agent's behavior with additional diversity while still achieving the primary task, we optionally combine the velocity-basse external reward Text with the intrinsic reward TPSD-
Given that FPSD exp( -x A is bounded in the range (0, 1], we design the external reward also have an upper bound of 1, ensuring balanced contribution of both rewards when the agent reaches optimal performance, as follows:
1.0
i 2 of
Text Tea( (vx) =
otherwise
This reward function assigns Tex = 1.0 when the agent's 's forward velocity er exceeds the threshold 8%, and increases linearly as Vr approaches the threshold from below. We set ey 0.5 for Ant and HalfCheetah, and oy_ = 1.0 for Humanoid, Hopper, and Walker2D.
Training the high-level policy for downstream tasks For downstream tasks, we train the high- level policy using PPO [80] while keeping the low-level skill policies frozen. This training utilizes task-specific reward and an additional observation, Susk, which are detailed in Appendix C.I. In all experiments, the high-level policy, >²' (T. 8msk, s) (or <> 8msk, 8) for baseline methods), selects skill every H environment steps, and the low-level policy then executes this skill for the subsequent H steps. The high-level policy is trained for 100k episodes using the hyperparameters listed in Table 4.
19

---

Training PSD combined with METRA METRA [67] learns an encoder &m and a skill policy [(a]
8, 2) that encourages transitions to deviate maximally along latent directions 2, while constraining the one-step latent distance to capture temporal coherence. As described in the original paper [67],
the constrained objective Eq. (9) optimized by maximizing the following components:
JMETRA. = (),s',,)~> [(0m(s') - om (s)) 2 + Am min (c. 1 - (l0m(s)) - 6m (s) (11)
JMETRA,A,, = Am (=) min (c, 1 - @l (s') pm (s) [s)
(12)
where Am is a Lagrange multiplier, updated during training via the dual gradient method to enforce the constraint.
naïve combination of PSD and METRA MEiRA-training their encoders independently and simply summing their intrinsic rewards-fails in practice. As explained in Section 2, the METRA objective strongly favors skills with the shortest possible period. This because shorter periods typically correspond to faster motions, which lead to larger per-step deviations in the latent space and thus yield higher values of (om (5') - (s) 2. Consequently, all discovered skills collapse into a single short-periodie behavior, undermining the diversity of the learned skill of PSD.
To address this issue, we condition each encoder on the other method's s skill variable by incorporating it as an additional input to the state. Specifically, we augment the input to ΦL with the skill variable from METRA, and the input to 6m with the period variable L from PSD, resulting in:
(2(88 &i (8) (m, L.).
This mutual conditioning allows each encoder to account for the temporal properties imposed by the other method, thereby regularizing their joint optimization and preventing skill collapse, For example, from the perspective of training (8, L). the METRA objective in Eqs. (11) and (12) encourages latent representations that exhibit large per-step deviations in the latent space while satisfying the periodicity determined by L.
By jointly optimizing both encoders with this conditioning, we obtain a policy π T(a s, 2, L) that can independently modulate both the temporal direction (i.e., the skill variable 2) and the temporal length (i.e., the period variable L.) in fully unsupervised manner. The full algorithm is described in Algorithm 3 and a complete list of hyperparameters is provided in Table
Latent Space Dimensionality As summarized in Table 2, we used (3, 6) -dimensional latent spaces across all embodiments, which we found to work well in practice. In contrast, a 2-dimensional latent space (i.e., a plane) led unstable performance for complex agents such as Ant or Humanoid. We hypothesize that, since the PSD) objective does not explicitly constrain the latent circles for each I, to lie on the same plane, having additional degrees of freedom allows different circles to occupy different planes. This, in turn, helps stabilize the embedding during training. Conversely, when the latent space restricted to only dimensions, this flexibility lost, which leads to instability.
C.3 Visualizations
PCA visualization for latent space As described in Section 3.2 and Appendix C.2, the circular latent space of PSD is not necessarily 2-dimensional. Given the PSD formulation, we map states to latent vectors (s) with 3 or more dimensions in practice to better capture periodicity. To visualize this circular latent space, as shown in Figure and our video, we apply Principal Component Analysis (PCA) to obtain 2-dimensional projection that clearly illustrates the underlying circular structure.
20

---

C.4 Full Algorithm of Adaptive Sampling Method
Algorithm 2 Adaptive Sampling Method
1: Initialize: policy π, encoder @, current sampling bound Tmin; Tmmax
2: updated_once _asi___ne____ r_menmmnee False
3: updated Sellated_one_eeee Lonce_mat max +- False
// Evaluate current bounds at Timin and Lemax
4: for each evaluation episode do
5: for L E {Lmin Lmax] do
6: Execute कa 8, L) for the entire episode
7: Compute cumulative reward 2t TPSD L))
8: end for
9: end for
10: Compute average reward Rimm. L
// Update current bounds
11: if RL > aT then
12: Limin f Limn -N opdated_once____ee
13:
<- True
14: end if
15: if R R > aT then
Limax L-max N
17:
updated_once__men +- True
18: end if
19: if RL < BT and updated once min True then
Liain Liiu +
21: end if
If RLLmxx BT and updated Cectlateddocce_eeeee True then
23:
Linxx Lmax -
end if
25:
return Lmiu: Lmax
C.5 Full Algorithm of PSD Combined with METRA
Algorithm 3 PSD combined with METRA
1: Initialize: policy PSD encoder é, METRA encoder om. sampling bound Lmin.max. replay
buffer D, Lagrange multiplier 21,2.
2: for each training epoch do
3: Update Lmins Lmax if AdaptiveSampling is enabled
// Environment interaction
4: for each episode the epoch do
5: Sample L ~ p(L) where L [Lmiin Imnax]
6: Sample 2 ~ p(z) where p(z) the skill prior of METRA
7: Execute <a s, L) for the entire episode, and store transitions (z, L, St, ag Se41) in D
8:
end for
// Update encoders and RL network
9: Update (1, 2) by maximizing JPSD. using samples (z, I,, St, 8t+1) from D
Update @im(s L.) by maximizing IMETRA,Om,... using samples (2, I, St, St+1, 81+4) from D
11: Compute intrinsic reward FPSD
Compute intrinsic reward 7METRA
13:
Update π (a 8, x, L) with OpSD 7PSD + "METRR using SAC
end for
21

---

C.6 Hyperparameters
Table Hyperparameters for training PSD.
Parameter
Value
Learning rate Discount factor 7
x 10
0.99
Optimizer
Adam [48] 8
of episodes per epoch
N of gradient steps per epoch
64
Replay buffer size
5 x 10
Minibatch size
1024 (ΦL). 256 (others)
Target smoothing coefficient
0.995
Entropy coefficient
Auto-tuned
Circular latent dimension d
{3, 6)
Output dimension of the positional encoding D
TeSD
10
JPSD
10-5
JPSD
0.5
5 (Ant, HallCheetahh, 10 (Humanoid, Hopper, Walker2D)
JPSD A1
5 (Ant, HalfCheetah),
JPSD X2
10 (Humanoid, Hopper, Walker2D)
N of hidden layers of hidden units per layer Step size of adaptive sampling N Adaptive sampling interval
2
1024
1 2000 episodes
N of evaluation episodes for adaptive sampling
5
Thresholds (a, (3) for adaptive sampling
(0.9, 0.4)
Table 3: Hyperparameters for training PSD with Pixel-based observation (others same as Table 2).
Parameter
Value
Replay buffer size
x 10
Minibatch size
512 (.)r.) 256 (others)
Circular latent dimension d Output dimension of the positional encoding D
[3, 6} 128
JPSD A
5 (Ant), 3 (HalfCheetah) (Ant), (HalfCheetah)
JPSD Aa
Encoder
CNN [51]
Random crop N of stacked frames
90 x 90 x + 84 x 3
of action repeat
22

---

Table 4: Hyperparameters for training the high-level policy using PPO.
Parameter
Value
Learning rate
3 10 (actor), 10 (critic)
Discount factor
0.99
Optimizer
Adam [48]
Skill duration H
10
N of episodes per epoch
N of gradient steps per epoch
80
Batch size
256
PPO clipping parameter
0.2
N of hidden layers
N of hidden units per layer
256
Table 5: Hyperparameters for training PSD Combined with METRA.
Parameter
Value
Learning rate
x 10-4
Discount factor 7
0.99
Optimizer
Adam [48] 8
N of episodes per epoch
N of gradient steps per epoch
64
Reward coefficient apsD
1.0
Replay buffer size
5 x 10
Minibatch size
1024 (Φt), 256 (others)
Target smoothing coefficient
0.995
Entropy coefficient
Auto-tuned
Skill dimension of METRA
2-D cont. (Ant), 1-D cont. (Walker2D)
Circular latent dimension
(Ant), (Walker2D)
Output dimension of the positional encoding D
6
FPSD K
10
JPSD
10-5
JeSD k
0.5 (Ant), (Walker2D)
JPSD A1
JPSD A2
(Ant), 10 (Walker2D)
IMETRA E
10 3
IMETRA Am
30
of hidden layers
2
N of hidden units per layer
512 (Ant), 1024 (Walker2D)
23

---

D Additional Experimental Results
D.1
Evolution of Learned Bounds via Adaptive Sampling Method
Bound Values
40
Ant
Ok
50k
100k
150k
200k
250k
300k
20
Bound Values
HalfCheetah
0k
50k
100k
150k
200k
250k
300k
60
Bound Values
Humanoid
0K
50k
100k
150k
200k
250k
300k
60
Bound Values
Hopper
0K
50k
100k
150k
200k
250k
300k
Bound Values
60
Walker2D
0k 50k
100k
150k
200k
250k
300k
Episode
Figure 9: Evolution of the Lmins Lmax during training, The figure shows how the period variable Liinn Lmax evolves over training episodes with the adaptive sampling method applied to the Ant, HalfCheetah, Humanoid, Hopper, and Walker2D environments. As training progresses, increasingly challenging periods are proposed the agent based on the average cumulative sum of resp, enabling the discovery of wider range of periodic behaviors.
In Figure 9, we visualize the evolution of the sampling range of during training with the adaptive sampling method. To prevent the period variable Lmins which must be positive integer, from becoming too small, we set the minimum value Lmin = 5.
Although training begins with single period value, the adaptive sampling method gradually proposes more challenging periods, enabling the agent to acquire skills across a broad range of dynamically feasible period lengths. Moreover, since training is conducted with the combined reward of Tex and TesD (as described in Appendix C.2), the agent learns to maintain a velocity above the target of while acquiring maximally diverse skills optimize the overall reward. A notable property of this method that even a proposed period initially rejected due to low performance, the agent may later learn to handle it training progresses. Overall, this method enables PSD autonomously discover wide range of periodic behaviors without requiring prior knowledge of agent-specific period scales.
24

---

D.2 Skill Interpolation
Ant
Walker2D
L=10
=22
250
502
500
L=12
L
L=13
L626
500
L=14
L=30
L=16
L=33
L=19
L-37
100
200
302
400
500
L=20
L=40
Figure 10: Trajectories of the skill policy (a 8, L) under different values of L. The figure shows representative joint trajectories of Ant (leff) and Walker2D (right) generated by the skill policy (aa s, L) under different values L. The blue trajectories are rollouts of the policy conditioned the final sampling candidates after convergence, while the magenta intermediate integer values between these candidates. Although the magenta trajectory does not perfectly satisfy the 2L periodicity, it still generalizes well, indicating that sampling strategy is effective and the circular representation PSD generalizes across diverse periods.
generated using
D.3 Additional Analysis of Pixel-based Observation Experiments
Comparing Figure 3 and Figure 7, we observe that the pixel-based HalfCheetah exhibits narrower and higher-frequency periodic behaviors than its state-based counterpart, despite having identical robot dynamics. We hypothesize that this due to the inability to differentiate periodic variations in the vertical direction from pixel observations. As shown in video, it is difficult to perceive z-axis variations from raw images, and this issue is exacerbated by random cropping. In contrast, in Mtttebbaed settings, the z-coordinate is explicitly provided the first dimension of the observation vector, making it easier for the neural network to recognize vertical changes. This suggests that the ability to represent vertical height enables PSD to learn longer-perion behaviors (e.g., jumping) in the state-based setting, as also visually demonstrated the video. On the other hand, the Ant environment, both basel and state-based observations provide similar levels of information, and thus result in similar walking behaviors.
25

---

D.4 Full Experimental Results of Figure 5
10
30
40
State value
10
1. 30
40
State valua
200
400
600
803
1,000
1200
1400
timestep
(a) Ant
1.
15
State value
400
500
600
702
L
L
L 11
L 15
State value
200
200
300
400
500
500
705
timestep
(b) heetah
State value
800
1000
1700
10
L 20
L 30
L 40
State value
200
400
600
800
1000
1200
1400
timestep
(c) Humanoid
26

---

10
30
State value
800
1020
1.
10
20
L 30
40
State value
200
400
620
8000 timestep
:000
1200
2405
(d) Hopper
L. =010
20
30
40
State value
200
400
690
810
1030
1200
1405
10
L
30
State valuue
820
8100
10000
1202
timestep
(e) Walker2D
Figure 11: Full experimental results of Figure 5. The figure shows the state trajectories and the 2D PCA projection of their latent encodings learned by PSD. Within single episode, we switch the period variable L at fixed time intervals.
27

---

D.5 Full Experimental Results of Figure 6
Relelive Error [%)
1-step distance
L-step distance
Ant
10
Relalive Error [96]
HalfCheetah
11
15
Relative Error
Humanoid
10
Relalive Error 1%6]
Hopper
10
20
30
40
Relative Eiror [96]
Walker2D
10
20
30
value
Figure 12: Full experimental results of Figure seeds). We compute the relative error of
learned temporal distance Rel. Error
Empirical average- Optimal value
100 [%].
Optimal value
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction match the contributions described and validated throughout the paper (see Introduction and Experiments).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our limitations are discussed in the Conclusion and Appendix, including generalization and scalability.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't secnowleddger in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: All theoretical assumptions and proofs are included in Appendix A, with a formal statement and verification of the main objective.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe necessary details for reproducing our experiments, including architecture, training procedure, and hyperparameters (see Experiments and Appendix C).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
30

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release the code with instructions and scripts to reproduce key experi-
ments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
conolicyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //npss wwtppblccccc//cccccccccccccc...e /ellic//uidee wtwggguidsss//e https//oooSSSSmsss...o..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and evaluation details including model architecture, optimizer, hyperparameters, and sampling procedures are provided in Experiments and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the standard deviation across multiple seeds in Table in the Experiment, as an estimate of variability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer:
[Yes]
Justification: We provide details on the computing resources used in our experiments in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https htt////nurrrp.rpeee wttppbbicc/tthcccsuddeeeeeeeeeed
Answer: [Yes]
Justification: Our research adheres to the NeurlPS Code of Ethics. It does not involve human subjects, sensitive data, or potentially harmful applications.
Guidelines:
The answer NA
that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: Our work is research in unsupervised reinforcement learning and does not involve real-world deployment, human subjects, or safety-critical systems. We found no direct societal impact based on current usage.
32

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve any high-risk models datasets. The experiments are conducted entirely in simulated environments and pose no foreseeable misuse concerns.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets (e.g., MuJoCo environments, baseline methods) are properly cited with their original papers and used according to their licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
33

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release code assets implementing PSD. The repository includes documen- tation, usage instructions.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mmoel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve any human subjects, and therefore no IRB or equivalent approval is required.
Guidelines:
34

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our research does not involve LLMs in the core method, experiments, or results. Any LLM-assisted editing, if used, did not influence the scientific content.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: : http://urrrppscccooeeeeeeeeeeeee c for what should or should not be described.
35


Reviews:
[{"review_id": "Tz7x11LNtJ", "reviewer": "Reviewer_B3mW", "summary": "In this paper, the authors propose Periodic Skill Discovery (PSD), a novel framework for unsupervised skill discovery in reinforcement learning that explicitly targets the learning of periodic behaviors. PSD maps states into a circular latent space, where movement along the circle corresponds to temporal progression, allowing the agent to represent and generate skills with controllable periodicity. The method involves jointly training an encoder that enforces geometric constraints in the latent space and a policy that maximizes a single-step intrinsic reward based on deviation from ideal circular transitions. The experiments evaluate the performance of the proposed method in various locomotion tasks, present ablation studies, and performance comparison with relevant existing techniques.", "questions": "1. How many distinct skills does PSD learn in practice? Is the number fixed, or is it implicitly determined by the range of the period variable? How does this number affect the downstream policy’s ability to select useful behaviors? \n2. How were the trajectories presented in Figure 5 selected?\n3. For the results shown in Table 1, were all methods trained with the same number of environment steps for both skill discovery and downstream high-level policy learning? How were hyperparameters selected across methods? Also, how were the skill sets chosen for the downstream tasks—did the high-level policy have access to the full set of discovered skills, or only a subset? \n4. The authors mention that an exploration reward was added to all methods for fair comparison. However, could this uniform addition unintentionally bias the results? Specifically, for methods already incentivizing exploration, could the added reward lead to excessive exploration or destabilize skill quality?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper introduces a compelling, and novel, idea of capturing periodic behaviors in unsupervised reinforcement learning. \n\nThe methodology is clearly presented, with helpful visualizations and geometric intuition that make the latent structure easy to understand.\n\n\nThe authors demonstrate the effectiveness of PSD across multiple MuJoCo tasks and provide comparisons to several baselines, both in terms of skill diversity and downstream task performance.\n\nA particularly interesting contribution is the adaptive sampling of the period variable, which allows the agent to dynamically expand the range of temporal patterns it can express. This aspect has strong practical potential for multi-timescale behaviors. However, the paper does not include ablation studies to isolate its impact. It is unclear how much performance or skill diversity would degrade without this component.\n\nWhile PSD is well-suited to locomotion tasks with cyclic structure, it remains unclear how well the method would generalize to settings with non-uniform or compositional sequences (e.g., behaviors like “walk → open door → pick up object” in a loop, or partially periodic domains with interruptions). \n\nWhile Table 1 shows promising improvements in downstream task performance using PSD-learned skills, the standard deviations are large, and results are averaged over only five seeds. Given the inherent variability of RL training, stronger statistical support (e.g., more seeds or hypothesis testing) would strengthen the claims.", "was_revised": true, "final_justification": "The authors clarified most of my concerns and, provided that some of that discussion makes it into the final version of the paper (if accepted), I believe it is a good contribution to the field.", "rebuttal": "Dear Reviewer B3mW,\n\nWe sincerely appreciate your constructive and insightful comments, which have been invaluable in helping us further strengthen our work.\n\nWe prepared our response below:\n\n---\n\n**Q1. Lack of ablation studies on the use of the adaptive sampling method to isolate its impact.**\n\n**A1-1.** We appreciate your comment. The adaptive sampling method is crucial for autonomously discovering diverse periods without supervision or prior knowledge. It operates by proposing progressively more challenging periods based on the agent's performance (average cumulative sum of $r_{psd}$). Without this method, the agent fails to learn a wide range of periods, even if those wide range of periods are sampled from the beginning of training. This is analogous to curriculum learning, where an agent masters a difficult task by tackling increasingly challenging goals, rather than being presented with the most difficult ones from the beginning.\n\n**A1-2.** To isolate its effect, we conducted an experiment in which a wide range of periods was sampled from the start, but **without the adaptive sampling** method. After training, we rolled out policies conditioned on each period $L$ and compared their sum of $r_{psd}$ with those from our full method using adaptive sampling.\n\n- Ant\n    \n    \n    |  | **L = 5** | **L = 16** | **L = 28** | **L = 40** |\n    | --- | --- | --- | --- | --- |\n    | **w/ adaptive** | **198.1 ± 1.2** | **188.1 ± 2.1** | 184.7 ± 2.9 | 179.5 ± 4.5 |\n    | **w/o adaptive** | 152.4 ± 3.7 | 187.8 ± 4.2 | **185.5 ± 2.2** | **183.2 ± 1.9** |\n\n- Halfcheetah\n    \n    \n    |  | **L = 5** | **L = 8** | **L = 11** | **L = 15** |\n    | --- | --- | --- | --- | --- |\n    | **w/ adaptive** | **187.8 ± 2.5** | **179.6 ± 3.4** | **177.3 ± 6.2** | **175.1 ± 2.2** |\n    | **w/o adaptive** | 185.0 ± 2.7 | 177.2 ± 3.3 | 175.9 ± 2.1 | 167.6 ± 6.3 |\n\n- Humanoid\n    \n    \n    |  | **L = 10** | **L = 26** | **L = 43** | **L = 60** |\n    | --- | --- | --- | --- | --- |\n    | **w/ adaptive** | **373.0 ± 8.4** | **351.9 ± 6.7** | **376.6 ± 9.5** | **385.2 ± 5.9** |\n    | **w/o adaptive** | 367.1 ± 2.5 | 115.5 ± 6.1 | 91.2 ± 14.7 | 100.9 ± 6.9 |\n\n- Hopper\n    \n    \n    |  | **L = 5** | **L = 20** | **L = 35** | **L = 50** |\n    | --- | --- | --- | --- | --- |\n    | **w/ adaptive** | **351.4 ± 4.9** | 360.1 ± 11.1 | **379.3 ± 9.6** | **353.3 ± 9.1** |\n    | **w/o adaptive** | 309.8 ± 25.6 | **367.1 ± 8.4** | 372.2 ± 2.9 | 337.6 ± 6.7 |\n\n- Walker2D\n    \n    \n    |  | **L = 5** | **L = 23** | **L = 41** | **L = 60** |\n    | --- | --- | --- | --- | --- |\n    | **w/ adaptive** | **379.6 ± 6.4** | 362.1 ± 12.2 | **366.6 ± 12.1** | **359.5 ± 10.4** |\n    | **w/o adaptive** | 342.7 ± 4.8 | **371.0 ± 4.6** | 345.4 ± 4.7 | 301.8 ± 8.2 |\n\nBased on the results, our adaptive sampling method is crucial for learning a wide range of high-quality skills, as it consistently outperforms the non-adaptive baseline. This is particularly evident at the extremes of the period range, which highlights its effectiveness in discovering a truly diverse skill repertoire.\n\n**Q2. Generalization of PSD to non-uniform or compositional sequences.**\n\n**A2.** Thank you for this insightful question. While the primary focus of this work, similar to other methods in unsupervised skill discovery, is on learning a rich repertoire of low-level skills, we believe that combining PSD with a hierarchical RL (HRL) framework could enable more complex, compositional tasks.\n\nFor example, consider a set of low-level policies $\\pi_{low}(a|s,z)$, where $z$ represents a low-level skill such as ‘pick’, ‘move’, or ‘place’ and is capable of both periodic and non-periodic behaviors. Based on this, we could then train a high-level policy $\\pi_{high}(z|s,L)$ with the PSD objective to encourage it to select skills $z$ as actions for creating complex, looped behaviors. This could be an interesting future direction.\n\n**Q3. Need for stronger statistical support for downstream task performance results.**\n\n**A3.** Thank you for your helpful suggestions. Following your valuable comment, we expanded our experiments from **5 to 10 seeds** to provide stronger statistical support.\n\n| **Downstream task** | **DIAYN** | **DADS** | **CSD** | **METRA** | **PSD (Ours)** |\n| --- | --- | --- | --- | --- | --- |\n| **HalfCheetah-hurdle** | 0.6 ± 0.5 | 0.9 ± 0.3 | 0.8 ± 0.6 | 1.9 ± 0.8 | **3.8 ± 2.0** |\n| **Walker2D-hurdle** | 2.6 ± 0.5 | 1.9 ± 0.3 | 4.1 ± 1.3 | 3.1 ± 0.5 | **5.4 ± 1.4** |\n| **HalfCheetah-friction** | 13.2 ± 3.4 | 12.4 ± 2.9 | 12.5 ± 3.8 | 30.1 ± 13.1 | **43.4 ± 19.1** |\n| **Walker2D-friction** | 4.6 ± 1.2 | 1.6 ± 0.1 | 5.3 ± 0.3 | 5.2 ± 1.6 | **8.7 ± 1.7** |\n\nFor the hurdle and friction tasks, we observed that PSD outperforms the baselines across all embodiments. We hypothesize that the (relatively) high standard deviation of PSD is attributable to the fact that it leverages a more diverse range of periodic behaviors as low-level skills compared to other baselines.\n\n**Q4. Number of distinct skills learned by PSD and its effect on downstream task performance.**\n\n**A4-1.** Thank you for the question. During the training of PSD, we uniformly sample 4 values of $L$ from the range $[L_{min}, L_{max}]$, including both bounds. We found this strategy sufficient, as our model can also achieve the desired periods for interpolated $L$ values after convergence. For more details, please refer to Appendices B.2 and C.2.\n\n**A4-2.** We appreciate your comment. Based on your thoughtful question, we conducted experiments on all downstream tasks, comparing PSD trained with 4 discrete skills and 7 discrete skills within the same bound $[L_{min}, L_{max}]$.\n\n| **Downstream task** | **PSD (w/ 4 skill)** | **PSD (w/ 7 skill)** |\n| --- | --- | --- |\n| **HalfCheetah-hurdle** | 3.8 ± 2.0 | **4.4 ± 1.9** |\n| **Walker2D-hurdle** | **5.4 ± 1.4** | 4.1 ± 1.2 |\n| **HalfCheetah-friction** | **43.4 ± 19.1** | 43.2 ± 13.6 |\n| **Walker2D-friction** | **8.7 ± 1.7** | 8.6 ± 0.6 |\n\nAs the results show, increasing the number of discrete skills sampled during training from 4 to 7 does not lead to a significant improvement in downstream task performance. Across all tasks, the performance of both settings is comparable within their standard deviations.\n\n**Q5. How were the trajectories presented in Figure 5 selected?**\n\n**A5.** If your question is about which state components were used to construct the trajectories, we selected the velocities and positions of joints of Ant and Walker2D that best highlight the periodicity. For clear visualization, we normalized these values to lie between -1 and 1. As described in the caption, we also switched the target period $L$ at fixed time intervals within a single episode.\n\n**Q6. Questions on training setup and skill selection for downstream tasks.**\n\n**A6-1.** We trained the low-level skill policy for all methods with the same number of environment steps for each embodiment: 40M for Ant and HalfCheetah, and 80M for Humanoid, Hopper, and Walker2D. We also closely followed the hyperparameters reported in their original papers.\n\nFor training the high-level policy for the downstream tasks, we ran 100k episodes across all experiments.\n\n**A6-2.** For all baseline methods, we designed the high-level policy to have access to the full set of discovered skills, matching the skill distribution $p(z)$ used during low-level policy training. For PSD, we uniformly sampled fixed number of $L$ values for $\\pi(a|s,L)$ from the range $[L_{min}, L_{max}]$.\n\n**Q7. Questions on the exploration reward.**\n\n**A7.** Thank you for raising this point. Since PSD is designed to learn *how* to behave within a task rather than directly address *what* to do, the specific task setting needs to be predefined when evaluating its performance. For this reason, we added a exploration reward to all methods for fair comparison. Importantly, this exploration reward did not negatively affect the quality of the skills required for downstream tasks. This type of setup where an auxiliary unsupervised reward is combined with the main learning objective is also widely used in prior work on intrinsic motivation [1, 2, 3].\n\n[1] Lee, Youngwoon, et al. \"Learning to coordinate manipulation skills via skill behavior diversification.\" *International conference on learning representations*. 2019.\n\n[2] Li, Jiahui, et al. \"Two Heads are Better Than One: A Simple Exploration Framework for Efficient Multi-Agent Reinforcement Learning.\" *NeurIPS*. 2023.\n\n[3] Bae, Junik, et al. \"TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations.\" *Conference on Robot Learning*. PMLR, 2025.\n\n---\n\nThank you again for your valuable and insightful review. We hope our responses have sufficiently addressed your questions, and we would be happy to provide any further clarification if needed."}, {"review_id": "iP96lEEVGW", "reviewer": "Reviewer_DTax", "summary": "The paper presents an unsupervised learning approach for learning locomotion policies that are periodic. This is achieved by constraining the latent space to follow a circular representation, using a set of terms that “encourage” the encoder to converge to such a structure. This way learning can sample from a range of periods that are adaptively bounded as learning progresses. The paper presents examples of learned policies on a range of different embodiments, commonly used for benchmarking, on locomotion tasks with a range of periods, and two downstream tasks. The downstream tasks are evaluated against a set of skill discovery methods.", "questions": "- Does the encoder see all the states, including the base position? Would it choose to ignore certain potentially non-periodic dimensions (such as the XY position) in favor of ones like the joint angles? Can it learn to produce periodic motions along those other dimensions too, for example, by moving in circles of various radius in XY space?\n\n- Visualizations on website could be more informative - maybe slow them down, show different colors for skills (like in Figure 8.).\n\n- What is the state space coverage without adding METRA? Does the algorithm on its own encourage state-covering trajectories? In Appendix B.2 it says that it uses an extrinsic velocity reward - is that necessary for learning useful behaviors?\n\n- A comparison of the state space coverage between PSD and the baselines would be good. From Figure 8 it seems that the XY coverage of PSD+METRA is worse than only METRA (based on their own results) - is this the case?\n\n- Similarly, a more in-depth analysis of the trade-offs of PSD+METRA would strengthen the claim that they can easily be combined. Is the loss worse for both methods when combining them?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n-To the best of my knowledge no other work has addressed unsupervised learning of periodic skills so this is a clear novelty and contribution for the work. \n-The paper is well written and the approach is clearly presented.\n-The approach is evaluated against other current skill discovery approaches and is shown to empirically work better on the benchmarks presented.\n-The experiments show that the method does learn periodic skills across a good range and can be combined with extrinsic rewards or other algorithms like METRA, which is another advantage.\n\nWeaknesses:\n- Visualizations on website could be more informative - maybe slow them down, show different colors for skills (like in Figure 8.). Examples videos of downstream task behaviors missing.", "was_revised": true, "final_justification": "Solid contribution.", "rebuttal": "Dear Reviewer DTax,\n\nThank you for your valuable and insightful comments. We have carefully considered your feedback and prepared our response below:\n\n---\n\n**Q1. Visualizations on website could be more informative.**\n\n**A1.** Thank you for your thoughtful suggestions! We will incorporate your feedback into the final version of our project page and paper. If you have any additional ideas to further improve the reader’s understanding, please feel free to share them, and we will be happy to include them.\n\n**Q2. Questions on the state space of the encoder.**\n\n**A2-1.** Thank you for your insightful question. The encoder's state representation does not include the global XY base position and consists only of proprioceptive states like joint angles, velocities, and heading angles, as described in Appendix B. This corresponds to the default observation space in standard Gym environments and is a common practice in prior work [1, 2, 3] on robot learning for generating diverse behaviors.\n\n**A2-2.** Furthermore, from the perspective of our downstream task setup, including the global XY position would be particularly problematic. The exploration reward we use encourages consistent forward movement, which makes the XY position an inherently non-periodic states. This would directly conflict with the PSD objective, which requires the latent representation to be identical after $2L$ steps, and would therefore hinder the learning process.\n\n**A2-3.** We verified this with an experiment on the Ant environment. When we removed the exploratory reward, provided the **full state (including the XY coordinates)**, and trained the agent using **only the PSD objective**, we observed that the agent vigorously wiggled in place, making body movements in sync with the period. This behavior is the result of the optimization satisfying our objective, as it makes **both the joint angles and the global XY coordinates periodic.** We will include this result in the final version of our project page.\n\nTherefore, PSD can successfully discover periodic skills even with the full state, which demonstrates the strength of the PSD objective.\n\n[1] Peng, Xue Bin, et al. \"Amp: Adversarial motion priors for stylized physics-based character control.\" *ACM Transactions on Graphics (ToG)* 40.4 (2021): 1-20.\n\n[2] Peng, Xue Bin, et al. \"Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters.\" *ACM Transactions On Graphics (ToG)* 41.4 (2022): 1-17.\n\n[3] Li, Chenhao, et al. \"FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning.\" *The Twelfth International Conference on Learning Representations*.\n\n**Q3. Questions on the state coverage of PSD.**\n\n**A3-1.** We appreciate your constructive comments. We would like to clarify that PSD is not designed to encourage spatially far-reaching, state-covering trajectories. Instead, its primary goal is to achieve **high coverage in the frequency domain** by autonomously discovering a diverse repertoire of skills with different periods, as demonstrated in Figure 3.\n\n**A3-2.** As mentioned in our response to Q2, an external reward is not necessary for learning periodic behaviors. Our method focuses on learning the **style** of motion (*how to behave*). The optional task reward simply provides a clear objective (*what to do*), which allows the various learned periodic styles to be demonstrated and evaluated more effectively.\n\n**Q4. From Figure 8 it seems that the XY coverage of PSD+METRA is worse than only METRA (based on their own results) - is this the case?**\n\n**A4.** Yes, your observation is correct. To quantify this, we measured the state space coverage by discretizing the XY plane (or X-axis for Walker2D) into a 1x1 grid and counting the number of unique cells visited over 32 random skill rollouts, averaged over 5 seeds.\n\nThe results confirm that the XY coverage of METRA+PSD is lower than that of METRA alone:\n\n| **State coverage** | **Ant** | **Walker2D** |\n| --- | --- | --- |\n| **METRA only** | 722 ± 32 | 20.3 ± 0.5 |\n| **METRA + PSD** | 337 ± 6 | 19.9 ± 2.1 |\n\nThe reason for this trade-off is detailed in our response to your next question (A5).\n\n**Q5.** Similarly, a more in-depth analysis of the trade-offs of PSD+METRA would strengthen the claim that they can easily be combined. Is the loss worse for both methods when combining them? \n\n**A5.** Thank you for your question. Conceptually, METRA and PSD align well, as both methods address temporal aspects of skills: METRA modulates the temporal **direction** ($z$), while PSD modulates the temporal **length** ($L$).\nHowever, as you suggest, combining them involves a trade-off. While our mutual conditioning approach ($\\phi_L(s, z), \\phi_m(s, L)$) solves the primary issue of skill collapse, it introduces a secondary trade-off related to representational capacity.\n\nBy conditioning each encoder on an additional skill variable, we increase the complexity of its input space. This can slightly diminish the individual performance of each component compared to when it is trained in isolation. Therefore, while the combined policy $\\pi(a | s, z, L)$ is significantly more versatile—**offering independent control over both skill direction and period**—this increased modality comes at the cost of a slight reduction in the peak performance of the individual METRA and PSD objectives compared to their standalone counterparts.\n\n---\n\nThank you again for your valuable and insightful review.\n\nAlso, please let us know if our responses have addressed your questions. If anything requires further clarification, please do not hesitate to let us know."}, {"review_id": "Olm6vHvTfV", "reviewer": "Reviewer_N2n9", "summary": "The authors observe that locomotion skills often require periodic motion patterns which is largely overlooked in unsupervised skill discovery methods. To address this, they propose Period Skill Discovery (PSD), a framework for discovering periodic behaviors in an unsupervised way. Given a randomly sampled duration variable $L$, PSD trains an encoder to map states into a circular latent space with periodicity $2L$.  An exploration policy is trained to move along the circular space in a predetermined frequency depending on $L$. The method is evaluated on a set of locomotion tasks using either state-based or pixel-based inputs. The authors qualitatively demonstrate that the learned behaviors exhibit distinct periodic patterns. Additionally, they show that these skills can support downstream task performance when combined with extrinsic rewards.", "questions": "1. Frequency domain analysis (Fig. 3): Could you provide an intuitive explanation for why a Fourier transform of PCA-projected, normalized skill trajectories is informative? What does this metric reveal about the learned behaviors?\n2. Robustness: Figures 3, 5, and 6 appear to be generated from a single random seed. Could you repeat these experiments across multiple seeds to assess the robustness of PSD? Similarly, Figure 6 presents histograms for just two $L$ values. How do these results vary across the full range of periodicities?\n3. Latent space dimensionality: How is the dimensionality of the latent space $S$ selected? How does it affect the resulting behaviors? Also, since the 2D visualizations are PCA projections, it would be helpful to note this directly in the figure captions.\n4. Sampling of $L$: The step-size adaptation for sampling L introduces several hyperparameters ($N$, $\\alpha$, $\\beta$) and lacks clear justification. Did the authors consider simpler alternatives? Can they comment on the impact of these hyperparameters?\n5. Pixel-based task setup: In pixel-based settings, robot limbs are differently colored. Was this necessary for the method to work? An ablation or explanation would clarify its impact.\n6. PSD + METRA setup: The combination with METRA is described as “fully unsupervised.” What aspect of vanilla PSD is supervised? Could METRA-PSD also be enhanced with extrinsic rewards for improved downstream performance?\n7. Method scope: Is PSD applicable beyond locomotion domains? How might it perform (or be adapted) for tasks like object manipulation? Experiments in non-locomotion domains could clarify the method’s broader applicability.", "limitations": "The limitations are only partially addressed. The paper includes a subsection on limitations, but it leans more toward outlining future work than identifying concrete shortcomings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- clear presentation, especially in the method section. Equations are well-integrated with intuitive descriptions.\n- strong qualitative visualizations that help illustrate the method and its motivation.\n- the use of circular latent states to encourage periodicity is well-motivated and original.\n- the method is evaluated against reasonable baselines from the unsupervised skill discovery literature.\n- the related work section is thorough and well-positioned.\n\nWeaknesses\n- the discrete step size adaptation of $L$ feels somewhat ad hoc and lacks theoretical justification.\n- behavioral analysis feels anecdotal. The experiments seem to be based on single-seed visualizations. A more systematic quantitative analysis across multiple seeds would strengthen the evaluation.\n- the focus on locomotion tasks may limit the method’s generality and significance.\n- missing ablations: Design choices, such as the dimensionality of the latent space $S$, the hyperparameters for sampling periodicity $L$, and the use of colored robot limbs (see questions below), are not systematically ablated or justified. Combined with the lack of discussion around where PSD may fail or underperform, this makes it difficult to assess the method’s true limitations and general applicability.", "was_revised": true, "final_justification": "The paper presents a solid contribution with a clearly written presentation, effective visualizations, and an original core idea. The proposed method is evaluated against reasonable baselines, and several of my initial concerns (e.g., step-size adaptation and missing ablations) were addressed through the explanations and additional experiments provided in the rebuttal.\nHowever, due to the method’s specific focus on locomotion tasks, I would have appreciated a more general quantitative assessment of the discovered skills beyond frequency-domain analysis. In particular, how variable the skill discovery is across seeds and how reliably useful skills are discovered remains somewhat unclear (especially for embodiments not tested on downstream tasks). This limits the understanding of the method’s applicability to new domains.\nOverall, while some open questions remain, I find that the reasons to accept the paper outweigh these concerns.", "rebuttal": "Dear Reviewer N2n9,\n\nWe are grateful for your careful reading and valuable suggestions.\n\nWe have summarized the weaknesses and questions you raised and provide our responses below:\n\n---\n\n**Q1. The discrete step size adaptation of $L$ feels somewhat ad hoc and lacks theoretical justification.**\n\n**A1.** In the standard MDP framework, time progresses in discrete steps. Consequently, $L$ in the PSD objective is intended to encode timestep information. Therefore, it should be a positive integer. To discover maximally diverse periods — while respecting the fact that the minimum unit of time is one step — we set the default step size of the adaptive sampling method to its minimal feasible value, $N=1$.\n\n**Q2. A more systematic quantitative analysis across multiple seeds would strengthen the evaluation. Could you repeat the experiments for Figures 3, 5, and 6 across multiple seeds to assess the robustness of PSD?**\n\n**A2.** Following the reviewer’s suggestion, we have conducted additional experiments to address this.\n\n**A2-1.** (Figure 3) To quantitatively compare the diversity of learned skills, we measured the frequency width — defined as the difference between the maximum and minimum frequencies of the skill trajectories (i.e., the gap between the highest and lowest frequency of bar in Figure 3) — and averaged the results over 3 seeds. A larger value for this metric indicates a broader range of periodic behaviors.\n\n|  | **Ant** | **HalfCheetah** | **Humanoid** | **Hopper** | **Walker2D** |\n| --- | --- | --- | --- | --- | --- |\n| **PSD (Ours)** | **9.1 ± 0.1** | **6.9 ± 0.2** | **6.1 ± 0.4** | **7.3 ± 1.6** | **8.7 ± 0.3** |\n| **METRA** | 2.1 ± 0.4 | 4.2 ± 0.2 | 2.0 ± 0.7 | 3.7 ± 0.5 | 3.5 ± 1.3 |\n| **CSD** | 2.3 ± 0.2 | 2.7 ± 0.3 | 1.7 ± 0.2 | 1.6 ± 0.4 | 4.2 ± 0.7 |\n\nAs the results show, PSD consistently discovers a much wider range of periodic behaviors compared to the baselines across all embodiments. The frequency width, which measures the diversity of the discovered periods, is significantly larger for PSD in every case.\n\nWe excluded DADS and DIAYN from this analysis as their policies collapse to a nearly static \"yoga pose\"—moving slightly at the beginning of an episode and then remaining stationary—which results in a frequency close to zero.\n\n**A2-2 & A2-3** (Figures 5 & 6). As you suggested, we have run the experiments for Figures 5 and 6 across multiple seeds to assess their robustness. While the NeurIPS rebuttal policy prevents us from including the new plots, we can summarize our findings.\n\nThe circular latent space structure shown in Figure 5 is learned robustly across all seeds. Interestingly, we often observe that different random seeds result in qualitatively different gaits for the same period, suggesting our method can find multiple diverse solutions. Our analysis for Figure 6 also confirms that the latent representation accurately reflects temporal distance, suggesting the learned distance converges to its optimal value.\n\nDue to the time/character limits of the rebuttal period, we will provide the detailed quantitative results for a wider range of $L$ values during the upcoming author-reviewer discussion period. Also, the full multi-seed visualizations for these figures will be included in the final version of our paper.\n\n**Q3. The focus on locomotion tasks may limit the method’s generality and significance.**\n\n**A3.** PSD objective is theoretically applicable to any domain that involves periodic components, as it does not require domain-specific knowledge for optimization. For instance, in manipulation settings, it can be applied to tasks such as rotating a cube with fingers, turning a valve wheel, or mixing a salad bowl. Moreover, given a set of low-level policies $\\pi_{low}(a|s,z)$, where $z$ represents a low-level skill such as ‘pick’, ‘move’, or ‘place’, a high-level policy $\\pi_{high}(z|s,L)$ can be trained with the PSD objective to select skills $z$ as actions for composing complex, looped behaviors. We believe this could be an interesting future extension of PSD.\n\n**Q4. Lacks clear justification for hyperparameters of adaptive sampling method.**\n\n**A4.** To justify our hyperparameter choices for the adaptive sampling method, we conducted an ablation study (averaged over 3 seeds) on the Ant environment. The results show a clear trade-off between the final range of discovered periods and the stability of the expansion process.\n\n| $(\\alpha, \\beta, N) = (0.4, 0.9, 1)$ | **episode = 0** | **episode = 100k** | **episode = 200k** | **episode = 300k** | **number of rollbacks** |\n| --- | --- | --- | --- | --- | --- |\n| upper bound | 20 ± 0 | 37.3 ± 6.1 | 41.6 ± 8.2 | 47.3 ± 4.1 | 2 ± 0.8 |\n| lower bound | 20 ± 0 | 5 ± 0 | 5 ± 0 | 5 ± 0 | 0 ± 0 |\n\n| $(\\alpha, \\beta, N) = (0.4, 0.9, 3)$ | **episode = 0** | **episode = 100k** | **episode = 200k** | **episode = 300k** | **number of rollbacks** |\n| --- | --- | --- | --- | --- | --- |\n| upper bound | 20 ± 0 | 40 ± 3.7 | 44 ± 2.4 | 47 ± 4.9 | 7.3 ± 3.2 |\n| lower bound | 20 ± 0 | 5 ± 0 | 5 ± 0 | 5 ± 0 | 0 ± 0 |\n\n| $(\\alpha, \\beta, N) = (0.5, 0.8, 1)$ | **episode = 0** | **episode = 100k** | **episode = 200k** | **episode = 300k** | **number of rollbacks** |\n| --- | --- | --- | --- | --- | --- |\n| upper bound | 20 ± 0 | 40.6 ± 5.3 | 47.3 ± 1.2 | 48.7 ± 3.2 | 17.3 ± 1.9 |\n| lower bound | 20 ± 0 | 5 ± 0 | 5 ± 0 | 5 ± 0 | 0 ± 0 |\n\n| $(\\alpha, \\beta, N) = (0.5, 0.8, 3)$ | **episode = 0** | **episode = 100k** | **episode = 200k** | **episode = 300k** | **number of rollbacks** |\n| --- | --- | --- | --- | --- | --- |\n| upper bound | 20 ± 0 | 42 ± 6.2 | 52 ± 5.1 | 42 ± 7.8 | 28.7 ± 4.8 |\n| lower bound | 20 ± 0 | 5 ± 0 | 5 ± 0 | 5 ± 0 | 0 ± 0 |\n\nFor instance, the less conservative threshold set ($\\alpha=0.5, \\beta=0.8$) with a larger step size ($N=3$) initially discovers the widest range. However, this aggressive search proves unstable, as shown by the high number of \"range rollbacks\" (28.7). This instability eventually causes the discovered range to collapse later in training (from 52 at 200k episodes to 42 at 300k). A high number of rollbacks generally indicates a potential for instability in long-term training.\n\nIn contrast, the setting ($\\alpha=0.4, \\beta=0.9, N=1)$ expands the range to a comparable margin (47.3) with minimal rollbacks (2.0), demonstrating a much more stable and efficient learning process. We checked that this parameter set performs well across all embodiments, and therefore selected it for providing the best balance between discovering a diverse skill repertoire and maintaining training stability.\n\n**Q5. Could you provide an intuitive explanation for why a Fourier transform of PCA-projected, normalized skill trajectories is informative? What does this metric reveal about the learned behaviors?**\n\n**A5.** For Figure 3, we normalized the skill trajectories for each dimension using statistics computed from random action rollouts. This was done to account for the different scales of the state dimensions and to visualize the periodicity of each dimension fairly. In addition, for the 1D FFT, we applied PCA to reduce the trajectories to a single dimension and visualized them in the frequency domain. This analysis revealed that, compared to the baselines, PSD exhibits behaviors spanning a wider range of frequencies, as indicated by the broader distribution of bars in the frequency domain chart.\n\n**Q6. How is the latent space dimensionality chosen and how does it affect the resulting behaviors?**\n\n**A6-1.** As shown in Table 2, we found that using {3, 6}-dimensional latent space works well in practice across all embodiments. However, we observed that a 2-dimensional latent space does not perform well for complex agents such as Ant or Humanoid. We hypothesize that, since the PSD objective does not explicitly constrain the circles for each $L$ to lie on the same plane, having additional degrees of freedom allows different circles to occupy different planes. This, in turn, helps stabilize the embedding during training. Conversely, when the latent space is restricted to only 2 dimensions (i.e., a plane), this flexibility is lost, which leads to instability.\n\n**A6-2.** Thank you for your suggestion to add a note that the 2D visualizations are the result of PCA projections! We will include this in the final version. If you have any additional ideas to further improve the reader’s understanding, please feel free to share them, and we will be happy to include them.\n\n\n**Q7. In pixel-based settings, robot limbs are differently colored. Was this necessary for the method to work? An ablation or explanation would clarify its impact.**\n\n**A7.** For symmetric agents like the Ant, limb coloration is a common practice to disambiguate limbs from pixel observations [1]. However, it was not strictly necessary for our specific Ant task (forward locomotion) or for the asymmetric Cheetah. Therefore, while helpful for feature learning, it is not an essential design choice for our method to work.\n\n[1] Park, Seohong, et al. \"Ogbench: Benchmarking offline goal-conditioned rl.\" arXiv preprint arXiv:2410.20092 (2024).\n\n**Q8. Discussion around where PSD may fail or underperform.**\n\n**A8.** PSD may underperform in settings with large, persistent external disturbances (e.g., constant interference from another agent) that make periodic behavior infeasible. We will add this discussion to our limitations section.\n\n\n**Q9. What aspect of vanilla PSD is supervised? Could METRA-PSD also be enhanced with extrinsic rewards for improved downstream performance?**\n\n**A9.** PSD itself is unsupervised as its objective is task-agnostic. The METRA-PSD setup is described as \"fully unsupervised\" because it combines two unsupervised methods without external supervision. As you suggest, extrinsic rewards can optionally be added to this framework to bias exploration, but this is a separate design choice.\n\n---\nThank you again for your valuable review. We hope our responses have addressed your questions."}, {"review_id": "2TwKMSrL8g", "reviewer": "Reviewer_qG2Z", "summary": "This work introduces a novel framework for unsupervised discovery of explicitly periodic skills in locomotion tasks called periodic skill discovery (PSD). This is done by learning a latent space in which, when embedded, trajectories are incentivised to follow a circular path with some known, fixed period. Conversely, an intrinsic reward is obtained from this latent embedding in which a period-conditioned policy is optimised with reinforcement learning to maximise the fit of the trajectory to a circle inside the latent space. These two approaches bootstrap off each other to result in the unsupervised discovery of periodic policies. The results section shows that, compared to prior unsupervised skill discovery work, PSD discovers locomotion skills with a greater diversity of periods.", "questions": "Main question (state space)\n- I think my most fundamental lack of understanding comes from the definition of what a state is. Presumably, the state used for the latent embedding does not include the global positional co-ordinates of the body? Otherwise a policy would presumably only be periodic if it returned to the same position. Is the state then made up of only the relative positions/rotations/velocities of the constituent parts of the body?\n- This feels like a fundamental limitation of the approach - can it still capture periodicity of a trajectory if only a subset of the state is periodic? e.g. if the global position is included in the state  \n\nOther minor points / questions\n- The problem statement (online unsupervised skill discovery) could be more clearly outlined in the introduction. If one was not familiar with the prior work in this area it is unclear exactly what the high level problem is and why this is useful.\n- Why is only the 'one-step' periodic constraint (corresponding to Eq. 6) used as the intrinsic reward while the 'diameter' constraint (corresponding to Eq. 5) is not?\n- Line 182 \"Since we want to discover maximally diverse period\" is either bad grammar or confusing at least\n- My understanding of section 3.4 is that the period limits essentially dynamically expand according to whether the policies on the limits are satisfying the periodic conditions. Is their a practical maximum here? It would seem like you could come up with policies of essentially unboundedly high period and with low periods of a single timestep (e.g. staying still).\n- This approach feels quite related to quality diversity methods (e.g. MAP elites [1]) where $L$ corresponds to a behavioural descriptor. Especially in regards to the main result alluding to how the diversity of periods is a useful thing. This link should be discussed.\n\n[1] - Jean-Baptiste Mouret, Jeff Clune. Illuminating search spaces by mapping elites. 2015.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n - The formulation of the periodic objective is intuitive, well explained and original\n - The paper is well presented and easy to read\n\nWeaknesses\n - I would consider the significance of the paper to be a practical inductive bias, however like all inductive biases it must be questioned if this is actually a useful thing or will eventually be surpassed by simply increasing the data, scale and compute of a more general method. Specifically, a generic unsupervised skill discovery method *should* be able to discover different gaits of different periods if appropriately conditioned on full trajectories of full state representations.\n- It seems like the method could not be applied to an offline dataset of trajectories, as the target period $L$ of a trajectory must be known for learning the latent space, limiting the applicability of the method to online approaches.\n- As far as I can tell the current setup cannot represent different gaits of the same period\n\nFor these reasons I am giving a borderline accept, but would be open to increasing my score - see questions section", "was_revised": true, "final_justification": "The authors response suitably clarified all the questions I raised.\nAs outlined in my original review and response, I think the most significant weakness of this method is how reliant it is on the given observation space to be entirely periodic - I do not think the method as it stands would generalise to pixel-based environments with arbitrary (i.e. non-repeating) terrain for instance. That being said, I do feel like the method as it stands is significant enough to warrant acceptance.", "rebuttal": "Dear Reviewer qG2Z,\n\nWe sincerely thank you for your detailed and insightful feedback.\nWe have prepared our responses below:\n\n---\n**Q1. The paper introduces a practical inductive bias. Will it be eventually be surpassed by simply increasing the data, scale and compute of a more general method?**\n\n**A1.**\nWe appreciate your comment. Even as generalist methods grow in data, scale, and compute, an explicit inductive bias for *periodicity* will continue to fill a role that pure scale alone cannot guarantee—much like convolution remains valuable in vision despite ever-larger transformers.\n\nAt the same time, this inductive bias is not at odds with future progress in large-scale models; it complements it. Since one of the PSD’s contribution is to discover a maximally diverse repertoire of useful periodic behavior in an unsupervised manner, it can be plugged in as a lightweight sub-module that a large, general model calls whenever rhythmic style or behavior-style rewards should be injected. In this way, the large model continues to handle broad task understanding and high-level decisions, while PSD provides a principled, sample-efficient mechanism for adding rhythmic nuance, thereby enhancing versatility rather than competing with scale.\n\n**Q2. Applicability of PSD to offline datasets.**\n\n**A2.** Thank you for your insightful comment regarding the applicability of PSD to offline datasets.\n\nThe main prerequisite is to know or estimate the period variable $L$ for each trajectory in the offline dataset, which we believe is a tractable problem.\n\nFor instance, we could estimate $L$ for a given trajectory using a Fourier transform to find its dominant frequency, similar to our analysis in Figure 3. Alternatively, one could add an auxiliary reconstruction loss to the existing PSD objective to find the period $L$ that best represents a trajectory's periodic features.\n\nOnce the period $L$ for each offline trajectory is estimated, an offline skill learning process is available. The recipe would be as follows:\n\n1. For each trajectory in the offline dataset, first estimate its period, $L$.\n2. Use these estimated (trajectory, L) pairs to train the PSD encoder and its circular latent space.\n3. Finally, with the learned representations, train an RL policy with $r_{psd}$ using a standard offline RL algorithm.\n\nAdditionally, an offline-to-online approach could allow the agent to discover skills with periods that **go beyond the range** covered by the original dataset, while remaining consistent with the behaviors it contains. Thank you again for the valuable suggestion.\n\n**Q3. As far as I can tell the current setup cannot represent different gaits of the same period.**\n\n**A3.** We appreciate your excellent comment. The primary contribution of this work lies in autonomously discovering a diverse range of *periods*, rather than directly targeting multimodality within a single period.\n\nHowever, your comment raises a very interesting direction for future work. Inspired by your idea, encoding the representation onto a more general geometry, such as the surface of a sphere (instead of a circle), would allow different gaits to be represented even for a single period. For example, different gaits of the same period could be encoded as distinct circles of the same circumference on the sphere's surface. We agree this is a very exciting future direction to explore.\n\n**Q4. Question about the state space of PSD.** \n\n**A4-1.** Your presumptions are correct. The default state representation for PSD does not include the global XY base position and consists only of proprioceptive states like joint angles and heading angles, as described in Appendix B. This corresponds to the default observation space in Gym environments and is a common practice in prior work [1, 2, 3] on robot learning for generating diverse behaviors.\n\n**A4-2.** When we trained the Ant agent with the full state (including the XY coordinates) using only the PSD objective, we observed that the agent vigorously wiggled in place, making body movements in sync with the period. This behavior is the result of the optimization satisfying our objective, as it makes both the joint angles and the global XY coordinates periodic.\n\n**A4-3.** As stated in our previous response, PSD can discover periodic behaviors even when the state includes the XY coordinates. However, a conflict arises when the agent must also satisfy a specific downstream task. If an external task reward encourages non-periodic changes in the XY position (e.g., by rewarding forward velocity), this would directly conflict with the PSD objective, which requires the latent representation to be identical after $2L$ steps, and would therefore hinder the learning process.\n\n[1] Peng, Xue Bin, et al. \"Amp: Adversarial motion priors for stylized physics-based character control.\" *ACM Transactions on Graphics (ToG)* 40.4 (2021): 1-20.\n\n[2] Peng, Xue Bin, et al. \"Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters.\" *ACM Transactions On Graphics (ToG)* 41.4 (2022): 1-17.\n\n[3] Li, Chenhao, et al. \"FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning.\" *The Twelfth International Conference on Learning Representations*.\n\n**Q5. Need a clear outline of the problem statement in the introduction for online unsupervised skill discovery.**\n\n**A5.** Thank you for your suggestion. We will add an explanation of the basic setup of unsupervised skill discovery, along with the problem it aims to address and its significance, in the second paragraph of the introduction.\n\n**Q6. Use of the one-step periodic constraint as the intrinsic reward instead of the diameter constraint.**\n\n**A6.** That is an excellent question regarding our reward design! We chose the one-step periodic constraint instead of the diameter constraint for two primary reasons: learning efficiency and compatibility with off-policy RL algorithms.\n\n**A6-1.** On a practical level, the one-step reward provides a dense, per-timestep learning signal, which is significantly more effective for policy training than the sparse reward from the diameter constraint that is only evaluated over an $L$-step horizon.\n\n**A6-2.** More fundamentally, the use of a single-step reward is crucial for general compatibility with standard off-policy algorithms (e.g., SAC, DDPG). In contrast, a reward based on the diameter constraint would be inherently a multi-step, sparse reward, as it depends on states that are $L$ timesteps apart. This choice was made to ensure that PSD is easy to use and broadly compatible with standard off-policy RL algorithms that learn from single-step transition tuples. However, as you suggest, a more advanced implementation could certainly be developed to handle the multi-step diameter constraint, perhaps by leveraging n-step returns, which is an interesting direction for future work.\n\nThus, we chose this approach to ensure a dense learning signal and broad applicability with standard off-policy RL algorithms.\n\n**Q7. Line 182 \"Since we want to discover maximally diverse period\" is either bad grammar or confusing at least.**\n\nThank you for your valuable suggestion. We found that the original wording was inconsistent with the tone of the paper.\n\nWe have revised the sentence as follows: \"**To enable the agent to discover a maximally diverse range of periods** without any prior knowledge of the agent-specific period ranges, we introduce an adaptive sampling method that …”\n\n**Q8. Is their a practical maximum of discovered period? It would seem like you could come up with policies of essentially unboundedly high period and with low periods of a single timestep (e.g. staying still).**\n\n**A8-1.** Thank you for the question. Your understanding is correct. the period’s upper & lower bounds dynamically expand based on whether the policies at the boundaries satisfy the periodic conditions.  \n\n**A8-2.** Regarding the practical maximum, while there is no hard-coded upper bound, the range is limited by what is **dynamically feasible** for the agent, especially when maintaining performance on a specific task. For example, there is likely a maximum period at which an agent can still maintain a forward velocity of 2m/s.\n\n**A8-3.** For the lower bound, since $L$ must be a positive integer in our objective, the shortest possible period corresponds to $L=1$, resulting in a period of 2 timesteps.\n\n**Q9. Connection to quality diversity methods such as MAP-Elites.**\n\nThank you for this excellent suggestion. You are correct that our approach has a link to quality diversity (QD) methods such as MAP-Elites. However, we would like to highlight a key difference. While this QD methods typically operate on a pre-defined, static grid, PSD dynamically adjusts the range of the descriptor ($L$) based on the agent’s performance (sum of reward) and the embodiment's dynamics.\n\nFollowing your suggestion, we will further investigate this connection and add a discussion of both the similarities and this key distinction to our Extended Related Work section.\n\n---\n\nThank you again for your valuable and insightful review. Also, please let us know if our responses have addressed your questions. If anything requires further clarification, please do not hesitate to let us know."}]

Author rebuttal:

Meta-review: This paper proposes an unsupervised skill discovery approach tailored to environments / robots where periodicity is helpful to solving relevant tasks, e.g. locomotion. The method trains an encoder of the desired periodic variables to encode to a time-profiled circular latent space through a novel skill discovery objective. The paper claims that the discovered skills are predictive over multiple horizons, can be applied to image observations, and can be combined with another skill discovery method. 

Reviewers perceived a variety of strengths, including the novelty, good motivation, and clear presentation.

Reviewers perceived a few weaknesses, including limited application to locomotion tasks, and that the method requires prior knowledge of what subset of the observation should be periodic (e.g. a global position XY in the observations was found to hurt performance).

Overall the paper appears to soundly demonstrate its claimed contributions and these claimed contributions are significant. I concur that the main weakness in the method is that it requires prior knowledge of what aspects of the observation should be periodic, a method that addressed this weakness would help enhance the applicability to other domains beyond locomotion.
Decision: Accept (poster)