FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning
Authors: Chenhao Li
Elijah Stanger-Jones
Steve Heim
Sangbae Kim
Venue: ICLR 2024 spotlight (2024)
Keywords: latent dynamics
motion representation and generation
representation learning
reinforcement learning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Motion trajectories offer reliable references for physics-based motion learning but suffer from sparsity, particularly in regions that lack sufficient data coverage. To address this challenge, we introduce a self-supervised, structured representation and generation method that extracts spatial-temporal relationships in periodic or quasi-periodic motions. The motion dynamics in a continuously parameterized latent space enable our method to enhance the interpolation and generalization capabilities of motion learning algorithms. The motion learning controller, informed by the motion parameterization, operates online tracking of a wide range of motions, including targets unseen during training. With a fallback mechanism, the controller dynamically adapts its tracking strategy and automatically resorts to safe action execution when a potentially risky target is proposed. By leveraging the identified spatial-temporal structure, our work opens new possibilities for future advancements in general motion representation and learning algorithms.

Published as a conference paper at ICLR 2024
FLD: FOURIER LATENT DYNAMICS FOR STRUCTURED MOTION REPRESENTATION AND LEARNING
Chenhao Li, Elijah Stanger-Jones, Steve Heim, Sangbae Kim
Department of Mechanical Engineering, Massachusetts Institute of Technology [chenhli, elijahsj, sheim, sangbae] ) @mi t edu
ABSTRACT
Motion trajectories offer reliable references for physics- based motion learning but suffer from sparsity, particularly in regions that lack sufficient data coverage. To address this challenge, we introduce seff-supervised, structured representation and generation method that extracts spatial-temporal relationships in periodic or quasi-periodic motions. The motion dynamics in a continuously parameterized latent space enable method to enhance the interpolation and generalization capabilities of motion learning algorithms. The motion learning controller, in- formed by the motion parameterization, operates online tracking of a wide range of motions, including targets unseen during training. With a fallback mechanism, the controller dynamically adapts its tracking strategy and automatically resorts to safe action execution when a potentially risky target is proposed. By leverag- ing the identified spatial-temporal structure, our work opens new possibilities for future advancements in general motion representation and learning algorithms.
INTRODUCTION
The availability of reference trajectories, such as motion capture data, has significantly propelled the advancement of motion learning techniques (Peng et all, 2018 Bergamin all 2019: Peng all 2021: 2022: Stal P022; all (023)a)) However, it is difficult to generalize policies using these techniques motions outside the distribution of the available data (Peng et al, 202T: al 20233). A core reason is that, while the trajectories in the data itself are induced by some dynamics of the system, the learned policies are typically trained to only replicate the data, instead of under- standing the underlying dynamics structure. In other words, the policies attempt to memorize the trajectory instances rather than learn to predict them systematically. Consequently, the gaps between trajectories present challenges for these models in accurately representing and learning motion in- terpolations or transitions, resulting in limited generalizations (Wiley & Hahn, 1997; Rose et all 1998). Moreover, the high nonlinearity and the embedded high-level similarity hinder data-driven methods from effectively identifying and modeling the dynamics of motion patterns (Peng et all 2018). Therefore, addressing these challenges requires systematic understanding and leveraging the structured nature of the motion space.
Instead of handling raw motion trajectories in long-horizon, high-dimensional state space, struc- tured representation methods introduce certain inductive biases during training and offer an efficient approach managing complex movements (Min Char, 2012: Lee et all (0ZIT). These methods focus on extracting the essential features and temporal dependencies of motions, enabling more ef- fective and compact representations (Lee et 2010; Levine all 2012). The ability to understand and capture the spatial-temporal structure of the motion space offers enhanced interpolation and gen- eralization capabilities that can augment training datasets and improve the effectiveness of motion generation algorithms (Holden ei all X0T: [scen et all 2018: Ibarz ald 2021). By uncovering and utilizing the underlying patterns and relationships within the motion space, continuous and rich sets of motions can be produced that progress realistically in smooth and temporally coherent manner (Siarke et all 2020; 2022: 2023).
In this work, present Fourier Latent Dynamics (FLD), generative extension to Periodic Autoen- coder (PAE) (Starke et all 2022) that extracts spatial-tempor relationships in periodic or quasi- periodic motions with novel predictive structure. FL.D efficiently represents high-dimensional

---

Published as a conference paper at ICLR 2024
trajectories by featuring motion dynamics in continuously parameterized latent space that accom- modates essential features and temporal dependencies natural motions. The enforcement of latent dynamics empowers FLD to enhance the proficiency and generalization capabilities of motion learn- ing algorithms with accurately described motion transitions and interpolations. The motion learning controllers, informed by the latent parameterization space of FLD,, demonstrate extended online tracking capability. A novel fallback mechanism enables the learning agent to dynamically adapt its tracking strategy, automatically identifying and responding to potentially risky targets by rejecting and reverting to safe action execution. Finally, combined with adaptive learning algorithms, FLD presents strong long-teny learning capabilities open- learning tasks, strategically navigating and advancing through novel target motions while avoiding unlearnable regions.
In summary, our contributions include: (i) self-supervised, structured representation and genera- tion method featuring continuously parameterized latent dynamics for periodic or quasi-periodic mo- tions. (ii) A motion learning and online tracking framework empowered by the latent dynamies with fallback mechanism. (iii) Supplementary analysis of long-term learning capability with adaptive target sampling on open-ended motion learning tasks. Supplementary videos and more details for this work are available https: /ttt://sttsssetteeeee woo.sstossssoeleeeeooooooooooooe gooole com wwwm//ieee/ce/1222111ddee
2 RELATED WORK
Motions are commonly described long-horizon trajectories in high- sighh-iimesional state space. How- ever, directly associating motions with raw trajectory instances yields highly inefficient representa- tions and poor generalization that fail capture motion features (Watter all 2015; |inn all 2016). In comparison, representing motions a structured manner allows learning algorithms to better comprehend the underlying patterns and relationships within the motion space.
While a straightforward approach is to parameterize motions from physical dynamics, determining explicit models with correct dynamics structures solely from kinematic observations can be challeng- ing without prior knowledge of the underlying physics (Li all, 2023a). Structured trajectory gener- ators address this issue by providing motion controllers with parameterized references with sufficient kinematic information. Some classical locomotion controllers parameterize the movement of each actuated degree of freedom by relying on cyclic open-loop trajectories described by sine curves (Man' et all 2018) or central pattern generators (Jsspertt, 2000; Gay et all, 2013 Dörfler & Bulld, 201
Shafee el all )02K)) Recent works enable dynamical adaptation of high-level motion by having the control policy directly modulate the parameters of the trajectory generator, thus modifying the dictated trajectories (Iscen all 2018 Lee el all 2020; Miki aLal 2022). In contrast to explicitly defined trajectory parameters, supervipe models such autoencoders explain motion evolu- tion in a latent space. These representation methods have shown success in controlling non- non-linear dynamical systems (Watter et all 2015), enabling complex decision-making (Ha & Schmidhubei. 2018), solving long-horizon tasks (Hatner et all, 2019), and imitating motion sequences (Berseth X A recent practice attempts to identify motion dynamics common latent space to foster temporal consistency between different dynamical systems (Kim 2020).
Another line of structured motion representation involves extracting trajectory features in the fre- quency domain. Frequency domain methods have been proposed for various motion-related tasks, including synthesis (Lm all 1994), editing (Bruderlin & Wilbiams 1995), stylization (Umuma et all 1995; Yumer & Mitra. 2016), and compression (Beaudoi et all 2007). To consider the correlation between different body parts, recent work on PAE constructs latent space using an autoencoder structure and applies frequency domain conversion as an inductive bias (Starke all 2022). The extracted latent parameters have been tested as effective full-body state representations in downstream motion learning tasks (Starke et all 2023). Despite such progress, PAE is restricted to representing local frames and is not fully exploited to express overall motions or predict them.
3 PRELIMINARIES
Despite the success of self-supervised learning schemes in solving complex tasks, existing self- supervised learning methods inevitably overlook the intrinsic periodicity in data (Yang et all, (0222). Less attention has been paid to designing algorithms that capture prevalent periodic or quasi-periodic temporal dynamics in robotic tasks. To this end, we explore representation methods with explicit
2

---

Published as a conference paper at ICLR 2024
account of periodicity inspired by PAE (Starke et all 2022) and develop generative capabilities thereon.
PAE addresses the challenges of learning the structure of the motion space, such as data sparsity and the highly nonlinear nature the space, by focusing on the periodicity of motions in the frequency domain. The structure of PAE is illustrated in Fig. STa We denote trajectory segments of length H in d-dimensional state space preceding time step by S, = (st- H. 8,) E RdxH as the input PAB. The autoencoder structure decomposes the input motions into latent channels that accom- modate lower-dimensional embedding &, E RexH of the motion input. A following differentiable Fast Fourier Transform obtains the frequency ft, amplitude at, and offset b, vectors of the latent trajectories, while the phase vector 6e is computed with separate fully connected layer. We denote this parameterization process with p. we have
2e enc(s:),
b, = p(z:),
(1)
where 6r. fr. ae. b, E R. Next, the reconstructed latent trajectory segments &, E Rex I are com- puted using sinusoidal functions parameterized by the latent vectors with
2,- = p(o fi, at b,) a sin (2x + o.)) +,,
(2)
where á denotes the reconstruction, T is the time window corresponding to the state transition horizon H. Finally, the network decodes the reconstructed latent trajectories 2, to the original motion space, and the reconstruction error is computed with respect to the original input
dec((.)) 2.), Lo MSE($,, Sc);
where s, E dx H1 and MSE denotes the Mean Squared Error. The network structure of PAE is described in Table L. We refer to the original work (Starke etal 2022) for more details.
PAE extracts multi-dimensional latent space from full-body motion data, effectively clustering motions and creating manifold in which computed feature distances provide more meaningful similarity measure compared to the original motion space as visualized in Fig. !!
4 APPROACH
4.1 PROBLEM FORMULATION
We consider the state space S and define a motion sequence (Sg, $1....) drawn from a reference dataset M trajectory of consecutive states E S. Our research focuses on creating physics- based learning controller capable of not only replicating motions prescribed by the reference dataset but also generating motions accordingly in response novel target inputs, thereby enhancing its generality across a wide range of motions beyond the reference dataset. To this end, we adopt two-stage training pipeline. In the first stage, efficient representation model is trained the reference dataset and a continuously parameterized latent space is obtained where novel motions can be synthesized by sampling the latent encodings. The second stage involves developing an effective learning algorithm that tracks the diverse generated target trajectories. In both motion representation and generation, we highlight the importance of identifying periodic or quasi-periodic changes in the underlying temporal progression of the motions that commonly describe robotic motor skills.
4.2 FOURIER LATENT DYNAMICS
By inspecting the parameters of the latent trajectorics of periodic or quasi-periodic motions encoded by PAE, observe that the frequency, amplitude, and offset vectors stay nearly time-invariant along the trajectories. We introduce the quasi-constant parameterization assumption.
Assumption latent trajectory (z,, Z1+1....) can be approximated by (2,, 2..1....) with bounded enror = zz 211, where 2, = plgv. f. a, b), Vt' E (t,t+1.....
Assumption holds with low approximation errors for periodic or quasi-periodic input motion tra- jectories, which yield constant frequency domain features. Since these latent features learedd, the assumption can be explicitly enforced. In the following context, we denote by o the latent state and f. a, the latent parameterization. Here, we introduce Fourier Latent Dynamics (FLD), which
3

---

Published as a conference paper at ICLR 2024
trajectory segment
Sp-1
Sp+
convalution enc
latent embedding
Lg
FC BN Atan2
FFT
Pr
8:
LD
sinusoidal reconstruction 9,
2i
2isn
convolution dec
dec
dec
reconstructed segments;
$/4x
FLD training pipeline. During training, latent dynamics are enforced to predict proceeding latent states and parameterizations. The prediction loss is computed in the original motion space with respect to the ground truth future states.
Figure I:
enforces reconstruction of over the complete trajectory by propagating latent dynamics parameter- ized by local state Φ and constant set of global parameterization f. a, and b.
We formalize the latent dynamics of FLD and its training process in Fig. M For a motion segment (8t- ,+......, 8t) whose latent trajectory segment is parameterized by Φt,, St, dr, and b,, we approximate the proceeding motion segment 8t+i = (8t- 11-1+:;1;... 8t+s) with the prediction & decoded from i-step forward propagation E'4; using the latent dynamics from time step t.
Z1+; p(0:= + iffit, ae, 8 dec(2(+:),
(4)
where A/ denotes the step time. The latent dynamics in Eq. E assumes locally constant latent param- eterizations and propagates latent states by advancing local phase increments. We can compute the prediction loss at time t + i. In fact, the local reconstruction process employed by PAE can be viewed as regression on a zero-step forward prediction using the latent dynamies. We can perform regressions on multi-step forward prediction by propagating the latent dynamics and define the total loss for training FLD with the maximum propagation horizon of N and a decay factor a,
L FLLD Σ
a'L,
L = MSE 4/4 8(+:).
Training with the FLD loss enforces Assump. M in local range of N steps. By choosing the appro- priate maximum propagation horizon and the decay factor, one can balance the tradeoff between the accuracy of local reconstructions and the globalness of latent parameterizations. In comparison to PAE (N = 0), which lacks temporal propagation structure and performs only local reconstruction with local parameterization, FLD dramatically reduces the dimensions needed express the entire trajectory, which facilitates both motion representation and generation. The latent dynamics also en- able autoregressive motion synthesis. Starting from initial latent state and parameterization, FLD generates future motion trajectories in smooth and temporally coherent manner by propagating the latent dynamics and continually decoding the predicted latent encodings following Eq.
For the following discussions, we consider training FLD on the reference dataset M and define the latent parameterization space θ C 3e encompassing the latent frequency, amplitude, and offset. Therefore, each motion trajectory can be exclusively represented by time-dependent la- tent state de IR that describes the local time indexing and constant latent parameterization 0 = (ʃ, a. b) E 3e that describes the global high-level features of the motion. We establish this idea with a schematic view the latent manifold induced by Φe and Suppl.
4.3 MOTION LEARNING
Given reference trajectories, physics-based motion learning algorithms train control policy that ac- tuates the joints the simulated character or robot and reproduces the instructed motion trajectories. FLD is able to represent rich sets of motions efficiently. In contrast to discrete handcrafted mo- tion indicators, the feature distances in the continuously parameterized latent space FLD provide learning algorithms more reasonable similarity measure between motions.
4

---

Published as a conference paper at ICLR 2024
skill sampler Pe
0
db
policy n
sinusoidal reconstruction 2
convolution dec
r
target
state s
Figure 2: System overview. During training, the latent states propagate under the latent dynamics and are reconstructed to policy tracking targets each step. The tracking reward n is computed as the distance between the target & and the measured states s.
4.3.1 POLICY TRAINING
At the beginning of each episode, set of latent parameterization 8g E R is sampled from a skill sampler po (e.g. a buffer of offline reference motion encodings, more variants and ablation studies are detailed in Suppl. A.2.C.. The latent state Φn E R is uniformly sampled from fixed range U. The step update of the latent vectors follows the latent dynamics in Eq.
0, 0,-1, Φ Φ At.
At each step, the latent state and the latent parameterization are used to reconstruct motion segment
$; (š , &t dec(2,) dec(p(é,, 0()),
(7)
whose most recent state & serves as tracking target for the learning environment at the current time step. The tracking reward encourages alignment with the target and is formulated in Suppl. A.2.
The latent state and parameterization are provided to the observation space to inform the policy about the motion and the specific frame it should be tracking. The policy observation and action space are described in Suppl. A.2.1 Figure A provides a schematic overview of the training pipeline, and algorithm overview is detailed in Algorithm II
4.3.2 ONLINE TRACKING AND FALLBACK MECHANISM
During the inference phase, the policy structure incorporates real-time motion input as tracking targets, irrespective their periodic or quasi-periodic nature. The latent parameterizations of the in- tended motion are obtained online using the FL.D encoder. Figure SI2 provides provvdd a schematic overview of the online tracking process. However, arbitrary reference inputs distant from the training distribu- tion result in limited tracking performance and potentially hazardous motor execution. Conse- quently, an online evaluation process is essential to assess the safety of a dictated motion, along with providing fallback mechanism to ensure the availability of an alternative safe target when necessary. To address this requirement, FLD naturally induces a process that leverages the central role of latent dynamics. Consider the input sequence consisting of trajectory segments si (s) # st) at each time step. These segments are stored in an input buffer I; (S Given the current latent state Φt, parameterization 0, and the input buffer I, our algorithm aims to determine the values of Φ and 0.+1 for the subsequent step.
ss) of length N.
In the absence of user input, represented by I₁ 0. the latent parameters simply propagate the latent dynamics based on Eq. M Conversely, when user input is present, we perform reconstruction and forward prediction on the state segments stored in the input buffer I,, based upon the earliest recorded segment s #1- The prediction is evaluated using the same loss metric L employed in Eq. 8 to measure the dissimilarity between predicted and actual state trajectories within I, This loss metric provides a quantification similarity in state transitions and system evolution. Consequently, small value of L indicates that the input motion adheres to comparable spatial-temporal relationships and exhibits periodicity akin to those observed in the training dataset.
the disparity in dynamics between motions, focusing on the
To determine the suitability of an input tracking target for acceptance or rejection, we establish a threshold EFLD derived from training statisties. When input motion is accepted, the updated
5

---

Published as a conference paper at ICLR 2024
input buffer 3
yes
3. no
prediction loss L
yes
LonD EFLO
Φe
n1
Pe1 1, p(enc(s)))
Pet1, Bett 0r fr 8:
(a) Target evaluation process
(b) Schematic overview
Figure 3: Online tracking and fallback mechanism. (a) The prediction loss L evaluated within imput buffer of user-proper pee-pproosee tracking targets. The mechanism accepts the proposal only when the prediction loss is below a threshold EFLD- (b) The proposed tracking targets (dashed curve) may contain risky states (dashed red dots). The fallback mechanism identifies these states and defaults them to safe alternatives (dashed red arrows and green dots) by propagating latent dynamics. Note that the real-time tracking trajectories are necessarily periodic or quasi-periodie.
tracking target is encoded from the latest state trajectory segment st within the input buffer I. This evaluation process is formulated in Fig. Gal Additionally, Fig. ST presents a schematic overview of the online tracking and fallback mechanism.
EXPERIMENTS
We evaluate FLD on the MIT Humanoid robot (Chignoli et all 2021), with which we show its applicability to state-of-the-art real-world robotic systems. We the human locomotion clips collected in Peng et al. (2018) retargeted to the joint space of our robot the reference motion dataset containing slow and fast jog. forward and backward run, slow and fast step place, left and right turn, and forward stride. We visualize some representative motions in Fig. Ny. Note that the motion labels are not observed during the training of the models and are only used for evaluation. In the motion learning experiments, we use Proximal Policy Optimization (PPO) (Schulman et all 20 in Isaac Gym (Rudin all 2022). Suppl. A. and Suppl. A.2 provide further training details.
5.1 STRUCTURED MOTION REPRESENTATION
We compare the motion embeddings of the reference dataset obtained from training FLD follow- ing Sec. g with different models, with parameters detailed in Suppl. A. The state space is specified in Suppl. A.I. After the computation of the latent manifold, we project the principal components of the phase features onto two-dimensional plane, outlined in (Starke et all 2022). We then compare the latent structure induced by FLD with that by PAE. Additionally, we adopt Variational Autoencoder (VAE) as commonly employed method for representing motions lower-dimensions space. Lastly, we plot the principal components of the original motion states for comprehensive analysis. We illustrate the latent embeddings acquired by these models in Fig. where each point corresponds to latent representation of trajectory segment input.
We elucidate the results by highlighting the inductive biases imposed during the training of these models. In the provided figures, samples from the same motion categories are assigned the same color; indicating close relationship between neighboring frames within the embedding. This spatial relationship implicitly enforced by the reconstruction process of all the models, promoting latent encodings of close frames to remain proximate in the latent space. However, the degree of temporal structure enforcement varies significantly, owing to the differing inductive biases.
Notably, FLD demonstrates the most consistent structure akin to concentric cycles, primarily due to the motion preedictree structure within the latent dynamics enforced by Eq. B The cycles depicted in the figures represent the primary period of individual motions. The angle around the center (latent state) signifies the timing, while the distance from the center (latent parameterization) represents the

---

Published as a conference paper at ICLR 2024
(a) Original
(b) VAB
(c) PAE
(d) FLD
Figure 4: Latent manifolds for different motions. Each color associated with a trajectory from motion type. The arrows denote the state evolution direction. FLD presents the strongest spatial- temporal relationships with explicit latent dynamics enforcement. PAE witnesses a similar but weaker pattern with local sinusoidal reconstruction. In comparison, VAE enables only spatial close- ness, and the trajectories of the original states are the least structured.
high-level features (e.g. velocity, direction, contact frequency, etc.) that remain consistent through- out the trajectory. This pattern reflects the strong temporal regularity captured by Assump. M. which preserves time-invariant global information regarding the overall motion. As PAE can viewed observe a weaker pattern in the latent FLD with zero-step latent propagation, in contrast, manifold PAE, where the consistency of high-level features holds only locally. Finally, the re- construction process employed in VAE training does not impose any specific constraints the temporal structure of system propagation. Consequently, the resulting latent representation, except for the direct encoding, exhibits the least structured characteristics among the models.
Powered by the latent dynamics, FLD offers a compact representation of high-dimensional motions by employing the time index vector o, and assuming high-level feature consistency 0 throughout each trajectory. Conversely, PAE encodes motion features only locally 0, = (())). The numbers of parameters of different models used to express a trajectory of length |r] are listed in Table M
Table 1: Motion representation parameters
Model
Original
VAE
PAE
FLD
# Parameters
dx
|r|
cx
(17l
H H+1) 4c x (|T H + 1)
4c
5.2 MOTION RECONSTRUCTION AND PREDICTION
We demonstrate the generality FLD reconstructing and predicting unseen motions during train- ing. Figure 55 (left) illustrates representative validation with diagonal run motion. At time f. 65, FLD undertakes motion reconstruction and prediction for future state transitions based on the most recent information S,, elaborated in Sec. Р.2. For comparison, we train PAE and feed-forward (FF) model with fully connected layers with the same input and output structure as FLD.
It evident that the motion predicted by FLD aligns with the actual trajectories. Partieularly in joint position evolution which presents strong sinusoidal periodicity, it exhibits the lowest relative error e. The superiority of FLD is especially pronounced in long-term prediction regions, where the other models accumulate significantly larger compounding errors. The effectiveness of FLD in accurately predicting motion for an extended horizon is attributed the latent dynamics enforced with an appropriate propagation horizon N in Eq. EL In the extreme case of N 0 (PAE), the relative error is larger due to the weaker temporal propagation structure. The result on the diagonal run trajectory demonstrates the ability of FLD to accurately predict future states despite not being exposed to this specific motion during training. This showcases the generalization capability of FLD, it effectively captures the underlying dynamics and temporal relationships inherent in the training dataset, which are prevalent and can be adapted to unseen motions. In comparison, the FF model training fails understand the spatial-temporal structure in the motions and results in strong
7

---

Published as a conference paper at ICLR 2024
FF
PAE
FLD
step run stride
2.5
b
[m
0.0
fleg
raad]
0.5 s
0
0.0
Garm [rad]
0.50
0.25
0
50 65
100
150
200
t [step]
Figure 5: Motion reconstruction and prediction of diagonal run trajectory (left). The solid and dashed curves denote the ground truth and predicted state evolution. The relative prediction error (vivid) of FF, PAE and FLD depicted with the axis indicating on the right. Latent offset (right) of step in place, forward run, and forward stride. Each radius denotes a latent channel.
slow jog
forward stride
spinkick
fast step
transition
slow jog
forward stridle
spinkick
fast step
2.5
[rad|[rac) ///s/ 11mm
0.0
3
0.0
2.5
2
Qleg
0
fast stop
transition
forward stride
rrad] [m/s]
2.5
Qleg v
-0.1
0
100
200
300
[step]
Figure 6: Motion tracking and fallback (top) and motion transition (bottom). The dashed curves denote the user-specified tracking target, and the solid ones denote the measured system states. The corresponding latent manifolds are depicted on the right side.
overfitting to the training dataset, thus limiting its generality. Moreover, the dedicated FF model solely propagates the states through autoregression and does not provide any data representation.
With the embedded motion motion-predieee predictive structure, the enhanced generality achieved by FLD is at- tributed to the well-shaped latent representation space, where sensible distances between motion patterns established. Figure El (right) depicts the latent offsets of step in place, forward run, and forward stride, where the parameterization of the intermediate motion (forward run) is distributed in between. This high-level understanding of motion similarity is further exemplified in Fig. SHI on different motion types (illustrated in Fig. (9) and forward run velocities.
5.3 MOTION TRACKING AND FALLBACK
Following Sec. we can learn a motion tracking controller that employs FLD parameterization space. We perform online tracking experiment where real-time user input of various motion types provided to the controller as tracking targets.
In the first example (Fig. R. top). switch the input motion different type every 100 time steps (indicated by vertical grey lines). Notably, one of the input motions, referred to as spinkick (visualized in Fig. ST77), lives far from the training distribution and is considered risky input. We
8

---

Published as a conference paper at ICLR 2024
observe that the controller achieves accurate user input tracking, as evidenced by the close alignment between the dictated (dashed) and measured (solid) states, except for the spinkick motion. At time t 200, when the proposed states of the spinkick motion received, FLD evaluates the latent dynamics loss LE and rejects the proposal. This decision is based on the limited similarity cal- culated between the proposed system evolution and the state propagation prevalent in the training dataset. In response to the rejected motion, the fallback mechanism is triggered, providing safe alter- native reference states that extend from the previous motion. Consequently, the controller continues to track the forward stride motion, with the actual reference states indicated by the dashed curves from the previous region.
Moreover, the controller demonstrates the ability to transition between different tracking targets smoothly. By considering the tracking of an arbitrary motion as a process of wandering between continuously parameterized periodic priors, FLD dynamically extracts essential characteristics of local approximates. To further understand the performance of FL.D and the learning agent on track- ing motions that fall into the gaps between trajectories captured in the reference dataset, we construct in the second example (Fig. E. bottom) a transition phase where the target motion parameterizations obtained from linear interpolation between the source and target motions. In particular, the in- terpolated movements exhibit a gradual evolution of high-level motion features, providing a clear and structured transition from high-frequency, low-velocity stepping low-frequee low-frequency, high-velocity striding sequences. This gradual evolution of motion features in the interpolated trajectories suggests that FLD is capable of capturing and preserving the essential temporal and spatial relationships of the underlying motions. It bridges the gap between different motion types and velocities, generating coherent and natural motion sequences that smoothly transition from one another.
5.4 EXTENDED DISCUSSION: SKILL SAMPLER DESIGN
Our experiment provides strong evidence that the motion learning policy informed by the FLD latent parameterization space effectively achieves motion in- in-betweening and coherent transitions that en- capsulate high-level behavior migrations. While this policy already shows remarkable performance when trained using targets derived from offline reference motions, its potential can be further ampli- fied with skill samplers that can continually propose novel training targets, which lead to enhanced tracking generality onto wider range of motions.
To this end, we perform ablation studies with different skill sampler implementations, including learning-progee progress-based based online curriculum (ALPGMM), and evaluate policy tracking general- ity beyond the reference dataset. The policy trained with A ALPGMM demonstrates the ability to acquire knowledge of the continuously parameterized latent space through interactions with the en- vironment and achieves enhanced performance in general motion tracking tasks as opposed to the fixed offline target sampling scheme (OFFLINE). We direct interested readers to Suppl. A.2. and Suppl. A.2.1 for supplementary experiments regarding this extended discussion.
6 CONCLUSION
In this work, we present FLD, novel self-supervised, structured representation and generation method that extracts spatial-temporal relationships in periodic or quasi-periodic motions. FLD ef- ficiently represents high-dimensional trajectories by featuring motion dynamics in a continuously parameterized latent space that accommodates essential features and temporal dependencies of nat- ural motions. Compared with models without explicitly enforced temporal structures, FLD signifi- cantly reduces the number of parameters required to express non-linear trajectories and generalizes accurate state transition prediction unseen motions. The enhanced generality by FLD is further confirmed with the high-level understanding of motion similarity by the latent parameterization space. The motion learning controllers, informed by the latent parameterization space, demonstrate extended online tracking capability. Our proposed fallback mechanism equips learning agents with the ability to dynamically adapt their tracking strategies, automatically recognizing and responding to potentially risky targets. Finally, our supplementary experiments on skill samplers and adaptive learning schemes reveal the long-term learning capabilities of FLD, enabling learning agents to strategically advance novel target motions while avoiding unlearnable regions. By leveraging the identified naatial-tempora structure, FLD opens up possibilities for future advancements in motion representation and learning algorithms.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
The experiment results presented in this work can be reproduced with the dataset and implementation details provided in Suppl. A. and Suppl. A22 The code has been open-sourced on the project page.
ETHICS STATEMENT
Our research on Fourier Latent Dynamics (FLD) involves constructing a meaningful parameteriza- tion to encode, compare, and predict paatialtemmorra relationships in data inputs. While focus on its applicability robot motion learning tasks, we acknowledge the existence of concerns about the potential misuse of the model broader range of data types.
From our experience, this approach is primarily constrained by the availability of rich training data, along with accurate input data during run-time. Our reliance data from motion-capture tech- niques, which necessitate a lab environment or simulations with direct ground-truth access, limits its practicality outside lab settings. However, this may change in the future with advancements in motion reconstruction from standard video feeds.
The model could also be used to generate plausible but fictitious actions for individuals, potentially leading to misinterpretations or misjudgments about their capabilities or intentions. If integrated into automated systems, FLD's predictive capabilities could influence decision-making processes. While this might enhance efficiency, it could pose risks if the model's predictions are inaccurate or based on biased data, leading to harmful decisions.
ACKNOWLEDGMENTS
We thank the members of the Biomimetic Robotics Lab for the helpful discussions and feedback the paper. We are grateful to MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources. This research was funded by NAVER Labs and Advanced Robotics Lab of LG Electronics Co., Ltd.
REFERENCES
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert Schapire. The nonstochastic multi- armed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.
Philippe Beaudoin, Pierre Poulin, and Michiel de Panne. Adapting wavelet compression to human motion capture clips. In Proceedings of Graphics Interface 2007, pp. 313-318, 2007.
Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. Drecon: data-driven responsive control of physics- based characters. ACM Transactions On Graphics (TOG), 38(6): 1-11, 2019.
Glen Berseth, Florian Golemo, and Christopher Pal. Towards learning to imitate from a single video demonstration. arXiv preprint arXiv:1901.07186. 2019.
Armin Bruderlin and Lance Williams. Motion signal processing. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pp. 97-104, 1995.
Matthew Chignoli, Donghyun Kim, Elijah Stanger-Jones, and Sangbae Kim. The mit humanoid robot: Design, motion planning, and control for acrobatic behaviors. In 2020 ZEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), pp. 1-8. IEEE, 2021.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the algorithm. Journal of the royal statistical society: series B (methodological), 39(1): 1-22, 1977.
Florian Dörfler and Francesco Bullo. Synchronization in complex networks of phase oscillators: A survey. Automatica, 1036)15539-1544 2014.
Chelsea Finn, Xin Yu Tan, Yan Duan. Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and
Automation (ICRA), pp. 512-519. IEEE, 2016.
10

---

Published as a conference paper at ICLR 2024
Sébastien Gay, José Santos- Victor, and Auke Ijspeert. Learning robot gait stability using neural networks as sensory feedback function for central pattern generators. In 2013 JEEE/RSJ interna- tional conference intelligent robots and systems, pp. 194- 201-201. Icee, 2013.
David Ha and Jürgen Schmidhuber. World models. arXiv preprini arXiv: 180:: 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1912v0/9/..0/110 2019.
Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan. Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learningg lessons have learned. The International Journal of Robotics Research, 40(4-5):698-7211 2021.
Auke Jan Ijspeert. Central pattern generators for locomotion control in animals and robots: review. Neural networks, 21(4):642-6) 653, 2008.
Atil Iscen, Ken Caluwaerts, Jie Tan, Tingnan Zhang, Erwin Coumans, Vikas Sindhwani, and Vincent Vanhoucke. Policies modulating trajectory generators. In Conference on Robot Learning, pp. 916- 926. PMLR, 2018.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
Nam Hee Kim, Zhaoming Xie, and Michiel Panne. Learning to correspond dynamical systems. In Learning for Dynamics and Control, pp. 105-117. PMLR, 2020.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
Seyoung Lee, Sunmin Lee, Yongwoo Lee, and Jehee Lee. Learning family of motor skills from single motion clip. ACM Transactions on Graphics (TOG), 1034):1-13, 2021.
Yongjoon Lee, Kevin Wampler, Gilbert Bernstein, Jovan Popović, and Zoran Popović. Motion fields for interactive character locomotion. In ACM SIGGRAPH Asia 2010 papers, pp. 1-8. 2010.
Sergey Levine, Jack M Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun. Continuous character control with -ow-dimesiiona embeddings. ACM Transactions on Graphics (TOG), 31 (4):1-10, 2012.
Chenhao Li, Sebastian Blaes, Pavel Kolev, Marin Vlastelica, Jonas Frey, and Georg Martius. Ver- satile skill control via self-supervised adversarial imitation of unlabeled mixed motions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2944-29500 IEEE, 2023a.
Chenhao Li, Marin Vlastelica, Sebastian Blaes, Jonas Frey, Felix Grimminger, and Georg Martius. Learning agile skills via adversarial imitation of rough partial demonstrations. In Conference on Robot Learning, pp. 342-3422-55 352.. PMLR, 2023b.
Zicheng Liu, Steven Gortler, and Michael F Cohen. Hierarchical spacetime control. Proceedings of the 21 annual conference Computer graphics and interactive techniques, pp. 35-42, 1994.
Manuel Lopes and Pierre ves Oudeyer. The strategic student approach for life-long exploration and learning. In 2012 IEEE international conference on development and learning and epigenetic robotics (ICDL), pp. 1-8. IEEE, 2012.
Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7 (62)xeabk2822, 2022.
Jianyuan Min and Jinxiang Chai. Motion graphs++ a compact generative model for semantic motion analysis and synthesis. ACM Transactions Graphics (TOG), (1 2012.
11

---

Published as a conference paper at ICLR 2024
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018.
Deepmimic: Example-
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee, Jie Tan, and Sergey Levine. Learn- ing agile robotic locomotion skills by imitating animals. arXiv preprint 2Xiv:2004.007844 2020.
Xue Bin Peng. Ze Ma. Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics- based character control. ACM Transactions on Graphies (ToG), 40(4):1-20, 2021.
Xue Bin Peng. Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 41()::1-17 2022.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. In Conference on Robot Learning, pp. $35-853. PMLR, 2020.
Carl Rasmussen. The infinite gaussian mixture model. Advances in neural information processing systems, 12, 1999.
Charles Rose, Michael F Cohen, and Bobby Bodenheimer. Verbs and adverbs: Multidimensionall motion interpolation. JEEE Computer Graphics and Applications, 18(5):32-40, 1998.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp. 91-100. PMLR, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Milad Shafice, Guillaume Bellegarda, and Auke Ijspeert. Deeptransition: Viability leads to the emer- gence of gait transitions in learning anticipatory quadrupedal locomotion skills. arXiv preprint arXiv:2306.074/9 2023.
Paul Starke, Sebastian Starke, Taku Komura, and Frank Steinicke. Motion in-betweening with phase manifolds. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1- 17, 2023.
Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. Local motion phases for learning multi-contact character movements. ACM Transactions on Graphics (TOG), 39(4):54 2020.
Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(:)1111 2022.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804. 10332, 2018.
Munetoshi Unuma, Ken Anjyo, and Ryozo Takeuchi. Fourier principles for emotion based human figure animation. In Proceedings of the 22nd annual conference on Computer graphics and inter- active techniques, pp. 91-96, 1995.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed control: locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
Douglas J Wiley and James K Hahn. Interpolation synthesis of articulated figure motion. IEEE Computer Graphics and Applications, 17(6):39-45, 1997.
Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, and Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. arXiv preprint 0:Xiv:2210.03115. 2022.
M Ersin Yumer and Niloy J Mitra. Spectral style transfer for human motion between independent actions. ACM Transactions Graphics (TOG), 35(4):1-8, 2016.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
MOTION REPRESENTATION DETAILS
A.1.1 STATE SPACE
The state space is composed of base linear and angular velocities v, w in the robot frame, measure- ment of the gravity vector in the robot frame and joint positions g as in Table S2
Table S2: Policy observation space
Entry
Symbol
Dimensions
base linear velocity base angular velocity
0:3
3:6
projected gravity
9
6:9
joint positions
9
9:27
A.1.2 REPRESENTATION TRAINING PARAMETERS
The learning networks and algorithm are implemented in PyTorch 1.10 with CUDA 12.0. Adam used as the optimizer for training the representation models. The information is summarized in Table S33
Table S3: Representation training parameters
Parameter
Symbol
Value
step time seconds
At
0.02
max iterations
5000
learning rate weight decay learning epochs
0.0001
0.0005
mini-batches
10
latent channels
C
trajectory segment length
H
51
FL.D propagation horizon
N
50
propagation decay
1.0
approximate training hours
Periodic Autoencoder
FLD shares the same network architecture PAE, whose encoder and decoder composed of ID convolutional layers. The periodicity in the latent trajectories is enforced by parameterizing each latent curve as sinusoidal function. While the latent frequency, amplitude, and offset computed with a differentiable real Fast Fourier Transform layer, the latent phase is determined using linear layer followed by Atan2 applied on 2D signed phase shifts on each channel. The network architecture is detailed in Table (S4)
Variational Autoencoder
VAB implemented as our baseline for representing motions in a lower-dimensional space. In our comparison. it admits the same input, output data structure and the same latent dimension as PAE. The network architecture detailed Table
Other baselines
The dedicated feed-forward model shares the same input and prediction output structure FLD, however, does not provide any representation of the motion it predicts. is trained to evaluate the motion synthesis performance of FL.D.
13

---

Published as a conference paper at ICLR 2024
0.5
trajectory segment
0.0
0.5
convolution enc
latent embedding
Lo
FC BN Atan2
FFT
1.6
d
0.8
0.0
sinusoidal reconstruction
1.2
convolution dec
0.3
-0.6
reconstructed S
50
100
150
[step]
(a) PAE structure.
Latent parameters along forward trajectory.
Figure S7: Motion representation using PAE. (a) PAE utilizes frequency domain analysis to extract the local periodicity of highly nonlinear motions. Latent features are constructed using sinusoidal functions. (b) Each color is associated with a distinct latent channel. Despite the fluctuation on two frequency channels, the latent frequency f. amplitude a, and offset stay nearly constant throughout the trajectory.
Table S4: PAE architecture
Network
Layer
Output size
Kernel size
Normalization
Activation
encoder
Convld
x
BN
ELU
Convld Convld
64 x 51
51
BN
ELU
x 51
51
BN
ELU
phase encoder
Linear
8 x 2
BN
Atan2
decoder
Convid Convid Convid
64 x 51
51
BN
ELU
x
BN
ELU
27 x 51
51
BN
ELU
The oracle classifier is trained to predict original motion classes from their latent parameterizations for evaluation purposes. trained provide better understanding of the adaptive curriculum learning migration in the latent parameterization space using privileged motion type information.
The network architectures of these models are presented in Table SG
A.1.3 LATENT MANIFOLD
Each motion within the set M maintains a consistent latent parameterization throughout its tra- jectory. Consequently, its latent representation can be visualized as circle (solid grey curves) in Fig. SNE where represents the distance from the center. The current state a specific time frame is denoted by the latent state di, which is depicted as the angle elapsed along the circle. The propa- gation of latent dynamics and motion evolution can thus be described as traveling around the circle. All motions within M correspond to collection of circles that span latent subspace, represented by the shaded rings. Therefore, we can interpolate or synthesize new motions by sampling within around this latent subspace. It is important to note that this latent subspace may be ill-defined for policy learning, as it may contain subregions where the decoded motions describe challenging unlearnable movements for the real learning system.
14

---

Published as a conference paper at ICLR 2024
Table S5: VAE architecture
Network
Type
Hidden
Activation
encoder
MLP
512, 256, 128
ReLU
mean encoder
Linear
standard deviation encoder
Linear
Softplus
decoder
MLP 128, 256, 512
ReLU
Table S6: Other baseline architectures
Network
Symbol
Type
Hidden
Activation
MLP
512, 512 1024, 512
ELU ELU
classifier
MLP
A.1.4 REFERENCE MOTIONS
A.1.5 SIMILARITY EVALUATION
Both figures clearly demonstrate a well-understood similarity between high-level motion features. In Fig. STa.l the latent representation of the intermediate motion (run, orange curves) is positioned between their neighboring representations (step in place and forward stride, red and green curves). A similar pattern is also identified in Fig. STOR, with more consistent latent representation within the same motion type (run forward), compared to more considerable structural discrepancy between different motion types in Fig. STOa As result of this well-understood spatial-temporal structure, sampling the latent representation space leads to realistic transitions and interpolations. This ability to fill in the gaps and generate rich set of natural motions from sparse offline datasets showcases the efficacy of FLD motion generation tasks. In summary, the automatically induced latent representation space θ in FLD plays crucial role in capturing the underlying spatial-temporal relationships and facilitating realistic motion generation through interpolation and transition. This advancement enables the algorithm greatly enriching available training datasets that improve the efficiency of the downstream learning process and the capability of learned policies.
learn from sparse offline data and generalize proficiently,
A.2 MOTION LEARNING DETAILS
A.2.1 OBSERVATION AND ACTION SPACE
Besides the state space, the policy observes additional information such as joint velocities i and last action a' in its observation space. Most importantly, the policy observes the instructed target motions by admitting their latent states and parameterizations. The detailed information together with the artificial noise added during training to increase the policy robustness presented in Table S7
The action space is of 18 dimensions and encodes the target joint position for each of the 18 actuators. The PD gains are set to 30.0 and 5.0, respectively.
A.2.2 POLICY TRAINING PARAMETERS
Adam is used as the optimizer for the policy and value function with an adaptive learning rate with a KL divergence target of 0.01. The policy runs at 50 Hz. All training done by collecting experiences from 4096 uncorrelated instances of the simulator in parallel. The information is summarized in Table SS
A.2.3 NETWORK ARCHITECTURE
The network structures of the learning policy ग and the value function V used PPO training are detailed in Table 5y
15

---

Published as a conference paper at ICLR 2024
M
(a) FLD latent manifold.
(b) Policy learned latent manifold.
Figure S8: Schematic view of the latent manifold induced by the latent state and latent parameter- ization of FLD. While the latent parameterization 0 determines which motion the current state is experiencing, the latent state Φe indicates the time index of the state frame on this motion. (a) Each motion is represented by solid grey circle. The shaded rings denote the collection of representa- tions of motions in the offline dataset M. (b) The grey circle denotes an unlearnable motion and the grey shaded ring denotes the unlearnable subspace. The green eircles represent learnable motions, with the dashed one denoting motion outside the offline dataset M but acquired during training. The green shaded ring denotes the motion region the policy eventually masters.
(a) Slow step place.
Forward run.
(c) Forward stride.
Figure S9: Representative motions in the training dataset. Base linear and angular displacement is integrated from velocity information.
A.2.4 DOMAIN RANDOMIZATION
In addition to observation noise, domain randomization applied during training to improve policy robustness for real-world application scenarios. On the one hand, the base mass of the parallel
16

---

Published as a conference paper at ICLR 2024
b
a
5
stop in place
forward run
forward stride
Latent representation different motion types.
a
900s IN
m/s
m/s
m/s
Latent representation of different forward velocities.
Figure S10: Similarity evaluation and motion interpolation in the latent representation space. The connected lines across the latent channels indicate the mean latent representation trajectories from the same motion same velocity. The relative locations between representations of related trajectories illustrate the well-understood similarity between high-level motion features.
Table S7: Policy observation space
Entry
Symbol
Dimensions
Noise level
base linear velocity
0:3
0.2
base angular velocity
3:6
0.05
projected gravity
6:9
0.05
joint positions
9
9:27
0.01
joint velocities
27:45
0.75
last actions
45:63
0.0
latent phase
sin ø
63:71
0.0
latent phase
Cos Ø
71:79
0.0
latent frequency
f
79:87
0.0
latent amplitude
a
87:95
0.0
latent offset
b 95:103
0.0
training instances is perturbed with an additional weight m' u(-0.5, 1.0), where i denotes uniform distribution. On the other hand, random pushing is also applied every 15 seconds on the robot base by forcing its horizontal linear velocity to randomly within Uay ~ ((550.5,00 0.5).
A.2.5 ALGORITHM OVERVIEW
17

---

Published as a conference paper at ICLR 2024
Table S8: Policy training parameters
Parameter
Symbol
Value
step time seconds
0.02
pellomeerfrmmanc buffer size
B
5000
max episode time seconds
20
iterations learning rate
20000
0.001
steps per iteration
24
learning epochs
5
mini-batches
KL divergence target
0.01
discount factor
?
0.99
clip range
0.2
entropy coefficient
0.01
parallel training environments
4096
number of seeds
approximate training hours
2
Table S9: Policy training architecture
Network
Symbol
Type
Hidden
Activation
policy
MLP 128, 128, 128
ELU ELU
value function
V
MLP 128, 128, 128
Algorithm M provides details of policy training. More information on the skill sampler and the skill-perfommance buffer B is given in Suppl. A.2.6
A.2.6 SKILL SAMPLERS
To achieve high tracking performance for various target motions, the design of the skill sampler pe critical. Our sampling strategy, which guides policy learning and enhances tracking, is shown in Fig. SMR. Training FLD the offline dataset M reveals latent parameterization spaces with unlearnable subspaces (grey dashed shaded ring), posing policy learning challenges. An effective skill sampler interacts with the environment during training identify and avoid these subspaces, and it should also explore novel motion targets, expanding performance boundaries. Figure SSH shows the desired performance region (green shaded ring) covering many learnable motions (green solid circles) in M and extending to new, effectively learned motions (green dashed circle).
We compared policy learning performance using four skill samplers: offline point pler (OFFLINE). offline Gaussian mixture model (GMM) sampler (GMM), random sampler (RANdOM), and absolute learning progress with GMM sampler (ALPGMM) (Portelas ctal (020). OFFLINE and GMM access the original offline dataset M, while RANDOM and ALPGMM di- rectly interact and navigate the latent space during online training. Their implementation details are presented in Suppl. A.2.G.
Offline point sampler
We can utilize the offline dataset encode trajectories into points 0 in the latent space θ adding them the buffer pa. During online training, random latent parameterization points are drawn from this buffer, and the offline trajectories are recovered through the generation process of FLD. However, this sampler is limited by dataset size and memory constraints.
Offline Gaussian Mixture Model sampler
This method uses a GMM (Rasmussen. (999) to parameterize the latent space O avoiding storing all points O. An offline GMM po is fitted using the Exppetation-Maxiiizaatie (EM) (Dempster all
18

---

Published as a conference paper at ICLR 2024
Algorithm Policy training
1: Input: FLD decoder dec
2: initialize skill sampler ps, skill-performance buffer
3: for learning iterations = 1, ...... do
4: sample latent states & and latent parameterizations
5: for environment steps 1, ..... do
6: generate motion tracking targets from Φ and 0 with FLD decoder dec calculate tracking reward ,
7:
8:
propagate latent states Φ according to Eq. G
9:
if reset then
10:
collect latent parameterizations and the tracking performance rf in skill-performance buffer B resample latent states & and latent parameterizations end if
11:
13:
end for
14:
update skill sampler pa with skill- skill-performance buffer
15:
update policy and value function with PPO another RL algorithm
16: end for
1977)) algorithm, with random points drawn during online training. The sampler's effectiveness depends on the dataset's quality, as it may struggle with unlearnalle or challenging motions.
Random sampler
Without access to the original dataset, the random sampler draws samples uniformly from the latent space confidence region. However, it overlooks the data structure and may generate unlcarnable targets due inefficient understanding of large sensorimotor spaces.
Absolute Learning Progress with Gaussian Mixture Models sampler
Identifying learnable motions in the latent space without the original dataset is challenging. A self- adaptive curriculum learning approach, essential in this context, adjusts the frequency of sampling target motions based on their potential for improved learning. This method parallels the strategic student problem (Lopes Oudeyer, 2012), focusing selecting tasks for maximal competence. We employ the ALP-GMM strategy for this purpose. ALP-GMM fits a GMM on previously sampled latent parameterizations, linked their ALP values. Sampling decisions are made using non- stochastic multi- aumei- bandit approach (Auer all 2002), with each Gaussian distribution as an arm and ALP as utility, steering the sampling towards high-ALP areas.
To get this parameterizztion ALP value alp(@), we follow the implementation from earlier work (Portelas et all (022)). For each newly sampled latent parameterization and associated track- ing performance v rem (0), the closest previously sampled latent parameterization with associated tracking performance 7 (0) is retrieved from skill-performance buffer B using the nearest neigh- bor algorithm. We then have
alp(0) =I (0) TT
(S8)
We use Faiss (lohnson 2019) for efficient vector similarity search find the nearest neighbors of the current latent parameterization in the kkill-performance buffer B. At each policy learning iteration, update the skill sampler by refitting the online GMM based the collected latent parameterizations and the corresponding ALP measure. We refer to the original work (Ronnals atall 2020) for more implementation details.
The design details of different skill samplers are presented in Table STU. For ALPGMM, we provide in Fig. SIlla schematic overview of its working pipeline.
19

---

Published as a conference paper at ICLR 2024
Table S10: Skill sampler parameters
Skill sampler
Parameter
Value
OFFLINE
features
24
buffer size
20000
GMM
features
24
components
8
covariance type
full
RANDOM
features
24
ALPGMM
features
25
minimum components
2
maximum components
10
information criterion
Bayesian
random sample rate
0.2
Axil-pperformance buffer size
5000
Aeill-performance collection interval
5
nearest neighbors
1
update interval
50
sample
env
alp-weighted GMM fitting
tracking performance
r0
0
alp
nearest neighbors
old tracking performance
T eoold
Figure S11: ALPGMM algorithm overview. The ALP is calculated on the latent parameterization of target motion samples on these samples weighted by their ALPP measure. New samples are drawn from the fitted GMM in the next iteration.
by comparing the old and new tracking performance, A GMM is then fitted
A.2.7 TRACKING PERFORMANCE
The tracking reward calculates the weighted sum of reward terms each dimension bounded in [0, 1].
T W.Ty + WwTw Wyyg Wgingn Tgteg
Wgerm
garm
(S9)
The tracking performance r" e [0, 1] computed by the normalized episodic tracking reward. The formulation of each tracking term is detailed as follows and their weights in Table STll
Table $11: Tracking reward weights
Weight
Wy
W.
Wgg
Wgter
W qarm
Value
1.0
1.0
1.0
1.0
1.0
Linear velocity
20

---

Published as a conference paper at ICLR 2024
To = -
(S10)
where o, = 0.2 denotes a temperature factor, and v denote the reconstructed target and current linear velocity.
Angular velocity
twe - ---10--11 -
(S11)
where ow 0.2 denotes temperature factor, @ and denote the reconstructed target and current angular velocity.
Projected gravity
Tg = 00100-911
(S12)
where og 1.0 denotes a temperature factor, 9 and g denote the reconstructed target and current projected gravity.
Leg position
(S13)
where Qiep 1.0 denotes temperature factor, qeeg and Qeg denote the reconstructed target and current leg position.
Arm position
Tarm -garm MM Jarm
(S14)
where farm 1.0 denotes a temperature factor, Garm and Qarm denote the reconstructed target and current arm position.
A.2.8 REGULARIZATION REWARDS
The regularization reward calculates the weighted sum of individual regularization reward terms.,
R WarTar + Wg. Nga + War
(S15)
The formulation of each regularization term is detailed as follows and their weights in Table STZ
Table S12: Regularization reward weights
Weight
War
Wgg
Way
Value
0..01
-2.5 x 10-7
-1.0 x 10-5
Action rate
= |a
2
(S16)
where a' and denote the previous and current actions.
Joint acceleration
6'
(S17)
g0
At
where j' and denote the previous and current joint velocity, At denotes the step time interval.
21

---

Published as a conference paper at ICLR 2024
LD
Φ
policy π
latent embedding z
a
convolution enc
user input sf
state
Figure S12: Online tracking. The latent states & and parameterization 0 are obtained from encoding accepted target proposals from user input si. cases where the user proposal is absent rejected, the latent dynamics step in and continuously provide fallback targets.
Joint torque
For
|T
(S18)
where T denotes the joint torques.
A.2.9
ONLINE TRACKING
Figure S provides a schematic overview of the online motion tracking pipeline.
A.2.10
ADAPTIVE CURRICULUM LEARNING
In addition to solely tracking targets in the offline dataset, we aim to address the challenge of learning continuous interpolation and transitions between motions the sparsely populated reference motion space.
To this end, we train FLD on the offline reference dataset M the first stage and obtain the latent parameterization space O. In the second stage of online motion learning, we generate tracking targets by sampling and decoding latent parameterizations with a skill sampler pa.
θ (f.a, a. b) pa A(O).
(S19)
Depending on the design of the skill sampler Po. the training datasets with FLD motion synthesis may describe different spaces of dictated reference motions used to train the policy. The resulting policy trained over this dataset induced by Pa is denoted by TTo
We distinguish the performance evaluation spectrums on closed-ended and open-ended learning. We define expert performance as the performance evaluation of learned tracking policy on the prescribed target motions. And we define general performance as the performance evaluation of leamed tracking policy on wide spectrum of target motions.
Therefore, the objective of motion learning can be written as
max Eorger (V O0)], Fpy Po
(S20)
where V(Tp. I0) denotes the performance metric of policy Tp on motion 0', and O' denotes the evaluation spectrum that differs in expert and general performances.
For expert and general tracking performances, we define the motion learning evaluation spectrum O' in Eq. S20 To evaluate the expert performance, we randomly sample test target motions from the collected latent parameterizations of the prescribed offline dataset M. For the general performance, the evaluation is determined by uniformly random samples from the continuous latent parameteri- zation space bounded by the confidence region induced by FLD training on M. The result five random seeds is presented in Table STA
The OFFLINE sampler, with direct access to the dataset M, generates trajectories closely resem- bling the target motion space, leading to high expert performance. However, does not use FLD's capability synthesize motions for broader tracking targets, limiting its generality to the policy
22

---

Published as a conference paper at ICLR 2024
Table S13: Tracking performance with proposed skill samplers.
Range
OFFLINE
GMM
RANDOM
ALPGMM
Expert
0.88 + 0.03
0.87 + 0.03
0.83 + 0.01
0.87 0.04
General
0.72 + 0.01
0.73 + 0.02 0.80 0.01
0.80 + 0.03
OFFLINE
GMM
RANDOM
ALPGMM
(+)
(
0.8
8
0.6
6
0.4
-4
0.2
10
Training iteration x103
Figure S13: Tracking performance and exploration factor on running motion targets.
network's understanding of spatial-temporal motion structures. The GMM sampler, similarly, uses parameterized latent representations of prescribed motions, achieving performance comparable to OFFLINE.
In contrast, the RANDOM sampler optimizes the policy for wide range of random motions without specific targeting, resulting less focused approach. ALPGMM, however, balances generaliza- tion with performance prescribed targets. It does this by learning from poliey-environment inter- actions and identifying useful target motions during training. The ALP measure guides its GMM sampling strategy, avoiding redundant targets and focusing promising, underexplored motions for significant performance improvements.
To understand this adaptive learning process, we maintain an online buffer of the latent parame- terization of rumning target motions selected by the skill samplers and evaluate the policies' average tracking performance on these motions. To quantify the online target search space enabled by the
skill samplers, we define an exploration factor (() as a channel-averaged ratio between the stan- dard deviation within the latent parameterizations of the online running target motion dg and that within the initial offline dataset.
(Ô)
E.
T 0e
(S21)
We plot the results in Fig. ST with the running tracking performance o! (Ô) in five random runs.
Note that the target motion space induced by different skill samplers and thus the spectrum the running tracking performance is calculated over is different. OFFLINE and GMM, with access to offline data, show strong tracking (bold curves) of prescribed motions but fail to explore beyond this dataset. keeping their exploration factors (thin curves) constant. contrast, RANDOM, de- spite lacking offline data and understanding of data structure, achieves similar performance due
23

---

Published as a conference paper at ICLR 2024
OFFLINE
GMM
RANDOM
ALPGMM
10%
60%
unlcarnable
0.8
0.6
1 0.4
0.2
0.0
10
15
20
Training iteration x10
Figure $14: Tracking performance on running motion targets under different mixtures of unlearn- able reference motions. The solid, dashed and dotted curves denote training with 0, 10% and 60% unlearnable motions the reference dataset, respectively.
FLD's well-structured latent space. However, this strategy may fail if the initial training data has unlearnalle components, as further discussed in Suppl. A.2.1
ALPGMM initially shows improved tracking performance, which gradually decreases it plores new motion regions, including under-explored or undefined areas. This expansion, evidenced by the increasing exploration factor, results in much larger motion coverage by training's 's end. This guided exploration ensures maintained expert performance on mastered motions in Table STA which would have been lost the sampling region grown only blindly. In our experiments, the curriculum learning open-ended. Future studies may look into practical regularization methods convergence. In summary, ALPGMM achieves high expert and
keep
constrain such exploration general performance by dynamically adapting its sampling distribution during training, resulting in broader coverage of the target motion space and improved motion learning generality.
A.2.11 UNLEARNABLE SUBSPACES
Similar to the discussion above, we investigate the average tracking performance over the running targets sampled the skill samplers under datasets containing different mixtures of unlearnable reference motions. To this end, FLD and the latent parameterization space are pre-trained on each dataset the first stage. We plot the mean of five random policy training runs on each dataset of 10% and 60% unlearnable motions in Fig. S14, respectively.
The solid curves indicate similar tracking performance evolution pattern as discussed above, where the latent parameterization space is not corrupted by the unlearnable motions. However, when the reference dataset contains unlearnable motions, blind random sampling strategies like RANDOM quickly fail. The presence of unlearnable motions distorts the latent parameterization space towards these challenging areas. Sampling strategies that rely heavily on direct access to the offline dataset (OFFLINE, GMM) or the ill-shaped space induced by it (RANDOM) may lead to substantial number of unreachable targets and significant learning difficulties. Especially with 60% motions unlearnable, policies trained with OFFLINE and GMM achieve less than half of the tracking perfor- mance in the fully learnable case, and those with RANDOM encounter drastic learning failure.
In contrast, ALPGMM facilitates policy learning by actively adapting its sampling distribution towards regions with high ALP measure, identifying promising targets and avoiding those with limited potential for improvement. This adaptive curriculum is particularly effective in cases where
24

---

Published as a conference paper at ICLR 2024
200
1000
Training iteration
10000
20000
0.0
0.5
1.00.0
0.5
1.0 4
0
alp
Figure S15: ALPGMM sampling strategy migration. Each dot in the visualization corresponds to the latent parameterization of a target motion, with colors representing the policy's tracking perfor- mance, the ALP measure, and the predicted motion type. Unlearnable components are colored grey with the label u. The ellipsoids represent online GMM clusters used by ALPGMM, with the color gradient indicating the mean ALP measure of samples from each cluster.
the latent parameterization space is heavily affected unlearnable components. In experiment with 60% unlearnable targets, ALPGMM produces the highest tracking performance among all skill samplers by focusing only on regions where the policy excels. The importance of this feature is highlighted in applications that require extracting as many learnable motions as possible from large, high-dimensional target datasets with unknown difficulty distributions.
To further understand the exploration and sampling strategy enabled by ALPGMM, we visualize the migration of the sampling region in a representative run with trajectories from an unlearnable motion type in Fig. SIS. To this end, we project to 2D plane the latent parameterization samples collected the skill-performance buffer B, which tracks the running target motions selected by ALPGMM, at different stages of training in Fig. STS For evaluation purposes, oracle classi- fier is trained to predict original motion classes from their latent parameterizations as detailed in Suppl. A.
Each dot represents the latent parameterization of target motion and colored according to the policy's 's performance r? in tracking this target, the ALP measure alp, and the motion type y pre- dicted by the oracle classifier in each column. Specifically, we color the unlearnable components grey with the label in the motion prediction results. In addition, plot the online GMM clusters
25

---

Published as a conference paper at ICLR 2024
0.5
0.0
0.5
7
4.
1.6
0.8
0.0
1.2
0.3
-0.6
50
100
150
[step]
Figure S16: Motion representation of a transition from a periodic motion to another using PAE. The quasi-constant motion parameterization assumption holds only locally.
employed by ALPGMM in ellipsoids with the color gradient indicating the mean ALP measure of samples from each cluster.
At the beginning (200 iterations) of learning, motion targets are initialized randomly. The policy is still undertrained at this early stage and thus tracks motions with generally low performance. As the learning proceeds (1000 iterations), the policy gradually improves its capability motions indicated by the increased performance and ALP measure on the colored samples. In the meantime, the policy also recognizes the unlearnable region (grey) where it fails to achieve better tracking performance and remains a low learning process. As ALPGMM biases its sampling to- ward Gaussian clusters with high ALP measure, the cluster on the unlearnable motions becomes silent, indicated by an ellipsoid with complete transparency. Further training (10000 iterations) en- larges the coverage of tracking targets indicated by the expanded sampling range centered around the mastered region. A gradient pattern in tracking performance and ALP measure is observed where higher values are achieved at points closer to the confidence region. Finally, more extended training time (20000 iterations) pushes ALPGMM to areas more distant from the initial target distribution, motivating the policy to focus on specific motions where the performance may be further improved. This demonstrates the efficacy of ALPGMM in navigating the latent parameterization space to focus on learnable regions.
the Iearnable
A.3 LIMITATIONS
A.3.1 QUASI-CONSTANT MOTION PARAMETERIZATION
FLD provides a solution to representing high-dimensional long-horizon motions in meaningful low dimensions. The training of FLD explicitly enforces latent dynamics that respect spatial-tempores relationships and identify the intrinsic transition patterns in periodic or quasi-periodic motions. How- ever, the propagation of such latent dynamics replies on the quasi-constant motion parameterization assumption (Assump. M., which holds well on periodic or quasi-periodic motions. When it comes motions containing less periodicity, we observe time-varying latent parameterization along the trajectory. As depicted in Fig. STA although the quasi-constant motion parameterization assump- tion may still hold locally in motions containing aperiodic transitions, the latent dynamics in the transition phase can be challenging determine. Thus, enforcing globally constant set of latent parameterization over the whole trajectory is likely to underparameterize the latent embeddings and result in inaccurate reconstruction. If the reconstruction loses too much information with respect to the original motion, additional techniques on period detection and data preprocessing that involve extra human effort have to be applied.
We may also view aperiodic motions as periodic ones with very long periods. This way, the length of the observation window needed to capture the transitions over whole period is also extended.
26

---

Published as a conference paper at ICLR 2024
(a) Jump
(h) Kick
(c) Spinkick
Figure S17: Representative motions that require specific reward functions to be learned.
This increases not only the computational complexity but also the reconstruction performance as FLD always performs global periodic latent approximation of the original motion.
A.3.2 MOTION-DEPENDENT REWARDS
With an appropriate sampling strategy, FLD is able to continually propose novel tracking targets online that enhance the generality of the learning policy. However, the adaptive curriculum learning methods implemented in this work assume that all such synthesized target motions can be learned with the same set reward functions. This does not necessarily hold true for large complex datasets where some motions may require specifically designed rewards to be invoked and picked up. Espe- cially for robotic systems, motion execution is strongly constrained by critical physical properties such non-trivial body part inertia and actuation limits such as motor torques. We exemplify some motions that require different sets of reward functions to be acquired in Fig. STZ. Additional efforts in designing target-dependent reward functions needed to push further the limit of tracking ca- pability of real systems. This may be especially challenging in low-level motor skill development, where such reward functions are strongly correlated with the sensorimotor spaces and physical limits of different embodiments.
A.3.3 LEARNING-PROGRESS BBSED EXPLORATION
In the adaptive curriculum learning process without privileged access to the offline data, ALPGMM searches for motion targets that yield the most significant ALP measure. However, these targets do not necessarily align with the intended reference motions. The policy may end up learning some interpolated motions that do not make sense terms of motor skills but yield high ALP measure during training. In these settings, learning learning-progress progress-based based exploration is allured to regions with the fastest growth in performance, giving up tracking challenging motions where progress requires more extended learning. Occasional random sampling mitigates this problem by reducing exploration's 's reliance on pure ALP measures. Without access to the offline data, additional careful analysis in locating truly useful motions in the latent parameterization space is needed.
27


Reviews:
[{"review_id": "mY57LR21Br", "reviewer": "Reviewer_RFvT", "summary": "This paper introduces a latent dynamics model and control policy to track periodic and quasi-periodic functions. The paper extends a PAE network (Starke 2022), which encodes a motion trajectory into a latent embedding used to generate a set of fourier coefficients. These coefficients are used to build a dynamics model assuming constant frequency, amplitude and offset, but time varying phase. The dynamics model predicts a future latent, which is decoded to produce a resultant motion. Control policies are also trained to generate frequency parameters that result in motion sequences that match some desired trajectory. A fallback mechanism compares the desired trajectory to the generated trajectory, and decides whether this is safe to follow (if both are similar), following a more conservative predicted motion if this is deemed unsafe.", "questions": "Figure 5 is confusing, I assume the right y axis is an error metric? Please label axis and caption to indicate these measures.\n\nFig S7 - this is used to motivate fixing frequency amplitude and bias, could you provide more detail on the motion encoded in this example? It seems that this would be motion dependent.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The latent dynamics model proposed seems of value to a broad class of periodic/ quasi periodic motions, and nicely extents the periodic autoencoder architecture (PAE - Starke 2022) to the motion generation use case.\n\nThe proposed approach allows for a natural fallback mechanism (detection of infeasible target motions, and fallback to sensible behaviours that lie within the training set.)\n\nThe proposed model produces expressive motions with a more compact trajectory representation than prior work.\n\nThe paper is very extensive, with a number of interesting ablations and visualisations.", "weaknesses": "The motion learning/ control policy part of this work needs a clearer problem formulation. It is unclear what the exact goal is here, and a lot is left to the readers to infer. I gather the goal is to learn to generate a series of motions that track a desired motion sequence, using the motion prediction, but it is unclear to me why you need to generate control parameters to do so, instead of just encoding the target motions directly.  A clearer problem description will help avoid confusion like this. \n\nMuch of the interesting work around control policies is in the appendices\n\nMissing related work:\n\nThis work appears closely related to a phase functioned neural network Holden et al. Phase-Functioned Neural Networks for Character Control, which computes network weights using a cyclic function controlled by phase. The proposed approach uses an autoencoder and has clear differences (PFNN does next state prediction), but the core idea is similar and PFNN also considers aspects like motion blending etc. The paper is mentioned in passing in the introduction, but I would recommend some discussion on this in the related work given the similarity in the core idea. \n\nMinor:\n\nThe term Generative Latent Dynamics is rather general, and not particularly descriptive of the proposed approach.\n\nThe paper is extensive, with a number of detailed appendices. This is good, but I found references to appendix figures in the main text body rather distracting.\n\nAssumption 1. I think it is worth pointing out earlier that since the latent space is learned, this can be enforced. \n\n5. Experiments - These experiments are motivated in terms of real world applicability to real robots, but character animation is not robot control, and this statement should be used with caution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FzRMsiXwCo", "reviewer": "Reviewer_itfQ", "summary": "In this paper, authors tackle the problem of motion representation. Inspired by prior work that takes into account periodicity of motion, the proposed method extends the periodic autoencoder to add a generative capability. Based on the insight that some elements do not change during periodic motions, the paper propose GLD, where subsequent latent representation is trained to be predicted, assuming phase can be advanced incrementally. Using the trained network, the method proposes to train to learn policies that aligns the predicted state and the state of the observation. Then during inference, the authors propose to filter out potentially dangerous or difficult states using a fallback mechanism. They compare the difference between the designated state and the actual predicted states, and if the error is significant, they propose to reject the designated state and fall back to the predicted state. The authors conduct various experiments to demonstrate the effectiveness of the actual latent space, as well as the proposed fallback mechanism.", "questions": "- What happens when no fallback strategy is employed? How are the resulting motions look with and without the fallback mechanism? Some comparison would be desirable.\n\n- Was there any drawback from introducing the assumption that some elements are generally constant throughout a sequence? Were there actions that did not present these characteristics?\n\n- Also, the action classes indicated by the colors in Fig.5 seems to be missing, making the evaluation of these latent spaces difficult. What do each color respond to?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposal to extend the phase autoencoder mechanism to predict the future state, based on the observation that some elements remain consistent throughout certain periodic motion, is very cleverly devised. The observation is used effectively to predict the upcoming $N$ segments, which can be seen from the prediction experiments conducted in Fig. 5. \n\n- The demonstrations in the supplementary material demonstrates that the fallback mechanism is effective at preventing undesired motion, and maintain the status quo as much as possible.\n\n- The authors applied the proposal to various tasks including motion tracking and motion transition. In both tasks, the proposed prediction method is able to interpolate between motions even when the fallback mechanism is triggered.", "weaknesses": "- The authors should make better effort to make the paper self-contained in the main manuscript, and not rely excessively on the supplementary material. There is a severe lack of details in the main manuscript, due to the authors moving them to the supplementary material. For example,\n1. One form of the skill sampler should be included in the main section. \n2. $\\mathcal{U}$ in Fig.2 is unclear from the main section.\n3. Section 5.4 seems unnecessary, as all the content is in the supplementary material.\n\n- The figures seem unorganized, as the readers are asked to refer to figures in a random order, including the supplementary material. The colored lines in Figures 5 and 6 is unclear. There should be some sort of explanation of each element in all the figures.\n\n- Despite the comparison of the actual latent space, the reconstruction accuracy seems to be missing. As I presume that the reconstruction accuracy falls as the prediction segment horizon $N$ increases, the authors should discuss the trade-off in comparison to existing methods. \n\n- The effectiveness of the fallback mechanism must be discussed more quantitatively. The authors only discuss the results in Section 5.3, but there is no concrete evidence that the mechanism worked well. There should be some statistics regarding if the motion prediction actually failed when the fallback mechanism was not introduced. Such objective evidence is lacking for the readers to decide whether it is effective or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DTNYiLg5Wj", "reviewer": "Reviewer_Sghv", "summary": "This paper presents GLD (Generative Latent Dynamics), a novel self-supervised representation and generation method that captures spatial-temporal relationships in periodic or quasi-periodic motions. GLD improves motion learning by incorporating motion dynamics into a parameterized latent space. The method tracks a wide range of motions, including unseen targets, and adapts to potentially risky targets. Furthermore, the paper presents experimental evidence (on the MIT Humanoid robot) showcasing GLD's effectiveness and long-term learning capabilities in open-ended motion learning tasks.  Additionally, the supplementary experiments demonstrate that GLD possesses long-term learning capabilities, which allow learning agents to strategically progress novel target motions while avoiding unlearnable regions.\n\nThe main contributions of this paper are:\n1. GLD (Generative Latent Dynamics) is a new method that extracts spatial-temporal relationships in periodic or quasi-periodic motions. It uses a novel self-supervised, structured representation and generation approach.\n2. GLD has demonstrated its effectiveness in open-ended motion learning tasks. It has long-term learning capabilities that enable learning agents to strategically advance novel target motions while avoiding unlearnable regions.\n3. An online tracking framework powered by GLD has a fallback mechanism. This enables learning agents to dynamically adapt their tracking strategies and automatically identify and respond to potentially risky targets.\n4. Recognition of spatial-temporal structures creates new possibilities for future motion representation and learning algorithms.\n\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nAfter reviewed the author's rebuttal, I think the authors have addressed most of my concerns, therefore; I have increased my score.", "questions": "1. Can GLD be extended to handle non-periodic motions?\n2. How does GLD perform in long-term learning tasks, and can it adapt to and learn new tasks in open environments?\n3. How can the stability and safety of GLD be ensured when applied in real-world scenarios?\n4. How does the GLD perform with noisy motion trajectories? It would be important to investigate how GLD performs in real-world scenarios where motion data may be corrupted or incomplete.\n5. Can GLD be extended to multi-agent motion learning? How does the computation complexity increase as the number of targets increases?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Strengths of the paper:\n1. Originality: The paper proposes GLD, a method that integrates motion dynamics in a parameterized latent space. It combines periodic autoencoders with generative latent dynamics, showcasing ingenuity in motion representation and learning.\n\n2. Quality: The paper showcases the creativity of periodic autoencoders with generative latent dynamics in motion representation and learning. It presents a well-designed experimental setup, comparing GLD's performance with state-of-the-art methods across various motion datasets, providing strong evidence for its effectiveness.\n\n3. Clarity: The paper is well-structured and presents its ideas in a clear manner. It begins with a thorough introduction that sets the context for the proposed method. The paper explains the underlying concepts and algorithms effectively, making it accessible for both experts and non-experts in the field. The experimental results are presented in an organized way, which helps readers understand the performance of GLD in various scenarios.\n\n4. Significance: The proposed method, GLD, has the potential to significantly improve motion representation and learning. It offers an efficient and effective way to generate structured motion patterns, improving the generalization capabilities of learning algorithms. By addressing challenges with raw motion trajectory data, GLD opens up new possibilities for advancements in motion representation and learning.\n\nIn summary, the paper presents a novel and original approach to motion representation and learning with GLD. It demonstrates its quality, clarity, and significance in the field. The paper's strengths lie in its creative combination of existing ideas, application to a new domain, and addressing limitations of prior results, making it a valuable contribution to the research community.", "weaknesses": "1. Limited Data Set: The paper heavily relies on a specific dataset for evaluation, which might not be representative of various motion patterns and scenarios. Hence, the proposed method's performance might not be generalizable to other datasets or real-world applications.\n\n2. Controller's Adaptability: Although the motion learning controller is designed to adapt its tracking strategy dynamically, the paper lacks a thorough analysis of the controller's adaptability in handling various motion patterns and unseen targets (for example, multiple intersecting targets). Further study could help establish the controller's robustness and versatility.\n\n3. Limited Adaptability to Other Domains: The paper discusses motion representation and learning in robotics. However, it may not be easily transferable to other domains, such as human motion analysis, due to differences in motion characteristics.\n\n4. Data Quality: The paper fails to address the possible negative effects of poor-quality data on the proposed GLD method's performance in real-world applications. Poor-quality data could potentially degrade the effectiveness and accuracy of the GLD method in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)