Continuity-Preserving  Convolutional Autoencoders for Learning Continuous Latent Dynamical Models from Images
Authors: Aiqing Zhu
Yuting Pan
Qianxiao Li
Venue: ICLR 2025 Poster (2025)
Keywords: Latent dynamical system
Autoencoders
Learning dynamics
Continuity-preserving
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Preserving continuity of latent states for learning continuous dynamics from discrete image observations

Abstract: Continuous dynamical systems are cornerstones of many scientific and engineering disciplines.
While machine learning offers powerful tools to model these systems from trajectory data, challenges arise when these trajectories are captured as images, resulting in pixel-level observations that are discrete in nature.
Consequently, a naive application of a convolutional autoencoder can result in latent coordinates that are discontinuous in time.
To resolve this, we propose continuity-preserving convolutional autoencoders (CpAEs) to learn continuous latent states and their corresponding continuous latent dynamical models from discrete image frames. 
We present a mathematical formulation for learning dynamics from image frames, which illustrates issues with previous approaches and motivates our methodology based on promoting the continuity of convolution filters, thereby preserving the continuity of the latent states.
This approach enables CpAEs to produce latent states that evolve continuously with the underlying dynamics, leading to more accurate latent dynamical models.
Extensive experiments across various scenarios demonstrate the effectiveness of CpAEs.

Published as a conference paper at ICLR 2025
CONTINUTTY-PESSERVING CONVOLUTIONAL AUTOENCODERS FOR LEARNING CONTINUOUS
LATENT DYNAMICAL MODELS FROM IMAGES
Aiqing Zhu', Yuting Pan Qianxiao Li
1 DDearrmmett of Mathematics, National University of Singapore Schooo of Computing, National University of Singapore
LTD
'Institute for Functional Intelligent Materials, National University of Singapore zaqênus edu. sg, pan. man.yutingeu... nus edu, gianxiao@nus.eddu
ABSTRACT
Continuous dynamical systems are cornerstones of many scientific and engineer- ing disciplines. While machine learning offers powerful tools model these sys- tems from trajectory data, challenges arise when these trajectories are captured as images, resulting in pixel-Ievel observations that are discrete nature. Con- sequently, naive application of a convolutional autoencoder can result in latent coordinates that are discontinuous time. To resolve this, we propose continuity- preserving convolutional autoencoders (CpAEs) to learn continuous latent states and their corresponding continuous latent dynamical models from discrete image frames. We present mathematical formulation for learning dynamics from im- age frames, which illustrates issues with previous approaches and motivates our methodology based promoting the continuity of convolution filters, thereby preserving the continuity of the latent states. This approach enables CpAFs to pro- duce latent states that evolve continuously with the underlying dynamics, leading to more accurate latent dynamical models. Extensive experiments across various scenarios demonstrate the effectiveness of CpAEs.
INTRODUCTION
Continuous dynamical systems, described by differential equations, are widely used scientific modeling tools across various biological, physical, and chemical processes. While traditionally described by mathematical models, the increasing availability of data has spurred the development of data-driven approaches (Brunton et al., 2016 Brunton & Kutz 2022; Schmidt & Lipson 2009). In particular, machine learning and neural networks have recently emerged as powerful tools, achieving remarkable success in tasks such as discovering (Chen et al. 2018 González Gariia et al. 1998
Raissi et al. 2018), predicting (Wang et al.
2021 | Wu & Xiu 2020; Xie et al.) 2004), and controlling
(Brunton & Kutz 2022 Chen et al. 2023 Zhong et al. 2020) continuous dynamical systems based on observed data.
Most methods for modeling dynamical systems are designed for observed data that already corre- spond to relevant state variables. However, in many scientific and engineering applications, we only have access to measurements that yield series of discrete image data (Botev et al. 2021 Chen et al. 2022]. When applied to image data, a common approach involves using autoencoders to encode nat- ural images located in high-di sigh-dimensional pixel space onto diwdemminiioall manifold (Greydanus) et al. 2019 Jin et al. 22023 Toth et al. 2020). is then assumed that the encoded sequences follow continuous paths governed by differential equation on this manifold, and machine learning meth- ods automatically capture the continuous dynamics (Botev et al.) [021] Toth et al. 2020). While this approach and assumption have been substantiated relatively simple tasks, complex visual patterns and dynamical behaviors remain challenging (Botev et al. 2021)
The continuous evolution of dynamical systems over time is fundamental characteristic of many fields. Machine leamning methods for capturing continuous dynamics from discrete data along con-
"Corresponding author.

---

Published as a conference paper at ICLR 2025
tinuous trajectories have advanced significantly, with robust, general general-purpose purpose algorithms now readily available (Chen et al. 2018 Krishnapriyan et al. 2023; Ott et al. 202I). However, image data pose key challenge since pixel coordinates often do not align with the continuous evolution of underly- ing dynamics, as illustrated in Fig. 1 Whereas latent states that evolve continuously over time are essential for discovering continuous latent dynamical systems, standard autoencoders often struggle to learn such valuable latent representations, as will be discussed in detail later.
Undertying dynamics
Pixel observations
Evolution of thi third posnt value
Illustration of pixel observations of continuous motion. A single pixel white square, initially located at the leftmost position, moves uniformly the right against black background (plotted in gray for clarity). Its position is recorded at each pixel step. This translational motion re- sults in pixel coordinates that first increase by one, then decrease by one, and finally remain constant. As an illustrative example, we show the evolution of the value at the third pixel position.
Figure 1:
We introduce preeeerving convolutional autoencoders (CpAEs) to learn continuously evolving latent states from discrete image frames. Our contributions are summarized as follows:
We propose mathematical formulation for learning continuous dynamics from image data to describe the continuity of latent states.
We establish a sufficient condition (Theorem 3.1). demonstrating that the latent states will evolve continuously with the underlying dynamics if the filters are Lipschitz continuous.
We introduce a regularizer (Eq. ( to promote the continuity of filters and, consequently, preserve the continuity of the latent states.
We perform several experiments across various scenarios to verify the effectiveness of the proposed method.
1.1 RELATED WORKS
Deep Autoencoder. An autoencoder (Baldi 2012 Ranzato et al. 2007 Rumelhart et 1985) is a type of neural network that encodes input data into compressed and meaningful representation, then decodes it to reconstruct the original input. A significant advancement in this area has been the development of Variational Autoencoders (VAFs) (Kingma & Welling 2014), which extend tradi- tional autoencoders by incorporating probabilistic modeling of the latent space. Autoencoders can be adapted and extended to various models, finding broad applications, including image generation and classification (Kingma & Welling 2014 Pu et al.) 20166). data clustering (Guo et al., 2017] Song et al. 2013), and anomaly detection (Gong et al. 2019 Zong et al.) 2018). In this paper, we focus on using autoencoders to learn latent dynamical models from images, and we propose novel continuity continuity-preserving preserving autoencoders that incorporate continuity prior for this task.
Discovering Dynamical model. Discovering dynamical models from observed time-series data To, -1, NN is a fundamental challenge in science. One effective strategy involves constructing a neural network model, denoted as N,, to learn a data-driven flow map that predicts the system's 's future states (Chen al., [022 Chen & Xiu| 2024 Wang al., 2021 Wu Xiul 2020). This model predicts the subsequent state Tn+1 based on the current state In. Alternatively, some researchers focus on modeling the governing function of the unknown differential equation (Chen et al.) 2018
Sonzález-Garcie et al. 1998 Raissi et al. 2018). Given an input In. In+1 is obtained by solving the NN-parameterized ODE at time At, starting from the initial condition V,, This approach can offer valuable insights into the system's 's dynamics. For instance, it enables the characterization of invariant distributions (Gu et al. 2023 Lin et al. 2023), energy landscapes (Chen et al.) 2023), and other essential properties (Qiu et al. (022). thereby expanding the scope of scientific investigation.
Learning dynamics from image observations. Numerous studies have explored the incorporation of classical mechanics-inspa innssieedddddiddddddee inductive biases, such as physical principles (Cranmer et al. 2020
Greydanus et al. 2019 Lutter et al. 2019 Yu et al. 2222 Zhang et al., (002),, geometry structures
(Eldred al. 2024 Jin al. 2020 Zhu et al. (022), and symmetries (Huh et al. 2020 Yang
ctt al.] 2024), into deep neural networks. While some of these models have been applied to learn
dynamics from images using autoencoders, they have mostly been tested on relatively simple visual
2

---

Published as a conference paper at ICLR 2025
patterns and dynamical behaviors. An enhanced approach (Botev al. 2021 Toth al. 2020)
involves employing VAEs to embed images, often resulting in improved predictive and generative performance. Given priority ensuring the continuous evolution of the latent states, this work focuses on deterministic autoencoders.
LEARNING LATENT DYNAMICAL MODELS USING AUTOENCODERS
This paper focuses on learning continuous latent dynamical models from sequential image observa- tions of an unknown system. Our dataset consists of discrete image frames sampled from multiple continuous trajectories, denoted as:
((X)) .x)) (X,"
X ,",,,,"),,, .))), X E (+++))(++))
(1)
where the superscript indicates the m-th trajectory and the subscript indicates the n-th time step of the trajectory. We assume that an underlying dynamical system governs the observed image time series, with its governing equations defined as follows:
= f(=,, =EZCRO
(2)
where denotes the set of states associated with the image observations, and assume that there a corresponding mapping I from the physical state space to the pixel space. Mathematically, the pixel observation time t, assumed satisfy X [(= where en is the state evolved from initial state = according Eq. (2). The true governing function f. the underlying physical states m and the mapping are all unknown. We further assume that the unknown governing function f Lipschitz continuous and bounded by My.
Figure Schematic of learn- ing latent dynamics. The en- coder takes images as inputs and infers the corresponding latent states; the decoder maps the la- tent states back to reconstruct the original images; the dynam- ical model outputs the subse- quent latent state Zn+1 based on the current state Z,..
Physical
Images
Encoder
Encodler
Encoder
Decoder
Decoder
Decoder R(ew)
Decador
Decoder
2(6))
2(t.)
x(aa)
Latent states:
Z
Dynamical
Dynamical model
Training
Prediction
A typical framework for this task comprises three components: an encoder, a dynamical model, and decoder. An illustration of this framework is shown Fig. 2 The goals are to: 1) learn an encoder that extract latent states consistent with the assumed latent dynamical system. 2) discover dynamical model that accurately captures the underlying latent dynamics, and 3) identify decoder capable of reconstructing the pixel observations.
The first objective is a prerequisite for the entire task; without no target dynamical system with assumed structures can be identified. Specifically, to learn meaningful continuous latent dynamical model, it essential that the extracted latent states are discrete samples of Lipschitz continuous tra- jectories. However, without constraints on the encoder, the extracted latent states may deviate from the assumed latent dynamical system. this paper. we focus on preserving the fundamental con- tinuity of the latent dynamical system. The encoder is carefully designed to ensure the continuous evolution of latent variables over time in this paper.
CONTINUTTY- AUTOENCODER
3.1
MATHEMATICAL FORMULATION
Given the discrete nature of image data in both space (pixels) and time, traditional definitions of continuity are not directly applicable. In this section, we propose a mathematical formulation to describe the continuity of latent states when learning continuous dynamics from images.
The formal mathematical definition of projecting the states of a dynamical system into images can be expressed in two steps:

---

Published as a conference paper at ICLR 2025
Unknown underlying dynamical system
Data
Networks
18
GOAL: 2 fa(2). leam fo
f(e),
D o E Rs idga 1pft-))
D
The mapping be continuous:
Physical state R!"
Object corresponding to a subset of [0, 1 a matrix in R (133)((2+5))
corresponding
Latent state.
Z E S(2)
It
Illustration of the mathematical formulation. The continuous dynamics of the system captured in pixel form. We learn latent dynamical models by encoding this recorded pixel data.
Figure 3:
Mapping a state to set of positions of all the particles constituting the objects: This mapping is represented as S(z) RD B(0, 1]2), where B(O,, 1]2) denotes the set of all Borel sets in [0, 1] S()) =: Ω2 is the set of positions of particles constituting the objects.
Discretizing the coordinate space into image signals, simplifying the images by setting background pixels to 0 and pixels containing objects to 1: Denote as the pixel size and (I + 1)6 1. Then we define the functional L6 B [ 1J²) (+++))x(+++1 as follows:
(1)) = to 1: if [i,d, (i + 1)ō] (ū2d, (62 + 1)8] n # 0,
otherwise,
where Q is a Borel set in [0, 1J2. [ denotes the element located at the (I+1 -2))- th row and (i1 + 1) column of the matrix (S2) E ((+++))(I++),
Finally, we define the observed image data for any state 2 as In o S(z). See Fig. for an illustration.
To reduce dimensionality, CNN typically employ parameter called stride to progressively down- sample feature maps. It is well-known that the convolution operation for two-dimensional input images can be expressed as follows (Goodfellow et al. 2016):
Input:
To E ((++))x(++)
J ΣΣu Σ
Hidden layers: [Lles
4- th Ho
Will 1= 1,..... L,, (3)
11=0
Output:
E(Io)= II.
Here 8i is the stride of the I-th layer; I E + + is the output feature map of the I-th layer, and we assume that ||ll O(1); W, E R(1+1)x(J++1) is the filter of I-th layer. For simplicity, we have omitted operations that do not significantly affect continuity, such activation layers. Additionally, to simplify index counting, assume that zero padding is applied only on side of the feature maps, meaning that [L 0 if Vi U/IT $4 + (1.2.....JJ)
Different resolutions 8 = 1/(1 + 1) correspond specific networks, where hyper- pece-parsmeters such as J/ should be adjusted accordingly. Therefore, we will also denote the feature map as If. the filter as W and the CNN as &g to emphasize their dependence on the pixel size. For the convenience of analysis, we assume the weights 8 of a standard CNN filter can be normalized to bounded function Wi 2 1-1, 1] that is independent of ö:
w b = W((10, 525)s1, where W(x) = if x é [0, J,
Here E is normalization coefficient ensure that || = O(1) and the last equation ensures that the filter function outputs when the index exceeds the defined size limits.
Now we are ready to provide the following definition of continuity, which serves relaxation of the traditional Lipschitz continuity.
Definition 3.1. A sequence of functions {gs(2) Z continuous if there exists a constant cg such that for all z1, 22 E Z, there exists a 8* such that if 8 < 8", then [1gs(21) (2(2)) Cgl|=1 22
R8 E {1/(I + 1)) is called 5-
is worth mentioning that pisce-wise constant approximation 9s of Lipschitz function g with parti- tion size 8 is ò-continuous. Moreover, if limg-0 g8 g. the d-continuity of ga is equivalent to the
4

---

Published as a conference paper at ICLR 2025
Time
Time
Underlying dynamics
Recorded dynamics by pixel
Observed underlying dynamics
Figure 4: Illustration of discrete nature of pixel observations for continuous motion. Considering a motion similar that depicted in Figure [ assume that the object occupies a very small volume and its motion is recorded in continuous time periods. The left side illustrates the underlying motion of the object, while the middle side shows the evolution of pixel values during the recording process. The right side depicts the observed motion derived from these pixel values, reflecting the discrete nature of pixel observations.
Lipschitz continuity of g- We adopt this definition due to the discrete nature of pixel observations, where slight variations in z might not be reflected in image, as will be illustrated in Section (32
As discussed previously, to learn continuous latent dynamical model, it is essential that the ex- tracted latent states Z Eg o Is S(z(t)) evolve continuously over time. Therefore, our objectives for the autoencoder
follows:
Find the encoders {Es} such that Es o Ig o S()) is 5-continuous.
* Find the decoders (Ds) such that D3 &g is approximately the identity mapping.
The second objective guarantees that the learned latent states are non-trivial, which is standard requirement for autoencoders. The first objective ensures that the latent states evolve continuously with the underlying dynamies. If it can be achieved, with an additional assumption that the pixel size 8* of the image is sufficiently small, then there exists a constant such that
|| 7n, Z = ||Eg- Is- o S(z.) - Eg. o Ig- o S(n+t1) S ee ||=n - Zn+ill S ce MyAL
In=00--.N. This inequality implies that the resulting latent variables { NN, m=1,--.M corresponding to the input images (I). are discrete samplings of Lipschitz continuous trajectories. This property allows us employ a continuous dynamical model for learning the latent dynamics and predicting future behavior by decoding the predicted latent states using the decoder. In the following sections, analyze why standard CNN encoders may fail to achieve the first objective and how our proposed method overcomes this limitation.
3.2 MODELING OF MOTION
To illustrate the rationale behind the modified definition of 5-continuity introduced in the previous section, we consider the example of rigid body motion in a wwo-dimensional plane. This type of motion, involving only translation and rotation, allows for a clear representation of the mapping S, which is essential for the subsequent analysis.
The equation of rigid body motion on two wwo-dimensional plane can be expressed as follows
& = f(z),
("), a' (ri..... ,R), z" (0I,... ,0k),
(4)
where K is the number of rigid bodies. The image corresponding to the state 2 is given by
S(z) = L0 (Sxx + Tkk 2 C IR?
(5)
where тA (k.2) and be, represent the translation and rotation of the object, respectively.
Details of this model can be found in Appendix A. Here assume that De (54) + -e C [0, 1 for & = -.... KK and that these sets are pairwise disjoint for all z E Z.
Suppose K = 1 and the motion only involves translation, we take Ω2 (0, wi x (0, w2] and further assume that w,/ is integer. this case, we have:
max jo. [s())))) (S(())), where 2 (f1.1, (122) and ÅI, jE2.jöSr1.
This implies that variations smaller than 8 may not be captured in the image, and thus, we can only track the dynamics of é, the piece-wise constant approximation of z, as illustrated in Fig. g Conse- quently, the function Eς o I& S(z) cannot be continuous under standard definitions of continuity. Using our definition, we can readily verify that (1) gs(=) = 2 8-continuous; (2) gz(2) = I8 S(2)
is not ō-continuous; and (3) 9 96 Z + a is 8-continuous if go Z + Id is 5-continuous and d dRd is Lipschitz continuous.
5

---

Published as a conference paper at ICLR 2025
(A)
(B)
(C)
Uniform motion
Two body
Single pendulum
Elastic pendulum
Double pendulum Swing slick
Polymer stretching
Figure 5: (A) Illustration convolution operation. The red boxes represent the filter size (I) or (() The blue box represents the object. The solid line indicates its initial position, while the dashed line represents its position after motion. (B) The variation of latent states divided by At for the two-body system, where the encoder is one-layer CNN with parameters uniformly sampled from [-1,1] 1]. The shaded region represents one standard deviation. (C) Examples of motion where the positions of the objects after variation only partially overlap with their positions before variation.
3.3 WHY STANDARD CNN AUTOENCODERS FAIL
Since Is S()) 3)) is not ô-continuous, the composition & I further restrictions is imposed on E. Next we illustrate this issue by example of uniform motion.
I S(z) is generally not ó-continuous no
Consider a vertical bar with a width of A undergoing uniform horizontal motion within the image, which is two-dimensional extension of the motion depicted in Fig. [ The equation governing this motion is 3 = 1, x0) = O. We suppose t E [0. 1/2] to ensure that the bar remains within the image. Let (2) (2, 2 + A (0, 1] and assume A/6 is an integer. The image of nō is of the form
I S (nd) (0(1+1)xx= ((1-1)x(4/--1) ((-1)x(---AA6))
where Xiz denotes the 12 matrix whose elements are all z. We can verify that if To Ig o S(nd), then [4,0 = Σ Σ [W1] 1.ji.ja == gs(n)).
Suppose the step size for the motion is set to 2A. We consider the two cases:
1) the CNN filter size is of constant size i.e. J1 O(1), then we have gx(22) 0 and g(0) 25
Bo = Σラ 0 ii W j26) as 6 + 0, where Bo * O otherwise I is all zero. Then for any constant c₁, > 0, if 24 < [Bol/r,, (2x(2 ((0) > cı, [2A]. This indicates that [Too)] is not 5-continuous, implying that the extracted latent states cannot be learned as continuous dynamics.
2) the CNN filter size increases as image resolution increases, i.e. J1 = (1/5), then we have
32/8
A/6
8
(2A) -96(0)
Σ
Σ (i(110,220e
Σ Wi(iid,j20)e
JAA'
ji 21=22/12200
11=0,1200
We assume Wi 526) are i..dd samples from uniform distribution on [-1, 1]. Then both (2A)) and g(0) are also independent samples from a uniform distribution on [-1, 1]. It follows
that (9x( (0) < Cy || for given Cg with probability zero as A - 0. This implies
that if no further restrictions is imposed on standard CNN filters, [fo is not ò-continuous and the extracted latent states do not exhibit continuous dynamics with probability one.
This calculation is schematically illustrated in the left panel of Fig. 5 It is noted that in scenarios involving small-volume objects and limited overlap in their positions between steps, standard CNN without additional constraints performs poorly in outputting continuous latent states. A numerical demonstration of this limitation for two-body systems is shown in the middle panel of Fig. Several examples of motions where this failure may occur are shown on the right side Fig. 5
3.4 QUANTIFYING CONTINUITY OF CNN AUTOENCODERS
In the aforementioned counterexample, it is evident that a sufficient condition for ensuring the con- tinuous evolution of latent states is that Wi is Lipschitz continuous. In this section, we will quantify the continuity of CNN autoencoders rigorously and extend this analysis more general scenarios. Assumption 3.1. There exists a positive constans MA and an integer L' < such that, if (iz, de)/LL//I'__ si} & [Ma. 1 - M.I2. then [ 0 I 1.... -
This assumption holds in scenarios where the objects of interest are well-captured and located within the central region of the image. Alternatively, can be satisfied by padding the input image with a sufficient number of zeros.

---

Published as a conference paper at ICLR 2025
With these preliminaries, next present the main theorem and provide its proof in Appendix A.2
Theorem 3.1. Assume that the underlying dynamical system is a rigid body motion ( on a two- dimensional plane. If Assumption 3.7 hold, let ew be constants satisfying
max [Wi(rn) Wi(y2)) S ow |1 -222l. 1=1,-
and if 8: 2or 1 =,,.. 1, then for any (2i 2)) 22 (22, E 2, we have
Cew
|[8 Is S(z1) E6 Is S(22))l S Cew || - z + 2 || -=éll as 8 + 0.
Here C is constant independent of 6 and z.
This theorem establishes a connection between the d-continuity of CNN encoder and the continuity of its filters. To ensure that the latent states evolve continuously with the underlying dynamics, is sufficient for the functions Wi representing the filters in the first few layers to Lipschitz.
3.5 METHOD TO PRESERVE CONTINUITY OF CNN AUTOENCODERS
In this section, discuss strategies promote continuity of W/ thereby ensuring that functions W, 1, 1,.,., L have small constant ew and & Ig S is 5-continuous.
Note the fact that max; max; [W(() 5, ( S ow, larger filters necessary to sure continuity. It is worth mentioning that using a filter with the same size as the input image in CNN is essentially equivalent to using fully connected neural network (FNN). While FNN coders are commonly employed in baseline methods, they are often insufficient for second objective of reconstructing images with complex visual patterns, as demonstrated Appendix AA.344
Focusing on images of size 3 x 128 x 256, the parameters of the downsampling layers within the encoder are detailed in Table [ The first three layers in this table use large filters (i.e., L 4). The remaining five layers are standard convolution layers used for extracting latent features.
Table 1: Architecture of the encoder in CpΛE
Layer
1
2
3
4
5
6
Filter size
12
12
12
4
4
4
(3,4)
Stride
2
2
2
2
(1.2)
As we only need the values of Wi on grids for computation, we recommend using the nonlocal op- erators method (Gilboa Osher, 2007 (2009), an image processing technique that promotes image continuity. This approach requires only the following regularizer for the filters:
J'JJ
J J- Σ 1=1
Σ
(W! W1 -W ) ((11,,20)) (116,
(6)
where \, is a weight hyperparameter, which is set to by default, l is positive and symmetric function and the parameter J > 1. Herein, we recommend employing the Gaussian kernel function k(x)- e - and setting j 1. And apply the regularizer Eq. (6 to the filters of the first three layers to penalize large ew and ensure the filters have appropriate continuity.
EXPERIMENTS
The benchmark methods used for comparison this section include existing dynamical models such as Neural ODEs (Chen et 20. Hamiltonian Neural Networks (HNNs) (Greydanus et al., 2019). and Symplectic Networks (SympNets) (Jin et al. 2020 2023), coupled with standard autoencoders. Neural ODEs incorporate only the continuity prior. In contrast, HNNs and SympNets are structured dynamical models that leverage prior knowledge of Hamiltonian systems. With the goal of obtaining latent states that closely aligns with the assumed dynamics, these methods (Greydanus et al. 2019 Jin et al.) (023) typically train the latent dynamical models and the autoencoders simultaneously by minimizing the following loss function:
C C= E
(T (ID E(x) II13 + 1I E(v) v12) E
0 {())1
7

---

Published as a conference paper at ICLR 2025
where T= (Xx, X ) 1=011 Mai,,,,-11 ,N-1, m=1, .M is the training dataset, d is the latent dynamical model detailed Appendix A.3.I
CpAEs able to learn latent states that evolve continuously with time. Thus, we propose to learn the latent states and their corresponding latent dynamical models separately. ODEs are not only continuous over time but also preserve orientation, characterized by the positive determinant of the Jacobian for the phase flow. Therefore, we employ VPNets (Zhu et al.] (022). which have the unit Jacobian determinant, for regularization. This regularization also helps penalize the large constant C' in Theorem 3.1
The loss function for CpAEs is defined as follows:
LCPAE = E(E.DET Do (x) x112
JR J.
where Tn is given by JR = An >n up o [(x) E I'D L.p o E(x) - v122 and Pop is small VpNet, ÅR is set to 1 by default. After training the CpAEs, we separately learn continuous dynamical model for the latent dynamics. this section, the latent model for CpAEs is chosen to be a Neural ODE.
We also compare our proposed method with the hybrid scheme of neural state variable (Chen al. 2022), a discrete model that has demonstrated impressive predictive accuracy. Note that the datasets used our experiments, except for the first one, are obtained from Chen et al. (2022). Following their preprocessing steps, we concatenate two consecutive frames to form the data points. Parameters of all methods can found in Appendix A.3.I
Assessing the fidelity of learned latent dynamical models remains an open challenge. The quality of background reconstruction can significantly impact the Pixel Mean Squared Error (PMSE). In contrast, if there is already overlap. large deviations in object position may not significantly increase PMSE. In this paper. we adopt a similar definition as in Jin et al. (2023); Botev compute the Valid Prediction Time (VPT) for evaluating model's 's predictive ability:
Botev et al. (2021)
VPT = arg max, {t S T PMSE(X,, X,) S VT t),
where E is a threshold parameter, X, is the ground truth and X is the prediction at time т. Here we T and 0.007 for the first three datasets, and = = 0.0015 for the last dataset. We found that once these thresholds are exceeded, there is a significant deviation in the predicted images. The VPT scores are averaged across all test trajectories. We also compute the Valid Prediction Frequency (VPF), which represents the frequency of test trajectories for which VPT 1. The code accompanying the experiments is publicly available at /nitss:///// :ttttp:://gitub.... github. com/ Aiqing Zhu/
CpAE
4.1 CONTINUITY OF LATENT STATES
We demonstrate the continuity using a simple circular motion dataset comprising 220 images (48 x 48) captured every 0.1 seconds along single trajectory, with 70 images for training and 150 for testing. A single hidden layer 48 x 48 CNN autoencoder with various regularizers is used to learn the latent states, followed by Neural ODE to model their dynamics. After training, we show the latent states of the test images and the predicted dynamics. More detail are given in Appendix A.4.1
Standard
AE Lg regularizer
AE + continuity regularizer (Eq.(6))
-5
-20
0.0
2.5
5.0
7.5 Time
10.0
32.5
15.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
Time
Time
Predicted latent dynamics component) Predicted latent dynamics (2nd component)
component) Latent states (2nd component)
Figure 6: The latent states and the corresponding learned dynamics derived from them
As shown in Fig. 6 neither the standard autoencoder nor the addition of a conventional L2 regu- larizer can extract continuously evolving latent states, leading to the failure of subsequent Neural ODE training. In contrast, the proposed continuity regularizer (6) ensures continuous latent state evolution, enabling the Neural ODE to effectively capture their dynamics.
8

---

Published as a conference paper at ICLR 2025
Method
CpAE
Hybrid scheme
AE +Neural ODE
AE+HNN
AE+SympNet
Dataset
VPT
VPF
VPT
VPF
VPT
VPF
VPT
VPF
VPT
VPF
Dumped pendulum
99.248.55
99.2
95.41:5.0
88.3
50.7431.2
23.3
Elastic pendulum
72.14:27.2
36.7
49.5-124.2
30.6::18.5
Double pendulum
69.1=31.5
40.0
46.842144
24.3113.8
11.014.2
15.1
0.0
Swing stick
57.4+200.4
11.1
13.7+5..1
0.0
14.4+7.5
0.0
24.1++14.5
0.0
14.8+12.2
0.0
Table The performance of four physical systems evaluated using the VPT and VPF metrics, All values are scaled by a factor of 100, with higher scores indicating better performance. VPT scores are reported as mean + standard deviation. We not report the performance of HNN and SympNet on the first two datasets, as their underlying systems are not Hamiltonian.
Ground truth
Ground truth
Prediction of CpAE
Prediction of CpAE
Prediction of hybrid scheme
Prediction of hybrid scherne
Prediction of CNN-AE Neural ODE
Prediction of CNN-AE Neural ODE
b-10/60 t-18/50 b-26/50 b=24/00 b-42/50 b-50r50 b-58/60
t=10/50
t-42/60
Damped pendulum
Elastic pendulum
Figure 7: Predictions for simulation data.
4.2 SIMULATION DATA
We then benchmark simulation datasets to show the enhanced prediction performance of CpAE.
Damped pendulum. This dataset consists of 1,200 trajectories, each containing 60 discrete data points-images of size 3 128 128-sampled time interval of seconds. The details of this system are provided in Appendix AA..2). The images the physical state maintain the form a single pendulum and are are generated using similar procedure in Chen et al. (2022).
Elastic pendulum. To verify the effectiveness of CpAEs on non-rigid motion, we consider the elastic double pendulum, where each pendulum
can stretch and contract. The dataset is gener- ated following the procedure outlined in Chen et al. (2022). It consists of 1,200 trajectories, each containing 60 data points-images of size 3 128 128-sampled at time intervals of te seconds.
Table and Fig. 7 demonstrate that both the hybrid scheme with neural state variables and the
proposed CpAEs accurately approximate the actual dynamics they evolve. In contrast, standard CNN autoencoders exhibit lower predictive accuracy due to their tendency to produce discontinuous latent states (as illustrated in Appendix A...3). while using continuous model learn the latent dynamics. Since hybrid schemes do not require continuous latent variables, the advantage of CpAEs less pronounced. Nevertheless, CpAEs provide distinct benefit by yielding continuous latent model, which is crucial for many scientific applications (Chen al.) 2023 Krishnapriyan et al.) 2023 Qiu et al. (022). This continuous latent model also facilitates tasks such as time reversal and interpolation between observed states (as shown in Appendix A.3.3)
4.3 REAL-WORLD DATA
To evaluate the model's performance on real-world systems, we perform experiments using the double pendulum and swing stick datasets from (Chen et al.) (022). Both datasets consist of images of size x 128 x 128 recorded at 60 fps. The double pendulum dataset contains 1, 200 trajectorics, each consisting of 60 discrete image frames, while the swing stick dataset includes 85 trajectories, each with 1, 212 discrete image frames.
success with simulated data, further show that CpAEs outperform baseline methods on real-world data, as illustrated Table [ and Fig. Although the double pendulum is Hamiltonian system, and SympNets well-suited for such systems (Jin et al. 2020), standard
Building

---

Published as a conference paper at ICLR 2025
Ground truth
Greund truth
Prediction of CoAE
Prediction of Cp4E
Prediction of hybrid schern
Prediction of hybrid scheme
Prediction of CNN-A Neural
Prediction CNN-AE Neural ODE
Prodiction CNNAA
Prodiction CNN-AE SympNet
I-10y50 1-18/80 1-26/50 t-34600
1-42/60 t=50/60 t-58660
1-2/60
b-10y50
1=18/50
1=26/60
1-34/60
1=42/60 t-50/60 t-58/00
Double pendulum
Swing stick
Figure 8: Predictions for real-world data.
autoencoders often struggle to learn latent variables that adhere strictly to Hamiltonian constraints, leading to predictions that deviate substantially from the true dynamics. This issue particularly pronounced for the double pendulum system.
SUMMARY
Leveraging known properties of data domain design efficient deep learning architectures has significantly advanced the field. When using a dynamic model with prior knowledge to learn hid- den dynamies from images, it essential that the extracted latent states follow dynamic system that aligns with the specified prior. In this paper, we introduce continuity-presed (CpAEs), novel approach designed to impose the continuity restriction on latent state represen- tations. Our main contribution is the mathematical formulation for learning dynamics from image frames, which illustrates the issue of discontinuity of the latent states extracted by standard CNN encoders. We then show that the output latent states of CNN encoders evolve continuously with the underlying dynamics, provided that the filters are Lipschitz continuous. Moreover, numerical experiments show that CpAEs outperform standard autoencoders.
autoencoders
In this paper, our continuity quantification is limited to rigid body motion a two-dimensional plane. Generalizing our analysis to include non-rigid motion and the projection of three-dimensional motion onto a two-dimensional planc presents significant challenges, which we leave for future re- search. While CpAEs show superior performance compared baseline approaches, it is important to note that they still fall short of effectively solving all tasks. As the visual complexity of the im- ages increases, particularly in the challenging swing-stick tasks where the dynamics are not fully characterized, and the images are captured against complex backgrounds with significant noise, we observe decline in performance, highlighting the need for more sophisticated approaches. Herein, we adopted the simplest method of promoting filter continuity by adding regularization. Future research could explore more effective methods, such as hypernetworks (Chauhan et al. (202)). to achieve this goal. Furthermore, our current research is exclusively focused on the weak prior of continuity within the context of CNN. Whereas numerous studies have explored incorporating clas- sical mechanics-inspired inductive biases into neural networks to construct dynamical models, promising direction would to develop autoencoders that explicitly impose other forms of prior knowledge. Moreover, we would like to investigate the effectiveness of advanced architectures, such as Vision Transformers (Dosovitskiy et al. 2021 and their variants (Sriwastawa & Arul Jothi 2024). autoencoders for learning dynamics from images in future work. This approach holds promise because the operation applied to each patch in ViTs can interpreted as convolution with a kernel size equal to the patch size, and the mathematical formulation introduced restricted CNNs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG3-RP-2022-028) A. Zhu is supported by the NRF fellowship (project No. NRF-NRIF113-201--0000
Pan is part the programme DesCartes and is supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
REFERENCES
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on unsupervised and transfer learning, pp. 37-49. JMLR Workshop and Conference Proceedings, 2012.
Aleksandar Botev, Andrew Jaegle, Peter Wirnsberger, Daniel Hennes, and Irina Higgins. Which priors matter? benchmarking models for learning latent dynamics. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, 2021.
Steven L Brunton and Nathan Kutz. Data-driven science and engineering: Machine learning, dynamical systems, and control. Cambridge University Press, 2022.
Steven L. Brunton, Joshua L. Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. USA, 113 (15):3932-3937, 2016.
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review hypernetworks deep learning. arXiv preprint arXiv:2306.06955 2023.
Boyuan Chen, Kuang Huang, Sunand Raghupathi, Ishaan Chandratreya, Qiang Du and Hod Lip- son. Automated discovery of fundamental variables hidden in experimental data. Nature Compu-
fational Science, 2(7):433-442, 2022.
Tianqi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In 32nd Conference Neural Information Processing Systems (NeurIPS 2018), pp- 6572-6583, 2018.
Xiaoli Chen, Beatrice W Soh, Zi-En Ooi, Elconore Vissol-Gaudin, Haijun Yu, Kostya S Novoselov, Kedar Hippalgaonkar, and Qianxiao Li. Constructing custom thermodynamics using deep learn-
ing. Nature Computational Science, pp. 1-20, 2023.
Yuan Chen and Dongbin Xiu. Learning stochastic dynamical system via flow map operator. Journal of Computational Physics, 508: 12984, 2024.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint #rXiv:2003.04633 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations (ICLR 2021). OpenRe- view.net, 2021.
Christopher Eldred, François Gay-Balman Sofiia Huraka, and Vakhtang Putkaradze. Lie-poisson neural networks (Ipnets): Data-based computing hamiltonian systems with symmetries. Neural Networks, 773:00612.. 2024.
Guy Gilboa and Stanley Osher. Nonlocal linear image regularization and supervised segmentation. Multiscale Modeling & Simulation, 6(2):595 2007.
Guy Gilboa and Stanley Osher. Nonlocal operators with applications to image processing. Multi- scale Modeling & Simulation, 100:51005-1028. 2009.

---

Published as a conference paper at ICLR 2025
Dong Gong, Lingqiao Liu, Vuong Le. Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1705-1714, 2019.
Raul Gonzâlez-Garefa, Ramiro Rico-Martinez, and Joannis Kevrekidis. Identification of dis- tributed parameter systems: A neural net based approach. Computers & Chemical Engineering, 22:8965-$968, 1998.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), pp. 15353-15363, 2019.
Yiqi Gu, John Harlim, Senwei Liang, and Haizhao Yang. Stationary density estimation of itô diffu- sions using deep learning. SIAM Journal on Numerical Analysis, 61(1):45 82, 2023.
Xifeng Guo, Xinwang Liu, En Zhu, and Jianping Yin. Deep clustering with convolutional au- toencoders. In Neural Information Processing: 24th International Conference, ICONIP 2017, Guangzhou, China, pp. 373-382. Springer, 2017.
In Huh, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Time-reversal symmetric ODE network. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020), pp. 19016-19027, 2020.
Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. Sympnets: Intrin- sic stracture-preserving symplectic networks for identifying hamiltonian systems. Neural Net- works, 132:166-178 2020.
Pengzhan Jin, Zhen Zhang, loannis G. Kevrekidis, and George Em Karniadakis. Learning poisson systems and trajectories of autonomous systems via poisson neural networks. IEEE Transactions on Neural Networks and Learning Systems, 34(11):8211-8283, 2023.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations (ICLR 20/4), 2014.
Aditi S Krishnapriyan, Alejandro F Queiruga, N Benjamin Erichson, and Michael W Mahoney. Learning continuous models for continuous physics. Communications Physics, 6(1):319, 2023.
Bo Lin, Qianxiao Li, and Weiqing Ren. Computing high-dimensional invariant distributions from noisy data. Journal of Compulational Physics, 474: 1 1783, 2023.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In 7th International Conference on Learning Representations (ICLR 20/9). OpenReview.net, 2019.
Katharina Ott, Prateek Katiyar, Philipp Hennig, and Michael Tiemann. Resnet after all: Neural ODEs and their numerical solution. In 9th International Conference on Learning Representations (ICLR 202 ), 2021.
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of images, labels and captions. In 30th Confer- ence on Neural Information Processing Systems (NIPS 2016), 2016.
Xiaojie Qiu, Yan Zhang, Jorge D Martin-Rufino, Chen Weng, Shayan Hosseinzadeh, Dian Yang. Angela N Pogson, Marco Y Hein, Kyung Hoi Joseph Min, Li Wang, et al. Mapping transcriptomic vector fields of single cells. Cell, 185(4))690-711, 2022.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Multistep neural networks for data- driven discovery of nonlinear dynamical systems. arXiv preprint 000100//0000//220000 2018.
Mare' Aurelio Ranzato, Fu Jie Huang, Y-Lan Boureau, and Yann LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In 2007 IEEE conference on computer vision and pattern recognition, pp. 1-8. IEFF, 2007.
12

---

Published as a conference paper at ICLR 2025
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science, 324(5923):81 2009.
Chunfeng Song, Feng Liu, Yongzhen Huang, Liang Wang, and Tieniu Tan. Auto-encoder based data clustering. In Progress in Pattern Recognition, Image Analysis, Computer Vision, and Appli- cations: 18th Iberoumerican Congress, CIARP 2013, pp. 117-124. Springer, 2013.
Asmi Sriwastawa and Angel Arul Jothi. Vision transformer and its variants for image classification in digital breast cancer histopathology: comparative study. Multimedia Tools and Applications, (((13):37731-39755 2024.
Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Alcksandar Botev, and Irina Higgins. Hamiltonian generative networks. In 8th International Conference on Learning Repre- sentations (ICLR 2020). OpenReview.net, 2020.
Rui Wang, Danielle Maddix, Christos Faloutsos, Yuyang Wang, and Rose Yu. Bridging physics- based and data-driven modeling for learning dynamical systems. In Learning for dynamics and control, pp. 385-398. PMLR, 2021.
Kailiang Wu and Dongbin Xiu. Data-driven deep learning of partial differential equations in modal space. Journal of Computational Physics, 408: 109300 2020.
Pinchen Xie, Roberto Car, and Weinan E. Ab initio generalized langevin equation. Proceedings of the National Academy of Sciences, 121(14)c230866812 121, 2024.
Jianke Yang, Wang Rao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry-informed gov- erning equation discovery. arXiv preprint 21X.::2400..6566 2024.
Haijun Yu, Xinyuan Tian, Weinan E, and Qianxiao Li. Onsagernet: Learning stable and interpretable
dynamics using generalized onsager principle. Physical Review Fluids, 6(11):1 11:4224 2021.
Zhen Zhang, Yeonjong Shin, and George Em Karniadakis. Gfinns: Generic formalism informed neural networks for deterministic and stochastic dynamical systems. Philosophical Transactions of the Royal Society A, (80((22)))20210020) 2022.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In 8th International Conference on Learning Representations (ICLR 2020). OpenReview.net, 2020.
Aiqing Zhu, Beibei Zhu, Jiawei Zhang, Yifa Tang, and Jian Liu. Vpnets: Volume-preserving neural networks for learning source-free dynamics. Journal of Computational and Applied Mathematics, 416: 1 14523, 2022.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national conference learning representations, 2018
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. RIGID MOTION MODELING
Here we consider rigid body motion, namely that all its particles maintain the same distance relative to each other. The position of the whole body can be represented by the imaginary translation and rotation that is needed to move the object from a reference placement its current placement
We consider the motion of a rigid body, where all particles maintain a fixed distance relative to each other. The position of the entire body can be described by the combined translation and rotation required to move it from reference placement to its current placement.
Let us first select a reference particle, denoted as A, typically chosen to coincide with the body's center of mass or centroid. When the rigid body is in its reference placement, the position vector of is denoted by r'A. For another particle B, its position vector can then expressed as:
r = 2 + - r = r + Ir's ('cls fg.. sin 0²)
where 1r r denotes the distance between points A and B, and O represents the direction angle of the vector rB -
When the rigid body is in its current placement, denote the position vector of by ra. Then the position vector of particle B can be written as
TB = FA + TB - TA = TA= rB - T.l(cos 0p. sin 0B)
Since the body is assumed to be rigid, the distance PB rAl = |r r remains invariant, and there exists an angle f such that 0g- = 02 + for all particles B. Thus, we have:
TB = T + TA r + Ir (cossos 0% + 0, sin 6% + 0)T
cos 0
sin 0
COs 0o sin 0
= T +TA T + 1r r
sin 0
cos
coss
sin 0
= PA - A +
(r A) + r..
sin cos
Thus, the position of the entire body can described using the translation of the refer- ence point r F r and the orientation of the body 0. Let Ω = {rl B the particle constitute the rigid body] denote the set of particle positions when the rigid body is in its reference placement. The set of positions corresponding to the state (r,0) in the current placement can then be expressed as:
S(z) r +b())
where r = -A - - represents the translation of the object, and 49 R22 - R22 defined as
cos 0 sin sin 0
s (r
(r r + +r
=
cos 0
which represents the rotation of the object. Given that there K objects, the rigid body motion on a two-dimensional plane can be given by Eq.
A.2 PROOF OF THEOREM 3.1
A.2.1
FUNCTION REPRESENTATION OF CNN AUTOENCODERS
For the convenience of analysis, we represent the feature maps and the filters of CNN autoencoder functions defined two-dimensional plane:
Input:
Io(
Hidden layers: [v,, 2) = Σ
I-1(1'11 3-=(0'-,=))00'''" .y)) I I= 1,...
(7)
Output:
Iz(y" ,²), 3² E L
14

---

Published as a conference paper at ICLR 2025
where the imput is defined as
[ S())li y E [116, (41 + 1)8) 1820, (82 + 1)6), i, 0, , if $ [0, IJp.
Toly, 2)
0,
l.e., To is a piece-wise constant approximation of the indicator function of the set S(z), with par- tition size of ò. Here, denote the evaluation set of thel-th layer as y C R2, which is determined by the parameter stride:
Y = {(1101, (261))
6, = II ==1
s,)6, 1 = ,,.,,
(8)
IR3 x I2a - R represent the filter of (-th layer, i.. (2(11,)28) if y E [110-11 (J1 1)) x [2201-1, (12 + 1)6(-1),
0i (v' v.
J1. 0,... JI,
otherwise.
Then it is straightforward to verify that Iz : I22 x Z + R represents the function corresponding to the feature map It, such that:
((i i20i), =) = ( = 0, TT.
Refer to Fig. 9 for an illustration. All functions here depend on 8. To avoid redundancy, we omit explicit notation of this dependence.
yi (yo, pe....>
y
{als Uls ...
I: 2 2)
Felpo +)
7e(ou +)
Ze(yu.
94
St
ve
du (or. Ra
(y 10
(yw-5x)
fafsu du (a(on.
far Be
fo Iea
(L
6 End
Pafsu
Ze(mi>
Tolas.
Ke(me
Kolou,
Zolov.
Ta(sn.
Ia 2)
Yd
Estos Maa MSi Mh, Bky --.
Y1 (2t. Dls,-.1 ...
Figure 9: Illustration of the convolution operation. The stride of halves the size of the feature map, and new evaluation set is generated by removing half of the elements from the previous one, specifically by deleting every other element. The blue dashed lines indicate operations, such as activation layers, that do not change the evaluation set.
Under Fq. 7 Assumption 3.1 can be expressed in the following equivalent forms. Assumption A.1. There exist constant MA such that if allb Ma. then
I(y 2) = O, Vy e (Y' u AY) n (Y' - A)).
Under Eq. 77. Theorem 3.1 has the following equivalent form.
Theorem A.1. Assume that the underlying dynamical system a rigid body motion on a two- dimensional plane. If A'AssumptionA. A. let co be constant satisfying
max max (or(())- ,,3/) 22 1-1 -111 1=1,-- L
||
1/2
|1- Vy 1-1 1-1 E ER?
1/2
ER²
ER²
1==
and if 8ı 2 for 1 = 1,...,.. - 1, then for any = (zf, +[), 22 = (z/2 E Z,
Ccs
[8 S(z)) E Is S(2))l S Ccs || ا/اl -
||/1-2/11
8 0.
Here C is constant independent of 8 and z.
15

---

Published as a conference paper at ICLR 2025
Given that max; ja choice, the normalization coefficients can be set as: 61 = 8 //|(2)),, = -2 for | = 2, L.
, ( I cs, must choose Ji (11/6). Under this
A.2.2 THE CASE OF L =1
We present the proof of Theorem A.1 for the case of L" 1.
Assume that the image of 2 can be represented as Sz)). Furthermore, suppose there exist volume- suuuee-rsseriing maps &a R2 + R2, dependent on A RD, such that (S)()) = S( + A) and | (x) - x|| I a ||||| for some constant c. It is straightforward verify that rigid body motions, including translation and rotation, satisfy these conditions.
Consider the general latent ODE
Let
(1'
1
Iols", 11) , ydy
/ [S(=)] ss()
Ó1 vdyyy
S(2) 82
we can verify that lims + [1 = 2. (2) In addition, we have
vz')dd - -SS ΦI Di(v.v"")dg) dgy S(z+A)
(10 A) -I ,2]
[S(z)
Ss()
1
Φi (2 ), v' 1 SS(=) Φ1(v0 doy
[(()]
S(=)
$ S())
S(z))acc ||||| S c |||
Taking such that for all < 8*, the following inequalities hold:
I (v 2) i. 2) c1 ||||,
[y A) , A) S cic ||||
These inequalities yield which concludes the proof.
( , A) Iz 30 ||||
A.2.3 THE CASE OF RIGID BODY TRANSLATIONAL MOTION
We now present the proof of Theorem A.I for the case of rigid body motion involving only transla- tion. We rewrite Eq. (7 in the following form emphasize the dependence of evaluation sets.
Input:
22)
Hidden layers: L+ 2|V ... v4)= Σ 2|11-11 ,..., "..,,,,,,,,,,))) 1 0, (9)
VEY!
Output:
In 1 ,... y") v² EY
Under the above expression, if y is not of form given in Eq. 8). we can still compute the output.
Definition A.I. Two evaluation sets Yt and ÿt are called equivalent, denoted Y! 8 if Vy E yd uylyyin Ý!, have I z/yl-1,.... Yo) 0.
We can readily check that two equivalent evaluation sets yield equivalent results for the convolution operation, i.e., if y i (1/+1, ,2
yi,,,,ii... ..") It,, (v/+1, 22/22, y
Y
Due to weight sharing, the convolution operation exhibits translational invariance. This property leads to the following characteristic of the function é representing the weights of the filter.
16

---

Published as a conference paper at ICLR 2025
Property A.1. The function @1 representing the weights of the convolution filter satisfies
2111" +A,
AA, A.y') 2111" A), A
For our motion model Eq. (4, if K = 1, the image of z is translationally invariant due to Eq. (5 i.e. To(y, + A) Toly A 2). Similar property holds for each feature map.
Lemma A.1. For motion model Eq. (4) with K L. the function It representing the feature map satisfies
I(y' 2 AIVi- ro) = Inl - A, 2|2/ - ,,..,,,'-
- A), VAER?
Proof.
The case when = 0 is obvious. Suppose now
Ily'; A|r , Int' - -,, A .., y - A),
then,
I41 41 2 A|r r
Σ I(y' 2 + y Y 10)0+4 ,1/t))
(by Eq. 2
M'EY!
E Inyy - -
r - 1) t
(by inductive hypothesis)
S'EY!
Σ 2 -A.---,.... y A)Ol+1 (5' + + ")
A.
(by taking g' =y¹ A)
-'EYIAA
= Σ -
L('' 21111=111 ,,.., y 1) (0' 1/11 A)EL
(by Property A.
(I /11 A, 2/Y A. ... y A).
(by Eq.
Hence the induction holds and the proof is completed.
As direct consequence, if y - A yi-1 yo A yo, we have the following corollary.
Here, we omit the notation of evaluation set for brevity.
Corollary A.1. For motion model Eq. (4 with K
1, and further assume that A E I22 satisfies
yl-1- yt- we have
I(y + D) = Inl' - 2))
Corollary A.2. For motion model Eq.
with K = 1, and further assume that A satisfies y
A y!1 then we have
, I(U 2)
Σ D¹
1- (0/()//++ + A,y - o1(y"-1,1) ,/)) EL.
lev!
Proof.
I(V 2 A) (1)
Σ (I-1 22 - 1- (1'"-1111'0
lev
= Σ
1-1(1/-1-,, A 2,)00 -,,1) Σ
I--1 -1 21) -1 -
g1-tEr1-1
Σ
1=101'1220010 A, -
Σ In'0-1 22)0(0000 De
E
1-1(8/11,
+ + (61
di -1' 1,11)) Et-
=
Mf 1ev
17

---

Published as a conference paper at ICLR 2025
For convenience, we introduce the following notations:
For satisfying y = y we denote
a1 (A) max
Ity', ,
Y'EY
For satisfying y A y we denote
(A) max max #'EY yf 'eyi
We let Mz and Ma be constants satisfying
Mr = max
maxmax max #E2
Iit'', 22)EE D'EY!,
Σ
MM = max max
Σ y-
y'EY'
Then we have the following property.
Lemma A.2. For motion model Eq.
4 with k= 1, let (A' be sequence defined recursively
=
by Á A and for L 1,. .....,
A is a variation satisfying Y (A Â') = Y Then
we have
a((A') S 1 + Σ M -ATT) ISISISL
k=0
Proof. Observing that (Á' - 1 satisfies the condition in Corollary A.2 we have
[Lv' = + - I(y =]] = [V , A 1-1 + (Å' - - - -
I + A x))
<
Σ
4- 1-1 + 1
A A ,-), s st
8f -E
Σ
I , -
01 1 .))
M--le
<Mrbi(A'- - X M
This estimate can be rewritten
ar (((A)) S Mzbi(A' -111+++ MM (-1)+Maa-11(2" (A
which yields
1-1 a(') S Maa-((A'1) + Σ Mc -A
and concludes the proof.
Lemma A.3. For motion model Eq. 4 with K 2 let A = (Ai,... Ax) with A E R
let (Á' (A).... Ak), A R2TT- be sequence defined recursively by A A and for
li=1 1= 1. 1, A is variation satisfying - (A) A Y for k 1,... K. If A 0,
then we have
(A)) ΣΣ
My - 1515L.
k=
Proof. Let (F1, TK) (Z1, ; and
K
Io ,2) = Σ Iz ( IT, = Σ
zk),
, - 1= 1,..,,
1=1
g--ler--1
18

---

Published as a conference paper at ICLR 2025
next prove that I((y' ΣK= (vy,2x) for 1,... L, by induction on l. Suppose now this statement holds for l, then we have
(y
Σ y'Evi
I( 200 (2)
Σ
Σ
( =x)
y'EY'
Swapping the order of summation leads to
K It ( 22))044111/ (v', y " Jel Σ I ;zk),
I+ ,)
Σ
Σ
9'EY
which completes the induction. Applying Lemma A.2 to I have
a'²) max J'EYi
2 (3 :2)
Σ
I + I (v(,
max
y'EY!
k=1
K
1-1
max a'EY
| ( 2 -1)
(v'.
MaM
- -A
k=
which concludes the proof.
By appropriately selecting the sequence (Â we immediately have the following corollary.
Corollary A.3. For motion model Eq. (4) with K 2 1, suppose A (A1, .Ak) satisfies
4 = (Ak,1, = & Tx,20) E TR? where TE.1, 1k.2 are integers.
(10)
Additionally, let Cus be constants satisfying bi and let 12.5 is the largest integer satisfying 18,5 S L and 2 S |kn for j 1,2 and
(i'1 s.) (is (12 5.)6) S clil00 + 10)
1, K. If we take st for 1 = 1,... L then have
1-1
K M Mr
a1(A)
S
Σ
A +
2 Ax.21
co
2
1=0
k=1
Proof. Let P/
(1/ 10 2)
E R2, where 1 the are integers defined recursively by
1 = Ik.1
$1.1 d mod 2' 1= 1= L. :2, 1 = O;
1/ mod? 2 1= L..... ,2, $0 0. We can readily check that the sequence (Á' =
2,2 TR.2,
d-1
satisfies Y ( Y
due to Assumption A. and Å 0. In addition, we have
(i_
1-i
1 1 ) =b (0L.1 - - )6, (h22 - k2 11)
k
bb (11/////--11 -2 00,
Lit / || 8 8 cs /
Scs
C
1
18 + Cs
24. 11,220,
2
where 12, is the largest integer satisfying k S I., 2
lit for j = 1, 2 and k: = 1.... K.
Finally, applying Lemma A.3 we have
L-1
K
at (A) < Σ
M
Mr
Co
A
+
CΦ
|
m
2 -
2
1=0
1=
19

---

Published as a conference paper at ICLR 2025
A.2.4 PROOF OF THEOREM A.1
With these results, we able provided the proof of Theorem A.1
Proof of Theorem A.1 Denote A
21
22.
A
22+
2/ and for any 8, denote
arg
min
A
A
A
A satisfies
10
Then have A (0, (5),0) and |
8. Subsequently, have
[ Ia ((1)) - S(2))
[8 o 1 + Eg 0 S(2))l
+ S(2
(0, A²) Eg I5 o (0, (0, Es o Is S(22 + (i ) + (0, Â)) - & I. S(z2 (Á. A'))
+
By applying the conclusion for case of I,a = to the first two components and Corollary A.3 to the third components, obtain
[[ Is 8(1)) - Eg Is S(12)l
<Cas || - ا Ccsd
My
Co
AR
Cp 2 -1
Av:21
2
Let 0. we conclude the proof.
A.3 ADDITIONAL EXPERIMENTAL DETAILS
A.3.1 MODEL DETAILS
Details of training. The models are trained with Adam optimization for 500 epochs, using learning rate of 0.001 and a batch size of 512. The data are divided into training, validation, and testing sets, with 80% used for training, 10% for validation, and 10% for testing.
Dimensions of latent states. For the two-body and damped pendulum systems, the latent state dimension was set to For the other datasets, latent state dimension was used. These values are intentionally larger than the theoretical minimum to enhance the model's expressiveness.
CNN autoencoder. The architecture of CNN encoder in this paper adapted from the setting provided by Chen et al. (2022). The encoder is 16-layer CNN, with the parameters of the down- sampling convolutional layers detailed Tables 3 and Each downsampling convolutional layer is followed by additional convolutional layer with same number of output channels, but 3 x 3 filter, a stride of 1, and zero padding of to enhance expressiveness. All convolutional layers are ac- companied by batch normalization layer and ReLU activation function. The final latent variable obtained through a linear layer.
Table 3: Architecture of the encoder in CpAE
Layer
5
6
7
Filter size
12
12
12
4
4
4
4
(3,4)
Stride
2
2
2
(1,2)
#Filters
16
32
64
64
64
64
64
64
Padding
1
1
1
Table 4: Architecture of the encoder in standard AE
Layer
5
6
7
Filter size
4
4
4
(3,4)
Stride
2
2
2
2
2
2
2
(1,2)
#Filters
64
64
64
64
64
Padding
1
1
1
20

---

Published as a conference paper at ICLR 2025
All autoencoders share the same decoder architecture, which leverages residual connections and multi-scale predictions to to improve reconstruction performance. In addition to the deconvolutional layers specified in Table []] each layer, except for the final one, is accompanied by upsampling operation. This upsampling operation is implemented using a transposed convolutional layer with a filter size of 4 x stride of 2, and Sigmoid activation function. The outputs of the deconvolutional layer and the upsampling layer are then concatenated along the channel dimension to form the input for the subsequent layer.
Table 5: Architecture of the decoder
Layer
6
3
2
Filter size
(3,4)
4
4
4
4
4
4
4
Stride
(1,2)
2
2
2
2
2
#Filters
64
64
64
64
64
32
16
3
Padding
1
1
1
Activation
ReLU
ReLU
ReLU
ReLU
ReLU
ReLU
ReLU
Sigmoid
Filters size. Larger filters are essential to ensure continuity. For filter in the 1-th layer of size Jj x Jr, let my = max (,0)) represent the largest weight value. The distance from the location of this maximum value the filter boundary must be less than [J1/2]ō. This
constraint leads to max; mm/((] I cw. Namely, max; max [Wi(116.126]
[J/2|8 I ow. In Theorem 3.1 we show that the constant for the translation component is given by 26"-r. As- suming 2 -1+ max; max,, -fa (1 528)
O(1), it follows that we only need to ensure 2" Jr (1/6). Based on this, we L 4 in our experiments. The images use are of size 3 x 128 256. This implies J; = 128 or 2 = 256, which translates to Ji 8 or Ji = 16. Taking the average of these values, we set
12.
VPNet. We employ very small VPNet for regularization. It comprises three linear layers, each consisting of two sublayers. The activation function employed Sigmoid.
Neural ODE. Neural Ordinary Differential Equations (Neural ODE) Chen et al. (0188 are contin- uous models by embedding neural networks into continuous dynamical systems. In this paper, we consider autonomous systems of first-order ordinary differential equations
y(t) = fe(v(t)),
v(0) = I,
di
The governing function fo in the Neural ODE is parameterized by FNN with two hidden layers, each containing 128 units and using a tanh activation function. Time discretization is performed using the explicit midpoint scheme.
SympNet. We utilize LA-SympNets with the default training configuration provided by (Jin et al, 2023). This setup includes three layers, each containing two sublayers, and employs the Sigmoid activation function.
HNN. The Hamiltonian in the HNN is parameterized by FNN with two hidden layers, each taining 128 units and using a tanh activation function. Time discretization is performed using the explicit midpoint scheme.
A.3.2 DETAILS OF DATASETS
Damped single pendulum. We consider a pendulum of length L and mass m, attached fixed point by rigid rod, free to swing under the influence of gravity. The moment of inertia of the pendulum is given by / 1/2 mLL Let 0 represent the angular position and 0 the angular velocity. The system's kinetic energy T and potential energy V are as follows:
1 10 - m
1
T
V -== mgL cos(O).
2
These energy yield the following second-order differential equation governing the system's dynam- ics:
3g
sin(
2L
21

---

Published as a conference paper at ICLR 2025
Graund truth
Ground
Barkward prediction of CpAF
Backward prediction of CpAE
20100 2-
t-1/G0
Figure 10: Backward predictions (left) and interpolations (right). Continuous models can accom- plish these tasks by changing the value or the sign of dt used in the integrator. This flexibility allows for both predicting past states and interpolating between known points.
True underlying states
Neural state variables
True underlying
Neural state variabless
states
CpAE
AE Neural DDC
W
M
M
20 UriOCoo
W
N
M
M
W
M
W
Pendulum
Elastic double pendulum
Figure 11: Evolution of the learned latent states for various autoencoders. CpABs are capable of generating latent states that evolve continuously with time.
Here we account for damping force proportional to the angular velocity, with damping coefficient k, the final equation of motion for the damped pendulum becomes:
3g sin( R...
2L
To simulate the physics equation of the single pendulum system, we set the pendulum mass m = 1, the pendulum length as 0. and the frictional constant & 0.8. For each trajectory, we randomly sample the initial angular position & and angular velocity Á.
A.3.3 ADDITIONAL RESULTS
Here we show the continuous models allows for both predicting past states and interpolating between known points (Fig. T0p, and visualize the evolution of the learned latent states (Fig. 11
22

---

Published as a conference paper at ICLR 2025
Ground truth
FNN:AE + SympNNl
FNN-AE Separable Hamiltonian
NN-AE Netrral ODE
t-15.0
Figure 12: Predictions for two-body dataset.
Ground truth
Ground truth
Reconstruction CNN-AE
Reconstruction of CNN-AE
}
Reconstruction FNN-AE
Reconstruction of FNN-AE
L-2/60
t-10/60
18/000
L-2660
L-34160
L-42/50 L-50/60 L-58/60
L-2/60
1-10/60
L-10/00
L-26/60 1-34/60
L-42/60 L-50/60 L-50/60
Damped pendulum
Elastic pendulum
Figure 13: Reconstructions for simulation data.
A.3.4 RECONSTRUCT IMAGE FROM EXACT LATENT STATES: FNN AUTOENCODER
We test the performance of FNN autoencoders using Two-body problem data. This dataset is gen- erated the same manner as in Jin et al. (003)) and used to illustrate the FNN autoencoder. consists of 100 observations-images of size 100 50-captured a time interval of 0.6 seconds along a single trajectory.
As shown in Fig. 12 all FNN-based methods, except HNN, accurately predict the ground truth within the given time interval for the two-body system data. Given the strong performance baseline set by these established methods, any further improvements are likely be incremental and may not sigmificantly surpass the current results.
However, this success does extend to datasets with more complex visual patterns. Fig. 13 illus- trates that FNN autoencoders fail to achieve complete reconstruction for the damped pendulum and elastic double pendulum datasets. Here, reconstruction indicates that the encoder extracts the current latent state from the input image, and the decoder maps it back reconstruct the same image. In this process, the decoder uses the exact latent state as input, with no dynamical information involved reflected in these reconstruction results. As discussed in Section [ the third goal of learning dy- namics from images is to identify a decoder capable of reconstructing pixel-level observations using the exact latent state input. Therefore, focus CNNs in this paper.
A.4 ABLATION STUDIES
A.4.1 DETAILS FOR CONTINUITY LATENT STATES
The autoencoder model consists of a single-layer CNN with 48 48 convolutional kernel and 32 output channels, followed by linear mapping to generate the latent states. This simple task and
23

---

Published as a conference paper at ICLR 2025
Ground
Prediction for AE continuity regularizer (Eq.(6))
Prediction for AE + La regularizer
Prediction for Standard AE
t=5.0
Figure 14: Predictions for circular motion data.
circular motion
VPT VPF for damped pendulum
10P
110
VPT
VPT and VPF
VPF
VPT
100
10r
95
0.5
1.0
A
ÅR
Figure 15:
Results evaluated using the VPT and VPF metrics of varying weights. All values scaled by a factor of 100, with higher scores indicating better performance. Note that the VPF metric reported for the circular motion dataset, consists of only a single trajectory.
model selection enable us to clearly and directly demonstrate the limitations of conventional CNNs in achieving 5-continuity. Herein, all autoencoders, regardless the applied regularizers, are trained using full-batch Adam optimization for 10 epochs with learning rate of 0.001. Once trained, Neural ODE is employed to model the dynamics the learned latent states. The governing function fo in the Neural ODE is parameterized by FNN with single hidden layers containing 64 units and using a tanh activation function.
As supplement to Fig. 6 the image prediction results presented in Fig. 14 Since neither the standard autoencoder nor the addition of conventional L2g regularizer can predict the evolution of latent states, they fail generate the predicted images. In contrast, the Neural ODE effectively captures the dynamics the latent states generated by the proposed CpAF, enabling accurate pre- dictions.
A.4.2 EFFECTS OF WEIGHTS
Here compare the performance when vary the weights of regularization terms Y and Ag-
The weight λ controls the trade-off between model complexity and the continuity of the filter. As shown the left side of Fig. [5 setting X too high penalizes overly complex models, which can lead to underfitting and suboptimal performance. Conversely, a lower λ reduces the influence of regularization, increasing the risk of discontinuity in the learned latent states. The weight An emphasizes orientation preservation (i.c., a positive determinant of the Jacobian) in the learned latent states. As shown the right side of Fig. 15 setting ÅR too high shifts the focus toward volume preservation (i.e., unit determinant of the Jacobian), which can cause underfitting and degrade model performance.
24


Reviews:
[{"review_id": "94BBrT0NHg", "reviewer": "Reviewer_5dSC", "summary": "This paper introduces Continuity-Preserving Autoencoders (CpAEs), aimed at learning continuous latent dynamical models from discrete image data. It addresses the challenge of modeling continuous dynamical systems from image trajectories by proposing a new autoencoder architecture that promotes continuity in the latent space. CpAEs are designed to maintain continuous evolution in latent states by enforcing Lipschitz continuity in convolutional filters, with experiments demonstrating improved performance over standard methods, particularly for complex dynamical systems.", "questions": "Would the method work on more complicated mechanical system beyond rigid bodies?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper identifies an important problem: standard autoencoders struggle to learn continuous latent states from discrete image observations of dynamical system. \n* The paper provides rigorous mathematical formulation and theoretical analysis, and a key theorem (theorem 3.1) that establishes conditions for continuous latent state evolution. \n* Empirical results demonstrate the effectiveness across multiple datasets and clear improvements over baseline methods.", "weaknesses": "* Theoretically, the analysis is limited to rigid body motion in 2D plane and does not fully address non-rigid motion cases. The analysis assumes simple transformations (translation and rotation) that may not capture real-world complexity. \n* The Assumption 3.1 about vanishing feature maps near image boundaries may be too restrictive, which may requires significant zero padding. \n* The requirement for large filter sizes (O($1/\\delta$)) could limit practical applications.\n* The significant drop in performance on the swing stick task (57.4% VPT vs 99.2% for damped pendulum) is not thoroughly analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uKLHsFUmXR", "reviewer": "Reviewer_UE7U", "summary": "The authors consider the setting where observations of a continuous dynamical system are captured by a sequence of images which are discrete in time and space. The challenge they claim to address is the misalignment of the evolution of images in pixel coordinates and the evolution of the underlying continuous time dynamical system. To address this challenge, the authors introduce the notion of delta-continuous, a relaxation of Lipschitz continuity where the ratio of the change in latent states and the change in continuous dynamical states is bounded by a constant. The term delta in the name delta-continuous refers to the size of the image pixels. As I understand, the claim is that if the pixels are too big then the delta-continuous condition fails and the latent states cannot be learned as a continuous dynamics. The authors subsequently introduce the continuity-preserving autoencoders (CpAEs) to encourage latent states to evolve continuously with the underlying dynamics. CpAEs do this using large filters in the first few layers, and a regularizer on the weights that encourage image continuity.", "questions": "I am unclear about the following experimental choices: 1) why do the authors compare to HNN architectures if the goal of the paper is to showcase the impact of the CpAE structure? 2) how does removing the VPNet regularizer impact CpAE performance? 3) how does using smaller weights in the initial layers impact CpAE performance? 4) if the CpAE ensures latent continuity why use a latent similarity loss (Line 399)? How much does this loss contribute to the observed performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "The paper is well organized and well written, although in some parts it might be easier to read if variable names were augmented with the concept they represent.", "weaknesses": "While the ideas presented are interesting, the impact of conventional CNNs failing to be delta-continuous is not clear to me. The experiments in the main paper (e.g., Figure 6) attempt to demonstrate the importance of delta-continuity by comparing the reconstructive ability of CpAE to other networks. I am unconvinced by this experiment for the following reasons: 1) it is not clear to me that reconstructive ability is an appropriate indicator of latent continuity; 2) it is not clear to me that using a fully-connected architecture as a baseline is appropriate since ‘[FNNs] are often insufficient for … reconstructing images with complex visual patterns’ (Line 346); 3) the conventional CNN baseline has comparable reconstructive ability to the proposed model. In my view, a more appropriate experiment is presented in Figure 10, pairing this experiment with a quantitative assessment would be more convincing (also I’m not sure what ‘Neural state variables’ are). This alone however, would only demonstrate that the resulting latent structure is more smooth with the CpAE structure. An additional experiment showing that a fixed dynamical model is better able to model the dynamics of CpAE states than AE states would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fqQYbB45o7", "reviewer": "Reviewer_NnyW", "summary": "This paper proposes continuity-preserving autoencoders (CpAEs) for learning continuous latent states from a series of images that represent a continuous dynamical system. The proposed autoencoders resolve problems from a naive usage of Convolutional Neural Networks due to the district nature of images. The key contributions are a mathematical formulation for learning continuous dynamics from image data, the sufficient condition for the latent states will evolve continuously with the underlying dynamics, a regularizer to promote continuity of filters and empirical results.", "questions": "- How did you determine the filter size for Table 1?\n- How much does the regularizer affect the results? Maybe the uplift in the result is just because of the filter sizes?\n- In Figure 6, there is no image of CpAE on the two bodies. Why?\n- Why are there no quantitative results for the two body data?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper presents an interesting mathematical formulation for learning continuous dynamical systems from images.\n- The method presents strong empirical results that show the proposed method works.", "weaknesses": "- There is a lack of comparison with AEs that are not based on standard CNNs in the results.\n- There are many assumptions that limit the formulation only to CNNs.\n- There is a lack of ablation studies on the proposed methods parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pTtzYB1dA5", "reviewer": "Reviewer_UqDE", "summary": "This paper introduces continuity-preserving autoencoders (CpAEs) to learn continuous dynamical models and continuous latent states from the discrete pixel-based frames of a video.\n\nFirst the authors present some theory showing that making the convolutional filters of the encoder Lipschitz continuous the states will evolve continuously. Then the authors introduce a regularizer to promote Lipschitz continuity.\n\nThe experiments show that by using CpAEs to learn continuous latent states, the learnt dynamical models are more accurate compared to standard CNN-based encoders that might produce discontinuous states .", "questions": "I have two questions on your modelling choices. What do you mean when you say:\n1. \"Given our priority on ensuring the fidelity of the latent dynamical model, this work focuses on deterministic autoencoders.\"\n2. \"CpAEs are able to learn latent states that closely align with the assumed continuous dynamics. Thus, we propose to learn the latent states and their corresponding latent dynamical models separately\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Learning dynamical systems from image data is an important topic relevant in many applications.\n2. Standard autoencoder are not forced to learn latent states that are continuous over time, which is a problem when trying to learn the underlying continuous model of the dynamics of the system. The paper presents an interesting theoretically grounded approach to enforce state continuity, which leads to improved performances in the considered experiments.\n3. This theoretical framework could be useful for the community to build on.", "weaknesses": "1. My main concern on this paper is that I am not convinced that the proposed ideas will generalize well to more realistic tasks for which standard CNN encoders are powerful enough. While this is also noted by the authors as a limitation in section 5, I feel that for this paper to have a bigger impact on the ICLR community it would be important to apply this method to more complex architectures (e.g. VAEs, ViT) that will hopefully improve performances at least on the \"swing stick\" experiment.\n\n2. The experimental section is missing some ablation studies to compare performances when varying some key parameters: 1) filter sizes and 2) regularizer parameters $\\lambda_J$ and $\\lambda_R$\n\n3. The mathematical formulation proposed in section 3.1 to project states to pixels is key to understand the rest of the paper but not very easy to follow. It could benefit from using a simple running example, for example focusing on a single particle moving in space (if you add this example, you could even consider moving the example in section 3.2 to the appendix as a multi-particle extension)\n\n4. Some notation is not introduced in the main text: $\\Omega$ in section 3.1, $K$ in section 3.2, $\\lambda_J$ in section 3.5, $\\lambda$ and $\\lambda_R$ in section 4", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)