Online Learning and Control of Complex Dynamical Systems from Sensory Input
Authors: Oumayma Bounou
Jean Ponce
Justin Carpentier
Venue: NeurIPS 2021 Poster (2021)
Keywords: Robotics and Control
Computer Vision

Abstract: Identifying an effective model of a dynamical system from sensory data and using it for future state prediction and control is challenging. Recent data-driven algorithms based on Koopman theory are a promising approach to this problem, but they typically never update the model once it has been identified from a relatively small set of observation, thus making long-term prediction and control difficult for realistic systems, in robotics or fluid mechanics for example. This paper introduces a novel method for learning an embedding of the state space with linear dynamics from sensory data. Unlike previous approaches, the dynamics model can be updated online and thus easily applied to systems with non-linear dynamics in the original configuration space.  The proposed approach is evaluated empirically on several classical dynamical systems and sensory modalities, with good performance on long-term prediction and control.

Online Learning and Control of Dynamical Systems from Sensory Input
Oumayma Bounou', Jean Ponce!:2, and Justin Carpentier'
Inria and Département d' Informatiiuu de I'Ecole Normale Supérieure, PSL Research University CCnter for Data Science, New York University
/oumayma.bounou, jean pennc, custin.carpentier) @ @inria..r
Abstract
Identifying an effective model of dynamical system from sensory data and using it for future state prediction and control is challenging. Recent data-driven algorithms based on Koopman theory are a promising approach to this problem, but they typically never update the model once it has been identified from relatively small set of observations, thus making long-term prediction and control difficult for realistic systems, in robotics fluid mechanics for example. This paper introduces a novel method for learning an embedding of the state space with linear dynamics from sensory data. Unlike previous approaches, the dynamics model can be updated online and thus easily applied to systems with non-linear dynamics in the original configuration space. The proposed approach evaluated empirically on several classical dynamical systems and sensory modalities, with good performance on long-term prediction and control.
Introduction
1.1
Context and motivation
Providing complex machines such robots with the capacity of learning effective models of their dynamics from their physical interactions with the world a key enabling factor for deploying them outside of laboratories and factories. Analytical models of these interactions through state- state-based representations are traditionally derived from physical principles and used solve challenging problems in various domains. They have been successfully deployed on heterogeneous systems such as robots or aircrafts/spacecrafts. Through the prism of control theory, these analytical models enable engineers to characterize and study the intrinsic dynamical properties of systems (contraction, stability, etc.), which are essential for system comprehension and for critical applications. context, optimal control theory [ and its online counterpart known model predictive control (2), exploit these models to plan and control complex dynamical systems (robots, rockets, etc.). Yet, the physical interactions of machines with the world are subject to complex natural phenomena, hardly describable by closed-fone form mathematical models.
this
By leveraging recent progress in machine learning, data-driven approaches 3 have emerged as an effective way of learning advanced models of complex physical phenomena such as turbulence in fluid mechanics ematical framework to learn and analyse complex dynamies. The overall objective of this formalism map the intrinsic state space of the system to typically the system dynamics evolve linearly. However, exhibiting such a map remains challenging, especially in the context of instrumented systems such as robots, which evolve in the world by exploiting their
3
5 6 In particular, the Koopman formalism 7 is an attractive and versatile math-
vector space where
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

sensory input. The sensory inputs used to build a representation of the internal state of the system and the world may be of various nature: images and force, temperature values or encoder readings, etc. Importantly, to be effective for real-world applications such as the control of robotic systems, learned dynamical representations should also offer online update capacities, to account for new measurements as they come, which is essential to get long-term accurate prediction capacities in changing environments, and update the internal dynamics model accordingly.
1.2 Related work
Learning dynamies from data. Data-driven approaches 88 9 based on the Koopman theory 7 and its approximation using the Dynamic Mode Decomposition (DMD) 3 have been used in the context
of building dynamical models from measurements 6 10
15
17 18] A first I6
group of approaches builds handcrafted functions to map from the measurements space to latent Such approaches are suitable when one specific system is being considered but cannot generalize to new systems, or to systems with varying physical properties. A second group of approaches learns a mapping from the measurements space to latent space in the form of parametric functions. These approaches are more flexible, and
space where the dynamics are forced to linear II0
[3
showed promising results for both prediction [4 [6] and control [6 [8] 19, [13), [66] and
[17 learn approximation of the DMD matrix in the form of trainable linear layer. While this setting is satisfying when only a single dynamical system is considered, it becomes inappropriate to build more generic models since the same matrix can not be used to approximate the dynamics of different systems. To address this issue, 66 14 18 look for an approximation of the DMD matrix as solution to a least-squares problem associated a linear model from a given trajectory of a system. In another line of work, [9) seek to identify a locally linear model of system's s dynamics in learned representation space, through identifying matrix to transition from one latent representation to the next one. However, their approach does not generalize to globally linear models.
trajectory of a system. This enables identifying
Learning and control of forced dynamics. An extension of the DMD framework to actuated systems for model identification and control purposes has been introduced in (20), where the transition from one time step to the next is modeled by two matrices; the previous DMD matrix and a control matrix. Building upon this framework, [6 16
18 focused on learning an embedding for actuated systems. O and [16] parametrize the control matrix as a linear layer that is learned along with the embedding, while authors in [18] estimate the control matrix similarly to the DMD matrix (i.e.; from an input system trajectory representation). but assume they have access to the true state of the system, instead of only measurements. although they do it on hand-crafted functions of either the measurements or the states, which they assume known in the case of (22). In this work, we look for both the DMD and control matrices as solutions to a single least-squares problem including state representations and control. However, we assume the true state of the studied systems unknown, and aim at leveraging sensory information to Iearn a linear representation of its dynamics. Contrary to the aforementioned approaches, we also introduce proximal reformulation of the standard DMD approach based on singular value decomposition (SVD) 23 to avoid introducing any extra hyperparameter of the rank truncation of the SVD that is required the classical DMD setting. To account for changing dynamics, [22] estimate initial DMD and control matrices from initial set of pairs of consecutive states of a given system, and perform active learning by updating their matrices with new executions of the system. In this work, we estimate matrices that are specific to a given trajectory, and update them with new measurements.
[12 1 21 22 also follow the same approach,
Future video prediction. The video prediction setting which is the focus of our paper has received a lot of attention in the past few years in computer vision, and various approaches for modeling the time evolution have been proposed. [24 25 26 do it through recurrent neural networks. Several approaches focused on predicting the next frame only conditioning on one previous frame [27 28 29 30]. However, image data does not contain any velocity information. Indeed, if not explicitly modeled, this information can only be obtained when considering at least two consecutive frames, which is the approach we follow in our work. Some approaches 30 31 model the multiplicity of future possible outcomes with variational approaches [32) however, this work, we focus on the
deterministic property of physical systems and only consider one possible future, like [33 34 35
2

---

d7l
12p 2m
L I 8r]
drl
T
0
A
Figure 1: General overview of the proposed pipeline: the measurements [di..... ... dr] are first encoded with 4y to codes [z1, [.1,.... 2r] Only the first codes used to estimate the linear system dynamics A arising from the lquaees minimization problem (3. Using this linear dynamics A and the last
code 2m. the last ' - m codes are predicted. The resulting reconstructed measurements [d,,... dr] obtained by decoding with Ve the embeddings of the actual measurements [Z,y...: 2m] and the predicted embeddings [Am+1:...;; Fr]
1.3
Contributions
We introduce an effective approach to learning a linear state representation directly from raw sensor input (e.g., images) in the form of an instance of a nonlinear latent embedding of state space and linear autoregressive model for that embedding [7]. Unlike classical approaches [G] 14
15 once learned, our model of the system dynamics can be updated online account for new measurements at very low cost. It also involves several key innovations: (1) Contrary to classical
16 17
18
36
machine learning approaches to this problem [27 31 37 our approach firmly grounded in (applied)
Koopman theory [4 38) and, as demonstrated in Section 3.4 it readily generalizes from identifying the dynamics of a system to actually controlling it. (2) Conversely, contrary to existing algorithms based on Koopman theory and dynamic mode decomposition B 39 40), show that online updates of the linear dynamics model can efficiently be implemented in our framework, leading to improved prediction results (Sections 3.2 and 3.3). (3) We introduce several technical improvements that make the proposed approach practical, including a block-companion matrix representation of the linear autoregressive model enabling the use of multiple frames to drive video synthesis (Section 2 and the use of robust, parameter-free proximal iterative refinement algorithm [41] for the least-squares estimation of this model which proves crucial in practice in the context of deep learning library such as PyTorch Our last main contribution is (4) an extensive experimental evaluation of the proposed approach on multiple problems from control theory (Section (..4. Our experiments include video simulations of complex dynamical systems for which we can provide accurate long-term predictions and that we can effectively control.
2 Online learning of system representations
Proposed approach: We propose to learn the dynamics of complex physical system from sensory inputs using a linear auto-regressive model for a nonlinear latent embedding of the corresponding state space [4] 6 12]. Concretely, we consider discrete-time dynamical system defined by some over some (arbitrary) state space X. In our setting,
transition function I. its effect can be in fact (partially) observed through measurements (elements of IR, video frames in our case) acquired in successive states along trajectories of the dynamics. Our aim to learn from such measurements (1) an implicit embedding ;p : X Z of the state space into some latent vector space Z using parametric encoder & IRP + Z of the corresponding data, and (2) a linear model A Z > Z of the dynamics in the latent space that, for any x in X, (LL)) Ay((x) In this setting, h = ((d) is the feature (pz) associated with the (unknown) state x in which d has been measured. There is reason to assume the dynamics L to be linear (which may be meaningless anyway since X' may not be a vector space, or more generally be endowed with a well-defined algebraic structure). However, we know from Koopman theory 7 that, when Z is taken be the set of all real (or vector-valued) functions over X, there exists a linear map P 2 - 2, called the Perron-Frobeniu operator 4 42 such that ((Lx) = Py(xx) for any x in X. Since Z is typically of infinite dimension in this case, P does not normally admit a finite representation in the form of matrix. In the practical case where the latent feature space Z is by design finite-dimensional, on the other hand, P is not guaranteed to be linear but, given m features 2.,..., Zm of dimension n, one can approximate its action using an n matrix A minimizing the loss:
unknown, but
[|
2m] A(E ........ 2m 2m-1]1
(I)
3

---

This approach is called Dynamic Mode Decomposition (DMD) [3] and, together with its extensions (20 39 43 has been successfully used study many dynamical systems. In our setting, the features are given by gathered in our case from some video clip. DMD typically operates on raw measurement data instead of a latent representation thereof. Learning the embedding function + requires some sort of supervisory information, and we follow the popular encoder/decoder framework [44] by also estimating the parameters of a decoding function V/ Z + -. such that & and V/ jointly minimize the regularized mean of the reconstruction error ||d - V o & (d)ll over some sample measurements. The full pipeline is composed of learnable encoder/decoder module mapping from the measurement space to a latent state space, combined with a linear state representation which is learned from a sequence of embedded measurements (see Fig. D. We now detail its different components.
((d,) (i 1...., ......,,, m.) and correspond to m successive measurements,
Measurement embedding. The goal of the autoencoder is learn an embedding of the measure- ments into latent space representation of the state space. While input measurements may be high-dimensional objects (e.g. images), autoencoders are often capable of learning effective yet compact representations of the corresponding state space. Given a sequence of T measurements [di.T], we are looking for two parametric functions: an encoder dg and a decoder V, parametrized by 0 E IR" and μ RR",, following the relation:
(d())) 2 and (x) de for all
Dynamic mode decomposition solves the least-squares problem associated with ①.. which can be reformulated as:
arg min A 2
Zz - AZ where Z1 = [ 2m-1 and Z2
2m]
(3)
standard approach to solve this problem makes use of the singular value decomposition of the matrix Z1 containing the collection of consecutive measurements, following the relation:
A" (21...., , = '2227
where Z is the pseudo-inverse of Zi. As in practice Z₁ is a singular matrix, ones needs to carefully choose given ainuularvaluu threshold, either to damp the solution or to cut the spectrum of Z1. Both solutions only provide approximate solutions of the original least-squares problem (3) Additionally, the SVD is differentiable only when all the singular values are different (45), which may not be the case in general and could lead to unstable behavior when performing (stochastic) gradient descent throw computational layers which include SVDs. To overcome this issue, we propose to rely on an alternative approach called proximal method of multipliers [1I. Instead of solving a classical least-squares problem, we suggest solving an equality-constranne constrained quadratic program of the form:
min AEB
2
PE s.t. Zz AZ1.
The proximal method of multipliers proximal term over the dual variables associated to the constraint Z₂- AZi, leading to:
41 augments the Lagrangian function associated to ( with a
L,(A,A.A))
1 IAP 2
Σ (2411 - - 11
X, 11.
(6)
2
where y the smoothing parameter over the dual, A are the multipliers associated to the constraints 2+41 Ar,, and X, is estimate of the multiplier d Such saddle-point reformulation can be iteratively solved through the the Karush- Kuhn-Tucker (KKT) system of cquations:
Ix 7
Zi
Ax A*
0
plp-1
Zg PA-
where In, corresponds to the identity matrix of dimension n. and A the stack of multipliers associated to each constraint 21+1 As. (7) is solved by iterative refinement until convergence to the fixed point solution. Typically, for small values of p (e.g. 10-8), less than dozen of iterations are needed to converge to the optimal solution and the approch converges for any value of p. The system of equations (7) always invertible thanks to the lower right block -pIr-1 and has a condition number close to the one of Z,. From computational perspective, such formulation can

---

be solved efficiently by exploiting the inherent sparsity of the 2x2 block matrix, using Cholesky decomposition of the KKT matrix. Contrary SVD, this approach can be easily differentiated by backward propagation of the gradient over the iterative process, which appears as an appealing feature for differentiable programming [46] in the context of deep learning.
Online dynamic mode decomposition for long-term prediction. Once built from a sequence of codes obtained from initial measurements, the DMD matrix A can be used predict the codes of future states given a known initial code. As classically done in [6 40 17
only the first m
measurements are used estimate A, and future codes are predicted as:
2m+t A'zm for t 2 1 s.t. A def A X
(8)
limes
In this framework, the matrix A is never updated, thus never using potential measurements, For long sequences, this might lead to inaccurate predictions as A remains static and is only estimated from codes of the first m measurements. While this is enough for simple systems (e.g, pendulum), becomes a limitation when more complex dynamical systems are considered (e.g, a cartpole). To overcome this issue, propose to update A when new measurements learned representation adaptive. We account for this during training by artificially extending the initial sequence used compute A with additional latent codes of measurements. Re-starting prediction from codes of the new measurement should force the prediction error to diverge less quickly. We refer the supplementary material for more details.
acquired, making the
Capturing dynamics effects. Until now, we have considered one-step linear dynamics: each encoding 2 only depends on the state 2;-1 linearly. However, most of the times, measurement at a given time is only a partial observation of the state of the system and does not contain enough information describe the full state of the system. For instance an image does not contain velocity information and at least two consecutive frames are needed to estimate it. To address this issue, we adopt& similar approach proposed in [14] and consider code 2, which linearly depends [Z
history of h previous codes zit
ze]. such that:
2++ A1 + A22
+A 24
24
(9)
where A; Ra,Xx, and A [A [A....., An). Like for Eq. [ Athh can be obtained by solving an augmented linear least-squares problem, following the same proximal approach as before. Note that this formulation is equivalent to the one in Eq. ( only augmented states [zt,.--, 2t,.. ++11 for
1 S 1 S m h + 1 would be considered, and the transition matrix would have a larger dimension of hn hn, depicting a comppccoonnnion structure of the form:
0
I
0
0
0
0
Ā
26+1 Azjh.
(10)
...
...
...
...
0
I
A
A2
An
Such scheme is classical when building models of dynamical systems from measurements and we refer to [47 for a comprehensive view.
Let us now consider an actuated system for which we have sequence of measurements d1:r. We assume the system was actuated with a sequence of control inputs W1:7-1 that we have access to. The goal is to learn a representation space of the states of the system, from the measurements [d1-r] and the control inputs [M.TT] such that in this representation space, the evolution is linear on both states and control inputs. Following the extension of the DMD approach to control 20 we look for a latent space and matrices AI,..., A₁ and B such that we have for all t:
Learning forced dynamics for control.
Z141
th = + Bu
(11)
Unlike (6]. we do not treat B as a learned parameter. We also perform least- deast-squares regression to find 4 and B following the same approach as before. Once these matrices are identified, they can be exploited within the standard linear quadratic regulator setting [ for control. This is possible because we make the assumption that independent of In the case where this assumption does not hold, one can refer to the iterative linear quadratic regulator setting [48].
5

---

Loss function. To learn the parameters associated with our pipeline, we minimize the empirical risk of the L2 reconstruction loss composed of two main terms: a first term associated to the autoencoder and second term accounting for the desired linear prediction in the state space. The full loss is given by:
N
m
La Ceg(tdur)eet..... N
(Po(()
Σ [=++1
|| v (A
Do 1)
Auta encuder loss
Prediction loss
(12)
To simplify the notations, we do not account for the terms associated to the history (Ath and z1" ). In the online setting, predictions in the latent space following measurement dj obtained time m are performed as A (dz) for I >> j.
3 Results
We apply our learnable framework of systems dynamics from sensory input challenging tasks. In particular, we address the tasks future state prediction and control from raw video data, including both unactuated (unforced dynamics) and actuated (forced dynamics) systems.
3.1 Experimental setup
Datasets. We generate three black-and-white 64 x 64 video datasets of classical dynamical systems using the Pinocchio library [49] for the simulation and Panda3D-viewer (50) for the rendering: a simple pendulum, a double pendulum] and cartpole. The cartpole we consider in our experiments composed a horizontal cart that is allowed one translation over an axis, to which a pole is attached and is allowed rotation over an orthogonal axis. For each of these systems, we consider both unactuated and actuated versions, and generate training datasets composed of 4000 trajectories of 5 seconds (T 100) in the first case, and 10 seconds (T 200) in the second, with measurements every 50 ms. In the case of unactuated systems, we consider systems with varying physical parameters (masses, lengths) while in the case of actuated systems, we consider systems with the same physical parameters. In both cases, trajectories are generated with different initial conditions (angular positions and velocities). More details the datasets' generation can be found in the supplementary material.
Architecture. The only learnable parameters in our model are those of the autoencoder. The encoder is made of 6 blocks of 3 3 convolutions followed by max-pooling, batch normalization and ReLu layers, except for the last block which does not have a ReLu layer. The decoder is a symmetric copy of the encoder. The dimension of the codes in latent space corresponds to the bottleneck size of the autoencoder, set to n, = 8 in all our experiments. More details on the architecture can be found in the supplementary material. Each learning problem takes about 3 hours to solve an Nvidia RTX6000 GPU.
Evaluation. The models of the unactuated systems are trained on 5 sequences and evaluated on 15s sequences, and those of actuated systems were trained on 10s sequences and evaluated on 20 sequences. Longer sequences are used for training actuated systems since in this case we seek to identify both dynamics (A) and control (B) matrices in latent space. For all systems, we have evaluated the prediction ability of trained models with the average RMSE loss over time. In Figures 3 the first vertical dashed line indicates the number of time steps used to estimate the dynamics matrix (and the control matrix in the of forced dynamics), and the second vertical dashed line indicates the duration for which the models were trained.
3.2 Long-term prediction of unforced dynamics
Figure shows the prediction error over an horizon of 15s for the pendulum and cartpole systems. For both systems, the error over the first time steps before the first vertical dashed line is the autoencoder error. Starting from the first vertical dashed line, the error is the prediction error. The red curve
'illustrated in the supplementary material.

---

GT
Ours offline
1kg. length Ours +
Partial updates
Ours + Full updates
Figure 2: Impact of online updates on the quality of the prediction for the cartpole. The first row shows ground truth images. The second row shows predicted frames without updates. The third row shows predicted frames with our model trained with partial updates. The last row shows predicted frames with online updates performed at each new measurement.
cefi Part di speliics Pul LEE000S
Offinn Dettal updelrs
Pull updones
Pastiggh pleo mn0e:
0.44
0.33
0.055
10.0
20
20.0
12.5
15.0
Time (a
Time 60
Figure 3: Average per-pixel RMSE loss over 15s prediction horizon. See text for details. Left: pendulum. Right: cartpole.
corresponds the where the matrix A is computed using fixed number of time steps and never updated. The blue curve corresponds to the case where the matrix is first computed using fixed number of time steps, then updated fixed intervals (every 750 ms for the pendulum, and every 50 ms for the cartpole). The green curve corresponds to the case where the matrix every 50ms. This case corresponds to predicting only step ahead, which is why the error is low and constant over time. For both systems, the prediction error grows with time when the matrix A is updated, which demonstrates the necessity of online updates if one wants to perform accurate long-term prediction. The impact of the online updates on the quality of predicted frames a cartpole can be seen in Fig. The qualitative predictions consistent with the evolution of the reconstruction error over time from Fig. (right).
updated
We compare our future prediction approach the standard method of learning the matrix A parameter of the model as in [17]. We train and evaluate both approaches on two different datasets of pendulums, the first one containing trajectories of the same pendulum (mass I = 0.6m) with different initial conditions, and the second one containing trajectories of different pendulums (mass m 1kg, length 0.3 1 0.8). Figure highlights the fact that computing as the solution of a least-square problem (3) leads to more accurate predictions than learning with fixed matrix A. In the supplementary material, we show that while the predictions are correct qualitatively on the first dataset even for long horizon, they are wrong on the second dataset as not even the first predicted frame is correct. This is expected as single matrix cannot be used to model the dynamics of systems with different physical parameters.
7

---

005
PVP225 2004 RYS2
0.005
10.0
15.2
Figure 4: Average per-pixel RMSE loss 15s prediction horizon. Left: pendulum with length 0.6 m. Right: pendulums with lengths varying from 0.3 to 0.8 m.
3.3 Long-term prediction of forced dynamics
A leamnd
Ours ettine Ours pamsilupcure
Outs
Ours partis updates
0.13
Mectoog pked MMSS
Aet 201r AYAE
0105
20.5
12.5
15.0
20.0
10
2.5
5.0
75
15.0
135
20.5
Teme 100
Time (0)
Figure 5: Average per-pixel RMSE loss over 20s prediction horizon. Left: actuated pendulum. Right: actuated cartpole.
We consider the case of controlled pendulum and cartpole systems. Figure shows shows the prediction error over a horizon of 20s of actuated pendulum and cartpole. The importance of (partial) online updates is demonstrated as without it the prediction error grows quickly over time.
We have compared our approach to the approach presented in (6. where the dynamics matrix A is the result of least-square problem similar to ours, while B is linear layer learned as trainable parameter along with the autoencoder parameters. We adapt their approach our setting in order to (1) include time-delay embeddings because our measurements are images, and (2) use an autoencoder tailored to our datasets, for fairer comparison. Even for offline prediction, our approach outperforms our implementation of [ in the case of the actuated pendulum (Fig. 5 left), which validates the need for learned specific control matrix per trajectory. Having constant B would imply that given sequence of control inputs has the same effect on the state of the system no matter when along applied, presupposing stationary regime of the states of the system, which is not the case for most real physical systems. This is all the more validated with the cartpole example in Fig- (right), for which the approach with static matrix B is unable to make accurate predictions in the 2.5s horizon, because of the chaotic regime of the system. More details on shorter-term prediction horizons of this approach can be found the supplementary material.
the trajectory
3.4 Control from video inputs
We demonstrate the effectiveness of our approach on the control of simulated physical systems directly from visual inputs. We consider here the case of controlling forced system such

---

pendulum. Such system is highly non-linear and does not exhibit any linear state representation. Using our pipeline, we learn a linear-state representation (A, B) which can then be plugged into
classic linear quadratic control settings []] similarly to [1 12 18 6
Let us assume the system is some initial configuration d1 in the measurements space, and our goal is to drive it to a target configuration dy given in the image space at time Te. We are looking for sequence of control inputs (U1,...,, WT that minimizes:
Tc
T Σ u Rue s.t. 2+41 = Azi + Bug and 21 (b)()), (13)
min Σ 2 - _(a-2))QQ22 2/) +
where Q and R are the standard symmetric positive definite matrices associated to the LQR setting.
Figure showw the trajectory obtained for driving the pendulum from the initial positions (red) to a larget configuration (green). As our representation is with delay embeddings with an history of 2 frames, we need constrain the QP problems on two initial conditions. It important to notice that we were not able to run this experiment with a single image history. Indeed, a pendulum and other mechanical systems are subject to gravity, which is a second-order information, which cannot be retrieved from a single image.
]
]
Figure 6: Illustration of pendulum control. Starting from two consecutive configurations (in red). control inputs are applied to the pendulum to force it to be in an inverted position in a horizon of 0.5 Final position is the target configuration (in green).
3.5 Limitations
In all the experiments, we use an embedding of the state of relatively small dimension n, = 8 compared to the configuration space dimension of the studied dynamical systems. Yet, while we manually choose this value, the issue of choosing the correct dimension of linear state space representations remains open. Additionally, all experiments were done simulated and low- dimensional systems without considering noise on the sensory inputs. A next step to our approach would be to include real systems or measurements of real systems of higher dimension (e.g, real video datasets). Finally, our approach motivated by the Koopman operator theory (7]. through the DMD approximation [3]. While this approximation works practice, no theoretical bound of this approximation that we are aware of has been established yet, and we believe that finding one remains an open problem. Some previous work 44
(52 discussed this issue from a spectral point of view by either showing that DMD (or EDMD) modes are good approximation of the true Koopman eigenfunctions for specific problems when these are known analytically, or that they provide a correct parametrization of the system. Again, as far as we know, assessing the accuracy of the DMD/EDMD approximation to the Koopman operator for nonlinear dynamics remains an open problem.
4 Conclusion
We have introduced trainable framework for learning embedding of the state space of physical systems into a latent space where the dynamics are linear, directly from raw sensor inputs (images). Contrary previous works, our computational framework allows learning system state representations

---

which are adapted to online updates, leading to more accurate predictions than offline models. This feature is also essential for online control of complex instrumented systems such as robots, where a compact state representation needs to be updated to account for dynamical changes in the environment. Additionally; our framework exhibits long-term prediction capabilities, an essential feature for planning and control. It is also able to learn complex embeddings for dynamical systems with varying parameters, which is an appealing feature for systems which operate in changing environmental conditions. We have applied it to future state prediction and control from input videos on several challenging dynamical systems. As future work, we plan to extend our approach to operate on a heterogeneous set of sensor imputs and apply it on real robotic systems for solving fine manipulation tasks by directly exploiting vision and tactile feedback, similar to the experimental setting developed in [37.. We plan to extend approach to 3D examples by accounting for multi-view measurements, in a similar spirit to what authors in 5 proposed.
Broader impact. This work is mostly a theoretical contribution to the online learning of complex dynamics from sensory input and its potential positive or negative impacts are similar to the ones of the field. Applications such as the learning of complex dynamics for machines interacting with the world or humans, among them manipulation or locomotion tasks in robotics, may lead to questionable use in surveillance military contexts. Yet, we believe that developing generic approaches to learning complex dynamics with structured and interpretable components is part of more general trend towards more structured learning algorithms. This could results in more explainable and efficient algorithms, a central topic for the ethical and ecological issues of AL.
5
Acknowledgements
We warmly thank Armand Jordana for fruitful discussions. This work was supported in part by the HPC resources from GENCI-IDRIS (Grand 2020-AD01 101 1263RI,,, the Inria/NYU collaboration, the Louis Vuitton/ENS chair on artificial intelligence and the French government under management of Agence Nationale de la Recherche as part of the "Investissements d' a'aveeni" program, reference ANR190P31A0001 (PRAIRIE 31A Institute).
References
[1] D. Liberzon, Calculus of variations and optimal control theory: concise introduction. Prince- ton university press, 2011.
[2] C. E. Garcia, D. M. Prett, and M. Morari, "Model predictive control: Theory and practice- survey." Automatica, vol. 25, no. 3, pp. 335-348, 1989.
[3] P. Schmid, "Dynamic mode decomposition of numerical and experimental data," Journal of Fluid Mechanics, pp. 5-28, 2010.
[4] S. L. Brunton, M. Budišié, E. Kaiser, and J. N. Kutz, "Modern koopman theory for dynamical systems," arXiv preprint arXiv:2 /2086, 2021.
[5] K. Taira, S. L. Brunton, S. T. Dawson, C. W. Rowley, T. Colonius, B. J. McKeon, O. T. Schmidt, S. Gordeyev, V. Theofilis, and L. Ukeiley, "Modal analysis of fluid flows: An overview," Aiaa Journal, vol. 55, no. 12, pp. 4013-4041, 2017.
[6] Morton, A. Jameson, M. J. Kochenderfer, and D. Witherden, "Deep dynamical modeling and control of unsteady fluid flows," in Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada (S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, eds.). pp. 9278-9288, 2018
[7] B. O. Koopman, "Hamiltonian systems and transformation in hilbert space," Proceedings of the National Academy of Sciences, vol. 17, 5, pp. 315-318, 1931.
[8] I. Mezić and A. Banaszuk, "Comparison of systems with complex behavior," " Physica D: Nonlinear Phenomena, vol. 197. no. 1-2, pp. 101-133, 2004.
[9] Mezic, "Spectral properties of dynamical systems, model reduction and decompositions." Nonlinear Dynamics, vol. 41, no. 1, pp. 309-325, 2005.
10

---

[10] S. L. Brunton, B. W. Brunton, J. L. Proctor, and J. N. Kutz, "Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control," PLOS ONE, vol. 11, c0150171, Feb 2016.
[11] I. Abraham, G. de la Torre, and T. Murphhy, "Model-based control using koopman operators," Robotics: Science and Systems XIII, Jul 2017.
[12] H. Arbabi, M. Korda, and I. Mezic, "A data-driven koopman model predictive control framework for nonlinear flows," 2018.
[13] D. Bruder, C. D. Remy, and R. Vasudevan, "Nonlinear system identification of soft robot dynamics using koopman operator theory," 2019.
[14] N. Takeishi, Y. Kawahara, and T. Yairi, "Learning koopman invariant subspaces for dynamic mode decomposition," in Advances in Neural Information Processing Systems (I. Guyon, U. V. Luxburg. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, Curran Associates, Inc., 2017.
[15] O. Azencot, N. Erichson, V. Lin, and M. Mahoney, "Forecasting sequential data using consistent Koopman autoencoders," in Proceedings of the 37th International Conference on Machine Learning (H. D. III and A. Singh, eds.). vol. 119 of Proceedings of Machine Learning Research, pp.
PMLR, 13-18 Jul 2020.
[16] Y. Xiao, X. Xu, and Q. Lin, "Cknet: convolutional neural network based on koopman operator for modeling latent dynamics from pixels," 2021.
[17] B. Lusch, J. N. Kutz, and S. L. Brunton, "Deep learning for universal linear embeddings of nonlinear dynamics," " Nature Communicationss vol. 9, Nov 2018.
[18] Y. Li, H. He, J. Wu, D. Katabi, and A. Torralba, "Learning compositional koopman operators for model-based control," 2020.
[19] M. Watter, J. T. Springenberg, J. Boedecker, and M. Riedmiller, "Embed control: A locally linear latent dynamics model for control from raw images," arXiv preprint 2015.
[20] J. Proctor, S. L. Brunton, and J. N. Kutz, "Dynamic mode decomposition with control," SIAM Journal on Applied Dynamical Systems, vol. 15, no. 1, pp. 142-161, 2016.
[21] S. L. Brunton, B. W. Brunton, J. L. Proctor, and J. N. Kutz, "Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control," PloS one, vol. 11,
2. p. e0150171 2016.
[22] I. Abraham and T. D. Murphey, "Active learning of dynamics for data-driven control using koopman operators," IEEE Transactions on Roboties, vol. 35, no. 5, pp. 1071-1083, 2019.
[23] N. Parikh and S. Boyd, "Proximal algorithms," Found. Trends Optim., vol. 1, p. 127-239, Jan.
2014.
[24] N. Srivastava, E. Mansimov, and R. Salakhudinov, "Unsupervised learning of video represen- tations using Istms," in International conference on machine learning, pp. 843-852, PMLR, 2015.
[25] I. Castrejon, N. Ballas, and A. Courville, "Improved conditional vrans for video prediction," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7608-7617, 2019.
[26] R. Villegas, A. Pathak, H. Kannan, D. Erhan, Q. V. Le, and H. Lee, "High fidelity video prediction with large stochastic recurrent neural networks," arXiv preprint /rxiv:991 1.1.00155 2019.
[27] Xue, Wu, K. Bouman, and W. T. Freeman, "Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks," arXiv preprint arXiv: 1607.22586, 2016.
[28] W. Lotter, G. Kreiman, and D. Cox, "Deep predictive coding networks for video prediction and unsupervised learning," arXiv preprint arXiv:1605.08100 2016.
[29] R. Villegas, J. Yang, Y. Zou, S. Sohn, X. Lin, and H. Lee, "Learning to generate long-term future via hierarchical prediction," in international conference on machine learning, pp. 3560-3569, PMLR, 2017.
[30] E. Denton and Fergus, "Stochastic video generation with a learned prior," in International Conference Machine Learning, pp. 1174-1183, PMLR, 2018.

---

[31] M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell, and S. Levine, "Stochastic variational video prediction," in ICLR, 2018.
[32] D. P. Kingma and M. Welling, "Auto-encoding variational bayes." arXiv preprint
arXiv:1312.6114, 2013.
[33] C. Finn, L. Goodfellow, and S. Levine, "Unsupervised learning for physical interaction through video prediction," in NeurlPS, 2016.
[34] M. Mathieu, C. Couprie, and Y. LeCun, "Deep multi-scale video prediction beyond mean square error," in ICLR, 2016.
[35] C. Vondrick, H. Pirsiavash, and A. Torralba, "Anticipating visual representations from unlabeled video," in CVPR, 2016,
[36] H. Tu, C. W. Rowley, D. M. Luchtenburg, S. L. Brunton, and N. Kutz, "On dynamic mode decomposition: Theory and applications." arXiv preprint arXiv:1312.0041, 2013.
[37] M. A. Lee. Y. Zhu, K. Srinivasan, Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. "Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks," " in 2019 International Conference on Robotics and Automation (ICRA), pp. 8943-8950, IEFE, 2019.
[38] M. Budišić, R. Mohr, and I. Mezić, "Applied koopmanism," Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 22, no. 4, p. 007510. 2012.
[39] J. H. Tu, C. W. Rowley, D. M. Luchtenburg. S. L. Brumton, and J. Nathan Kutz, "On dynamic mode decomposition: Theory and applications," Journal of Computational Dynamics, vol. 1,
no. 2. p. 391-421, 2014.
[40] M. O. Williams, I. G. Kevrekidis, and C. W. Rowley, "A data-driven approximation of the koopman operator: Extending dynamic mode decomposition." Journal of Nonlinear Science, vol. 25, p. 1307-1346, Jun 2015.
[41] R. T. Rockafellar, "AAumeetedd Lagrangians and applications of the proximal point algorithm in convex programming." Mathematics of operations research, vol. no. 2, pp. 97-116, 1976.
[42] S. Klus, P. Koltai, and C. Schütte, "On the numerical approximation of the perron- proron-frobenius and koopman operator," " arXiv preprint arXiv: 1511155225500 1.09977 2015.
[43] M. O. Williams, I. G. Kevrekidis, and C. W. Rowley, "A data approximation of the koopman operator: Extending dynamic mode decomposition," Journal of Nonlinear Science, vol. 25, no. 6, pp. 1307-1346, 2015.
[44] G. E. Hinton and R. S. Zemel, "Autoencoders, minimum description length, and helmholtz free energy." Advances in neural information processing systems, vol. 6, pp. 3-10, 1994.
[45] M. B. Giles, "Collected matrix derivative results for forward and reverse mode algorithmic differentiation." in Advances in Automatic Differentiation, pp. 35-44, Springer, 2008
[46] V. Roulet and Z. Harchaoui, "Differentiable programming à la moreau," arXiv preprint
arXwv2201..15458, 2020.
[47] L. Ljung, System Identification, pp. 163-173. Boston, MA: Birkhäuser Boston, 1998.
[48] W. Li and E. Todorov, "Iterative linear quadratic regulator design for nonlinear biological movement systems."" in ICINCO (I), pp. 222-229, Citeseer, 2004.
[49] J. Carpentier, G. Saurel, G. Buondonno, J. Mirabel, F. Lamiraux, O. Stasse, and N. Mansard, "The pinocchio c++ library fast and flexible implementation of rigid body dynamics algo- rithms and their analytical derivatives," in IEEE International Symposium on System Integrations (SII), 2019.
[50] I. Kalevatykh, "panda3d viewer." http::///ttuuub.oom/lalalarrr//p 2019.
viewer
[51] M. Korda and I. Mezić, "Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control," Automatica, vol. 93, p. 149-160, Jul 2018.
[52] H. Zhang, S. Dawson, C. W. Rowley, E. A. Deem, and L. N. Cattafesta, "Evaluating the accuracy of the dynamic mode decomposition." arXiv preprint arXiv: /7 10.0044445 2017.
[53] Y. Labbe, J. Carpentier, M. Aubry, and J. Sivie, "Cosypose: Consistent multi-view multi-object 6d pose estimation," Proceedings of the European Conference on Computer Vision (ECCV), 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] see supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see supplemental material
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "lKiMJ7bu_YO", "reviewer": "Reviewer_9oAb", "summary": "This work proposed an approach of online learning and control of nonlinear systems relying on the insights from Koopman operator theory, and included empirical examples.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The importance of online nonlinear control problem is clear and should be studied.\nHowever, for this work, the reviewer has several major concerns.\n\nMajor concerns\n1) The overall approach is very similar to embed to control [https://arxiv.org/pdf/1506.07365.pdf].\n   Embed to control primarily considers locally linear model over latent space, but it also mentions that it can be easily applied to globally linear cases.\n   It tackles the problem of controlling agents from raw image inputs and studies the empirical results thoroughly.\n   Therefore, except for the fact that the current work explicitly says it is doing online updates, the reviewer hardly sees any difference from the line of work mentioned above.  (see [https://arxiv.org/pdf/1910.08264.pdf] as well)\n   \n2) For online update aspects; the difficulty of online learning for control problems basically stems from the difficulty of exploration.\n   Without exploration, for example, [https://arxiv.org/pdf/1703.04680.pdf] this work studies convergence of EDMD under certain conditions, and it is straightforward to use this idea to study the trade-off between the amount of data and approximation accuracy, which can then be used for robust control for example.\n   Also, [https://arxiv.org/pdf/1906.05194.pdf] this work already studies active learning of Koopman operator for optimal control problem under certain conditions, which is basically about the online learning.\n   If the current work considers so carefully about the online learning problem either experimentally or theoretically that it gives us more insights than the line of work mentioned above, those should be clearly stated (the reviewer thinks it is not clarified in the current form)\n   \n3) The author(s) claim that this work is for controlling complex dynamical systems; however, as long as the underlying dynamics is pendulum, it would be hard to justify this claim.\n   Also, it is not always possible to write a nonlinear controlled system by  z_t+1 = Az_t + Bu+t even if the space of z is carefully selected.  (B should be dependent on x in general)\n   The advantage of this linear model is the ease of computing optimal control using Riccati equation; however, if this transformation cannot be applied to general nonlinear systems, it is hard to say that this formulation can be applied to complex dynamical systems.  Then, what are the advantages of this approach over existing model-based RL (which includes online algorithm as well)?\n   \nHere are some minor comments\n1) line 251;  what is \"??\" here?  there are some minor typos, please check them again.\n\n-------------\nreview updated after the initial response", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations could be discussed more (for example, the concerns the reviewer mentioned above)\nSocial impact is discussed well.", "review_text": "", "extra_scores": {"ethical_concerns": "The reviewer believes there is no significant ethical issue in this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "enbKA43lq6", "reviewer": "Reviewer_3pLg", "summary": "This paper discusses the problem of controlling a dynamic system based on raw sensory inputs. Their method uses an encoder/decoder to map the sensor information (video frames in this paper) to a learned embedded space. Next, it proposes to build a linear function that evolves the dynamic system’s states in the embedded space. This paper evaluates the performance of the proposed pipeline on three classic control problems: cart-pole, single pendulum, and double pendulum.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the proposed pipeline seems reasonable, the method is evaluated on three classical control problems only. In my opinion, a NeurIPS paper needs more competitive examples.\n- It is a bit too much to call cart-pole/(double) pendulum “complex dynamical systems” in the title/abstract/introduction. Please consider toning down the writing in title/abstract/introduction.\n- The discussion on Eqn. (3) and (5) would be clearer if some context about the dimension of n and m could be given. As far as I understand, Eqn. (3) and (5) are two quite different optimization problems that should be used in different scenarios: if AZ_1 = Z_2 is an under-determined system (e.g., n >> m), then Eqn. (5) is preferred. On the other hand, if AZ_1 = Z_2 is a overdetermined system (n << m), one should stick to Eqn. (3) because the equality condition AZ_1 = Z_2 in Eqn. (5) may not be feasible at all. Which case is this paper faced with?\n- From what I understand, all three examples (cart-pole, pendulum, and double pendulum) deal with 2D dynamics only because their motions are constrained on 2D planes, although they can be rendered as if they are 3D scenes. I would like to see a 3D example (e.g., quadrotor, ant, cheetah, or humanoid robots) in simulation, and more importantly, I am curious to see if the pipeline from video frames to embedded space still works for a 3D task. I would guess this would create some additional challenges due to the ambiguities in understanding 3D scenes from 2D images.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper did a good job of discussing its limitations in its low-dimensional, simulation-only experiments. I agree with these limitations, but instead of leaving them as future work, I would suggest trying to solve them in this submission so that it could be a competitive paper.\n\nI do not see major societal and ethical issues from the proposed method, and I appreciate that the authors explicitly discuss them in the paper.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "94fVaVojtj", "reviewer": "Reviewer_1SAM", "summary": "This paper, inspired by (approximations of) Koopman operator theory, proposes an adaptive algorithm that learns a (controlled) dynamics model online. The learned model consists of an encoder and estimated (and continuously updated) A, B matrices. The method is shown to work well (with low RMSE and good control performance) on several low-level control tasks with images as the observations (hence the encoder).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think that this paper is interested and could be accepted for NeurIPS, however there are some improvements that could be done:\n\n* Discussion of algorithmic limitations. Experiments could be extended to higher DOF control systems to address this issue. Failures/problems with auto-encoder should be mentioned, how/when do these effect the introduced approach? In 3.4 how would the approach work if one were learning and controlling at the same time? How would some of the hyperparameters influence the results if not specified well? e.g. h, \\rho, \n\n* How would the approach work if B(x) also depended on x, with x the image or if B(z), if z is the latent variable? In control theory, the control affine dynamics x_{t+1} = A(x) x_{t} + B(x) u_{t} is very popular.\n\n* How does minimizing (12) work together with the least squares estimation for A, B? Maybe including algorithm (with the full loop) could clarify this.\n\n* Koopman operator theory seems almost tangential to the introduced approach. How does this theory influence/guide the introduced approximate approach? When could the approximations be problematic, does the theory offer any insights?\n\n* Comparisons: are there other control/model learning papers (also those that don't use autoencoders?) that work on images? Could these be included in the comparisons?\n\nMinor comments:\n\n- The title is too general, please change to restrict to the specific method that is introduced (adaptively learning latent linear models from images)\n\n- There are a few mistakes, some hindering the readability, please go over the manuscript carefully.\n\n- Captions are not adequate, the figures need to be self-explanatory. Check also line 251 for missing figure.\n\n===== UPDATE =====\n\nAfter reading the rebuttal and re-reading the paper, I still think that the paper can be accepted. The authors have replied well to most points raised also. However I cannot raise my score more because one of my main concerns, that of the Koopman operator theory not guiding the introduced methodology, was not answered sufficiently. For future work (regardless whether the paper is accepted or not), I would advise the authors to look at when or if Koopman operator theory could guide the design of approximate frameworks. When could features be sufficient for a control task? Can we know if by constraining the features and the dynamics model within some parametric form + dimensionality, we are not sacrificing much (i.e. staying 'close' to the operator P in some metric?)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: From the SAC. This is an instance where the rebuttal and the discussion worked. While the original decision for this paper was to not accept, it is being raised to a recommended accept. The primary reason is the quality of the rebuttal, and the useful technical discussion between authors and reviewers that ensued that seems to have been revealing. To the authors: I trust that you will take all reviewer feedback into account and most importantly, that all of the things in your rebuttal and discussion that were promised will be done in the next version of the paper. 
Decision: Accept (Poster)