Online Learning and Control of Complex Dynamical Systems from Sensory Input

1 Introduction . 1.1 Context and motivation . Providing complex machines such as robots with the capacity of learning effective models of their dynamics from their physical interactions with the world is a key enabling factor for deploying them outside of laboratories and factories . Analytical models of these interactions through state-based representations are traditionally derived from physical principles and used to solve challenging problems in various domains . They have been successfully deployed on heterogeneous systems such as robots or aircrafts/spacecrafts . Through the prism of control theory , these analytical models enable engineers to characterize and study the intrinsic dynamical properties of systems ( contraction , stability , etc . ) , which are essential for system comprehension and for critical applications . In this context , optimal control theory [ 1 ] and its online counterpart known as model predictive control [ 2 ] , exploit these models to plan and control complex dynamical systems ( robots , rockets , etc. ) . Yet , the physical interactions of machines with the world are subject to complex natural phenomena , hardly describable by closed-form mathematical models . By leveraging recent progress in machine learning , data-driven approaches [ 3 , 4 ] have emerged as an effective way of learning advanced models of complex physical phenomena such as turbulence in fluid mechanics [ 3 , 5 , 6 ] . In particular , the Koopman formalism [ 7 ] is an attractive and versatile mathematical framework to learn and analyse complex dynamics . The overall objective of this formalism is to map the intrinsic state space of the system to a typically infinite-dimensional vector space where the system dynamics evolve linearly . However , exhibiting such a map remains challenging , especially in the context of instrumented systems such as robots , which evolve in the world by exploiting their 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . sensory input . The sensory inputs used to build a representation of the internal state of the system and the world may be of various nature : images and force , temperature values or encoder readings , etc . Importantly , to be effective for real-world applications such as the control of robotic systems , learned dynamical representations should also offer online update capacities , to account for new measurements as they come , which is essential to get long-term accurate prediction capacities in changing environments , and update the internal dynamics model accordingly . 1.2 Related work . Learning dynamics from data . Data-driven approaches [ 8 , 9 ] based on the Koopman theory [ 7 ] and its approximation using the Dynamic Mode Decomposition ( DMD ) [ 3 ] have been used in the context of building dynamical models from measurements [ 6 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] . A first group of approaches builds handcrafted functions to map from the measurements space to a latent space where the dynamics are forced to be linear [ 10 , 11 , 12 , 13 ] . Such approaches are suitable when one specific system is being considered but can not generalize to new systems , or to systems with varying physical properties . A second group of approaches learns a mapping from the measurements space to a latent space in the form of parametric functions . These approaches are more flexible , and showed promising results for both prediction [ 14 , 15 , 16 , 17 ] and control [ 6 , 18 , 19 ] . [ 15 ] , [ 16 ] and [ 17 ] learn an approximation of the DMD matrix in the form of a trainable linear layer . While this setting is satisfying when only a single dynamical system is considered , it becomes inappropriate to build more generic models since the same matrix can not be used to approximate the dynamics of different systems . To address this issue , [ 6 , 14 , 18 ] look for an approximation of the DMD matrix as a solution to a least-squares problem associated to a trajectory of a system . This enables identifying a linear model from a given trajectory of a system . In another line of work , [ 19 ] seek to identify a locally linear model of a system ’ s dynamics in a learned representation space , through identifying a matrix to transition from one latent representation to the next one . However , their approach does not generalize to globally linear models . Learning and control of forced dynamics . An extension of the DMD framework to actuated systems for model identification and control purposes has been introduced in [ 20 ] , where the transition from one time step to the next is modeled by two matrices : the previous DMD matrix and a control matrix . Building upon this framework , [ 6 , 16 , 18 ] focused on learning an embedding for actuated systems . [ 6 ] and [ 16 ] parametrize the control matrix as a linear layer that is learned along with the embedding , while authors in [ 18 ] estimate the control matrix similarly to the DMD matrix ( i.e . ; from an input system trajectory representation ) , but assume they have access to the true state of the system , instead of only to measurements . [ 12 , 13 , 21 , 22 ] also follow the same approach , although they do it on hand-crafted functions of either the measurements or the states , which they assume known in the case of [ 22 ] . In this work , we look for both the DMD and control matrices as solutions to a single least-squares problem including state representations and control . However , we assume the true state of the studied systems is unknown , and aim at leveraging sensory information to learn a linear representation of its dynamics . Contrary to the aforementioned approaches , we also introduce a proximal reformulation of the standard DMD approach based on a singular value decomposition ( SVD ) [ 23 ] to avoid introducing any extra hyperparameter of the rank truncation of the SVD that is required in the classical DMD setting . To account for changing dynamics , [ 22 ] estimate initial DMD and control matrices from an initial set of pairs of consecutive states of a given system , and perform active learning by updating their matrices with new executions of the system . In this work , we estimate matrices that are specific to a given trajectory , and update them with new measurements . Future video prediction . The video prediction setting which is the focus of our paper has received a lot of attention in the past few years in computer vision , and various approaches for modeling the time evolution have been proposed . [ 24 , 25 , 26 ] do it through recurrent neural networks . Several approaches focused on predicting the next frame only conditioning on one previous frame [ 27 , 28 , 29 , 30 ] . However , image data does not contain any velocity information . Indeed , if not explicitly modeled , this information can only be obtained when considering at least two consecutive frames , which is the approach we follow in our work . Some approaches [ 30 , 31 ] model the multiplicity of future possible outcomes with variational approaches [ 32 ] however , in this work , we focus on the deterministic property of physical systems and only consider one possible future , like [ 33 , 34 , 35 ] . 1.3 Contributions . We introduce an effective approach to learning a linear state representation directly from raw sensor input ( e.g. , images ) in the form of an instance of a nonlinear latent embedding of a state space and a linear autoregressive model for that embedding [ 17 ] . Unlike classical approaches [ 6 , 14 , 15 , 16 , 17 , 18 , 36 ] , once learned , our model of the system dynamics can be updated online to account for new measurements at very low cost . It also involves several key innovations : ( 1 ) Contrary to classical machine learning approaches to this problem [ 27 , 31 , 37 ] , our approach is firmly grounded in ( applied ) Koopman theory [ 4 , 7 , 38 ] and , as demonstrated in Section 3.4 , it readily generalizes from identifying the dynamics of a system to actually controlling it . ( 2 ) Conversely , contrary to existing algorithms based on Koopman theory and dynamic mode decomposition [ 3 , 39 , 40 ] , we show that online updates of the linear dynamics model can efficiently be implemented in our framework , leading to improved prediction results ( Sections 3.2 and 3.3 ) . ( 3 ) We introduce several technical improvements that make the proposed approach practical , including a block-companion matrix representation of the linear autoregressive model enabling the use of multiple frames to drive video synthesis ( Section 2 ) , and the use of a robust , parameter-free proximal iterative refinement algorithm [ 41 ] for the least-squares estimation of this model which proves crucial in practice in the context of a deep learning library such as PyTorch . Our last main contribution is ( 4 ) an extensive experimental evaluation of the proposed approach on multiple problems from control theory ( Section 3.4 ) . Our experiments include video simulations of complex dynamical systems for which we can provide accurate long-term predictions and that we can effectively control . 2 Online learning of system representations . Proposed approach : We propose to learn the dynamics of a complex physical system from sensory inputs using a linear auto-regressive model for a nonlinear latent embedding of the corresponding state space [ 4 , 6 , 12 ] . Concretely , we consider a discrete-time dynamical system defined by some transition function L : X → X over some ( arbitrary ) state space X . In our setting , L is unknown , but its effect can be in fact ( partially ) observed through measurements ( elements of Rp , video frames in our case ) acquired in successive states along trajectories of the dynamics . Our aim is to learn from such measurements ( 1 ) an implicit embedding ϕ : X → Z of the state space into some latent vector space Z using a parametric encoder Φ : Rp → Z of the corresponding data , and ( 2 ) a linear model A : Z → Z of the dynamics in the latent space so that , for any x in X , ϕ ( Lx ) = Aϕ ( x ) . In this setting , h = Φ ( d ) is the feature ϕ ( x ) associated with the ( unknown ) state x in which d has been measured . There is no reason to assume the dynamics L to be linear ( which may be meaningless anyway since X may not be a vector space , or more generally be endowed with a well-defined algebraic structure ) . However , we know from Koopman theory [ 7 ] that , when Z is taken to be the set of all real ( or vector-valued ) functions over X , there exists a linear map P : Z → Z , called the Perron-Frobenius operator [ 4 , 42 ] such that ϕ ( Lx ) = Pϕ ( x ) for any x in X . Since Z is typically of infinite dimension in this case , P does not normally admit a finite representation in the form of a matrix . In the practical case where the latent feature space Z is by design finite-dimensional , on the other hand , P is not guaranteed to be linear but , given m features z1 , . . . , zm of dimension n , one can approximate its action using an n× n matrix A minimizing the loss : ‖ [ z2 , . . . , zm ] −A [ z1 , . . . , zm−1 ] ‖2F . ( 1 ) This approach is called Dynamic Mode Decomposition ( DMD ) [ 3 ] and , together with its extensions [ 20 , 39 , 43 ] , it has been successfully used to study many dynamical systems . In our setting , the features are given by zi = Φ ( di ) ( i = 1 , . . . , m ) and correspond to m successive measurements , gathered in our case from some video clip . DMD typically operates on raw measurement data instead of a latent representation thereof . Learning the embedding function Φ requires some sort of supervisory information , and we follow the popular encoder/decoder framework [ 44 ] by also estimating the parameters of a decoding function Ψ : Z → Rd such that Φ and Ψ jointly minimize the regularized mean of the reconstruction error ‖d−Ψ◦Φ ( d ) ‖22 over some sample of measurements . The full pipeline is composed of a learnable encoder/decoder module mapping from the measurement space to a latent state space , combined with a linear state representation which is learned from a sequence of embedded measurements ( see Fig . 1 ) . We now detail its different components . Measurement embedding . The goal of the autoencoder is to learn an embedding of the measurements into a latent space representation of the state space . While input measurements may be high-dimensional objects ( e.g . images ) , autoencoders are often capable of learning effective yet compact representations of the corresponding state space . Given a sequence of T measurements [ d1 : T ] , we are looking for two parametric functions : an encoder Φθ and a decoder Ψµ parametrized by θ ∈ Rnθ and µ ∈ Rnµ , following the relation : Φθ ( dt ) = zt and Ψµ ( zt ) = d̂t for all t. ( 2 ) Dynamic mode decomposition solves the least-squares problem associated with ( 1 ) , which can be reformulated as : arg min A 1 2 ‖Z2 −AZ1‖2F where Z1 = [ z1 , . . . , zm−1 ] and Z2 = [ z2 , . . . , zm ] . ( 3 ) A standard approach to solve this problem makes use of the singular value decomposition of the matrix Z1 containing the collection of consecutive measurements , following the relation : A∗ ( z1 , . . . , zm ) = Z2Z + 1 , ( 4 ) where Z+1 is the pseudo-inverse of Z1 . As in practice Z1 is a singular matrix , ones needs to carefully choose a given singular-value threshold , either to damp the solution or to cut the spectrum of Z1 . Both solutions only provide approximate solutions of the original least-squares problem ( 3 ) . Additionally , the SVD is differentiable only when all the singular values are different [ 45 ] , which may not be the case in general and could lead to unstable behavior when performing ( stochastic ) gradient descent throw computational layers which include SVDs . To overcome this issue , we propose to rely on an alternative approach called proximal method of multipliers [ 41 ] . Instead of solving a classical least-squares problem , we suggest solving an equality-constrained quadratic program of the form : min A∈Rnz×nz 1 2 ‖A‖2F s.t . Z2 = AZ1 . ( 5 ) The proximal method of multipliers [ 41 ] augments the Lagrangian function associated to ( 5 ) with a proximal term over the dual variables associated to the constraint Z2 = AZ1 , leading to : Lρ ( A , Λ , Λ − ) = 1 2 ‖A‖2F + T−1∑ t=1 λTi ( zt+1 −Azt ) − ρ 2 ‖λt − λ−t ‖22 , ( 6 ) where ρ is the smoothing parameter over the dual , λt are the multipliers associated to the constraints zt+1 − Azt , and λ−t is an estimate of the multiplier λt . Such a saddle-point reformulation can be iteratively solved through the the Karush-Kuhn-Tucker ( KKT ) system of equations : [ Inz Z1 ZT1 −ρIT−1 ] [ Ak Λk ] = [ 0 Z2 − ρΛk−1 ] , ( 7 ) where Inz corresponds to the identity matrix of dimension nz and Λ k is the stack of multipliers associated to each constraint zt+1 = Azt . ( 7 ) is solved by iterative refinement until convergence to the fixed point solution . Typically , for small values of ρ ( e.g . 10−8 ) , less than a dozen of iterations are needed to converge to the optimal solution and the approch converges for any value of ρ . The system of equations ( 7 ) is always invertible thanks to the lower right block −ρIT−1 and has a condition number close to the one of Z1 . From a computational perspective , such a formulation can be solved efficiently by exploiting the inherent sparsity of the 2x2 block matrix , using a Cholesky decomposition of the KKT matrix . Contrary to SVD , this approach can be easily differentiated by backward propagation of the gradient over the iterative process , which appears as an appealing feature for differentiable programming [ 46 ] in the context of deep learning . Online dynamic mode decomposition for long-term prediction . Once built from a sequence of codes obtained from initial measurements , the DMD matrix A can be used to predict the codes of future states given a known initial code . As classically done in [ 6 , 40 , 17 ] , only the first m measurements are used to estimate A , and future codes are predicted as : ẑm+t = A tzm for t ≥ 1 s.t . At def = A× · · · ×A︸ ︷︷ ︸ t times . ( 8 ) In this framework , the matrix A is never updated , thus never using potential new measurements . For long sequences , this might lead to inaccurate predictions as A remains static and is only estimated from codes of the first m measurements . While this is enough for simple systems ( e.g , a pendulum ) , it becomes a limitation when more complex dynamical systems are considered ( e.g , a cartpole ) . To overcome this issue , we propose to update A when new measurements are acquired , making the learned representation adaptive . We account for this during training by artificially extending the initial sequence used to compute A with additional latent codes of measurements . Re-starting prediction from codes of the new measurement should force the prediction error to diverge less quickly . We refer to the supplementary material for more details . Capturing dynamics effects . Until now , we have considered one-step linear dynamics : each encoding zt only depends on the state zt−1 linearly . However , most of the times , a measurement at a given time is only a partial observation of the state of the system and does not contain enough information to describe the full state of the system . For instance an image does not contain velocity information and at least two consecutive frames are needed to estimate it . To address this issue , we adopt a similar approach as proposed in [ 14 ] and consider a code zt which linearly depends on a history of h previous codes z : ht = [ zt−h+1 , . . . , zt ] , such that : zt+1 = A1zt−h+1 +A2zt−h+2 + · · ·+Ahzt = A : hz : ht , ( 9 ) where Ai ∈ Rnz×nz and A : h = [ A1 , . . . , Ah ] . Like for Eq . ( 1 ) , A : h can be obtained by solving an augmented linear least-squares problem , following the same proximal approach as before . Note that this formulation is equivalent to the one in Eq . ( 1 ) , only augmented states [ zt , . . . , ztm−h+1 ] for 1 ≤ t ≤ m− h+ 1 would be considered , and the transition matrix would have a larger dimension of hn× hn , depicting a block-companion structure of the form : Ã = 0 I 0 . . . 0 0 0 I 0 . . . . . . . . . . . . . . . . . . 0 . . . . . . 0 I A1 A2 . . . . . . Ah s.t . z : ht+1 = Ãz : ht . ( 10 ) Such a scheme is classical when building models of dynamical systems from measurements and we refer to [ 47 ] for a comprehensive view . Learning forced dynamics for control . Let us now consider an actuated system for which we have a sequence of measurements d1 : T . We assume the system was actuated with a sequence of control inputs u1 : T−1 that we have access to . The goal is to learn a representation space of the states of the system , from the measurements [ d1 : T ] and the control inputs [ u1 : T ] such that in this representation space , the evolution is linear on both states and control inputs . Following the extension of the DMD approach to control [ 20 ] , we look for a latent space and matrices A1 , . . . , Ah and B such that we have for all t : zt+1 = A : hz : ht +But . ( 11 ) Unlike [ 6 ] , we do not treat B as a learned parameter . We also perform least-squares regression to find A : h and B following the same approach as before . Once these matrices are identified , they can be exploited within the standard linear quadratic regulator setting [ 1 ] for control . This is possible because we make the assumption that B is independent of z . In the case where this assumption does not hold , one can refer to the iterative linear quadratic regulator setting [ 48 ] . Loss function . To learn the parameters associated with our pipeline , we minimize the empirical risk of the L2 reconstruction loss composed of two main terms : a first term associated to the autoencoder and a second term accounting for the desired linear prediction in the state space . The full loss is given by : Lθ , µ ( { d1 : T } i=1 , ... , N ) = 1 N N∑ i=1 m∑ t=1 ‖dit −Ψµ ( Φθ ( dit ) ) ‖22︸ ︷︷ ︸ Auto-encoder loss + T∑ t=m+1 ‖dit −Ψµ ( At−mi Φθ ( d i m ) ) ‖22︸ ︷︷ ︸ Prediction loss . ( 12 ) To simplify the notations , we do not account for the terms associated to the history ( A : h and z : ht ) . In the online setting , predictions in the latent space following a measurement dj obtained at time j > m are performed as At−jΦθ ( dj ) for t > j .

Summary: This paper, inspired by (approximations of) Koopman operator theory, proposes an adaptive algorithm that learns a (controlled) dynamics model online. The learned model consists of an encoder and estimated (and continuously updated) A, B matrices. The method is shown to work well (with low RMSE and good control performance) on several low-level control tasks with images as the observations (hence the encoder).