Learning System Dynamics from Sensory Input under Optimal Control Principles
Authors: Oumayma Bounou
Jean Ponce
Justin Carpentier
Venue: Submitted to ICLR 2023 (2023)

Abstract: Identifying the underlying dynamics of actuated physical systems from sensory input is of high interest in control, robotics, and engineering in general. In the context of control problems, existing approaches decouple the construction of the feature space where the dynamics identification process occurs from the target control tasks, potentially leading to a mismatch between feature and real state spaces: the systems may not be controllable in feature space, and synthesized controls may not be applicable in the state space. Borrowing from the Koopman formalism, we propose instead to learn an embedding of both the states and con- trols in feature spaces where the dynamics are linear, and to include the target control task in the learning objective in the form of a differentiable and robust optimal control problem. We validate this approach with simulation experiments of systems with non-linear dynamics, demonstrating that the controls obtained in feature space can be used to drive the corresponding physical systems and that the learned model can serve for future state prediction.

Under review as a conference paper at ICLR 2023
LEARNING SYSTEM DYNAMICS FROM SENSORY INPUT UNDER OPTIMAL CONTROL PRINCIPLES
Anonymous authors Paper under double-blind review
ABSTRACT
Identifying the underlying dynamics of actuated physical systems from sensory input is of high interest control, robotics, and engineering general. the context of control problems, existing approaches decouple the construction of the feature space where the dynamics identification process occurs from the target control tasks, potentially leading to mismatch between feature and real state spaces: the systems may not be controllable in feature space, and synthesized controls may not be applicable in the state space. Borrowing from the Koopman formalism, we propose instead to learn an embedding of both the states and con- trols in feature spaces where the dynamics are linear, and include the target control task in the learning objective in the form of a differentiable and robust optimal control problem. We validate this approach with simulation experiments of systems with non- non-linear dynamics, demonstrating that the controls obtained in feature space can be used drive the corresponding physical systems and that the learned model can serve for future state prediction.
INTRODUCTION
The study of dynamical systems is key element in understanding most physical phenomena. Such systems are ruled by ordinary differential equations of state variables that contain enough infor- mation to describe and determine their behavior, and analytical models of these systems tionally derived as solutions of the differential equations in question. However, it is hard to fully model mathematically most real-life phenomena for several reasons: they may have very complex dynamics with complex and constantly changing interactions with the environment, and the state of the physical systems involved may be unknown not fully observable. On the other hand, the physical systems themselves, if not their internal states, can be observed with sensory data provid- ing implicit information about the underlying (and unknown) states. Thus, leveraging measurement data is natural, and is actually done in wide range of approaches which build representations of systems from past measurements in the form of feature spaces (Brunton et al., 201 Arbabi al.) 2018 Bruder et al. 2019: Brunton et al.) [02T). These models are of high practical interest since they enable compact representations compared to the density of measurements (c.g., when measure- ments are images). They also enable lifting the state of the system
tradi-
higher dimensional space where predictive models can be built. However, even when effective, these estimated models and feature spaces remain highly uninterpretable, and using them in solving control problems remains challenging. Linear models on the contrary are easily interpretable, and enable exact and effective control when coupled with LQR solvers. In particular, the Koopman operator theory (Koopman) 1931 has gained a lot of interest recently (Proctor et al. 2016;) Brunton et al. 2016b Abraham al. 2017, Morton al., [0188] Korda & Mezić [018;; Arbabi et al. [2018) Brunton et al. 2021). l). It guarantees the existence of a linear (if typically
dimeniiodimessioaaa) representation of the dy- namies of the observables (vector-valued functions) defined over the state space. Finite dimensional approximations have been proposed, and dynamic mode decomposition (DMD) (Schmid, 2010) is of particular interest in this context. In DMD, an approximation of the Perron-Frobenius operator, adjoint to the Koopman operator, is constructed in the form of matrix that ensures the transition from one observation to the next. Proctor et al.] (2016) first extended the use of DMD to actuated systems and modeled the system dynamics as linear function of the state representation and the
control. Several works have built upon this approximation (Morton et al.] 2018 Li et al.] 22020) and various methods for estimating the corresponding operators have been proposed (Morton al.

---

Under review as a conference paper at ICLR 2023
2018 Xiao et al. 2021: Li et al., 2020). In all these works, these operators are constructed solve a prediction task, assuming the controls are known and, once obtained, they are used in a control task, typically an LQR problem. However, there two main issues with this decoupled approach. First, the learned features may not be adapted to the control task, since they were not trained for it. Thus, the modeled dynamics are not guaranteed to be effective when used as (linear) constraints to minimize a given (quadratic) cost. We believe that including the control problem in the learning process should help learning features that are well suited for both prediction and control. Second, when looking for a couple of matrices that satisfy the desired linearity property in the feature space, it is assumed that the dynamics are linear in the real controls. This is a strong assumption that not necessarily satisfied, and is not justified by Koopman operator theory. In addition, the dimension of the observations representations is usually larger than that of the (unknwon) states, resulting in representation that has potentially more degrees of freedom than the real system. Thus, if the di- mension of the control in feature space remains that of the original system, the system might not be controllable anymore. This is justified by the Kalman criterion for controllability (Kalman (964) which is that the controllability matrix is full rank. This is facilitated by having observation and con- trol feature spaces of similar dimensions. To address this issue, we propose also lift the controls to higher dimensional space in order to avoid rank-deficiency issues in the controllability matrix. To summarize, we learn in this work representation of dynamical systems from measurements where a linear model of their dynamics can be identified and used to predict their behaviour and control them. In this representation space, both states and controls are lifted to higher dimension, which makes the system linearly controllable in this space. We are able to do so by directly includ- ing a control task in the learning objective, which allows us to learn representations that are better suited for control. We include experiments that show the effectiveness of our approach in controlling pendulum and cartpole systems in simulation.
METHOD
2.1
CONTROLLED DYNAMICAL SYSTEMS
Actuated dynamical systems are dynamical systems whose state z(t) E X C R follows a differen- tial equation of the form (t) = f (x((t, u(t)) where f is (non-linear) function, and u(t) a control input in a control space U C IRP. In this work, we study discrete time actuated dynamical systems, ie., systems whose discrete state x, X C R follows an equation of the type
I++1 f( uz),
where ie in U C RP is control vector. In practical settings, the model ft is unknown, and the states 2e are unknown or only partially observable through sensors 9t- In this work, we consider measurements d, = gr (2z), and seek to learn encodings of the unknown states of the system from these measurements.
2.2 APPROACH
We want to learn encodings of both the states and the controls that have three properties: first, the code for the system state at time f, constructed from system measurements at that time i, should contain enough information to capture the behavior of the system at that time. Second, we want the dynamics to be linear in codes space, even though they may be arbitrary in the original state space. This is motivated in part by the Koopman linear representation of arbitrary dynamics for non- actuated systems. Third, we want the system to be controllable in the learned representation space. Because of the first and second properties, the system dynamics are lifted to higher dimensional space. In such a space, the system representation has more degrees of freedom, and might not be controllable with the original controls anymore. In our approach, we propose to also learn an encoding of the controls through a second autoencoder, in order to lift the controls to a higher dimensional space.
Encoding, Let us consider a dynamical system governed by Eq. (I.. We assume that the states ø; and the model g are unknown, and that we only have access to sequence of T measurements de in I C HRCxxXX XW of the system (images in our case). We want to learn the parameters of encoders p I R and 1/> + d such that:
2e p(di), a (ue),, for t 1, I...,T.
(2)
2

---

Under review as a conference paper at ICLR 2023
dd......dp
2 Pp...... 2xd
RM+1D.....
lå...... dz}
Φs y
A, B
$ Y
(it....lyy My-1
lg...., Mml
[aAD 2-11
_a 0-
l2M+D 2..
LQR
Figure I: Method. Measurements de and controls ue are input respectively to the image and control encoders. The first M elements encoded and used estimate dynamics matrices A and B. Codes from M + to T are obtained in two ways: (1) as solutions to the LQR problem of driving the system from its configuration at time M to its configuration at time in T M time steps, and (2) as predictions obtained with the forward linear model associated with the matrices (A, B). All codes 24 and C₂ are then input respectively to the image and control decoders, and image and control reconstructions d and ún are obtained.
The encoders sp, learned in an auto-encoding fashion jointly with their decoder counterparts E= + and V Id ll such that de (2x) and ù, (ce).
Linear model estimation. Based on Koopman operator theory, we want the dynamics to be linear in the representation space. We use an approach similar to the DMD approximation look for dynamics and control matrices A and B. Our goal is to learn representation space where the dynamics are linear, i.e., we want to find matrices and B such that
2++1 Az, Be, for all
(3)
To do so, split the sequence of T measurements in subsequences. The first one, from 1 to M (where M is an integer lower than 7') is used to estimate the matrices A and B, and the second one, from M to T, used verify that the dynamics matrices enable prediction. Formally, A and B are estimated using 21 ....... 2M and C1, , as
(A, B) arg min
244 Pzt Qcal
(P.Q)
To solve this problem, we follow
Li et al.
2020 and Bounou et al. (2021): defining the matrices
Z1 = [2] 1M- Zo = [22 12.... ]. and
Ci ... CM 1/. we solve
(A, B) arg min 12 PZ QCI (P.Q)
(5)
Because the matrices Zi. Zg and C are constructed from learned encodings, they are not guaran- teed to be full-rank, making the linear least-squares problem (5) ill-posed. To avoid unstabilities during training and ensure we always obtain solution, we use the proximal method of multipliers (Rockafellar 1976) and solve iteratively the well well-posed problem
(A++1 B*+1)) arg min
1/ - P7 - QCI2 LIIP A + 9 (P.Q)
B
(6)
until convergence, starting from random initial matrices A and Ba Unlike other regularization and Q, this iterative procedure guaranteed to
schemes such as adding 1.2 regularizations converge to solution to the original problem (5) (Rockafellar) 1976 Parikh & Boyd 2014), and not to solution to a shifted problem as may happen with L2 regularizations.
Future representation prediction. The future code values 2M+1, 2T are obtained using the forward linear model:
2e+1 Az, Ba, for & in M....
3

---

Under review as a conference paper at ICLR 2023
This step ensures that the learned encoders (p; & and the matrices A and B are suited for prediction.
LQR control task. Finally, we want the system to be controllable in codes space. The goal is to drive the system from initial configuration in the encoding space to target in that same space. We formalize this as an LQR problem:
T
(2, 2) = arg min
Σ 2m M
(2 1--7)(((2= - 22) + of Ra + (2r -27) 27)) 27)
(8a)
2,0
s.t. 2141 Azt Bea,
2M = ((dM)
(8b)
27 = (((r))
where Q and Q are symmetric positive semi-definite state cost matrices, and R is a symmetric positive definite control cost matrix. Here, 2M is the initial configuration of the system, and 27 is want the system reach. Both encode measurements of the system in initial and target configurations (e.g., if the system is pendulum and the measurements are images, 2M and zi are encodings of the images of the pendulum the original position and at the desired target position.). The quadratic cost in Eq. (8a) has a tracking term that pushes the variables 2 to be close to the target 27 all along the control trajectory.
the target configuration
and (8) lead to three situations: from 1 to M, the variables 2n are otbained directly by encoding the corresponding measurements di...., das- From M + to T, they are obtained by the forward linear model Ze+1 = Az, + BeN, using encodings of the known controls MM++,..- Ur. Finally, +......., êJ are obtained by solving the LQQR problem of driving the system in the encoding space from EM to the target ê7. There are two types of control features:
Supervision. Equations
control encodings CI,... Car from Eq. (2), and solutions to the LQR problem ĆM+1,... êT from problem (8). In all cases, the codes 2, and should match the original measurements once decoded:
d d,...,, dr +(2)), ((2)))),,,,)) 4 i...... Or (r))....... (cr))
(10)
Training objective. Our model is trained to minimize the empirical risk of the loss:
T-1
Lo((d', u') )i=.....N))
क ((p())1
Σ
$ (Aip(d)) BBb 12
M
Measurements AE loss
Prediction loss
Iu V (ab(()))
Σ
M=++1
Controls AE loss
LQR r000 loss
LQR controls Joss
(11)
where 0 is the set of parameters of the encoders &o and and decoders + and V. The first term uses the auto-encoder measurements loss and ensures that the encoder learns a representation with enough information about the system given timestep reconstruct the observations. The second one uses prediction loss which ensures that the estimated matrices (A, B) enable the prediction of future codes in the latent space (that are then decoded to the measurement space). The third term uses the auto-encoder controls loss and ensures that the control encoding can be mapped back its original space. The last two terms are associated with the control task: they ensure that the decodings of the optimal solutions to the LQR match the original optimal measurements and control trajectories. Since the different terms in the loss are not in the same scale nor the same unit, we add scaling coefficients before each one of them for training.
LQR formulation with delayed coordinates. In practice, raw measurement data may not contain sufficient information for future prediction. It is the case for example when we observe a single image of an oscillating pendulum at a given time step and we can not tell from this image whether the pendulum is going up down. Instead, encodings of at least two consecutive measurements are needed to predict the future code. This case is easily handled using the classical trick of using augmented feature representations associated with multiple frames, similarly to what is done in
4

---

Under review as a conference paper at ICLR 2023
Takeishi al. (2017). Let define the augmented vector 2 2 h+1 ...27 2. where the number of consecutive measurements we choose to consider. The new dynamics equation
becomes 2t+11 = A =f Be,, where A [A] An]. To be in the formalism of Eq.
8b). we define the augmented matrices Å and B as
0
1
0
0
0
1
0
0
=
.... 0
(12)
A1
Ah
B
where is of size (hn x hn) and B is of size (hn d). We also define augmented initial and terminal
constraints =1th = M-
*M and ž = [
2 both of size In.
With these notations, the new dynamics equation becomes:
E+1 AZ, + Beg
(13)
2.3 RELATED WORK
Data-driven approaches based on the Koopman operator theory
Koopman, 1931 have gained in
popularity in the context of building dynamical models directly from mcasurements (Morton et al.,
2018 Brunton et al. 2016a Abraham al., 2017 Arbabi et al.,
2018
Bruder et al. 2019 Takeishi
et al.
2017| Azencot et al. 2020 Xiao et al., 2021 Lusch et al.
2018 Li et al.) 2020). Borrowing
from the Perron-Frobenius operator, adjoint to the Koopman operator (Brunton al. 2021] Klus al.
2015), these approaches are grounded in the formalism enabled by the dynamic mode decom-
position (DMD) [Schmid] 2010). this context, several approaches have been proposed to construct representation space with linear dynamics. Such space can either be handcrafted (Brunton al.
2016a
Abraham et al.] 2017 Arbabi et al.] [0188] Bruder et al.] 2019) or learned (Takeishi et al,
2017 Lusch et al. 2018 8 Morton al. 2018 Azencot al. 2020 Li al. 2020] Xiao et al.,
2021 both cases, the representation space is built in the form of a mapping from the measure-
ments space latent space. The DMD framework has also been extended to actuated systems for model identification and control purposes in Proctor et al. (2016). Based on this extension, Brunton
al. (2016b); Morton al. (2018); Arbabi et al. (2018); Bruder al. (2019); Li et al. (2020) and Bounou et al. (2021) seek to identify dynamics and control matrices, either using learned represen-
tation spaces or hand-crafted ones, and then use the obtained matrices solve control problems. In another line of work, Watter et al. (215)) seek to identify time-dependent locally linear model of a system's i dynamies in a learned embedding space. However, their approach does not generalize to globally linear models. In all these approaches, the control task is decoupled from the representation space construction, potentially leading to a mismatch between the representation space and the real state space. To address this issue, propose to include control task in the learning framework in order to learn a dynamical system representation which is forced to be controllable. In particular, we leverage differentiable optimization techniques from Amos al. (2018) and Roulet & Harchaoui 2022) in order to include a differentiable LQR controller in the learning framework.
3 RESULTS
Single pendulum dataset
Multiple pendulume uns dataset
a
= 10"
a = 15"
a =
(
10
a = 15
T. 30
95
100
100
55
84
95
T. 35
38
80
92
21
40
61
T. 40
17
36
15
26
T. 45
3
6
12
6
12
Table 1: Success rates of pendulum control given various thresholds and different control horizons T., An experiment is considered success when the pole reaches its target position within range of +-a degrees. Success rates are expressed in % and were evaluated on 1000 control tasks.

---

Under review as a conference paper at ICLR 2023
Control sequence for the pendulum
Distance to target for the pendulum
140
for control-tesk
lgr, cortroooto
Control in N.m
Angular distance
100
-2
80
gt
ky_ contral-task
20
bp
0.0
0.2
0.4
0.6 0.0
0.2
0.4
0.6
Time in
Time in
Figure 2: LQR problem of taking a pendulum from an initial position to a target position in 0.6 s (or 30 time steps.). Left: control trajectories in N.m. Right: distance target angular position in degrees.
GT
Model trained for prediction
Model trained
prediction
and control
M
Figure 3: Pendulum. Video rendering of the trajectory of the controlled system. In the first row, the system controlled with the ground truth controls. In the second row, it controlled with control solutions to the LQR problem using the model trained for prediction only. In the last row, it is controlled using control solutions the LQR problem using the model trained for prediction and control. All sequences are subsampled (we show frame over 5).
3.1 DATASETS
We generate datasets of optimal trajectories for pendulum and cartpole. For each system, specify initial and terminal conditions. We use Pinocchio (Carpentier et al. 2015-2021| to simu- late geometric models of the systems, and Crocoddyl (Mastalli et al.] 2020) to solve the optimal control problem taking the systems from the initial condition to the terminal condition. For the pendulum, the initial and terminal conditions are initial and target angular positions and velocities 4. and target positions are sampled uniformly between =F and 4. We choose such distributions of initial and target positions ensure the optimal control trajectories we obtain are not trivial. The cartpole we consider in our experiments is a cart translating along a horizontal axis. A pole is attached to it, and it can rotate over an axis orthogonal to both itself and the cart axis. For the cartpole, the initial and terminal configurations are initial and target Cartesian positions and velocities for the tip of the pole. For all trajectories, the target position the cartpole one where the pole is up and vertical. Both datasets contain different trajectories of pendulums and cartpoles with different physical parameters. For the pendulums, the pole length is sampled uniformly between 0.5 m and 0.8 m, and the mass is sampled uniformly between 0.5 kg and 2 kg. For the cartpoles, the carts cylinders of radius 0.1 and of
3m for the pole. Initial positions are sampled uniformly between I and ix

---

Under review as a conference paper at ICLR 2023
100
Distance to (deg)
Distance to (deg)
077
hortzon
horizon
Figure 4: Pendulum. Average distance to target as function of the control horizon. Left. Model trained on multiple trajectories of single pendulum. Right. Model trained on multiple trajectories of pendulums with various masses and pole lengths.
Pola cislarce to goal (ceg)
Cart distance to geal (m)
120
2.0
100
1.5
1.5
20
0.3
0.6
0.7 Time horizon (s)
6.8
0.0
0.7
3.8
Time horizon (s)
Figure 5: Cartpole. Average distance target function of the control horizon. Left. Pole
distance to target. Right. Cart distance to target.
length and mass sampled uniformly respectively between 0.3 m and 0.7 m and kg and kg. The pole length is fixed at 0.7 m, and its mass is sampled uniformly between 0.5 kg and 2 kg. Then, each trajectory is generated from different initial and target conditions. We render both generated systems using pandašd-viewer (Kalevatykh) 2019) into black and white videos of frames of 64 x 64 pixels. We train our models on 4000 videos 10 seconds (200 frames), and test them on 1000 videos the same duration. Systems different in both sets: their phy physical parameters are diverse, and initial and target configurations are random. Thus trajectories in both sets are different.
3.2
ARCHITECTURE
The image and control autoencoders are the only functions with trainable parameters in our proposed model (the dynamics matrices A and B are latent variables). The image autoencoder is made of 6 convolutional blocks encoder, where each convolutional block except for the last one is composed by 3 convolution layer, max-pooling layer, batch normalization layer and ReLU layer. The last block does not have batch normalization layer nor RELU layer. The decoder is symmetric the encoder (with transposed convolutional layers instead of convolutional layers). The encoder of the control autoencoder is made of a dense layer followed by tangent hyperbolic (tanh) non- linearity. The decoder is also symmetric to the encoder. We learn one unique autoencoder for all the pendulum systems, and another unique autoencoder for all the cartpole systems. We the differentiable LQR solver introduced in im|Amos et al. (2018), to which we add proximal regularization terms to ensure stability during training, similarly to Bounou al. (202I) for the estimation stage. The LQR cost matrices Q and R are set to the identity matrix. We train our models during 200 epochs single Tesla V V100 GPU.
3.3 LEARNING DYNAMICS WITH CONTROL TASK IN THE OBJECTIVE
Given measurements of system performing an optimal trajectory and the associated optimal con- trols, we learn a representation space where the system dynamics are forced be linear, and exploit this linearity to solve LQR control problems in this space. Solutions of the LQR problem are then used to control the original system. Since the trajectories are optimal the measurements space (by construction during the datasets generation), also look for optimal control trajectories in the

---

Under review as a conference paper at ICLR 2023
representation space. In our case, the measurements are videos, and the overall task is supervised in the following way: given sequences length T and a time index M smaller than T, codes 2 and Ct from 1 M are encodings of the measurements d, and the controls u, from 1 M: 2, (pd,) and Cr = (b4)). We then compute the codes 2ạ from M + to in two ways. First, we use the forward linear model 2++1 Az, (T)( using the known ue that encoded, and starting from = : This is to ensure the encoders p; & and the matrices (A, B) are suited for prediction. Next, solve the LQR problem associated with the task of taking the system from M to (in T M) time steps. This is to ensure the encoders p. t and the matrices (A, B) are suited for con- trol. The definition of codes from M + to T is consistent in both ways: first, the codes verify the dynamics constraints by construction; second, they are solution of the LQR problem and are thus feasible (i.e., they also verify the dynamics constraints).
3.3.1 CONTROL RESULTS
Contri
posi trajectory
position
Controlin
-2
Anr
120
Distmce
-12
1.4
Time
06
03
0.2
04
06
Figure 6: LQR problem of taking cartpole system from an initial position to target position in 0.6s (or 30 time steps.). Left: control trajectories. Middle: Angular position of the pole over time. Right: Cart distance to its target position over time.
i
GT
Model trained for prediction only
T
Model trained for prediction and control
T
M
Figure 7: Cartpole. Video rendering the trajectory of the controlled system. In the first row, the system is controlled with the ground truth controls. In the second row, it is controlled with control solutions to the LQR problem using the model trained for prediction only. In the last row, it is controlled using control solutions the LQR problem using the model trained for prediction and control. All sequences are subsampled (we show frame over 5).
Figure 2 (respectively Figure shows control trajectories (left) and the resulting controlled system trajectories (right). The control trajectories are obtained by solving the LQR problem taking a pendulum (respectively a cartpole) from initial position (f M) to target one (t 7") in 30 time steps (0.6 s). The initial and target positions are directly specified in the form of images, enabling image-based control. The ground truth controls (in green solid lines) correspond to the optimal controls in the original space. The LQR controls (in dashed-lines, both blue and red) are obtained by solving the LQR control problem of Eq. (8) in the encoding space, then decoding the
8

---

Under review as a conference paper at ICLR 2023
obtained sequence back to the measurements space with the learned controls decoder L. The blue dashed-lines correspond to controls obtained with model that is trained for both future prediction while the red dashed-lines correspond to controls obtained with model that is trained for future prediction only Igr-no-control-task We see that in the case where the model is trained for both prediction and control (lgr-control-task), the decoded controls are very close to the ground truth ones, In addition, these decoded controls are effective in driving the systems to their target position. This is not the case anymore with the model trained for prediction only, for which the decoded controls are very different from the ground truth ones, and consequently fail in driving the systems to their target position. qualitative example of the effectiveness of including the control task in the learning framework can be seen in Fig. 3 and it is all the more noticeable in the case of the cartpole in Fig. 7 which not surprising since its dynamics are more chaotic than the pendulum's. s. Figure 4 shows the average over 1000 samples of the distance to the target angular position (in degrees) as a function of the control horizon (in s) for the pendulum. In the left plot, the model is trained on trajectories of one single pendulum, whereas in the right plot, it is trained on trajectories of pendulums with various physical parameters. In both cases, including the control task in the learning framework (Iqr-control-task in red) leads significant improvement of the control performance, We also report success rates of the models trained with a control task in table The model trained (and evaluated) on a dataset with trajectories of a single pendulum has higher success rates than the model trained (and evaluated) on a dataset with trajectories from multiple pendulums, but the latter still achieves satisfactory results, especially on short control horizons. Figure 5 shows the average over 1000 samples of the distance to target pole and cart positions for the cartpole. We show the average distance of the pole angle in the left plot, and the average distance of the cart position in the right plot. The appraoch trained without control taks Ilg--no-control-tast completely fails when applied to the cartpole, especially on longer control horizons.
and control (Igr-control-taske
4 CONCLUSION
In this work, we introduce approach to jointly learn and control the dynamics of nonlinear dynamical system from raw observations directly. We notably leverage the Koopman formalism to leam an encoding
space where the system dynamics are linear. To enforce the controllabilty of the system in this learned representation space, we include a control task in our learning framework to ensure the learned representation is adapted control. Experimentally, we show that this joint strategy is effective in controlling complex dynamical systems such as pendulums and cartpoles directly from raw images. In particular, we show that separating the dynamics learning process from the control task leads to infeasible solutions in practice, which are avoided with the combined strategy. In future work, we plan to apply our approach to more complex dynamical systems such as robotic arms to control their movement from raw visual inputs directly, using images to specify target objectives. In addition, we plan to leverage self-supervised approaches to avoid using optimal trajectories examples during training.
REFERENCES
Ian Abraham, Gerardo de la Torre, and Todd Murphey. Model-based control using koopman op- erators. Robotics: Science and Systems XIII, Jul 2017. doi: 10.116077rss22017.xi.... URL
http: / wwt.coi..oio www.oo..................
2017 XIII. 052.
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J 7ico Kolter. Differentiable MPC for end-to-end planning and control. Advances in neural information processing systems, 31, 2018.
Hassan Arbabi, Milan Korda, and Igor Mezić. A data-driven koopman model predictive control framework for nonlinear partial differential equations. In 2018 IEEE Conference on Decision and Control (CDC), pp. 6409 6409-6414, IEEE, 2018.
Omri Azencot, N. Benjamin Erichson, Vanessa Lin, and Michael Mahoney, Forecasting sequen- tial data using consistent koopman autoencoders. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pp. 475 485. PMLR, 13-18 Jul 2020. URL http: 7/proceed i ngs. . mll wwl...pras/vvll / http://1111///eect2220a html

---

Under review as a conference paper at ICLR 2023
Oumayma Bounou, Jean Ponce, and Justin Carpentier, Online learning and control of complex dynamical systems from sensory input. Advances in Neural Information Processing Systems, 34, 2021.
Daniel Bruder, David Remy, and Ram Vasudevan. Nonlinear system identification of soft robot dynamics using koopman operator theory. In 2019 International Conference Robotics and Automation (ICRA), pp. 6244-6250. IEEE, 2019.
Steven L. Brunton, Bingni W. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PLOS ONE, 1102): Feb 2016a. ISSN 1932-6203, doi: h11.7///oomall..on.0.50000
URL
doi http:///a...d..rrg1111117777/ journna pone 0150171
Steven I. Brunton, Bingni W Brunton, Joshua I. Proctor, and Nathan Kutz. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PloS one, 11(2):0150171, 2016b.
Steven I. Brunton, Marko Budišić, Eurika Kaiser, and Nathan Kutz. Modern koopman theory for dynamical systems. arXiv preprint arXiv:2 /2086, 2021.
Justin Carpentier, Florian Valenza, Nicolas Mansard, et al. Pinocchio: fast forward and inverse dynamics for poly-articulated systems. https://saattnaaaaaa.ggghhbohoop 2015-2021.
Kalevatykh, panda3d viewer. http://ss.ggggaooooooo////aaaaaa 2019.
Rudolf Emil Kalman. When is linear control system optimal? Journal of Basic Engineering, 1964.
Stefan Klus, Péter Koltai, and Christof Schütte. On the numerical approximation of the perron- frobenius and koopman operator. arXiv preprint a1Xiv:151..05597,, 2015.
B. O. Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 17(5):11-318, 1931. ISSN 0027-8424. doi: 10.1073/pnas.77.5. 315. URL htthst::: //ww: www. pnas www.://///oooeennooooooo
Milan Korda and Igor Mezić. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. Aulomatica, 93:149-160, Jul 2018. ISSN 0005- doi:
a0toaiaato 8.03.046. URL
doi 10. 10...0166/ /0tp://0.......tttttttttaaaaaaaa a.w.mmataaattttaicaaaaaaa
2018. 03.8.03.000 046
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koop- man operators for model-based control. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
attps://openzett wwt.eeewwweeeeeee..em......eee.e forum? ?dd -d--- H1 dzA4tPr
Bethany Lusch, J. Nathan Kutz, and Steven I.. Brunton. Deep learning for universal lin- ear embeddings of nonlinear dynamics. Nature Communications, 9(1), Nov 2018. ISSN 2041-1723. doi: 10. 10.1038/411467-01--0722 s41467 -18-0018-1100000 07210-0
doi..dii..oo 1ttp://xx....1...11100.01.11....
Carlos Mastalli, Rohan Budhiraja, Wolfgang Merkt, Guilhem Saurel, Bilal Hammoud, Maximilien Naveau, Justin Carpentier, Ludovic Righetti, Sethu Vijayakumar, and Nicolas Mansard. Crocod- dyl: An Efficient and Versatile Framework for Multi-Contact Optimal Control. In IEEE Interna- tional Conference on Robotics and Automation (ICRA), 2020.
Jeremy Morton, Antony Jameson, Mykel J. Kochenderfer, and Freddie D. Witherden. Deep dynamical modeling and control of unsteady fluid flows. Samy Bengio, Hanna M. Wal- lach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Anmual Conference on Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9278 9288, 2018. URL / nwwrips......................... 220ca0094400233555577ee633333333
10

---

Under review as a conference paper at ICLR 2023
Neal Parikh and Stephen Boyd. Proximal algorithms. Found. Trends Optim., 1(3):127-23)) January 2014. ISSN 2167-3888.
doi: 10.1561/2400000003.
URL
2400000003
Joshua L Proctor, Steven L Brunton, and Nathan Kutz. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems, 15(1):142-161, 2016.
R Tyrrell Rockafellar. Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Mathematics of operations research, 1(2):97-116, 1976.
Vincent Roulet and Zaid Harchaoui. Differentiable programming à la moreau. In ICASSP 2022- 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 1498-3502 IEEE, 2022.
Peter Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics, pp. 5-28, 2010. doi: 10.1017/s00221 12010001217.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httppp:::ppooceeaaiiiiio
nwwrips...... cc/ /t0pr//201//EElee/3333333333 http:///....32222..0566666666282
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL ht https: http::/esss//....c..g...g.ss www.iiinnnn.n......ieee.m.....ee www.nnenris..ooo
Yongqian Xiao, Xin Xu, and Qianli Lin. Cknet: A convolutional neural network based on koopman operator for modeling latent dynamics from pixels. CoRR, abs/2102.10055, 2021. URL https:
//axxiv.cmm org. abs 2202. 100.0...0003200 10205


Reviews:
[{"review_id": "t8Y4EFfbmB", "reviewer": "Reviewer_kRhs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method of learning system dynamics from raw sensory input based on Koopman operator theory. Both state and control variables are embedded into a latent space with an encoder-decoder structure. The system dynamics in latent space are forced to be linear. Apart from loss terms measuring the encoding and prediction capabilities, the authors add two losses measuring how well the latent dynamics are controllable using an LQR controller. This is novel compared to prior work.\nThe approach is validated on two simulated systems: pendulum and cartpole. The authors compare their proposed method to a baseline without the additional control-related loss terms and show qualitatively that their method can control both environments, whereas the baseline fails.", "summary_of_the_review": "Due to the weak experimental support and my concerns regarding the chosen control approach I think the paper should be rejected. However, I welcome the authors add additional quantitative evaluations to support their claims.", "strength_and_weaknesses": "Strength:\n- The paper is clear, conscise and easy to understand.\n- The authors propose two novel changes to koopman-based learning of system dynamics: First, they couple the prediction and control task and learn a representation that has to satisfy both. Second, the authors remove the requirement of the original controls being linear in latent space. These are valuable contributions.\n\nWeaknesses:\n1.  The experimental validation seems incomplete and purely qualitative. The authors do not compare against any prior method. Even though these prior methods do not train on the control task directly, they also test the control task directly or learn an actor-critic policy on top of the learned dynamics. DCKNet for example also tests the cartpole. I am missing a quantitative comparison to prior approaches.\nThe authors also only show a simple successful control trajectory. I would like to see a success rate when randomizing the initial condition of the system. \n2. The chosen control strategy, namely LQR, seems problematic for the control problems tested against. Both the pendulum and cartpole are typically considered difficult due to their inherent underactuation. In both cases the pole cannot simply be put into the upright configuration without performing a complex swingup motion. To the best of my knowledge LQR should be incapable of solving such a control problem. In the two control trajectories the authors show in their paper the pole directly moves to the upright configuration.\nTherefore, I would like the authors to clarify if the presented pendulum is underactuated. The shown control trajectory for the cartpole starts in an initial configuration which seems to have enough energy such that a direct swingup is possible. Therefore I would like to see the experiment starting from randomized initial conditions with a success rate.\n3. The approach the authors propose requires \"expert trajectories\" because the representation is learned such that the LQR controller in latent space mimics an optimal controller. For more complex systems gathering \"expert trajectories\" is difficult. Could the authors comment on how this impacts the scalability of their approach to more complex environments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bxqdGB5nm1R", "reviewer": "Reviewer_1Vdd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "this paper suggests to simultaneously learn state representation from images and to fit the LQR solution in this state representation (and linear latent dynamics) to the externally provided optimal control signal.  ", "summary_of_the_review": "The concerns in \"Strength And Weaknesses\" prevent me from recommending this paper for publication in ICLR.\n", "strength_and_weaknesses": "1. I do not understand the rational of this work. If you have optimal control, u, you have a solution. Why to calculate it again via LQR?\n\n2. I do not see any connection to \"the Koopman formalism\" except for the fact that the latent dynamics is represented by matrices (linear dynamics), and the Koopman operator is a linear operator. \n\n3. What is the original objective? Eq.11 is a sample based approximation of the original objective. \n\n4.  Eq.11 combines terms with different units and scales. Are the measurement loss and control loss \non the same scale? The later is in units of actions. The former is in different unites. \n\n5. Directly relevant works (e.g,. Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, Martin Riedmiller) are not mentioned.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Gw9edsl6fO4", "reviewer": "Reviewer_72mB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a latent dynamics learning (or system identification) scenario from very high dimensional observations (images). Inspired by the Koopman operator theory, the authors try to find a latent space where the dynamics is linear. and use this dynamics to control two simple dynamical systems (pendulum toss-up and cartpole) with LQR. Unlike other Koopman-theory inspired approaches, the authors include the (LQR) control problem inside the training scenario and show examples where not including results in failure to control the system successfully.", "summary_of_the_review": "In general I believe this paper could be accepted, however the experimental evidence is not extensive. I would be willing to increase my score if the authors provide more discussions and quantify their approach more.", "strength_and_weaknesses": "Strengths:\n\n* The paper is well written and motivated, and the presented approach makes sense. The authors show some successful examples where their contribution leads to successful control,\n\nWeaknesses:\n\n* The paper however suffers from lack of extensive experimental support. The authors only show sample sets of images and a few plots where the method leads to an improvement. The effect on the overall test data (1000 videos of 10 seconds each on both the pendulum and the cartpole systems) is not mentioned, nor is it quantified (have I missed it???)\n\n* Some comparative discussion is missing. How much of the improvement is due to the added two cost functions coming from LQR, and how much of it is due to the choice of the network and other details? Didn't the previous Koopman-theory-inspired approaches train autoencoders that led to successful control? As pendulum is the simplest nonlinear system I assume there were successful pendulum toss-ups reported previously. What was the difference there? \n\n* Ablation studies are missing. What is the effect of the the networks chosen, and is it difficult to train? I assume looking at eq. (11) that training with it successful controllers must be quite difficult and prone to issues (local minima, poor performance, nonconvergence etc.)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject