Patent Description:
In many real-life applications, it is desirable to learn a model of the dynamics of a physical system. This may be especially desirable when the physical system can only be observed as a highly unstructured environment, e.g., as frames from a video camera instead of physically relevant quantities as angles, angular velocities etc. In such examples, neural networks are a promising approach to learn the transformation from observation space to the latent, physically meaningful low-dimensional space of the system dynamics of the physical system, and may also be referred to as machine learned dynamics models.

An example of such a real-life application is the manufacturing of electronic components, where two parts may need to be connected by first bonding a copper wire to one component, then bending the wire into place and, finally, bonding the copper wire to the second component. The bending of the wire may be performed by an arm of a robot. To make this procedure adaptive to the individual components, the bending process cannot be hardcoded but is preferably learned based on sensor input. Typically, sensors that measure the physically relevant low-dimensional state of the system (e.g., angles, angular velocities, joint positions. ) cannot be added to the production system if it is already in place, as they must be built into the machine itself. In contrast, it may be rather straightforward to add an external sensor, as e.g. a camera system, to the manufacturing station. This video data this camera system may be used to learn the bending process of the copper wire.

Disadvantageously, it is very hard to learn the dynamics of a robotic arm from video data using a neural network, since the characteristic long-term behavior of the robotic arm, e.g., stability at certain equilibrium points, can typically not be built automatically into a neural network. The same applies to the modeling of the dynamics of other types of physical systems, including but not limited to robots, vehicles, buildings, domestic appliances, power tools, manufacturing machines, etc., from observed states of the physical system.

Namely, when using a generic neural network to model the dynamics of a physical system, little can be guaranteed about the behavior of the learned neural network. For example, it is difficult to determine the stability properties of a learned neural network. This has led to the need for enormous data sets for training such neural networks in order to ensure that the learned dynamics of the neural network have correct long-term behavior.

While some recent work [<NUM>], [<NUM>], [<NUM>], has begun to consider stability properties of neural networks, it has typically done so by "softly" enforcing stability as an additional loss term on the training data. Consequently, such approaches can say little about the stability of the learned neural network for unseen states, referring to states of the physical system which are not 'seen' during training. Disadvantageously, such approaches are not optimally suited for real-life applications in which such unseen states may occur.

<NPL>, disclose a design of model predictive control (MPC) systems for nonlinear processes that utilize an ensemble of recurrent neural network (RNN) models to predict nonlinear dynamics.

It would be desirable to train a dynamics model to learn the dynamics of a physical system, for example the forward dynamics of a robotic arm, in a manner which ensures a globally stable modeling of the dynamics of the physical system.

In accordance with a first aspect of the invention, a machine learning system and corresponding computer-implemented method are provided for training a dynamics model to learn dynamics of a physical system, as defined by claim <NUM> and <NUM>, respectively. In accordance with a further aspect of the invention, a computer-implemented method and a corresponding control or monitoring system are provided for controlling or monitoring a physical system using a machine learned modelling of dynamics of the physical system, as defined by claim <NUM> and <NUM>, respectively. In accordance with a further aspect of the invention, as defined by claim <NUM>, a computer-readable medium is provided comprising data representing instructions arranged to cause a processor to perform the computer-implemented method of claim <NUM> or <NUM>.

The state of the physical system may be expressed in various ways. For example, in the above example of a robotic arm, the state of the robotic arm may be expressed as angles, angular velocities, joint positions, etc., or in general may be represented by known physical quantities. The training data may contain direct observations of such known physical quantities. In other examples, the state of the physical system may be observed in a high-dimensional observation space, e.g., as video data, which may be transformed to a latent, physically meaningful low-dimensional space using a transformation. In this case, the states of the physical system and thereby the training data may be represented as latent samples in the obtained low-dimensional space, and may, but do not need to, correspond to known physical quantities. It is known to learn such transformation by machine learning, e.g., using Variational Auto Encoders (VAEs). In some examples, the transformation may be learned by the dynamics model itself in that the training data may comprise indirect observations of physically meaningful quantities, e.g., in a high-dimensional observation space.

It is noted that, in embodiments which are not encompassed by the wording of the claims, the dynamics of a physical system may in many cases also be learned from a state of the environment of the physical system, instead or in addition to learning the dynamics from the state of the physical system itself. For example, the steering dynamics of an autonomous vehicle may be learned from a video camera mounted in the autonomous vehicle which is directed towards the road ahead of the vehicle.

To train the dynamics model, training data is accessed which defines a time-series of states x(t) of the physical system. It is known per se to train a dynamics model to learn the dynamics of a physical system using such training data. Namely, the training data provides time-sequential pairs of states, with each pair being formed by a current state and a future state of the physical system, with the former being an input and the latter being a target output of the dynamics model in the model's training.

The above measures ensure global stability of the learned dynamics model f(x(t)) by way of a Lyapunov function V(x(t)). Namely, the stability of any nonlinear system can be shown via the existence of a Lyapunov function, which is a particular type of function that connects a descent in its function values along trajectories of the dynamics f with the different types of stability. A dynamics system is globally stable if and only if there exists a continuously differentiable positive definite function V (i.e., V(<NUM>) = <NUM>, V(x) > <NUM> ∀x ≠ <NUM>) that decreases along the trajectories of f, i.e., <MAT> ∇V(x(t))Tf(x(t)) < <NUM>. In a time-discrete variant of the above, the initial state x<NUM> and f(x) may together define a state evolution x(t), t ∈ [<NUM>; T] or x<NUM>, x<NUM>,. , xT in discrete-time. This state evolution may represent a trajectory of f(x), and decreasing along all trajectories of f may mean that for any x<NUM> and any T, it holds that: VV(x(t)) < <NUM> or V(xt+<NUM>) < V(xt).

Given a learned dynamics model, the Lyapunov function may be learned as well, e.g., using a neural network. However, instead of learning a dynamics model and attempting to separately verify its stability via a (learned) Lyapunov function, the above measures jointly learn a dynamics model and Lyapunov function, where the dynamics is inherently constrained to be stable everywhere in the state space according to the Lyapunov function. Accordingly, the learned dynamics model is stable even for states which are not seen during training. Compared to known approaches for learning dynamics models, the above measures provide a learned dynamics model which is inherently globally stable, instead of being merely incentivized during the training to be globally stable, e.g., via a loss function. The above measures provide a learned dynamics model which is highly suitable for real-life applications in which a physical system may assume a state which was unseen during training as the learned dynamics model is inherently guaranteed to be stable.

In some examples, the dynamics model may be integrated into deep architectures, for example an architecture in which a variational auto-encoder transforms observations from sensor data to a latent, physically meaningful low-dimensional space of the system dynamics of the physical system and which may be used as input to the learned dynamics model to obtain a prediction of a future state of the physical system. This prediction may be used in various control or monitoring applications, for example to control a robotic arm to bend a wire, to control an autonomous vehicle to steer and/or brake, etc..

The processor subsystem is configured to jointly learn the dynamics model and the Lyapunov function by learning as the dynamics model (f(xt)) a projection (Proj) of nominal dynamics (f̂(xt)) of the physical system, which are learned based on the time-series of states (xt), onto a function that fulfills the Lyapunov condition as defined by the Lyapunov function (V(xt)). Here, the nominal dynamics f̂(xf) may represent the system dynamics which are learned directly and 'naively' from the training data before projecting, and which may for example be learned by a neural network. During training, the final dynamics f(xt) may then be evaluated by projection of f̂(xt) onto the set of dynamics that make V(xt) a Lyapunov function to ensure global asymptotic stability.

The projection (Proj) is an orthogonal projection onto a halfspace. For an arbitrary (e.g., non-stable) nominal dynamics f̂(xt) and a given positive definite function V(x) > <NUM>, V(<NUM>) = <NUM>, V(x) > <NUM> for x ≠ <NUM>, global asymptotic stability may be ensured by projecting f̂ to f(x) = Proj(f̂(x), {f: ∇V(x)T f ≤ αV(x)}) = f̂(x) - <MAT> where Proj(·,·) may denote an orthogonal projection onto a halfspace. For example, automatic differentiation tools may be used to define V, such that including VV in the definition of f(xt) may be done straightforwardly. Thereby, f(xt) may be trained straightforwardly, similar to the training of any other dynamics model.

Optionally, the dynamics model (f(xt)) comprises a neural network for learning the nominal dynamics (f̂(xt)) of the physical system based on the time-sequential pairs of states (xt). In addition to the machine learnable Lyapunov function V(xt), the dynamics model may comprise a neural network for learning the nominal dynamics f̂(xt). The dynamics model may thus be comprised of two different neural networks.

Optionally, the machine learnable Lyapunov function (V(xt)) is represented at least in part by an input-convex neural network (ICNN, g(x)). The Lyapunov function V(xt) may also be learned and represented by a neural network. By specifically selecting an input-convex neural network g(x) which enforces the condition that g(x) is convex in its inputs x, it may be ensured that V(xt)) has only one global optimum, which is one of the properties of the Lyapunov function which is to be learned by the input-convex neural network.

Optionally, the input-convex neural network (ICNN, g(x)) comprises a smoothed variant of a rectified linear unit (ReLu) for each of the activation functions. Using a smoothed variant of the ReLu may ensure that V(xt) is continuously differentiable. For example, a linear ReLu may be modified to provide a quadratic function in [<NUM>, d].

Optionally, the Lyapunov function (V(xt)) is shifted to establish a zero value of the Lyapunov function (V(xt) = <NUM>) at an equilibrium point (xt = <NUM>) of the Lyapunov function, and wherein the learning of the Lyapunov function comprises using a quadratic regularization term to ensure strict positive definitiveness of the Lyapunov function. Positive definitiveness is another property of the Lyapunov function which is to be learned by the input-convex neural network, and which may be ensured by the above measures.

Optionally, the processor subsystem is configured to, when learning the input-convex neural network (ICNN, g(x)) on the training data (xt), apply a continuously differentiable invertible function (F(x)) to the training data (xt) and use an output of the continuously differential invertible function as an input (g(F(x)) to the input-convex neural network. Invertibility may ensure that the sublevel sets of V, which may be by definition convex sets, map to contiguous regions of the composite function g ∘ F, thus ensuring that no local optima exist in the composed function and thus that V has only one optimum.

Optionally, the output device is an actuator associated with the physical system, and the processor subsystem is configured to control the physical system by providing control data to the actuator which is based on the inferred future state (xt+<NUM>) of the physical system. For example, the actuator may be used to control a vehicle, such as an autonomous or semi-autonomous vehicle, a robot, a manufacturing machine, a building, etc..

Optionally, the control or monitoring system is one of: a vehicle control system, a robotics control system, a manufacturing control system, a building control system. Such types of control systems greatly benefit from using a dynamics model which has been learned to be globally stable by being jointly learned with a Lyapunov function.

The following describes with reference to <FIG> the training of a learnable dynamics model which comprises a learnable Lyapunov function to establish global stability of the learned dynamics model, then describes with reference to <FIG> the dynamics model and its training in more detail, then describes with reference to <FIG> applications of the dynamics model which include a comparison to a non-stable dynamics model, and with reference to <FIG> the use of the learned dynamics model for the control or monitoring of a physical system, such as an (semi-)autonomous vehicle.

<FIG> shows a system <NUM> for training a dynamics model to learn dynamics of a physical system by learning to infer a future state of the physical system and/or its environment based on a current state of the physical system and/or its environment.

The system <NUM> comprises an input interface for accessing training data <NUM> representing a time-series of states of the physical system and for accessing model data <NUM> defining a machine learnable dynamics model which comprises a machine learnable Lyapunov function. For example, as also illustrated in <FIG>, the input interface may be constituted by a data storage interface <NUM> which may access the training data <NUM> and the model data <NUM> from a data storage <NUM>. For example, the data storage interface <NUM> may be a memory interface or a persistent storage interface, e.g., a hard disk or an SSD interface, but also a personal, local or wide area network interface such as a Bluetooth, Zigbee or Wi-Fi interface or an ethernet or fiberoptic interface. The data storage <NUM> may be an internal data storage of the system <NUM>, such as a hard drive or SSD, but also an external data storage, e.g., a network-accessible data storage. In some embodiments, the training data <NUM> and the model data <NUM> may each be accessed from a different data storage, e.g., via a different subsystem of the data storage interface <NUM>. Each subsystem may be of a type as is described above for the data storage interface <NUM>.

The system <NUM> further comprises a processor subsystem <NUM> which is configured to, during operation of the system <NUM>, to learn the dynamics model based on time-sequential pairs of states so as to learn to infer a future state of the physical system based on the current state, wherein said learning is constrained to provide a globally stable modelling of the dynamics of the physical system by jointly learning the dynamics model and the Lyapunov function so that values of the learned Lyapunov function decrease along all trajectories of states inferred by the learned dynamics model. This dynamics model and its training will be further described with reference to <FIG>.

The system <NUM> may further comprise an output interface for outputting trained model data <NUM> representing the learned (or 'trained') dynamics model. For example, as also illustrated in <FIG>, the output interface may be constituted by the data storage interface <NUM>, with said interface being in these embodiments an input/output ('IO') interface, via which the trained model data <NUM> may be stored in the data storage <NUM>. For example, the model data <NUM> defining the 'untrained' dynamics model may during or after the training be replaced, at least in part, by the model data <NUM> of the trained dynamics model, in that the parameters of the dynamics model, such as weights and other types of parameters of neural networks, may be adapted to reflect the training on the training data <NUM>. This is also illustrated in <FIG> by the reference numerals <NUM>, <NUM> referring to the same data record on the data storage <NUM>. In other embodiments, the trained model data <NUM> may be stored separately from the model data <NUM> defining the 'untrained' dynamics model. In some embodiments, the output interface may be separate from the data storage interface <NUM>, but may in general be of a type as described above for the data storage interface <NUM>.

<FIG> shows a computer-implemented method <NUM> for training a dynamics model to learn dynamics of a physical system by learning infer a future state of the physical system based on a current state of the physical system and/or its environment. The method <NUM> may correspond to an operation of the system <NUM> of <FIG>, but does not need to, in that it may also correspond to an operation of another type of system, apparatus or device or in that it may correspond to a computer program.

The computer-implemented method <NUM> is shown to comprise, in a step titled "ACCESSING TRAINING DATA", accessing <NUM> training data representing a time-series of states of the physical system and/or its environment, in a step titled "ACCESSING MODEL DATA DEFINING DYNAMICS MODEL", accessing <NUM> model data defining a machine learnable dynamics model which comprises a machine learnable Lyapunov function, in a step titled "LEARNING DYNAMICS MODEL BASED ON TRAINING DATA", learning <NUM> the dynamics model based on time-sequential pairs of states so as to learn to infer a future state of the physical system and/or its environment based on the current state, wherein said learning step <NUM> comprises, in a step or sub-step titled "JOINT LEARNING OF DYNAMICS MODEL WITH LYAPUNOV FUNCTION", constraining the learning to provide a globally stable modelling of the dynamics of the physical system by jointly learning <NUM> the dynamics model and the Lyapunov function so that values of the learned Lyapunov function decrease along all trajectories of states inferred by the learned dynamics model.

The following examples describe the machine learnable dynamics model, including the training thereof, in more detail. Instead of using a neural network, a Gaussian process or another machine learnable model type may be used to learn the nominal dynamics of the physical system and to learn the Lyapunov function.

Briefly speaking, the following deals with learning continuous time dynamical systems, but also applies mutatis mutandis to discrete time dynamical systems. That is, given a state at time <MAT>, the time-derivative of the state is modelled as <MAT> for some function <MAT>. Modeling the time evolution of such dynamical systems (or their counterparts with control inputs ẋ(t) = f(x(t), u(t)) for <MAT>) is a foundational problem, with applications in reinforcement learning, control, forecasting, and many other settings. Owing to their representational power, neural networks have long been a natural choice for modeling the above function. However, when using a generic neural network to model dynamics in this setting, very little can be guaranteed about the behavior of the learned system. For example, it is very difficult to say anything about the stability properties of a learned model (informally, the tendency of the system to remain within some invariant bounded set). While some recent work has begun to consider stability properties of neural networks, see the aforementioned [<NUM>, <NUM>, <NUM>], such work has typically done so by ("softly") enforcing stability as an additional loss term on the training data. Consequently, such work can say little about the stability of the system in unseen states.

In this specification, an approach is described to learning neural network dynamics that are provably stable over the entirety of the state space. To do so, the system dynamics and a Lyapunov function may be jointly learned. This stability may be a hard constraint imposed upon the model. Accordingly, unlike recent approaches, stability may not be enforced via an imposed loss function but built directly into the dynamics of the model (e.g., even a randomly initialized model in the described model class will be provably stable everywhere in state space). For that purpose, a Lyapunov function may be provided, which may be based on an input convex neural network, and which may ensure global exponential stability to an equilibrium point while still allowing for expressive dynamics.

Using these methods, the following demonstrates learning the dynamics of physical models such as n-link pendulums, e.g., with reference to <FIG>, and shows a substantial improvement over generic networks. It is also shown how such dynamics models can be integrated into larger network systems to learn dynamics over complex output spaces. In particular, it is shown with reference to <FIG> how to combine the model with a variational auto-encoder (VAE) to learn dynamic 'video textures'.

The following primarily considers the setting of autonomous dynamics ẋ(t) = f(x(t)) for <MAT>. While the techniques described in this specification are applicable to the dynamics with control as well, the following discussion focuses on the autonomous case for ease of explanation. A system is defined to be globally asymptotically stable (for simplicity, around the equilibrium point xe = <NUM>) if we have x(t) → <NUM> as t → ∞ for any initial state <MAT>; f is locally asymptotically stable if the same holds but only for x(<NUM>) ∈ B where B is some bounded set containing the origin. Similarly, f is globally (locally, respectively) exponentially stable (e.g., converges to the equilibrium "exponentially quickly") if <MAT> for some constants m, α ≥ <NUM> for any <MAT> (B, respectively).

The area of Lyapunov theory establishes the connection between the various types of stability mentioned above and descent according to a particular type of function known as a Lyapunov function. Specifically, let <MAT> be a continuously differentiable positive definite function, i.e., V(x) > <NUM> for x ≠ <NUM> and V(<NUM>) = <NUM>. Lyapunov analysis says that f is stable (according to the definitions above) if and only if we can find some function V as above such the value of this function is decreasing along trajectories generated by f. Formally, this is the condition that the time derivative V̇(x(t)) < <NUM>, i.e., <MAT>.

This condition should hold for all <MAT> or for all x(t) ∈ B to ensure global or local stability respectively. Similarly, f is globally asymptotically stable if and only if there exists positive definite V such that <MAT>.

Showing that these conditions imply the various forms of stability is relatively straightforward, but showing the converse (that any stable system must obey this property for some V) is relatively more complex. However, we are largely concerned with the "simpler" of these two directions, as the goal is to enforce conditions that ensure stability.

Briefly speaking, for an arbitrary (non-stable) nominal dynamics f̂ and a given positive definite function V(x) ≥ <NUM>, V(<NUM>) = <NUM>, V(x) > <NUM> for x ≠ <NUM>, we ensure global asymptotic stability by projecting f̂ to <MAT> where Proj(·,·) denotes the orthogonal projection onto a halfspace. For example, automatic differentiation tools may be used to define/implement V such that including VV in the definition of f can be done straightforwardly. Overall, the neural network for f may be trained straightforwardly, similar to the training for any other function.

The following provides a more detailed explanation of the above while referring to <FIG>. The intuition of at least some of the techniques described in this specification may be summarized as follows: instead of learning a dynamics function and attempting to separately verify its stability via a Lyapunov function, a dynamics model and Lyapunov function are jointly learned, where the modeling of the dynamics may be inherently constrained to be stable (everywhere in the state space) according to the Lyapunov function. Specifically, following the principles mentioned above, let <MAT> denote a "nominal" dynamics model referring to a dynamics model which is directly learned on the training data, e.g., without incorporating the Lyapunov function, and let <MAT> be a positive definite function: V(x) ≥ <NUM> for x ≠ <NUM> and V(<NUM>) = <NUM>. Then in order to (provably, globally) ensure that a dynamics function is stable, ḟ is projected such that it satisfies the condition <MAT> i.e., one may define the dynamics as <MAT> where Proj(x; C) denotes the orthogonal projection of x onto the point C, and where the second equation follows from the analytical projection of a point onto a halfspace. V may for example be defined using automatic differentiation tools. Accordingly, the gradient VV term may be included into the definition of f, obtaining a composite neural network for f (e.g., comprising a first neural network for ḟ and a second neural network for the Lyapunov function) which may be trained just like any other function.

<FIG> provide an illustration of the above. Each of <FIG> shows a contour plot <NUM>-<NUM> of a Lyapunov function of a stable dynamics system, in which <FIG> illustrates the trajectory of the Lyapunov function by way of an arrow pointing in a direction of increasing value of the Lyapunov function. Let g(x) = <MAT>. <FIG> illustrates f̂(x) having a component g(x) not in the half space and which may be subtracted to obtain f(x), and <FIG> illustrates f̂(x) being in the half space and returned unchanged.

As indicated previously, V may need to be positive definite, but additionally V may be preferred to have no local optima except <NUM>. This is due to Lyapunov decrease condition: one may desire to guarantee stability to the equilibrium point x = <NUM>, yet the decrease condition imposed upon the dynamics may mean that V may be decreasing along trajectories of f. If V has a local optimum away from the origin, the dynamics may in theory, but may not need to in practice, get stuck in this location; this may manifest itself by the <MAT> term going to zero, which may result in the dynamics becoming undefined at the optima. The above conditions may be enforced by design decisions as described below.

The Lyapunov function V may be learned and represented by a neural network. To ensure the required properties of V, the following considerations may apply. More specifically, one may prefer to ensure (i) that V has only one (global) optimum (without loss of generality assumed to be V(<NUM>) = <NUM>), (ii) that V is positive definite, e.g., V(x) > <NUM> for x ≠ <NUM> and (iii) that V is continuously differentiable.

Firstly, property (i) may be enforced by representing V with an input-convex neural network (ICNN) g, which may be defined by: <MAT> where Wi may be real-valued weights mapping from inputs to the i + <NUM> layer activations; Uimay be positive weights mapping previously layer activations zi to the next layer; bi may be real-valued biases; and σi may be convex, monotonically non-decreasing non-linear activations, such as the ReLU or smooth variants thereof. With this formulation, g is convex in x, and any convex function can be approximated by such networks.

Secondly, it may be desirable to ensure that V is positive definite. Representing V with an ICNN makes V convex, i.e., there is only one optimum, however it does not enforce V(<NUM>) = <NUM>. There are multiple ways to ensure V(<NUM>) = <NUM>. For example, one could find the global optimum of g and shift it or by removing the biases terms. However, shifting requires finding the optimum during training, which may be computationally expensive while removing the biases may put limitations on the class of variables that can be approximated. As an alternative approach, g may be shifted to ensure V(<NUM>) = <NUM>, while a little quadratic regularization term may be added to ensure positive definiteness, e.g., as: <MAT> where σk may be a positive, convex, non-decreasing function with σk(<NUM>) = <NUM> and ε is a small constant, for example selected such that <MAT>.

Thirdly, it may be desirable to ensure that V is continuously differentiable, for example by using a smoothed version <NUM> of the ReLU activations, as shown in <FIG>, that replaces the linear ReLU <NUM>, as shown in <FIG>, by a quadratic function in [<NUM>, d].

Additionally, warping to the input space may be used, such that the input to g is first mapped by any continuously differentiable, invertible function. Namely, although convexity may ensure that the Lyapunov function have no local optima, this may be a sufficient but not necessary condition, and indeed requiring a strongly convex Lyapunov function may impose too strict a requirement upon the learned dynamics. For this reason, the input to the ICNN function g(x) above may be optionally preceded by any continuously differentiable invertible function <MAT>, e.g., using <MAT> as the Lyapunov function. Invertibility may ensure that the sublevel sets of V (which are convex sets, by definition) may map to contiguous regions of the composite function g ∘ F, thus ensuring that no local optima exist in this composed function.

The above-described dynamics model may be demonstrated on several example problems described with reference to <FIG>. <FIG> relate to the modelling of a physically-based dynamical system, specifically the n-link pendulum. A damped, rigid n-link pendulum's state x may be described by the angular position θi and angular velocity θ̇i of each link i. In the following, f̂ may be a 2n-<NUM>-<NUM>-2n fully connected neural network, and V a <NUM>n-<NUM>-<NUM>-<NUM> ICNN, with properties described previously. The models are trained with pairs of data (x, x) produced by the symbolic algebra solver sympy.

In <FIG>, the simulated dynamics for the case of a simple damped pendulum (i.e., with n = <NUM>) are shown by a streamplot <NUM>, while <FIG> shows a streamplot <NUM> corresponding to the dynamics learned by the dynamics model and with <FIG> showing a contour plot <NUM> of the learned Lyapunov function. As can be seen, the dynamics may be learned to accurately predict motion of the pendulum even over long time periods.

The learned dynamics may also be evaluated quantitatively while varying n and the time horizon of simulation. <FIG> shows a graph <NUM> showing the prediction error <NUM> of the angular position θ and angular velocity θ̇ for an <NUM>-link pendulum as a function of timesteps <NUM>, while <FIG> shows a graph <NUM> showing the average prediction error <NUM> over <NUM> timesteps as a function of a number of links <NUM> of the pendulum. Both graphs <NUM>, <NUM> show the errors for both a 'simple' prior art model <NUM> and a learned dynamics model <NUM> of a type as described in this specification. While both the simple and the stable models show increasing mean error at the start of the trajectory, the dynamics model <NUM> is able to capture the contraction in the physical system (implied by conservation of energy) and in fact exhibits decreasing error towards the end of the simulation (true and simulated dynamics are both stable). In comparison, the error in the simple model increases.

<FIG> relate to another example in which the dynamics model as described in this specification may be used for stable video texture generation, for example using a Variational Autoencoder (VAE) to learn an encoding for images and the stable dynamics model to learn the dynamics in encoding-space. Given a sequence of frames (y<NUM>, y<NUM>,. ), the overall network may be fed a frame at time t and may be trained to reconstruct the frames at time t and t + <NUM>. Specifically, a VAE may be defined by the encoder <MAT> giving mean and variance µ, <MAT>, latent state <MAT> and decoder <MAT>, yt ≈ d(zt). The overall network may be trained to minimize both the standard VAE loss (reconstruction error plus a KL divergence term), but also to minimize the reconstruction loss of a next predicted state.

The evolution of the latent dynamics may be modeled at zt+<NUM> ≈ f(zt), or more precisely yt+<NUM> ≈ d(f(zt)). In other words, as also illustrated in <FIG> which shows a structure of the video texture generation network in which the encoder e and decoder d form a Variational Autoencoder and in which the stable dynamics model f is trained together with the decoder d to predict a next frame in a video texture, the overall network may be trained minimize: <MAT>.

The overall network may be trained on pairs of successive frames sampled from videos. To generate video textures, the dynamics model may be seeded with the encoding of a single frame and the dynamics model may be numerically integrated to obtain a trajectory. The VAE decoder may convert each step of the trajectory into a frame.

<FIG> shows trajectories projected onto a two-dimensional plane for, from left to right, three different runs of the dynamics model with the stability constraint and for a dynamics model without the stability constraint in the form of a generic neural network. In this example, the true latent space is <NUM>-dimensional, and the trajectories may be projected onto a two-dimensional plane for display. For the non-stable model, the dynamics quickly diverge and produce a static image, whereas for the stable model, it is possible to generate different (stable) trajectories that keep generating realistic images over long time horizons.

<FIG> shows a system <NUM> for controlling or monitoring a physical system using a machine learned modelling of dynamics of the physical system. The system <NUM> comprises an input interface <NUM> for accessing trained model data <NUM> representing a learned dynamics model as may be generated by the system <NUM> of <FIG> or the method <NUM> of <FIG> or as described elsewhere. For example, as also illustrated in <FIG>, the input interface may be constituted by a data storage interface <NUM> which may access the trained model data <NUM> from a data storage <NUM>. In general, the input interface <NUM> and the data storage <NUM> may be of a same type as described with reference to <FIG> for the input interface <NUM> and the data storage <NUM>. <FIG> further shows the data storage <NUM> comprising sensor data <NUM> representing an observation of a current state of the physical system. As described elsewhere, in some embodiments, the sensor data <NUM> may also be received directly from a sensor <NUM> via a sensor interface <NUM> of via another type of interface instead of being accessed from the data storage <NUM> via the data storage interface <NUM>.

The system <NUM> further comprises a processor subsystem <NUM> which may be configured to, during operation of the system <NUM>, infer the future state of the physical system based on the observation of the current state of the physical system as obtained from the sensor data <NUM>, and using an output interface as described elsewhere, provide output data <NUM> to an output device to enable the control or monitoring of the physical system based on the inferred future state, as also described elsewhere in this specification.

It will be appreciated that the same considerations and implementation options apply for the processor subsystem <NUM> as for the processor subsystem <NUM> of <FIG>. It will be further appreciated that the same considerations and implementation options may in general apply to the system <NUM> as for the system <NUM> of <FIG>, unless otherwise noted.

<FIG> further shows various optional components of the system <NUM>. For example, in some embodiments, the system <NUM> may comprise a sensor data interface <NUM> for directly accessing sensor data <NUM> acquired by a sensor <NUM> in an environment <NUM>. The sensor <NUM> may but does not need to be part of the system <NUM>. The sensor <NUM> may have any suitable form, such as an image sensor, a lidar sensor, a radar sensor, a pressure sensor, a contain temperature sensor, etc. In some embodiments, the sensor data <NUM> may sensor measurements of different physical quantities in that it may be obtained from two or more different sensors sensing different physical quantities. The sensor data interface <NUM> may have any suitable form corresponding in type to the type of sensor, including but not limited to a low-level communication interface, e.g., based on I2C or SPI data communication, or a data storage interface of a type as described above for the data storage interface <NUM>.

In some embodiments, the system <NUM> may comprise an actuator interface <NUM> for providing control data <NUM> to an actuator <NUM> in the environment <NUM>. Such control data <NUM> may be generated by the processor subsystem <NUM> to control the actuator <NUM> based on one or more inferred future states of the physical system, which may be generated using the learned dynamics model. For example, the actuator may be an electric, hydraulic, pneumatic, thermal, magnetic and/or mechanical actuator. Specific yet non-limiting examples include electrical motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servomechanisms, solenoids, stepper motors, etc. Such type of control is described with reference to <FIG> for an (semi-)autonomous vehicle.

In other embodiments (not shown in <FIG>), the system <NUM> may comprise an output interface to a rendering device, such as a display, a light source, a loudspeaker, a vibration motor, etc., which may be used to generate a sensory perceptible output signal which may be generated based on one or more inferred future states of the physical system. The sensory perceptible output signal may be directly indicative of the inferred future states of the physical system, but may also represent a derived sensory perceptible output signal, e.g., for use in guidance, navigation or other type of control of the physical system.

In general, each system described in this specification, including but not limited to the system <NUM> of <FIG> and the system <NUM> of <FIG>, may be embodied as, or in, a single device or apparatus, such as a workstation or a server. The device may be an embedded device. The device or apparatus may comprise one or more microprocessors which execute appropriate software. For example, the processor subsystem of the respective system may be embodied by a single Central Processing Unit (CPU), but also by a combination or system of such CPUs and/or other types of processing units. The software may have been downloaded and/or stored in a corresponding memory, e.g., a volatile memory such as RAM or a non-volatile memory such as Flash. Alternatively, the processor subsystem of the respective system may be implemented in the device or apparatus in the form of programmable logic, e.g., as a Field-Programmable Gate Array (FPGA). In general, each functional unit of the respective system may be implemented in the form of a circuit. The respective system may also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as distributed local or cloud-based servers. In some embodiments, the system <NUM> may be part of vehicle, robot or similar physical entity, and/or may be represent a control system configured to control the physical entity.

<FIG> shows an example of the above, in that the system <NUM> is shown to be a control system of an (semi-)autonomous vehicle <NUM> operating in an environment <NUM>. The autonomous vehicle <NUM> may be autonomous in that it may comprise an autonomous driving system or a driving assistant system, with the latter also being referred to as a semiautonomous system. The autonomous vehicle <NUM> may for example incorporate the system <NUM> to control the steering and the braking of the autonomous vehicle based on sensor data obtained from a video camera <NUM> integrated into the vehicle <NUM>. For example, the system <NUM> may control an electric motor <NUM> to perform (regenerative) braking in case the autonomous vehicle <NUM> is expected to collide with a traffic participant. The system <NUM> may control the steering and/or braking to avoid collision with the traffic participant. For that purpose, the system <NUM> may infer a future state of the vehicle with respect its environment, including the traffic participant, based on the sensor data obtained from the video camera. If the future state of the vehicle, e.g., its future position relative to the traffic participant, is expected to result in a collision, the system <NUM> may take corresponding action.

<FIG> shows a computer-implemented method <NUM> for controlling or monitoring a physical system using a machine learned modelling of dynamics of the physical system. The method <NUM> may correspond to an operation of the system <NUM> of <FIG>, but may alternatively also be performed using or by any other system, apparatus or device.

The method <NUM> is shown to comprise, in a step titled "ACCESSING SENSOR DATA", accessing <NUM> sensor data representing an observation of a current state of the physical system, in a step titled "ACCESSING MODEL DATA DEFINING LEARNED DYNAMICS MODEL", accessing <NUM> model data defining a machine learned dynamics model as described elsewhere in this specification, e.g., as obtained from the system <NUM> or method <NUM> of <FIG>, in a step titled "INFERRING FUTURE STATE FROM OBSERVED CURRENT STATE USING MODEL", inferring <NUM> the future state of the physical system based on the observation of the current state of the physical system as obtained from the sensor data, and in a step titled "PROVIDING OUTPUT DATA BASED ON INFERRED FUTURE STATE", using an output interface to an output device which is used in the control or monitoring of the physical system, providing <NUM> output data to the output device to enable the control or monitoring of the physical system based on said inferred future state. As also illustrated in <FIG>, instructions for the computer, e.g., executable code, may be stored on a computer readable medium <NUM>, e.g., in the form of a series <NUM> of machine-readable physical marks and/or as a series of elements having different electrical, e.g., magnetic, or optical properties or values. The executable code may be stored in a transitory or non-transitory manner. Examples of computer readable mediums include memory devices, optical storage devices, integrated circuits, servers, online software, etc. <FIG> shows an optical disc <NUM>. In an alternative embodiment, the computer readable medium <NUM> may comprise trained model data <NUM> defining a machine learned dynamics model as described elsewhere in this specification.

Claim 1:
A machine learning system (<NUM>) for training a dynamics model to learn dynamics of a physical system, in particular forward dynamics of a robotic arm, vehicle, domestic appliances, power tool or a manufacturing machine, by learning to infer a future state of the physical system based on a current state of the physical system, the machine learning system comprising:
- an input interface (<NUM>) for accessing:
- training data (<NUM>) comprising a time-series of states (xt) of the physical system;
- model data (<NUM>) defining a machine learnable dynamics model (f(xt)) which comprises a machine learnable Lyapunov function (V(xt));
- a processor subsystem (<NUM>) configured to learn the dynamics model based on the training data time-sequential pairs of states (xt) so as to learn to infer a future state (xt+<NUM> = f(xt)) of the physical system based on the current state (xt),
wherein said learning is constrained to provide a globally stable modelling of the dynamics of the physical system by jointly learning the dynamics model and the Lyapunov function so that values of the learned Lyapunov function decrease (V(xt+<NUM>) < V(xt)) along all trajectories of states (x<NUM>, x<NUM>, ...,xT) inferred by the learned dynamics model,
wherein the processor subsystem (<NUM>) is further configured to jointly learn the dynamics model and the Lyapunov function by learning as the dynamics model (f(xt)) a projection (Proj) of nominal dynamics (f̂(xt)) of the physical system, which are learned based on the time-series of states (xt), onto a function that fulfills the Lyapunov condition as defined by the Lyapunov function (V(xt)),
wherein the projection (Proj) is an orthogonal projection onto a halfspace.