Paper Info

Title: Interpretable reduced-order modeling with time-scale separation Interpretable reduced-order modeling with time-scale separation
Publish Date: 7 March 2023
Author List: Sebastian Kaltenbach (from CSE-Lab, ETH Zurich, Harvard SEAS), Phaedon-Stelios Koutsourelakis (from CSE-Lab, ETH Zurich, Harvard SEAS), Petros Koumoutsakos (from CSE-Lab, ETH Zurich, Harvard SEAS), Harvard Seas (from CSE-Lab, ETH Zurich, Harvard SEAS)

Figure

FIG. 5. Comparison between the phase-space of the reference solution (left) and the phase-space of the predictions
FIG. 7. Comparison between predictions and reference solutions for a new initial condition fort = 1.25, 3.75, 7.5, 12.5, 20, 30  (from left to right and top to down).We note that with longer prediction time the uncertainty bounds increases.Despite the chaotic nature of the KS equation, the predictive posterior mean is close to the reference solution for t ≤ 12.5

abstract

Partial Differential Equations (PDEs) with high dimensionality are commonly encountered in computational physics and engineering. However, finding solutions for these PDEs can be computationally expensive, making model-order reduction crucial. We propose such a data-driven scheme that automates the identification of the time-scales involved and, can produce stable predictions forward in time as well as under different initial conditions not included in the training data.
To this end, we combine a non-linear autoencoder architecture with a time-continuous model for the latent dynamics in the complex space. It readily allows for the inclusion of sparse and irregularly sampled training data. The learned, latent dynamics are interpretable and reveal the different temporal scales involved.
We show that this data-driven scheme can automatically learn the independent processes that decompose a system of linear ODEs along the eigenvectors of the system's matrix. Apart from this, we demonstrate the applicability of the proposed framework in a hidden Markov Model and the (discretized) Kuramoto-Shivashinsky (KS) equation.
Additionally, we propose a probabilistic version, which captures predictive uncertainties and further improves upon the results of the deterministic framework.

INTRODUCTION

High-fidelity simulations of critical phenomena such as ocean dynamics and epidemics have become essential for decision-making. They are based on physically-motivated PDEs expressing system dynamics that span multiple spatiotemporal scales and which necessitate cumbersome computations . In recent years there is increased attention to the development of data-driven models that can accelerate the solution of these PDEs as well as reveal salient, lower-dimensional features that control the long-term evolution.
In most cases, data-driven reduced-order models are not interpretable. In particular, models based on neural networks despite good predictive capabilities , they offer a black-box description of the system dynamics. A possible remedy is applying a symbolic regression to the learned neural network representation , but this adds additional computational cost due to the two-step procedure.
A number of frameworks such as SINDy allows to learn interpretable dynamics but it relies on the a-priori availability of lower-dimensional descriptors and of time-derivatives which can be very noisy for both simulation and experimental data. Other frameworks are tailored to specific problems such as molecular dynamics .
Here, we present a framework that only needs the value of the observables, and not their derivatives, as training data and is capable of identifying interpretable latent dynamics. The deployment of interpretable latent dynamics ensures that conservation of important properties of that are reflected in the reduced-order model .
The present method is related to approaches based on the Koopman-operator extended Dynamic Mode Decomposition (eDMD) but uses continuous complex-valued latent space dynamics and only requires one scalar variable per latent dimension to describe the latent space dynamics. Therefore we do not have to enforce any parametrizations on the Koopman matrix .
The time-continuous formulation moreover allows to incorporate sparse and irregularly sampled training data and fast generation of predictions after the training phase. By using a complex-valued latent space we can also incorporate harmonic effects and reduce the number of latent variables needed. Linear and non-linear autoencoders are used to map the observed, high-dimensional time-series to the lower-dimensional, latent representation and we identify simultaneously the autoencoder as well as the latent dynamics by optimizing a combined loss function.
Hence the to tasks of dimensionality reduction and discovery of the reduced dynamics are unified while other frameworks treat the two parts separately . Apart from using an architecture based on autoencoders to identify the latent space, projection-based methods could also be employed . We are also proposing a probabilistic version of our algorithm ) that makes use of probabilistic Slow Feature Analysis .
This allows for a latent representation that arart from being time-continuous, can quantify the predictive uncertainty and hierarchically decompose the dynamics into their pertinent scales while promoting the discovery of slow processes that control the system's evolution over long time horizons. The rest of the paper is structured as follows: We introduce the methodological framework as well as algorithmic details in section II.
Particular focus is paid on the interpretability of the inferred lower-dimensional dynamics. In section III we present three numerical illustrations, i.e. a system of linear ODEs, a hidden Markov Model and the discretized KS-equation. We then present in section IV the probabilistic extension of the framework and apply it to the KS-equation.
We conclude with a summary and a short discussion about possible next steps. We introduce the autoencoders deployed in this work, followed by the interpretable latent space dynamic and discuss the training process. We consider data from high-dimensional time series x n ∈ R f with n = 1, ..., T . We remark that the intervals between the different states do not need to be uniformly spaced.

Autoencoder

A core assumption of the method is that each high-dimensional state x n can be compressed to a lower-dimensional representation z n ∈ C c with c << f . We identify this lower-dimensional representation by an autoencoder consisiting of a parameterized encoder and decoder. The encoder maps the high-dimensional representation to the latent space as:
The latent space is complex-valued. The decoder reconstructs the high-dimensional representation based on the latent variables as: We denote the parameters of the encoder as well as the decoder by θ. As discussed later in Section II C, both set of parameters are optimized simultaneously during training and therefore there is no need for differentiating them.

Interpretable Latent Space Dynamics

We employ a propagator in the latent space to capture the reduced-order dynamics of the system. In contrast to other time-extended variational autoencoder frameworks, our representation uses complex valued latent variables. In addition the latent variables are treated independently. The latter feature enables us to have an interpretable latent dynamics as well as a model that is especially suitable for being trained in the Small Data regime due to the small number of required parameters.
This is in contrast to temporal propagators such as LSTMs . For each dimension i of the latent variable z we are using the following continuous ODE in the complex plane: By solving this ODE, we can define the operator: Interpretable reduced-order modeling with time-scale separation Here, λ is a vector containing all the individual λ's and ∆t n indicates the time-step between the latent states.
The symbol is used to indicate a component-wise multiplication. We remark that the latent variables and the parameter governing the temporal evolution are complex numbers and their role in describing the system dynamics is similar to that of an eigenvalue. The real part is associated with growth and decay whereas the imaginary part is representing the periodic component.
This approach has similarities with the Koopman-operator based methods and the extended dynamic mode decomposition . In contrast to the methods mentioned before we are using a continuous formulation in the latent space that allows us to incorporate scarce and irregularly sampled training data and directly rely on complex numbers in the latent space.

Training and Predictions

We optimize a loss function that combines both a reconstruction loss as well as a loss associated with the error of our learned propagator in the latent space: (5) We note that we could directly incorporate mini-batch training by only taking the summation over a subset of the N available training data.
For new predictions of unseen states, we use the encoder to generate a latent representations which is then advanced in time by the learned propagator. At a designated time step we are using the decoder to reconstruct the high-dimensional solution. We applied our algorithm to three systems. First, we show that the algorithm is capable of exactly reproducing the solution of a linear ODE and to identify its eigenvalues.
Afterwards we are applying the framework to a high-dimensional process generated by a complex latent dynamics, which is correctly identified. As a final test case, we are applying the algorithm to a Kuramoto Shivashinski (KS) equation. Interpretable reduced-order modeling with time-scale separation

Linear ODE

We are considering a two-dimensional ODE system for x = y 1 y 2 : Based on the obtained training data we run our algorithm using a linear encoder and decoder structure as well as two latent variables z. The loss function was optimized using the Adam algorithm . As we consider a linear ODE we can analytically compute the eigenvalues involved and compare it with the parameters λ identified by our algorithm.
We observe in Figure that the algorithm was able to recover the correct values, i.e. the eigenvalues 7 and 3 of the given linear ODE. The system does not have a periodic component and the two imaginary parts correctly go to zero, whereas the real parts converge to the reference value. Moreover we are also able to identify for the linear mapping between our latent variables z and the training data a matrix consisting of a multiple of the eigenvectors (1,1) and (1,-1) and thus the correct solution.
This example was chosen to show that the algorithm is able to quickly identify the exact solution of a linear ODE in terms of its linearly independent components.

Hidden multiscale dynamics

We consider eight-dimensional synthetic time series data produced by an underlying twodimensional complex valued process. In particular, the data points x are generated by first solving for the temporal evolution for the two complex-valued processes p 1 and p 2 and than mapping to the eight-dimensional space by using a randomly sampled linear mapping W .
One of the two processes used to generate the data is chosen to be much slower than the other one and both processes have a periodic component. dp 2 dt = (−0.9 + 1.5i)p 2 (8) As training data we consider 40 time series with 150 data points each, obtained by simulating the described processes for a maximum of t = 15 s and then sampling from the obtained data points.
Hence the training data consists of: • 40 time-series • with each consisting 150 observations of the x at a uniform time-step ∆t = 0.0025 The autoencoder obtained consists of one linear layer for both the decoder as well as the encoder. The model is trained for 5000 iterations using the Adam optimizer and a learning rate of 10 −3 .
The results for the convergence of the parameters λ 1 and λ 2 can be found in Figure . We note that the process which is slower decaying and thus more responsible for the long-term evolution of the system has a higher convergence rate than the faster process. With the obtained parameters λ as well as the trained autoencoder, we compute predictions based on the last time step used for training, i.e. we apply the encoder to obtain our latent representation and than use the latent dynamics to advance the latent representation in time.
Afterwards, we employ the decoder to reconstruct the full high-dimensional system. The results can be found in Figure and show very good agreement between predictions and reference data. This example shows that our model is successfully able to carry out dimensionality reduction and moreover indicates that the convergence rate between latent processes can be different.
The latter is relevant when training models as for accurate predictions all latent processes and their dynamics should be converged.

Kuramoto-Sivashinsky

Finally, we applied our algorithm to the KS equation and aim to identify a reduced-order model for the solution u(y, t): We employed periodic boundary conditions, µ = 1 and a domain size y ∈ [0, 22]. For this domain-size, the KS-equation exhibits a structurally stable chaotic attractor as discussed in The black lines divides the area for which training data was given from the area without raining data.
; . The equation is discretized in space using a discretization step of 22 64 resulting in a state vector x of dimension 64 and a nonlinear system of coupled ODEs. This is solved using a stiff fourth-order solver  We employed a non-linear encoder and decoder with four fully-connected layers each and ReLU-activation functions as well as Dropout Layers between the fully-connected layers.
We trained the model for 200000 iterations using Adam and a learning rate of 5 • 10 4 and assuming a five-dimensional latent space. We obtained the λ's in Figure . Four latent variables have λ's close to zero and thus a slow temporal dynamic that is responsible for the long-term evolution whereas one latent variable is quickly decaying.
Based on the obtained parameters, we do predictions based on an unseen initial condition not contained in the training data. We are able to reconstruct the correct phase space based on our predictions despite only using a very limited amount of training data. The results for the phase space can be seen in Figure .
Although the small-scale fluctuations in the temporal dynamics are not well captured, the model identifies the correct manifold which has a good accuracy compared to the reference solution. All phase-spaces were obtained by using a finite-difference operator on the data or predictions. These results are in accordance Interpretable reduced-order modeling with time-scale separation with whose LSTM-based temporal dynamic model was also able to find the correct phase space but not to track the actual dynamics for long-term predictions.
Our model is not able to account for noise in the temporal evolution and thus dealing with chaotic, small-scale fluctuations is challenging. We believe that a probabilistic version of our algorithm could be advantageous here. This section contains a fully probabilistic formulation for the deterministic model discussed before.
We replace the Autoencoder with a Variational Autoencoder and the ODE in the latent space with a SDE. The loss function which we optimize is the Evidence-Lower Bound (ELBO).

Model Structure

We postulate the following relations for our probabilistic model using an Ornstein-Uhlenbeck (OU) for each dimension i of the latent space and a Wiener process W t in the latent space: We again assume that the latent variables z t are complex-valued and a priori independent. Complex variables were chosen as their evolution includes a harmonic components which are observed in many physical systems.
We assume an initial conditions z 0,i ∼ CN (0, σ 2 0,i ). The total parameters associated with the latent space dynamics of our model are thus {σ 2 0,i , σ 2 i , λ i } c i=1 and will be denoted by θ together with all parameters responsible for the decoder mapping G (see next section). These parameters along with the state variables z t have to be inferred from the data x t .
Based on probabilistic Slow Feature Analysis (SFA) , we set σ 2 i = 2; (λ j ) and σ 2 0,i = 1. As a consequence, a priori, the latent dynamics are stationary. A derivation and reasoning for this choice can be found in Appendix A. Hence the only independent parameters are the λ i , the imaginary part of which can account for periodic effects in the latent dynamics.

Variational Autoencoder

We employ a variational autoencoder to account for a probabilistic mappings from the lower-dimensional representation z n to the high-dimensional system x n . In particular we are employing a probabilistic decoder The encoder is used to infer the state variables z based on the given data and thus defined in the inference and learning section.

Inference and Learning

Given the probabilistic relations , our goal is to infer the latent variables z 0:T as well as all model parameters θ. We follow a hybrid Bayesian approach in which the posterior of the state variables is approximated using amortized Variational Inference and Maximum-A-Posteriori (MAP) point-estimates for θ are computed.
The application of Bayes' rule for each data sequence x 0:T leads to the following posterior: where p(θ) denotes the prior on the model parameters. In the context of variational inference, we use the following factorization of the approximate posterior i.e. we infer only the mean µ and variance σ for each state variable based on the given data points.
This conditional density used for inference is the encoder-counterpart to the probabilistic decoder defined in the section before. It can be readily shown that the optimal parameter values are found by maximizing the Evidence Lower Bound (ELBO) F(q φ (z 0:T ), θ) which is derived in Appendix B. We compute Monte Carlo estimates of the gradient of the ELBO with respect to φ and θ with the help of the reparametrization trick and carry out stochastic optimization with the ADAM algorithm .

Results for the probabilistic extension

We applied our probabilistic version to the KS-equation. We used the same settings as for the deterministic approach but considered up to 10 complex latent variables. The obtained λ's are in Figure . The probabilistic model allows us to quantify the uncertainty in predictions. In Figure predictions for various time-steps and the respective uncertainty bounds are shown for an unseen initial condition.
Due to the chaotic nature of the KS-equation and the small amount of training data, the underlying linear dynamic of our model is only able to capture the full dynamics for a limited time horizon. Fortunately, due to the probabilistic approach the model is capable of capturing chaotic fluctuations with increasingly wide uncertainty bounds.
We also computed the phase space representation for the KS-equation based on the predictions obtained by our model and compare it with the reference solution. The probabilistic model identifies the correct manifold with a better accuracy than the deterministic model. As some of the small-scale fluctuations are accounted as noise, the resulting manifold is more concentrated at the origin and the obtained values are slightly smaller than the reference manifold although their shape is very similar.

Question: How does the framework capture the reduced-order dynamics?
Answer: By using a propagator in the latent space.