# CD-ROM – Complementary Deep - Reduced Order Model

Emmanuel Menier, Michele Alessandro Bucci, Mouadh Yagoubi, Lionel

Mathelin, Marc Schoenauer

► To cite this version:

Emmanuel Menier, Michele Alessandro Bucci, Mouadh Yagoubi, Lionel Mathelin, Marc Schoenauer.
CD-ROM – Complementary Deep - Reduced Order Model. IUTAM Symposium on Data-driven modeling and optimization in fluid mechanics 2022, Jun 2022, Aarhus, Denmark. hal-04405482

HAL Id: hal-04405482

https://hal.science/hal-04405482v1

Submitted on 19 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# CD-ROM

## Complementary Deep - Reduced Order Model

E. Menier<fnref n="1" />,<fnref n="2" /> M.A. Bucci<fnref n="2" /> M. Yagoubi<fnref n="1" /> L. Mathelin<fnref n="3" />

M. Schoenauer<fnref n="2" />

<figure><img src="image_2.png" /></figure>

AIRBUS

<figure><img src="image_3.png" /></figure>

MICHELIN

<figure><img src="image_4.png" /></figure>

Air Liquide

<figure><img src="image_5.png" /></figure>

eDF

SNCF

<footnote n="1">¹IRT - SystemX, Palaiseau, France</footnote>

<footnote n="2">²TAU, Inria / LISN, Université Paris-Saclay / CNRS, Orsay, France</footnote>

<footnote n="3">³LISN, Université Paris-Saclay & CNRS, Orsay, France</footnote>
* Proper Orthogonal Decomposition can be used to extract the principal modes (V) from a physical simulation :

<figure><img src="image_6.png" /></figure>

* The flow approximates as a linear combination of a reduced number (r) of modes :

$$u(x, t) \approx \bar{u} = V(x)\alpha(t) \\ V \in \mathbb{R}^{n \times r}, \alpha \in \mathbb{R}^r$$
* An equation for the coefficients of this simplified form is obtained through Galerkin Projection of the original system :

$$ \frac{\partial u(x, t)}{\partial t} = g(u(x, t)), \quad V^T u = \alpha \\ \implies \frac{d\alpha}{dt} = V^T g(u(x, t)) $$

* The dynamics $g()$ of the reduced model are computed from the approximate solution, which introduces error :

$$ \frac{d\alpha}{dt} \approx V^T g(\bar{u}) \neq V^T g(u(x, t)) $$
* In the case of a laminar 2-D cylinder flow, a 3 equation model can be used to capture the transient dynamics<fnref n="1" />:

<figure><img src="image_7.png" /></figure>

<footnote n="1">¹ B. R. Noack et al., "A hierarchy of low-dimensional models for the transient and post-transient cylinder wake", Journal of Fluid Mechanics 497, 335-363 (2003).</footnote>
* To address classical POD-Galerkin model's shortcomings, we propose to add a correction term to their dynamics:

$$ \frac{d\alpha}{dt} = V^T g(u) = V^T g(\bar{u}) + f $$

* This correction term depends on information outside of the POD basis:

## Corrected ROM:

$$ g(u) = g(VV^T u + (I - VV^T)u) \approx g(V\alpha) + g'_{V\alpha}(I - VV^T)u $$
* Following Takens's theorem, we can retrieve the information lost during the projection on the POD basis by considering the past states of the system :

<figure><img src="image_8.png" /></figure>
* Delay Differential Equations can be used to aggregate information from the past in a time continuous manner :

$$ \frac{dx}{dt} = f(t, x, y), \quad y(t) = \int_{-\infty}^{t} x(\tau)e^{\lambda(\tau-t)}d\tau, \quad \lambda \in \mathbb{R}_{+} $$

* These equations are solved as an augmented ODE system :

$$ \begin{aligned} \frac{dx}{dt} &= f(t, x, y) \\ \frac{dy}{dt} &= x - \lambda y \end{aligned} $$
* We use Neural Networks to learn the ROM's correction from the memory $y$:

$$ \frac{d\alpha}{dt} = V^T g(V\alpha) + \mathcal{NN}(y) \\ \frac{dy}{dt} = \alpha - \lambda y $$

* The information available in memory $y$ is limited by its dimension:

$$ y(t) = \int_{-\infty}^{t} \alpha(\tau) e^{\lambda(\tau-t)} d\tau \implies \dim(y) = \dim(\alpha) $$
* To address the bottleneck in the memory, we propose to add an encoder network to expand the dimension of the memory :

$$ \frac{d\alpha}{dt} = V^T g(V\alpha) + NN(y) \\ \frac{dy}{dt} = E(\alpha) - \lambda y $$

* Leading to the final CD-ROM formulation :

<figure><img src="image_9.png" /></figure>
* Training data for the true ROM trajectories can be obtained from the snapshot data :

$$ \hat{\alpha}_{t_i} = V^T u_{DNS,t_i} $$

* The model can then be trained through adjoint backpropagation<fnref n="2" />:

$$ \mathcal{L}(\alpha) = \sum_{t_0}^{t_n} \|\alpha_{t_i} - \hat{\alpha}_{t_i}\|_2 $$

<figure><img src="image_10.png" /></figure>

<footnote n="2">² R. T. Q. Chen et al., "Neural ordinary differential equations", (2019).</footnote>
Cylinder Results

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>
* We tested the approach on a more challenging case, the chaotic pinball<fnref n="3" />:

<figure><img src="image_13.png" /></figure>

* Reminder : 8 modes are enough to capture 99 % of the cylinder flow's energy.

<footnote n="3">³ N. Deng et al., "Low-order model for successive bifurcations of the fluidic pinball", Journal of Fluid Mechanics 884, 10.1017/jfm.2019.959 (2019).</footnote>
* Taking a very reduced number ($r = 10$) of modes for the pinball flow yields a very unstable model.

<figure><img src="image_14.png" /></figure>
Pinball Results

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>
* The test trajectory presents statistics similar to the original attractor :

<figure><img src="image_17.png" /></figure>

Probability density function of the amplitude of each mode over a trajectory of ≈ 1000 seconds. The CD-ROM results are presented in blue while the DNS is in black.
* The test trajectory presents statistics similar to the original attractor :

<caption>Pinball : Correlation Dimension and Maximum Lyapunov Exponent</caption>

Correlation dimension

<figure><img src="image_18.png" /><figcaption>Mean trajectory divergence against time</figcaption></figure>

<figure><img src="image_19.png" /></figure>
* We can learn a different $\lambda$ for each dimension of the memory :

$$y(t) = \int_{-\infty}^{t} E(x(s))e^{\Lambda(s-t)}ds, \Lambda = \begin{bmatrix} \lambda_1 & & & \\ & \ddots & & \\ & & \lambda_N & \\ & & & \end{bmatrix}, N = \text{dim}(y)$$

* Each $\lambda_i$ defines the time horizon of each memory dimension, which can be computed as :

$$\tau_i = \frac{1}{\lambda_i}$$
* Sine waves are a good test-case to assess the ability of the CD-ROM to learn the time-horizons :

<figure><img src="image_20.png" /></figure>

* With a wave of period $T$, using a delay of $\tau = \frac{T}{4}$ yields a perfect circle in phase space.
## Trajectory Fit

* We use a superposition of two waves of periods $T_1 = 50$, $T_2 = 168$:

<figure><img src="image_21.png" /></figure>

* We need a memory of dimension 3 to represent the dynamics of the wave.
* Starting from varying initial conditions, the model learns similar horizons.

<figure><img src="image_22.png" /></figure>

* $\tau_1 \approx 50/4$, $\tau_2 \approx 168/4$, $\tau_3 \approx 168$
Summary :

* We propose a novel, time-continuous, neural architecture for modeling partially observable systems.

* It provides a computationally inexpensive and data-efficient correction to POD-Galerkin models, illustrated here in different flow configurations.

* Relevant and physically consistent time-constants are extracted from observations of the system.

* CD-ROM: Complementary Deep - Reduced Order model, Menier *et al.*, https://arxiv.org/abs/2202.10746
References

<figure><img src="image_23.png" /></figure>

B. R. Noack, K. Afanasiev, M. Morzyński, et al., "A hierarchy of low-dimensional models for the transient and post-transient cylinder wake", Journal of Fluid Mechanics 497, 335-363 (2003).

<figure><img src="image_24.png" /></figure>

R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, "Neural ordinary differential equations", (2019).

<figure><img src="image_25.png" /></figure>

N. Deng, B. R. Noack, M. Morzynski, et al., "Low-order model for successive bifurcations of the fluidic pinball", Journal of Fluid Mechanics 884, 10.1017/jfm.2019.959 (2019).
Annex Slides
* Although the CD-ROM increases the computational costs, it remains less expensive than bigger models :

<figure><img src="image_26.png" /></figure>

<table><thead><tr><th colspan="2">Integration times</th></tr></thead><tbody><tr><td>Uncorrected 10 modes</td><td>30 ms</td></tr><tr><td>Uncorrected 173 modes</td><td>550 ms</td></tr><tr><td>CD-ROM 10 modes</td><td>330 ms</td></tr></tbody></table>

Performance and computational cost of different models on a point outside the CD-ROM training basis.
* This model can then be progressively corrected and stabilised through our approach :

<figure><img src="image_27.png" /></figure>
* Echo State Networks are based on random matrices $W_{in}$ and $W$ :

$$ 
\begin{gathered}
Y_{t+1} = \tanh(W_{in} \cdot x_t + W \cdot Y_t) \\
\text{s.t. } \rho(W) < 1
\end{gathered}
$$

* Only the output operator is optimised to map the output sequence (O) to the memories :

$$ 
\begin{gathered}
x_t = W_{out} \cdot Y_t \\
W_{out} = Y^\dagger O
\end{gathered}
$$
• Ignoring the non-linearity, the ESN becomes a linear dynamical system :

$$ 
\begin{aligned}
Y_{t+1} &= W_{in} \cdot x_t + W \cdot Y_t \\
&= Y_t + \Delta_t \frac{dY}{dt}
\end{aligned}
\quad \Rightarrow \quad
\frac{dY}{dt} = \frac{W_{in}}{\Delta_t} \cdot x(t) + \frac{W - I}{\Delta_t} Y(t)
$$

• The operator $\tilde{W} = \frac{W-I}{\Delta_t}$ eigenvalues have negative real part. It plays the same memory dissipation role as the $\Lambda$ matrix in our architecture :

$$ \text{ESN :} \qquad \frac{dY}{dt} = \frac{W_{in}}{\Delta_t} \cdot x + \frac{W-I}{\Delta_t} Y $$

$$ \text{Deep ROM :} \qquad \frac{dY}{dt} = \text{Enc}(x) - \Lambda Y $$
* With Neural ODEs<fnref n="4" />, we can back-propagate gradients through the simulation steps:

<figure><img src="image_28.png" /></figure>

$$ \frac{dz(t)}{dt} = f(z, t; \theta) $$

$$ a(t) = \left. \frac{dL}{dz} \right|_{t} $$

$$ \frac{da(t)}{dt} = -a(t) \frac{\partial f(z(t), t; \theta)}{\partial z} $$

$$ \frac{dL}{d\theta} = - \int_{t_0}^{t_1} a(t) \frac{\partial f(z(t), t; \theta)}{\partial \theta} $$

<footnote n="4">⁴ R. T. Q. Chen et al., "Neural ordinary differential equations", (2019).</footnote>
* In our case, the Neural ODE corresponds to the complete ROM :

$$z = \begin{pmatrix} \alpha \\ y \end{pmatrix}$$

$$f(z, t; \theta) = \frac{1}{\bar{R}e} \bar{K} \alpha - \alpha^T \bar{N} \alpha + \mathcal{NN}(y; \theta_1) \\ E(\alpha; \theta_2) - \Lambda y$$

* Because the model is fully expressed with differentiable objects, the required jacobians can be easily evaluated through automatic differentiation.
Cylinder Hyperparameters

<table><tr><td colspan="2">ReLU activation</td></tr><tr><td>Encoder Network</td><td></td></tr><tr><td>Layer 1</td><td>3 neurons</td></tr><tr><td>Layer 2</td><td>11 neurons</td></tr><tr><td>Layer 3</td><td>19 neurons</td></tr><tr><td>Layer 4</td><td>27 neurons</td></tr><tr><td>Correction Network</td><td></td></tr><tr><td>Layer 1</td><td>30 neurons</td></tr><tr><td>Layer 2</td><td>30 neurons</td></tr><tr><td>Layer 3</td><td>30 neurons</td></tr><tr><td>Layer 4</td><td>3 neurons</td></tr></table>
# Pinball Hyperparameters

<table><tr><th colspan="2">Swish activation</th></tr><tr><th colspan="2">Encoder Network</th></tr><tr><td>Layer 1</td><td>10 neurons</td></tr><tr><td>Layer 2</td><td>36 neurons</td></tr><tr><td>Layer 3</td><td>63 neurons</td></tr><tr><td>Layer 4</td><td>90 neurons</td></tr></table>

<table><tr><th colspan="2">Correction Network</th></tr><tr><td>Layer 1</td><td>100 neurons</td></tr><tr><td>Layer 2</td><td>500 neurons</td></tr><tr><td>Layer 3</td><td>500 neurons</td></tr><tr><td>Layer 4</td><td>10 neurons</td></tr></table>
* Discrete time delays are unsuitable for classical time integrators :

<figure><img src="image_29.png" /></figure>