Lyapunov Regularized Forecaster

Rong Zheng
University of California San Diegor2zheng@ucsd.edu

Sicun Gao
University of California San Diegosicung@ucsd.edu

Rose Yu∗
University of California San Diegoroseyu@ucsd.edu

Abstract
Turbulent flow prediction plays a crucial role in climate change prediction. Especially, the long-term prediction of turbulent flow is a primary and promising goal
for its future development and attracts more attention from researchers. However,
because the Navier-Stokes equations on which turbulent flow relies are chaoticsystems, imperceptible initial differences can lead to large differences in futurestates, making the long-term prediction extremely difficult, even for the state-ofthe-art turbulence prediction model Turbulent-Flow Net (TF-Net) that introduces a
trainable bispectral decomposition and combines the temporal properties of turbulence with spatial modeling. Realizing that the error propagation leads to severe
instability over time in long-term prediction, we propose a time-based Lyapunov
regularizer2 to the loss function of TF-Net to avoid training error propagation andimprove the trained long-term prediction. The comparison experiment shows that
our Lyapunov-regularized forecaster does have more stable long-term predictions.

1Introduction

Computational Fluid Dynamics (CFD) is a cross-cutting science in which fluid mechanics andcomputer science are interwoven, and it utilizes physical rules and statistics on time series and spatial
distributions to compute and thus obtain approximate solutions to fluid control equations. CFD is
widely used with great success in various fields and industries. CFD is also at the heart of climate
simulation and is directly helpful in understanding and predicting climate change, in which theprediction of turbulent flow under the control of Navier-Stokes equations plays an important role inclimate prediction. Predicting turbulent flow is one of the most challenging problems in science andengineering. Navier-Stokes equations belong to a chaotic system. That is, as a strongly nonlinear
high-dimensional system, it brings huge degrees of freedom and multi-scale coupling. It is classified
as a statistical physics problem far from equilibrium, and the sensitivity to small perturbations anderrors makes the accuracy of long-term predictions very difficult to achieve.
Current deep learning utilizes a large amount of statistical data with known physical laws forturbulence to improve the long-term prediction of turbulence. For example, Turbulent-Flow Net (TFNet) [4], a data-driven deep learning model, introduces a trainable bispectral decomposition using
the popular and reliable Reynolds-averaged Navier-Stokes (RANS) and Large Eddy Simulation (LES)
nonlinear coupled models and obtains good prediction results by combining the temporal propertiesof turbulence with spatial modeling. However, turbulence prediction is extremely error-sensitive andeach step of the prediction depends on a large amount of data derived from the velocity field of the∗2Corresponding author: Rose Yu (roseyu@ucsd.edu)
Our code and models are available on Github

Machine Learning and the Physical Sciences workshop, NeurIPS 2022.

historical steps. The error propagation leads to severe instability over time in long-term prediction.
Some models achieve robust stability by adding training noise, space-based loss constraints, anddataset augmentation [3]. Differently, we plan to add a time-based regularizer to constrain the model
during training to further improve the stability of its long-term prediction.
We assume turbulence prediction as a continuous-time dynamical system, and the Lyapunov function
is a direct tool to determine the stability of this dynamical system. In this paper, we use TF-Net as
a turbulence prediction model and model the time evolution of turbulence prediction by adding a
Lyapunov regularizer that exploits the properties of temporal dynamical systems to further enhance
its long-term prediction effect. We observe that with the addition of the Lyapunov regularizer, weobtain more accurate long-time series predictions with an outstanding improvement in the test RMSE
compared to the unregularized TF-Net.

2Methodology

We develop the Lyapunov regularized turbulent flow forecaster training scheme by using TF-Net as
the prediction model and adding our Lyapunov regularizer to the loss function. The purpose of ourregularizer is to maintain the stability of the long-term prediction accuracy. We want the long-term
prediction to be an asymptotically stable autonomous system. That is, the difference of v̂t and vtdecreases toward 0 as time t increases, where v̂t is the predicted two-dimensional velocity field attime t and vt is the corresponding ground truth. Based on the theorem on asymptotic stability ofLyapunov functions, we wantdd< 0 for all v̂t ̸= vt
V̇ (x(t)) = V (x(t)) = V (v̂t − vt )(1)
= 0 only if v̂t = vt
dt
dt
2.1

Lyapunov Function

We assume the Mean Square Error (MSE) of the prediction and ground truth at time t as the Lyapunovfunction,V (x(t)) = ||x(t)||22 = ||v̂t − vt ||22(2)where x(t) = v̂t − vt . Also, V satisfies the conditions of Lyapunov function that V (x) > 0 for allx ̸= 0 and V (0) = 0.
Then, we use the finite difference of V (x(t + 1)) − V (x(t)) to approximate V̇ , which is used as apart of the regularizer,V̇ (x(t)) ≈ V (x(t + 1)) − V (x(t)) = ||v̂t+1 − vt+1 ||22 − ||v̂t − vt ||22(3)While training, the Lyapunov regularizer reduces the difference in prediction error over long-termtime steps, which constrains the cumulative error caused by the long-term prediction to increase
slowly over time steps.
2.2

Wrappers

As our goal is to treat V̇ as a convex constraint by maintaining its negativity, we wrap V̇ with the log
barrier function,1
ϕlog (x(t)) = − log(−V̇ (x(t)))(4)α
where α is a large positive number that increases the optimization accuracy and avoids the unnecessary
minimization of V̇ .
Wrapping V̇ with ReLU is another solution that helps maintain the negativity,
ϕrelu (x(t)) = ReLU (V̇ (x(t)))(5)

It keeps the gradient of V̇ for back-propagation only if the derivative is positive, which guides V̇ todecrease toward negative.
22.3

Weight Scheduler

Without an appropriate weight scheduler, the regularizer may increase the prediction error in shortterm prediction steps. Because at early stages, the prediction errors are low but the increment slope issteep, which may cause the weight of the Lyapunov regularizer to be higher than the prediction loss.
Thus, we add a scheduler based on the prediction MSE to balance the weight of the regularizer,
w(x(t)) =1
2
1 + e−s·(||x(t)||2 −m)(6)which is an adjusted sigmoid function. As s is a big positive constant, the weight approaches to 0
when the prediction MSE ||x(t)||22 is less than the threshold m and approaches to 1 when the MSE
is greater than m. Since the prediction MSE is low at early steps and high at later steps due to theteacher-forcing training method, the regularization weight at early steps will be approximately zero
but approximately one at later steps with an appropriate m.
2.4

Lyapunov Regularizer

Utilizing Equation (5) as the constant weight and Equation (4) as the constraint, we can get the
Lyapunov regularizer
Llya =TN X
Xitw(x(t))ϕ(x(t)) =TN X
Xiw(v̂(t) − v(t))ϕ(v̂(t) − v(t))(7)

twhere N is the batch size and T is the prediction length.

3Experiments

3.1

Data

Our experimental data are the turbulent-flow velocity fields in two-dimensional space simulated bythe Lattice Boltzmann Method [1]. The relevant physical parameters of the simulation are properly
set, including Prandtl number = 0.71, Rayleigh number = 2.5 × 108 , and the maximum Mach
number = 0.1. The simulation provides 2000 frames of two-dimensional velocity fields with thespatial resolution 1792 × 256. We divide each 1792 × 256 frame into seven 256 × 256 sub-regions
and downsample them into 64 × 64 images. Then, we use a sliding window with a length of 100 togenerate 9,870 samples, and we select 6,000 for training, 1,700 for validation, and 2,170 for testing.
The data is normalized for training, validation, and testing.
3.2

Results

In the comparison test, TF-Net is used as the prediction model for the turbulent flow predictiontask. We compare our results with TF-Net without our regularizer and a state-of-the-art Lyapunov
regularization baseline [2].
The baseline model contains a trainable Lyapunov function, which consists of an input-convex neuralnetwork (ICNN). We connect ICNN with the input and output of TF-Net and apply the loss functionof the baseline while training. The baseline model is tuned based on the averages of validation resultsof the turbulent-flow data.
For our regularizer, we implement two versions of ϕ(x(t)) in Llya : one uses log barrier ϕlog inEquation (4), and another uses ReLU ϕrelu in Equation (5). We set α = 103 in ϕlog . We fix s = 300
in weight scheduler w(x(t)) and tune m within the range of 0.5 and 0.8 for each of the two ϕ’s. Whiletraining, we select multiple samples of turbulent flow sequences as one batch, and train sequenceswith one-step prediction and teacher-forcing method. We add the Lyapunov regularizer Llya tothe original loss function of TF-Net to obtain the regularized forecaster. All models are fixed at
100 training epochs. Every two models are trained using one Nvidia A6000, and the training takes
approximately 1.5 hours to complete.
We utilize RMSE as the evaluation metric for quantifying the prediction performance of the following
60 frames. All test results are averaged over four runs with random initialization. The comparison
3(a) RMSE(b) Prediction Examples

Figure 1: (a) RMSE of Turbulent Flow Forecasting, where vertical lines are the standard deviations;(b) Ground Truth (Target) and the predicted velocity fields generated by various methods, including
TF-Net, ICNN and the proposed LRF at prediction step of 60.

results are shown in Figure 1(a), from which we can observe that our Lyapunov regularizers canprovide distinct performance improvements.
With Equation (3) we can see that the role of the regularizer is to make the long-term prediction
error curve smoother. Fortunately, when we directly add the regularizer described in Equation (3)combined with the wrapper in Equation (4) to the loss function of the prediction model, the long-termRMSE decreases instead of simply raising the short-term RMSE to make the curve smoother. We
further added the weight scheduler so that the short-time RMSE is not negatively affected, resulting
in the curve in Figure 1(a). The visualization of the ground-truth and predicted velocity fields are inFigure 1(b) and Figure 2 in Appendix.

4Limitations

In fact, we observed that if we train TF-Net with our Lyapunov regularizer but without the weightscheduler, the RMSEs after time steps 20 are smoother and lower than the one with the weightscheduler. However, the short-term prediction RMSEs are high to further improve the overallsmoothness to cooperate with the constraint of V̇ . Therefore, although the scheduler w can reduce
the short-term prediction RMSE, it may have some negative impacts on the long-term prediction
steps. In the future, we will develop a more efficient pair of weight scheduler and Lyapunov functions
to thoroughly exploit the advantages of Lyapunov regularization in the long-term prediction part.

5Conclusion

In this work, we use TF-Net, which combines the properties of turbulence physics and big data,
as a prediction model and add to it a Lyapunov regularizer that exploits the properties of temporal
dynamical systems to improve the accuracy of long-term predictions. We choose the prediction errorof the velocity field at a time point t as the Lyapunov function and approximate the derivative of itwith respect to time to serve as a regularizer for the long-time prediction. To make it act as a constraintfunction, we wrap it using a barrier function or a rectifier. We further add a weight scheduler toattenuate its negative effect on short-term prediction steps. Through comparison experiments weobserve an effective long-time prediction boosting effect. After further enhancements to our Lyapunov
regularizer, this regularization might be helpful for the long-term prediction of climate change.
4

Broader Impact
Currently, the application of Lyapunov Regularize Forecaster is limited to climate change prediction.
We discovered that our Lyapunov regularizer does not have any targeted prerequisite on the prediction
model and prediction data type. Therefore, we will enhance the regularizer from a generalization
perspective and test its effectiveness in other rule-bounded prediction domains. For example, it
may improve the accuracy of long-term fire ignition simulation on grassland, which helps wildfire
prevention and control. It may also promote the reliability and safety of autonomous drone forindustrial or agricultural sites.

Acknowledgement
We thank Professor Sicun Gao for providing helpful suggestions regarding the further enhancementof the Lyapunov model. We also thank Ray Wang for advice on preprocessing the dataset.

References[1] D. B. Chirila. Towards lattice Boltzmann models for climate sciences: The GeLB programming
language with applications. PhD thesis, Universität Bremen, 2018.
[2] J. Z. Kolter and G. Manek. Learning stable deep dynamics models. Advances in neural
information processing systems, 32, 2019.
[3] K. Stachenfeld, D. B. Fielding, D. Kochkov, M. Cranmer, T. Pfaff, J. Godwin, C. Cui, S. Ho,P. Battaglia, and A. Sanchez-Gonzalez. Learned coarse models for efficient turbulence simulation.
arXiv preprint arXiv:2112.15275, 2021.
[4] R. Wang, K. Kashinath, M. Mustafa, A. Albert, and R. Yu. Towards physics-informed deeplearning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining, pages 1457–1466, 2020.

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes](b) Did you describe the limitations of your work? [Yes] See Section 4(c) Did you discuss any potential negative societal impacts of your work? [N/A](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes] All authors have read and confirmed
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A](b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We will makethe source code publicly available later(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)? [Yes] See Section 3(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See Figure 1(a)(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] See Section 3.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
5(a) If your work uses existing assets, did you cite the creators? [Yes] All existing code,
data, models are listed in References and cited in text(b) Did you mention the license of the assets? [N/A](c) Did you include any new assets either in the supplemental material or as a URL? [N/A](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [Yes] The data is publicly available(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A](b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]6

AVisualization(a) T + 10(b) T + 20(c) T + 30(d) T + 60

Figure 2: Ground Truth (Target), and the predicted velocity fields v of TF-Net, Baseline (ICNN), andRegularizers (LRF-ReLU, LRF-Log); at time T + 10, T + 20, T + 30, and T + 60, where input
sequence has length T7