Model Reduction with Memory and the Machine Learning of Dynamical Systems

The well-known Mori-Zwanzig theory tells us that model reduction leads to memory effect. For a long time, modeling the memory effect accurately and efficiently has been an important but nearly impossible task in developing a good reduced model. In this work, we explore a natural analogy between recurrent neural networks and the Mori-Zwanzig formalism to establish a systematic approach for developing reduced models with memory. Two training models-a direct training model and a dynamically coupled training model-are proposed and compared. We apply these methods to the Kuramoto-Sivashinsky equation and the Navier-Stokes equation. Numerical experiments show that the proposed method can produce reduced model with good performance on both short-term prediction and long-term statistical properties. In science and engineering, many high-dimensional dynamical systems are too complicated to solve in detail. Nor is it necessary since usually we are only interested in a small subset of the variables representing the gross behavior of the system. Therefore, it is useful to develop reduced models which can approximate the variables of interest without solving the full system. This is the celebrated model reduction problem. Even though model reduction has been widely explored in many fields, to this day there is still a lack of systematic and reliable methodologies for model reduction. One has to rely on uncontrolled approximations in order to move things forward. On the other hand, there is in principle a rather solid starting point, the Mori-Zwanzig (M-Z) theory, for performing model reduction , . In M-Z, the effect of unresolved variables on resolved ones is represented as a memory and a noise term, giving rise to the so-called generalized Langevin equation (GLE). Solving the GLE accurately is almost equivalent to solving the full system, because the memory kernel and noise terms contain the full information for the unresolved variables. This means that the M-Z theory does not directly lead to a reduction of complexity or the computational cost. However, it does provide a starting point for making approximations. In this regard, we mention in particular the t-model proposed by Chorin et al . In  reduced models of the viscous Burgers equation and 3-dimensional Navier-Stokes equation were developed by analytically approximating the memory kernel in the GLE using the trapezoidal integration scheme. Li and E  developed approximate boundary conditions for molecular dynamics using linear approximation of the M-Z formalism. In , auxiliary variables are used to deal with the non-Markovian dynamics of the GLE. Despite all of these efforts, it is fair to say that there is still a lack of systematic and reliable procedure for approximating the GLE. In fact, dealing with the memory terms explicitly does not seem to be a promising approach for deriving systematic and reliable approximations to the GLE. ∗The Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ 08544, USA †Department of Mechanics and Aerospace Engineering, Southern University of Science and Technology, Shenzhen 518055, Peoples Republic of China ‡Beijing Institute of Big Data Research, Beijing, 100871, P.R. China 1 ar X iv :1 80 8. 04 25 8v 1 [ cs .L G ] 1 0 A ug 2 01 8 One of the most successful approaches for representing memory effects has been the recurrent neural networks (RNN) in machine learning. Indeed there is a natural analogy between RNN and M-Z. The hidden states in RNN can be viewed as a reduced representation of the unresolved variables in M-Z. We can then view RNN as a way of performing dimension reduction in the space of the unresolved variables. In this paper, we explore the possibility of performing model reduction using RNNs. We will limit ourselves to the situation when the original model is in the form of a conservative partial differential equation (PDE), the reduced model is an averaged version of the original PDE. The crux of the matter is then the accurate representation of the unresolved flux term. We propose two kinds of models. In the first kind, the unresolved flux terms in the equation are learned from data. This flux model is then used in the averaged equation to form the reduced model. We call this the direct training model. A second approach, which we call the coupled training model, is to train the neural network together with the averaged equation. From the viewpoint of machine learning, the objective in the direct training model is to fit the unresolved flux. The objective in the coupled training model is to fit the resolved variables (the averaged quantities). For application, we focus on the Kuramoto-Sivashinsky (K-S) equation and the Navier-Stokes (N-S) equation. The K-S equation writes as ∂u ∂t + 1 2 ∂u ∂x + ∂u ∂x2 + ∂u ∂x4 = 0, x ∈ R, t > 0; (1) u(x, t) = u(x+ L, t), u(x, 0) = g(x). (2) We are interested in a low-pass filtered solution of the K-S equation, ū, and want to develop a reduced system for ū. In general, ū can be written as the convolution of u with a low pass filter G(y):