Title: LEARNING PARTIALLY OBSERVABLE PDE DYNAMICS WITH NEURAL NETWORKS

Abstract: Spatio-Temporal processes bear a central importance in many applied scientific fields. Generally, differential equations are used to describe these processes. In this work, we address the problem of learning spatio-temporal dynamics with neural networks when only partial information on the system's state is available. Taking inspiration from the dynamical system approach, we outline a general framework in which complex dynamics generated by families of differential equations can be learned in a principled way. Two models are derived from this framework. We demonstrate how they can be applied in practice by considering the problem of forecasting fluid flows. We show how the underlying equations fit into our formalism and evaluate our method by comparing with standard baselines.

1. INTRODUCTION:

We consider the problem of learning complex dynamics typical of physical systems using deep learning models. Modeling these dynamics is central to almost any problem in applied physics. The classical approach to this problem is model based : the dynamics of the physical system is usually formulated using Partial Differential Equations (PDEs) describing the evolution of state variables. PDEs are then solved through some clever numerical scheme (as in Meerschaert & Tadjeran (2006) for example). This methodology is used in many application domains like climate science, financial engineering, etc.. Although it has been extensively developed, several challenges remain open. Modeling physical phenomena accurately can be tedious, if not unfeasible : the relations between state variables are often complex and, even when those are known, specifying a functional form for the dynamics can be difficult (Lions et al. (1992) , Peebles (1980) ). Moreover, designing accurate, robust and computationally feasible numerical schemes for PDEs is usually a difficult problem for which current solutions rely on extensive domain specific background.

The availability of huge data quantities gathered from different types of sensors or simulations is a strong incentive to develop data driven approaches, as an alternative or complement to standard model based methods. For many real world problems, the underlying physics of the phenomenon is not always known in its entirety and data-driven methods, inspired by our current physical knowledge, appear as a particularly promising alternative to model based approaches, see Bezenac et al. (2018) , Ling et al. (2016) , Stewart & Ermon (2016) or Rudy et al. (2017b) for some examples. On the other hand, mature fields such as differential equations could provide machine learning with useful insights and intuitions. We advocate here a ML approach incorporating structural priors inspired from such a well developed domain.

We consider the problem of learning dynamical systems where available observations only provide partial information on the system state and dynamics which is a typical scenario in real world applications, be it in weather forecasting, financial engineering, or even video prediction. The objective is to learn from such observations the underlying system dynamics so as to forecast future observations. We focus on data generated by complex highly non-linear differential equations and analyze whether these dynamics can be learned by Deep Learning models. This context is particularly interesting since simulations relying on PDEs are essential in many applied disciplines and since this allows us to perform controlled experiments providing access to all the variables of the problem.

Our contributions are the following :

• We propose a general framework for modeling partially observed dynamical systems governed by PDEs with neural networks. This framework is inspired by the way physicists incorporate observations into their forecasting pipeline. It should be flexible enough to accommodate a large class of modeling problems other than forecasting observations. • From this framework we derive two instances of training (PT and JT), with two different levels of prior injection, and two methods for multi-step forecasting (SSE and MSRE). • As a case study, we consider two PDEs describing the evolution of fluid flow, namely the Euler and Navier-Stokes equations, and show how they can be formulated within this formalism. We study the performance and behaviour of our models on the problem of forecasting fluid flow using two common baseline models and focusing on the interplay between injection of prior information and performance.

We introduce our framework in section 2, detail the framework and its instances in section 3 and show how PDEs fit within this framework in section 4. Experimental results and analysis are described in section 5 and recent related work is reviewed in section 6.

2. BACKGROUND AND PROBLEM SETTING:

2.1 LEARNING FROM PARTIALLY OBSERVED DATA

A dynamical system in our context can be broadly defined as a function X with values in Ω which obeys a set of differential equations. We consider spatio-temporal dynamics for which X can be written as a function of (t, x) ∈ R × R d where t and x are respectively the time and space variables.

The spatial vector-valued function X t contains the quantities of interest describing a studied physical system at time t and gives the evolution in time and space of the system's state. More precisely, we will be interested in differential equations which can be written as :

EQUATION

where X t denotes the state vector at time t, D x X t the spatial derivatives of X t and F is a function with values in Ω (common instances for F involve the gradient operator ∇X, the Laplacian ∇ 2 X,...). By specifying a particular structure on F , one can describe the dynamics of many different classes of systems : an important example is that of the Navier-Stokes equations from fluid dynamics which bear a central importance in many applied sciences and which will be our application example.

2.1.1. NEURAL IMPLEMENTATION OF NUMERICAL SCHEMES:

Solving differential equations usually requires using a numerical scheme in order to approximate the solution. Finite difference methods for example make use of discrete approximations of the derivatives. Let us suppose available a sequence of states {(X t ) t }, each X t being a function of the space variable x, X t and X t+δt being separated by a short time interval δt for all t. A straightforward discretization of equation 1 is the forward Euler scheme :

X t+δt -X t ≈ δt • F (X t , D x X t )

where δt is the discretization timestep.

Using a parameterized neural network r θ , we can train it in order to fit the relation above :

X t+δt ≈ X t + δt • r θ (X t , D x X t )

This is reminiscent of the skip connection structures used in the residual block of ResNet or in some RNN models. Linear multi-step methods extend this idea by performing several integration steps.

A simple multi-step extension of the Euler scheme could then be :

X t+kδt ≈ X t + δt k-1 l=0 r θ (X t+lδt , D x X t+lδt )

This sequential process could be implemented by a ResNet architecture so that the latter could be used as a parametric model for approximating solutions of differential equations.

Moreover, spatial derivatives of X t at a given point x can classically be expressed with finite difference schemes over the values of the function at neighbouring points V x :

DX t (x) = K({X t (x ) | x ∈ V x })

where K is a linear function. Thus, we can express each component of D x X t as a convolution over X t , motivating the use of convolutional networks. Thus, given a sufficiently large training set of state sequences and supposing that we are able to learn their dynamics, we could, just by knowing the initial state of our system, use the learned operator and forecast its behaviour for any time horizon.

One could constrain the structure of K to express a particular set of derivatives. This would be a way to add additional structure and knowledge into the forecasting operator. One could also adopt a more hybrid approach such as in Long et al. (2018) .

For the examples we studied in this paper, we use standard convolutional ResNets as we wanted to disentangle as much as possible the contributions of the elements we introduce, particularly when injecting priors into the estimator. However, it would be interesting in future research to analyze the effects of more structural constraints added into the forecasting operator.

2.1.2. LINKING STATES TO PARTIAL OBSERVATIONS:

In realistic settings, the state is generally only partially observed e.g., when studying the ocean's circulation, variables contained in the system's state such as temperature or salinity are observable while others such as velocity or pressure are not. In other words, the measured data is only a projection of the complete state X. We model this measurement process with a fixed operator H linking the system's state X t to the corresponding observation Y t :

Y t = H(X t )

In the following, H is supposed to be known, fixed and differentiable. This setting is the one which we focus on in this paper.

3. FRAMEWORK:

To summarize, observations Y t are supposed to be generated by a state space model with variable X t and the main objective is the prediction of future observations Y t+k for a certain horizon k. The general form of the model for a prediction at horizon 1 is then :

EQUATION

We make the simplifying hypothesis that the information loss through the projection H is so that a state X t can be deterministically reconstructed using a long enough sequence. In other words, there exists an integer k and an operator E such that :

EQUATION

For more details on this assumption, refer to appendix C.

For now, let us consider only one step predictions : given a sequence of initial observations (Y 1 , ..., Y k ), we want to find the best estimation Y k+1 for the observation Y k+1 , where observations Y are supposed to be generated an underlying process X. We propose a framework where predicting Y k+1 is decomposed into three steps :

• Prediction of the current state : Given past observations, we use a learnable operator e ω parameterized by ω to produce a (complete) state vector X t .

• Prediction of the future state : X t is given as input to a second learnable operator f θ parameterized by θ which produces the next state X t+1 .

• Projection of the future observation : X t+1 is mapped onto the space of observations using operator H to predict future observation Y t+1 . This is summarized in the following system :

EQUATION

The learning problems amounts to optimizing the following loss :

EQUATION

)

where d is the loss metric and e and f are functions belonging to appropriate parametric families. Time lag parameter k can be set up by cross-validation. As discussed in section 2, a natural choice for f is a convolutional residual network modeling the system's dynamics whereas, for e, designing the right family of functions will depend on the structure of the studied data.

3.1. FORECASTING MODELS:

Let us now introduce two instances of the framework described above. For clarity, we first consider one step predictions and then introduce the multi-step extension in section 3.2.

3.1.1. JOINT TRAINING OF ESTIMATION AND FORECASTING:

When the information about the underlying process is limited to observations, the straightforward way to solve the minimization problem equation 5 is to train e and f jointly. Prior information here is limited to the architectures of e and f as well as that of the pipeline equation 4 and the fixed operator

EQUATION

i be a training set of observation sequences, each sequence being identified by index i. The loss corresponding to sequence i is then :

L = d H(f θ (e ω (Y (i) t-k+1 , ..., Y (i) t ))), Y (i) t+1

With this approach only observations are needed for training without any additional supervision. On the other hand, there is no reason for it to have any interpretable structure or correspond to the physical modelling of the studied system.

3.1.2. PRE-TRAINING THE STATE ESTIMATOR:

It is not uncommon that some form of additional knowledge on the system state is available and can be exploited.

• It is sometimes possible to access to a limited number of examples of state sequences through costly measurements. Then pre-training of e and f could be beneficial, its success depending on the complexity of the studied dynamics and on the information loss associated with H. • More often it is possible to have access to a noisy or approximate version of the states X t through simulations. Complex simulators and generated data are available in several domains : it is common for models to be developed using simulated data and then to be adapted to real observations. A lighter alternative is to use a simplified model which allows fast and cheap simulations. Generated sequences can then be used e.g. for pre-training e and f before fine-tuning with real observations or more complex simulations. In both cases, the simplified dynamic plays the role of a prior over the underlying process. In order to illustrate and evaluate this idea, we will consider the Euler equations as a simplified version of the more complex Navier Stokes equations in the experiments (section 5.3).

3.2. MULTI-STEP FORECASTING:

We present below two strategies that could be used for prediction at horizon l > 1. In order to forecast l 2 steps ahead, once the state at time t has been the estimated with X t using e ω , we may apply one of the following strategies : < l a t e x i t s h a 1 _ b a s e 6 4 = " F 7 g D Z 6 i U 8 b T j L s P a

Y t k+1 < l a t e x i t s h a 1 _ b a s e 6 4 = " F I 2 X q Q Q + i m 8 R O t t n K O X o C I u + 4 V o = " > A A A B 8 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I g l g S E f R Y 9 O K x g v 2 i D W W z 3 b R L N 5 u w O x F K 6 K / w 4 k E R r / 4 c b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y R S G H T d b 2 d l d W 1 9 Y 7 O w V d z e 2 d 3 b L x 0 c N k y c a s b r L J a x b g X U c C k U r 6 N A y V u J 5 j Q K J G 8 G o 7 t p v f n E t R G x e s R

X q Q Q + i m 8 R O t t n K O X o C I u + 4 V o = " > A A A B 8 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I g l g S E f R Y 9 O K x g v 2 i D W W z 3 b R L N 5 u w O x F K 6 K / w 4 k E R r / 4 c b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y R S G H T d b 2 d l d W 1 9 Y 7 O w V d z e 2 d 3 b L x 0 c N k y c a s b r L J a x b g X U c C k U r 6 N A y V u J 5 j Q K J G 8 G o 7 t p v f n E t R G x e s R

X q Q Q + i m 8 R O t t n K O X o C I u + 4 V o = " > A A A B 8 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I g l g S E f R Y 9 O K x g v 2 i D W W z 3 b R L N 5 u w O x F K 6 K / w 4 k E R r / 4 c b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y R S G H T d b 2 d l d W 1 9 Y 7 O w V d z e 2 d 3 b L x 0 c N k y c a s b r L J a x b g X U c C k U r 6 N A y V u J 5 j Q K J G 8 G o 7 t p v f n E t R G x e s R

X q Q Q + i m 8 R O t t n K O X o C I u + 4 V o = " > A A A B 8 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I g l g S E f R Y 9 O K x g v 2 i D W W z 3 b R L N 5 u w O x F K 6 K / w 4 k E R r / 4 c b / 4 b t 2 0 O 2 v r C w s M 7 M + z M G y R S G H T d b 2 d l d W 1 9 Y 7 O w V d z e 2 d 3 b L x 0 c N k y c a s b r L J a x b g X U c C k U r 6 N A y V u J 5 j Q K J G 8 G o 7 t p v f n E t R G x e s R

E x R 5 u f v / 6 w U = " > A A A B 6 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j R f s h b S i b 7 a Z d u t m E 3 Y l Q Q n + C F w + K e P U X e f P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u F t f W N z a 3 i d m l n d 2 / / o H x 4 1 D J x q h l v s l j G u h N Q w 6 V Q v I k C J e 8 k m t M o k L w d j G 9 m 9 f Y T 1 0 b E 6 g E n C f c j O l Q i F I y i t e 4 f + 9 g v V 9 y q O x d Z B S + H C u R q 9 M t f v U H M 0 o g r Z J I a 0 / X c B P 2 M a h R M 8 m m p l x q e U D a m Q 9 6 1 q G j E j Z / N V 5 2 S M + s M S B h r + x S S u f t 7 I q O R M Z M o s J 0 R x Z F Z r s 3 M / 2 r d F M M r P x M q S Z E r t v g o T C X B m M z u J g O h O U M 5 s U C Z F n Z X w k Z U U 4 Y 2 n Z I N w V s + e R V a F 1 X P 8 t 1 l p X 6 d x 1 G E E z i F c / C g B n W 4 h Q Y 0 g c E Q n u E V 3 h z p v D j v z s e i t e D k M 8 f w R 8 7 n D 0 I E j c Q = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F 7 g D Z 6 i U 8 b T j L s P a E x R 5 u f v / 6 w U = " > A A A B 6 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j R f s h b S i b 7 a Z d u t m E 3 Y l Q Q n + C F w + K e P U X e f P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u F t f W N z a 3 i d m l n d 2 / / o H x 4 1 D J x q h l v s l j G u h N Q w 6 V Q v I k C J e 8 k m t M o k L w d j G 9 m 9 f Y T 1 0 b E 6 g E n C f c j O l Q i F I y i t e 4 f + 9 g v V 9 y q O x d Z B S + H C u R q 9 M t f v U H M 0 o g r Z J I a 0 / X c B P 2 M a h R M 8 m m p l x q e U D a m Q 9 6 1 q G j E j Z / N V 5 2 S M + s M S B h r + x S S u f t 7 I q O R M Z M o s J 0 R x Z F Z r s 3 M / 2 r d F M M r P x M q S Z E r t v g o T C X B m M z u J g O h O U M 5 s U C Z F n Z X w k Z U U 4 Y 2 n Z I N w V s + e R V a F 1 X P 8 t 1 l p X 6 d x 1 G E E z i F c / C g B n W 4 h Q Y 0 g c E Q n u E V 3 h z p v D j v z s e i t e D k M 8 f w R 8 7 n D 0 I E j c Q = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F 7 g D Z 6 i U 8 b T j L s P a E x R 5 u f v / 6 w U = " > A A A B 6 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j R f s h b S i b 7 a Z d u t m E 3 Y l Q Q n + C F w + K e P U X e f P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u F t f W N z a 3 i d m l n d 2 / / o H x 4 1 D J x q h l v s l j G u h N Q w 6 V Q v I k C J e 8 k m t M o k L w d j G 9 m 9 f Y T 1 0 b E 6 g E n C f c j O l Q i F I y i t e 4 f + 9 g v V 9 y q O x d Z B S + H C u R q 9 M t f v U H M 0 o g r Z J I a 0 / X c B P 2 M a h R M 8 m m p l x q e U D a m Q 9 6 1 q G j E j Z / N V 5 2 S M + s M S B h r + x S S u f t 7 I q O R M Z M o s J 0 R x Z F Z r s 3 M / 2 r d F M M r P x M q S Z E r t v g o T C X B m M z u J g O h O U M 5 s U C Z F n Z X w k Z U U 4 Y 2 n Z I N w V s + e R V a F 1 X P 8 t 1 l p X 6 d x 1 G E E z i F c / C g B n W 4 h Q Y 0 g c E Q n u E V 3 h z p v D j v z s e i t e D k M 8 f w R 8 7 n D 0 I E j c Q = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F 7 g D Z 6 i U 8 b T j L s P a E x R 5 u f v / 6 w U = " > A A A B 6 n i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 F Q S E e q x 6 M V j R f s h b S i b 7 a Z d u t m E 3 Y l Q Q n + C F w + K e P U X e f P f u G 1 z 0 N Y X F h 7 e m W F n 3 i C R w q D r f j u F t f W N z a 3 i d m l n d 2 / / o H x 4 1 D J x q h l v s l j G u h N Q w 6 V Q v I k C J e 8 k m t M o k L w d j G 9 m 9 f Y T 1 0 b E 6 g E n C f c j O l Q i F I y i t e 4 f + 9 g v V 9 y q O x d Z B S + H C u R q 9 M t f v U H M 0 o g r Z J I a 0 / X c B P 2 M a h R M 8 m m p l x q e U D a m Q 9 6 1 q G j E j Z / N V 5 2 S M + s M S B h r + x S S u f t 7 I q O R M Z M o s J 0 R x Z F Z r s 3 M / 2 r d F M M r P x M q S Z E r t v g o T C X B m M z u J g O h O U M 5 s U C Z F n Z X w k Z U U 4 Y 2 n Z I N w V s + e R V a F 1 X P 8 t 1 l p X 6 d x 1 G E E z i F c / C g B n W 4 h Q Y 0 g c E Q n u E V 3 h z p v D j v z s e i t e D k M 8 f w R 8 7 n D 0 I E j c Q = < / l a t e x i t > Xt < l a t e x i t s h a 1 _ b a s e 6 4 = " y B i G P l U R I e I P n k T u 7 H a U v F o t z d o = " > A A A B 8 n i c b Z B N S 8 N A E I Y 3 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g q 2 F N J T N Z t I u 3 W T D 7 k Q o o T / D i w d F v P p r v P l v 3 L Y 5 a O s L C w / v z L A z b 5 h J Y d B 1 v 5 3 K 2 v r G 5 l Z 1 u 7 a z u 7 d / U D 8 8 6 h q V a w 4 d r q T S v Z A Z k C K F D g q U 0 M s 0 s C S U 8 B i O b 2 f 1 x y f Q R q j 0 A S c Z B A k b p i I W n K G 1 / D 4 K G U H R m w 5 w U G + 4 T X c u u g p e C Q 1 S q j 2 o f / U j x f M E U u S S G e N 7 b o Z B w T Q K L m F a 6 + c G M s b H b A i + x Z Q l Y I J i v v K U n l k n o r H S 9 q V I 5 + 7 v i Y I l x k y S 0 H Y m D E d m u T Y z / 6 v 5 O c b X Q S H S L E d I + e K j O J c U F Z 3 d T y O h g a O c W G B c C 7 s r 5 S O m G U e b U s 2 G 4 C 2 f v A r d i 6 Z n + f 6 y 0 b o p 4 6 i S E 3 J K z o l H r k i L 3 J E 2 6 R B O F H k m r + T N Q e f F e X c + F q 0 V p 5 w 5 J n / k f P 4 A p F + R e Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y B i G P l U R I e I P n k T u 7 H a U v F o t z d o = " > A A A B 8 n i c b Z B N S 8 N A E I Y 3 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g q 2 F N J T N Z t I u 3 W T D 7 k Q o o T / D i w d F v P p r v P l v 3 L Y 5 a O s L C w / v z L A z b 5 h J Y d B 1 v 5 3 K 2 v r G 5 l Z 1 u 7 a z u 7 d / U D 8 8 6 h q V a w 4 d r q T S v Z A Z k C K F D g q U 0 M s 0 s C S U 8 B i O b 2 f 1 x y f Q R q j 0 A S c Z B A k b p i I W n K G 1 / D 4 K G U H R m w 5 w U G + 4 T X c u u g p e C Q 1 S q j 2 o f / U j x f M E U u S S G e N 7 b o Z B w T Q K L m F a 6 + c G M s b H b A i + x Z Q l Y I J i v v K U n l k n o r H S 9 q V I 5 + 7 v i Y I l x k y S 0 H Y m D E d m u T Y z / 6 v 5 O c b X Q S H S L E d I + e K j O J c U F Z 3 d T y O h g a O c W G B c C 7 s r 5 S O m G U e b U s 2 G 4 C 2 f v A r d i 6 Z n + f 6 y 0 b o p 4 6 i S E 3 J K z o l H r k i L 3 J E 2 6 R B O F H k m r + T N Q e f F e X c + F q 0 V p 5 w 5 J n / k f P 4 A p F + R e Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y B i G P l U R I e I P n k T u 7 H a U v F o t z d o = " > A A A B 8 n i c b Z B N S 8 N A E I Y 3 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g q 2 F N J T N Z t I u 3 W T D 7 k Q o o T / D i w d F v P p r v P l v 3 L Y 5 a O s L C w / v z L A z b 5 h J Y d B 1 v 5 3 K 2 v r G 5 l Z 1 u 7 a z u 7 d / U D 8 8 6 h q V a w 4 d r q T S v Z A Z k C K F D g q U 0 M s 0 s C S U 8 B i O b 2 f 1 x y f Q R q j 0 A S c Z B A k b p i I W n K G 1 / D 4 K G U H R m w 5 w U G + 4 T X c u u g p e C Q 1 S q j 2 o f / U j x f M E U u S S G e N 7 b o Z B w T Q K L m F a 6 + c G M s b H b A i + x Z Q l Y I J i v v K U n l k n o r H S 9 q V I 5 + 7 v i Y I l x k y S 0 H Y m D E d m u T Y z / 6 v 5 O c b X Q S H S L E d I + e K j O J c U F Z 3 d T y O h g a O c W G B c C 7 s r 5 S O m G U e b U s 2 G 4 C 2 f v A r d i 6 Z n + f 6 y 0 b o p 4 6 i S E 3 J K z o l H r k i L 3 J E 2 6 R B O F H k m r + T N Q e f F e X c + F q 0 V p 5 w 5 J n / k f P 4 A p F + R e Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y B i G P l U R I e I P n k T u 7 H a U v F o t z d o = " > A A A B 8 n i c b Z B N S 8 N A E I Y 3 9 a v W r 6 p H L 4 t F 8 F Q S E f R Y 9 O K x g q 2 F N J T N Z t I u 3 W T D 7 k Q o o T / D i w d F v P p r v P l v 3 L Y 5 a O s L C w / v z L A z b 5 h J Y d B 1 v 5 3 K 2 v r G 5 l Z 1 u 7 a z u 7 d / U D 8 8 6 h q V a w 4 d r q T S v Z A Z k C K F D g q U 0 M s 0 s C S U 8 B i O b 2 f 1 x y f Q R q j 0 A S c Z B A k b p i I W n K G 1 / D 4 K G U H R m w 5 w U G + 4 T X c u u g p e C Q 1 S q j 2 o f / U j x f M E U u S S G e N 7 b o Z B w T Q K L m F a 6 + c G M s b H b A i + x Z Q l Y I J i v v K U n l k n o r H S 9 q V I 5 + 7 v i Y I l x k y S 0 H Y m D E d m u T Y z / 6 v 5 O c b X Q S H S L E d I + e K j O J c U F Z 3 d T y O h g a O c W G B c C 7 s r 5 S O m G U e b U s 2 G 4 C 2 f v A r d i 6 Z n + f 6 y 0 b o p 4 6 i S E 3 J K z o l H r k i L 3 J E 2 6 R B O F H k m r + T N Q e f F e X c + F q 0 V p 5 w 5 J n / k f P 4 A p F + R e Q = = < / l a t e x i t > Xt+1 < l a t e x i t s h a 1 _ b a s e 6 4 = " p M u / H P v 6 R x e V D L q W 7 K b 5 U 8 X e 7 g E = " > A A A B 9 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F E I S S i K D H o h e P F e w H t K F s t t t 2 6 W Y T d y e F E v I 7 v H h Q x K s / x p v / x m 2 b g 7 a + s P D w z g w z + w a x F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 6 a J E s 1 4 g 0 U y 0 u 2 A G i 6 F 4 g 0 U K H k 7 1 p y G g e S t Y H w 3 q 7 c m X B s R q U e c x t w P 6 V C J g W A U r e V 3 R x T T d t Z L 8 c L L e u W K W 3 X n I q v g 5 V C B X P V e + a v b j 1 g S c o V M U m M 6 n h u j n 1 K N g k m e l b q J 4 T F l Y z r k H Y u K h t z 4 6 f z o j J x Z p 0 8 G k b Z P I Z m 7 v y d S G h o z D Q P b G V I c m e X a z P y v 1 k l w c O O n Q s U J c s U W i w a J J B i R W Q K k L z R n K K c W K N P C 3 k r Y i G r K 0 O Z U s i F 4 y 1 9 e h e Z l 1 b P 8 c F W p 3 e Z x F O E E T u E c P L i G G t x D H R r A 4

A m e 4 R X e n I n z 4 r w 7 H 4 v W g p P P H M M f O Z 8 / t 9 S S D A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " p M u / H P v 6 R x e V D L q W 7 K b 5 U 8 X e 7 g E = "

> A A A B 9 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F E I S S i K D H o h e P F e w H t K F s t t t 2 6 W Y T d y e F E v I 7 v H h Q x K s / x p v / x m 2 b g 7 a + s P D w z g w z + w a x F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 6 a J E s 1 4 g 0 U y 0 u 2 A G i 6 F 4 g 0 U K H k 7 1 p y G g e S t Y H w 3 q 7 c m X B s R q U e c x t w P 6 V C J g W A U r e V 3 R x T T d t Z L 8 c L L e u W K W 3 X n I q v g 5 V C B X P V e + a v b j 1 g S c o V M U m M 6 n h u j n 1 K N g k m e l b q J 4 T F l Y z r k H Y u K h t z 4 6 f z o j J x Z p 0 8 G k b Z P I Z m 7 v y d S G h o z D Q P b G V I c m e X a z P y v 1 k l w c O O n Q s U J c s U W i w a J J B i R W Q K k L z R n K K c W K N P C 3 k r Y i G r K 0 O Z U s i F 4 y 1 9 e h e Z l 1 b P 8 c F W p 3 e Z x F O E E T u E c P L i G G t x D H R r A 4

A m e 4 R X e n I n z 4 r w 7 H 4 v W g p P P H M M f O Z 8 / t 9 S S D A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " p M u / H P v 6 R x e V D L q W 7 K b 5 U 8 X e 7 g E = "

> A A A B 9 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F E I S S i K D H o h e P F e w H t K F s t t t 2 6 W Y T d y e F E v I 7 v H h Q x K s / x p v / x m 2 b g 7 a + s P D w z g w z + w a x F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 6 a J E s 1 4 g 0 U y 0 u 2 A G i 6 F 4 g 0 U K H k 7 1 p y G g e S t Y H w 3 q 7 c m X B s R q U e c x t w P 6 V C J g W A U r e V 3 R x T T d t Z L 8 c L L e u W K W 3 X n I q v g 5 V C B X P V e + a v b j 1 g S c o V M U m M 6 n h u j n 1 K N g k m e l b q J 4 T F l Y z r k H Y u K h t z 4 6 f z o j J x Z p 0 8 G k b Z P I Z m 7 v y d S G h o z D Q P b G V I c m e X a z P y v 1 k l w c O O n Q s U J c s U W i w a J J B i R W Q K k L z R n K K c W K N P C 3 k r Y i G r K 0 O Z U s i F 4 y 1 9 e h e Z l 1 b P 8 c F W p 3 e Z x F O E E T u E c P L i G G t x D H R r A 4

A m e 4 R X e n I n z 4 r w 7 H 4 v W g p P P H M M f O Z 8 / t 9 S S D A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " p M u / H P v 6 R x e V D L q W 7 K b 5 U 8 X e 7 g E = "

> A A A B 9 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F E I S S i K D H o h e P F e w H t K F s t t t 2 6 W Y T d y e F E v I 7 v H h Q x K s / x p v / x m 2 b g 7 a + s P D w z g w z + w a x F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 6 a J E s 1 4 g 0 U y 0 u 2 A G i 6 F 4 g 0 U K H k 7 1 p y G g e S t Y H w 3 q 7 c m X B s R q U e c x t w P 6 V C J g W A U r e V 3 R x T T d t Z L 8 c L L e u W K W 3 X n I q v g 5 V C B X P V e + a v b j 1 g S c o V M U m M 6 n h u j n 1 K N g k m e l b q J 4 T F l Y z r k H Y u K h t z 4 6 f z o j J x Z p 0 8 G k b Z P I Z m 7 v y d S G h o z D Q P b G V I c m e X a z P y v 1 k l w c O O n Q s U J c s U W i w a J J B i R W Q K k L z R n K K c W K N P C 3 k r Y i G r K 0 O Z U s i F 4 y 1 9 e h e Z l 1 b P 8 c F W p 3 e Z x F O E E T u E c P L i G G t x D H R r A 4

A m e 4 R X e n I n z 4 r w 7 H 4 v W g p P P H M M f O Z 8 / t 9 S S D A = = < / l a t e x i t >

Ŷt+1:

< l a t e x i t s h a 1 _ b a s e 6 4 = " s k U 5 i l 6 Z P G

A P d x E j E S 9 C W 9 f x 0 E s = " > A A A B 9 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I g l A S E f R Y 9 O K x g v 2 Q N p T N d t s u 3 W z i 7 q R Q Q n 6 H F w + K e P X H e P P f u G 1 z 0 N Y X F h 7 e m W F m 3 y C W w q D r f j s r q 2 v r G 5 u F r e L 2 z u 7 e f u n g s G G i R D N e Z 5 G M d C u g h k u h e B 0 F S t 6 K N a d h I H k z G N 1 O 6 8 0 x 1 0 Z E 6 g E n M f d D O l C i L x h F a / m d I c X 0 M e u m e O 5 l 3 V L Z r b g z k W X w c i h D r l q 3 9 N X p R S w J u U I m q T F t z 4 3 R T 6 l G w S T P i p 3 E 8 J i y E R 3 w t k V F Q 2 7 8 d H Z 0 R k 6 t 0 y P 9 S N u n k M z c 3 x M p D Y 2 Z h I H t D C k O z W J t a v 5 X a y f Y v / Z T o e I E u W L z R f 1 E E o z I N A H S E 5 o z l B M L l G l h b y V s S D V l a H M q 2 h C 8 x S 8 v Q + O i 4 l m + v y x X b / I 4 C n A M J 3 A G H l x B F e 6 g B n V g 8 A T P 8 A p v z t h 5 c d 6 d j 3 n r i p P P H M E f O Z 8 / u V + S D Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " s k U 5 i l 6 Z P G A P d x E j E S 9 C W 9 f x 0 E s = " > A A A B 9 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I g l A S E f R Y 9 O K x g v 2 Q N p T N d t s u 3 W z i 7 q R Q Q n 6 H F w + K e P X H e P P f u G 1 z 0 N Y X F h 7 e m W F m 3 y C W w q D r f j s r q 2 v r G 5 u F r e L 2 z u 7 e f u n g s G G i R D N e Z 5 G M d C u g h k u h e B 0 F S t 6 K N a d h I H k z G N 1 O 6 8 0 x 1 0 Z E 6 g E n M f d D O l C i L x h F a / m d I c X 0 M e u m e O 5 l 3 V L Z r b g z k W X w c i h D r l q 3 9 N X p R S w J u U I m q T F t z 4 3 R T 6 l G w S T P i p 3 E 8 J i y E R 3 w t k V F Q 2 7 8 d H Z 0 R k 6 t 0 y P 9 S N u n k M z c 3 x M p D Y 2 Z h I H t D C k O z W J t a v 5 X a y f Y v / Z T o e I E u W L z R f 1 E E o z I N A H S E 5 o z l B M L l G l h b y V s S D V l a H M q 2 h C 8 x S 8 v Q + O i 4 l m + v y x X b / I 4 C n A M J 3 A G H l x B F e 6 g B n V g 8 A T P 8 A p v z t h 5 c d 6 d j 3 n r i p P P H M E f O Z 8 / u V + S D Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " s k U 5 i l 6 Z P G A P d x E j E S 9 C W 9 f x 0 E s = " > A A A B 9 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I g l A S E f R Y 9 O K x g v 2 Q N p T N d t s u 3 W z i 7 q R Q Q n 6 H F w + K e P X H e P P f u G 1 z 0 N Y X F h 7 e m W F m 3 y C W w q D r f j s r q 2 v r G 5 u F r e L 2 z u 7 e f u n g s G G i R D N e Z 5 G M d C u g h k u h e B 0 F S t 6 K N a d h I H k z G N 1 O 6 8 0 x 1 0 Z E 6 g E n M f d D O l C i L x h F a / m d I c X 0 M e u m e O 5 l 3 V L Z r b g z k W X w c i h D r l q 3 9 N X p R S w J u U I m q T F t z 4 3 R T 6 l G w S T P i p 3 E 8 J i y E R 3 w t k V F Q 2 7 8 d H Z 0 R k 6 t 0 y P 9 S N u n k M z c 3 x M p D Y 2 Z h I H t D C k O z W J t a v 5 X a y f Y v / Z T o e I E u W L z R f 1 E E o z I N A H S E 5 o z l B M L l G l h b y V s S D V l a H M q 2 h C 8 x S 8 v Q + O i 4 l m + v y x X b / I 4 C n A M J 3 A G H l x

B F e 6 g B n V g 8 A T P 8 A p v z t h 5 c d 6 d j 3 n r i p P P H M E f O Z 8 / u V + S D Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " s k U 5 i l 6 Z P G 

A P d x E j E S 9 C W 9 f x 0 E s = " > A A A B 9 H i c b Z B N S 8 N A E I Y n f t b 6 V f X o Z b E I g l A S E f R Y 9 O K x g v 2 Q N p T N d t s u 3 W z i 7 q R Q Q n 6 H F w + K e P X H e P P f u G 1 z 0 N Y X F h 7 e m W F m 3 y C W w q D r f j s r q 2 v r G 5 u F r e L 2 z u 7 e f u n g s G G i R D N e Z 5 G M d C u g h k u h e B 0 F S t 6 K N a d h I H k z G N 1 O 6 8 0 x 1 0 Z E 6 g E n M f d D O l C i L x h F a / m d I c X 0 M e u m e O 5 l 3 V L Z r b g z k W X w c i h D r l q 3 9 N X p R S w J u U I m q T F t z 4 3 R T 6 l G w S T P i p 3 E 8 J i y E R 3 w t k V F Q 2 7 8 d H Z 0 R k 6 t 0 y P 9 S N u n k M z c 3 x M p D Y 2 Z h I H t D C k O z W J t a v 5 X a y f Y v / Z T o e I E u W L z R f 1 E E o z I N A H S E 5 o z l B M L l G l h b y V s S D V l a H M q 2 h C 8 x S 8 v Q + O i 4 l m + v y x X b / I 4 C n A M J 3 A G H l x B F

U W z Z I k b Y G j E v + h o = " > A A A B 7 X i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i Q i 6 L L o x m U F + 4 A 2 l M l 0 0 o 6 d Z M L M j V B C / 8 G N C 0 X c + j / u / B s n b R b a e m D g c M 4 9 z L 0 n S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 j U o 1 4 y 2 m p N L d g B o u R c x b K F D y b q I 5 j Q L J O 8 H k N v c 7 T 1 w b o e I H n C b c j + g o F q F g F K 3 U 7 g 8 V m s q g W n P r 7 h x k l X g F q U G B 5 q D 6 Z Y M s j X i M T F J j e p 6 b o J 9 R j Y J J P q v 0 U 8 M T y i Z 0 x H u W x j T i x s / m 2 8 7 I m V W G J F T a v h j J X P 2 d y G h k z D Q K 7 G R E c W y W v V z 8 z + u l G F 7 7 m Y i T F H n M F h + F q S S o S H 4 6 G Q r N G c q p J Z R p Y X c l b E w 1 Z W g L y k v w l k 9 e J e 2 L u m f 5 / W W t c V P U U Y Y T O I V z 8 O A K G n A H T W g B g 0 d 4 h l d 4 c 5 T z 4 r w 7 H 4 v R k l N k j u E P n M 8 f J 6 O O 1 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S s r N a 5 n e 1 f M 1 U W z Z I k b Y G j E v + h o = " > A A A B 7 X i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i Q i 6 L L o x m U F + 4 A 2 l M l 0 0 o 6 d Z M L M j V B C / 8 G N C 0 X c + j / u / B s n b R b a e m D g c M 4 9 z L 0 n S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 j U o 1 4 y 2 m p N L d g B o u R c x b K F D y b q I 5 j Q L J O 8 H k N v c 7 T 1 w b o e I H n C b c j + g o F q F g F K 3 U 7 g 8 V m s q g W n P r 7 h x k l X g F q U G B 5 q D 6 Z Y M s j X i M T F J j e p 6 b o J 9 R j Y J J P q v 0 U 8 M T y i Z 0 x H u W x j T i x s / m 2 8 7 I m V W G J F T a v h j J X P 2 d y G h k z D Q K 7 G R E c W y W v V z 8 z + u l G F 7 7 m Y i T F H n M F h + F q S S o S H 4 6 G Q r N G c q p J Z R p Y X c l b E w 1 Z W g L y k v w l k 9 e J e 2 L u m f 5 / W W t c V P U U Y Y T O I V z 8 O A K G n A H T W g B g 0 d 4 h l d 4 c 5 T z 4 r w 7 H 4 v R k l N k j u E P n M 8 f J 6 O O 1 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S s r N a 5 n e 1 f M 1 U W z Z I k b Y G j E v + h o = " > A A A B 7 X i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i Q i 6 L L o x m U F + 4 A 2 l M l 0 0 o 6 d Z M L M j V B C / 8 G N C 0 X c + j / u / B s n b R b a e m D g c M 4 9 z L 0 n S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 j U o 1 4 y 2 m p N L d g B o u R c x b K F D y b q I 5 j Q L J O 8 H k N v c 7 T 1 w b o e I H n C b c j + g o F q F g F K 3 U 7 g 8 V m s q g W n P r 7 h x k l X g F q U G B 5 q D 6 Z Y M s j X i M T F J j e p 6 b o J 9 R j Y J J P q v 0 U 8 M T y i Z 0 x H u W x j T i x s / m 2 8 7 I m V W G J F T a v h j J X P 2 d y G h k z D Q K 7 G R E c W y W v V z 8 z + u l G F 7 7 m Y i T F H n M F h + F q S S o S H 4 6 G Q r N G c q p J Z R p Y X c l b E w 1 Z W g L y k v w l k 9 e J e 2 L u m f 5 / W W t c V P U U Y Y T O I V z 8 O A K G n A H T W g B g 0 d 4 h l d 4 c 5 T z 4 r w 7 H 4 v R k l N k j u E P n M 8 f J 6 O O 1 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S s r N a 5 n e 1 f M 1 U W z Z I k b Y G j E v + h o = " > A A A B 7 X i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i Q i 6 L L o x m U F + 4 A 2 l M l 0 0 o 6 d Z M L M j V B C / 8 G N C 0 X c + j / u / B s n b R b a e m D g c M 4 9 z L 0 n S K Q w 6 L r f T m l t f W N z q 7 x d 2 d n d 2 z + o H h 6 1 j U o 1 4 y 2 m p N L d g B o u R c x b K F D y b q I 5 j Q L J O 8 H k N v c 7 T 1 w b o e I H n C b c j + g o F q F g F K 3 U 7 g 8 V m s q g W n P r 7 h x k l X g F q U G B 5 q D 6 Z Y M s j X i M T F J j e p 6 b o J 9 R j Y J J P q v 0 U 8 M T y i Z 0 x H u W x j T i x s / m 2 8 7 I m V W G J F T a v h j J X P 2 d y G h k z D Q K 7 G R E c W y W v V z 8 z + u l G F 7 7 m Y i T F H n M F h + F q S S o S H 4 6 G Q r N G c q p J Z R p Y X c l b E w 1 Z W g L y k v w l k 9 e J e 2 L u m f 5 / W W t c V P U U Y Y T O I V z 8 O A K G n A H T W g B g 0 d 4 h l d 4 c 5 T z 4 r w 7 H 4 v R k l N k j u E P n M 8 f J 6 O O 1 g = = < / l a t e x i t > f ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 P b J Y Q C 2 D p Z 4 0 u i + E Q o A 4 X Q + 8 b U = " > A A A B 8 X i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 8 E i e C q J C H o s e v F Y w X 5 g W 8 p m O 2 m X b j Z h d y K U 0 H / h x Y M i X v 0 3 3 v w 3 b t s c t P W F h Y d 3 Z t i Z N 0 i k M O R 5 3 0 5 h b X 1 j c 6 u 4 X d r Z 3 d s / K B 8 e N U 2 c a o 4 N H s t Y t w N m U A q F D R I k s Z 1 o Z F E g s R W M b 2 f 1 1 h N q I 2 L 1 Q J M E e x E b K h E K z s h a j 2 E / 6 9 I I i U 3 7 5 Y p X 9 e Z y V 8 H P o Q K 5 6 v 3 y V 3 c Q 8 z R C R V w y Y z q + l 1 A v Y 5 o E l z g t d V O D C e N j N s S O R c U i N L 1 s v v H U P b P O w A 1 j b Z 8 i d + 7 + n s h Y Z M w k C m x n x G h k l m s z 8 7 9 a J 6 X w u p c J l a S E i i 8 + C l P p U u z O z n c H Q i M n O b H A u B Z 2

V 5 e P m G a c b E g l G 4 K / f P I q N C + q v u X 7 y 0 r t J o + j C C d w C u f g w x X U 4 A 7 q 0 A A O C p 7 h F d 4 c 4 7 w 4 7 8 7 H o r X g 5 D P H 8 E f O 5 w / m u 5 E N < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 P

b J Y Q C 2 D p Z 4 0 u i + E Q o A 4 X Q + 8 b U = " > A A A B 8 X i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 8 E i e C q J C H o s e v F Y w X 5 g W 8 p m O 2 m X b j Z h d y K U 0 H / h x Y M i X v 0 3 3 v w 3 b t s c t P W F h Y d 3 Z t i Z N 0 i k M O R 5 3 0 5 h b X 1 j c 6 u 4 X d r Z 3 d s / K B 8 e N U 2 c a o 4 N H s t Y t w N m U A q F D R I k s Z 1 o Z F E g s R W M b 2 f 1 1 h N q I 2 L 1 Q J M E e x E b K h E K z s h a j 2 E / 6 9 I I i U 3 7 5 Y p X 9 e Z y V 8 H P o Q K 5 6 v 3 y V 3 c Q 8 z R C R V w y Y z q + l 1 A v Y 5 o E l z g t d V O D C e N j N s S O R c U i N L 1 s v v H U P b P O w A 1 j b Z 8 i d + 7 + n s h Y Z M w k C m x n x G h k l m s z 8 7 9 a J 6 X w u p c J l a S E i i 8 + C l P p U u z O z n c H Q i M n O b H A u B Z 2

V 5 e P m G a c b E g l G 4 K / f P I q N C + q v u X 7 y 0 r t J o + j C C d w C u f g w x X U 4 A 7 q 0 A A O C p 7 h F d 4 c 4 7 w 4 7 8 7 H o r X g 5 D P H 8 E f O 5 w / m u 5 E N < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 P

b J Y Q C 2 D p Z 4 0 u i + E Q o A 4 X Q + 8 b U = " > A A A B 8 X i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 8 E i e C q J C H o s e v F Y w X 5 g W 8 p m O 2 m X b j Z h d y K U 0 H / h x Y M i X v 0 3 3 v w 3 b t s c t P W F h Y d 3 Z t i Z N 0 i k M O R 5 3 0 5 h b X 1 j c 6 u 4 X d r Z 3 d s / K B 8 e N U 2 c a o 4 N H s t Y t w N m U A q F D R I k s Z 1 o Z F E g s R W M b 2 f 1 1 h N q I 2 L 1 Q J M E e x E b K h E K z s h a j 2 E / 6 9 I I i U 3 7 5 Y p X 9 e Z y V 8 H P o Q K 5 6 v 3 y V 3 c Q 8 z R C R V w y Y z q + l 1 A v Y 5 o E l z g t d V O D C e N j N s S O R c U i N L 1 s v v H U P b P O w A 1 j b Z 8 i d + 7 + n s h Y Z M w k C m x n x G h k l m s z 8 7 9 a J 6 X w u p c J l a S E i i 8 + C l P p U u z O z n c H Q i M n O b H A u B Z 2

V 5 e P m G a c b E g l G 4 K / f P I q N C + q v u X 7 y 0 r t J o + j C C d w C u f g w x X U 4 A 7 q 0 A A O C p 7 h F d 4 c 4 7 w 4 7 8 7 H o r X g 5 D P H 8 E f O 5 w / m u 5 E N < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 P

b J Y Q C 2 D p Z 4 0 u i + E Q o A 4 X Q + 8 b U = " > A A A B 8 X i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 8 E i e C q J C H o s e v F Y w X 5 g W 8 p m O 2 m X b j Z h d y K U 0 H / h x Y M i X v 0 3 3 v w 3 b t s c t P W F h Y d 3 Z t i Z N 0 i k M O R 5 3 0 5 h b X 1 j c 6 u 4 X d r Z 3 d s / K B 8 e N U 2 c a o 4 N H s t Y t w N m U A q F D R I k s Z 1 o Z F E g s R W M b 2 f 1 1 h N q I 2 L 1 Q J M E e x E b K h E K z s h a j 2 E / 6 9 I I i U 3 7 5 Y p X 9 e Z y V 8 H P o Q K 5 6 v 3 y V 3 c Q 8 z R C R V w y Y z q + l 1 A v Y 5 o E l z g t d V O D C e N j N s S O R c U i N L 1 s v v H U P b P O w A 1 j b Z 8 i d + 7 + n s h Y Z M w k C m x n x G h k l m s z 8 7 9 a J 6 X w u p c J l a S E i i 8 + C l P p U u z O z n c H Q i M n O b H A u B Z 2

V 5 e P m G a c b E g l G 4 K / f P I q N C + q v u X 7 y 0 r t J o + j C C d w C u f g w x X U 4 A 7 q 0 A A O C p 7 h F d 4 c 4 7 w 4 7 8 7 H o r X g 5 D P H 8 E f O 5 w / m u 5 E N < / l a t e x i t > H < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 • Single State Estimation (SSE) : the estimation operator e is used only once in order to obtain an initial state X t for the forecasting operator:

V 2 1 3 g O k z v R h Z 9 0 W d k h L r P Z l D p o = " > A A A B 8 n i c b V D L S s N A F L 2 p r 1 p f V Z d u g k V w V R I R d F l 0 0 2 U F + 4 A 2 l M l 0 0 g 6 d z I S Z G 6 G E f o Y b F 4 q 4 9 W v c + T d O 2 i y 0 9 c D A 4 Z x 7 m X N P m A h u 0 P O + n d L G 5 t b 2 T n m 3 s r d / c H h U P T 7 p G J V q y t p U C a V 7 I T F M c M n a y F G w X q I Z i U P B u u H 0 P v e 7 T 0 w b r u Q j z h I W x G Q s e c Q p Q S v 1 B z H B C S U i a 8 6 H 1 Z p X 9 x Z w 1 4 l f k B o U a A 2 r X 4 O R o m n M J F J B j O n 7 X o J B R j R y K t i 8 M k g N S w i d k j H r W y p J z E y Q L S L P 3 Q u r j N x I a f s k u g v 1 9 0 Z G Y m N m c W g n 8 4 h m 1 c v F / 7 x + i t F t k H G Z p M g k X X 4 U p c J F 5 e b 3 u y O u G U U x s 4 R Q z W 1 W l 0 6 I J h R t S x V b g r 9 6 8 j r p X N V 9 y x + u a 4 2 7 o o 4 y n M E 5 X I I P N 9 C A J r S g D R Q U P M M r v D n o v D j v z s d y t O Q U O 6 f w B 8 7 n D 3 r z k V 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 V 2 1 3 g O k z v R h Z 9 0 W d k h L r P Z l D p o = " > A A A B 8 n i c b V D L S s N A F L 2 p r 1 p f V Z d u g k V w V R I R d F l 0 0 2 U F + 4 A 2 l M l 0 0 g 6 d z I S Z G 6 G E f o Y b F 4 q 4 9 W v c + T d O 2 i y 0 9 c D A 4 Z x 7 m X N P m A h u 0 P O + n d L G 5 t b 2 T n m 3 s r d / c H h U P T 7 p G J V q y t p U C a V 7 I T F M c M n a y F G w X q I Z i U P B u u H 0 P v e 7 T 0 w b r u Q j z h I W x G Q s e c Q p Q S v 1 B z H B C S U i a 8 6 H 1 Z p X 9 x Z w 1 4 l f k B o U a A 2 r X 4 O R o m n M J F J B j O n 7 X o J B R j R y K t i 8 M k g N S w i d k j H r W y p J z E y Q L S L P 3 Q u r j N x I a f s k u g v 1 9 0 Z G Y m N m c W g n 8 4 h m 1 c v F / 7 x + i t F t k H G Z p M g k X X 4 U p c J F 5 e b 3 u y O u G U U x s 4 R Q z W 1 W l 0 6 I J h R t S x V b g r 9 6 8 j r p X N V 9 y x + u a 4 2 7 o o 4 y n M E 5 X I I P N 9 C A J r S g D R Q U P M M r v D n o v D j v z s d y t O Q U O 6 f w B 8 7 n D 3 r z k V 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 V 2 1 3 g O k z v R h Z 9 0 W d k h L r P Z l D p o = " > A A A B 8 n i c b V D L S s N A F L 2 p r 1 p f V Z d u g k V w V R I R d F l 0 0 2 U F + 4 A 2 l M l 0 0 g 6 d z I S Z G 6 G E f o Y b F 4 q 4 9 W v c + T d O 2 i y 0 9 c D A 4 Z x 7 m X N P m A h u 0 P O + n d L G 5 t b 2 T n m 3 s r d / c H h U P T 7 p G J V q y t p U C a V 7 I T F M c M n a y F G w X q I Z i U P B u u H 0 P v e 7 T 0 w b r u Q j z h I W x G Q s e c Q p Q S v 1 B z H B C S U i a 8 6 H 1 Z p X 9 x Z w 1 4 l f k B o U a A 2 r X 4 O R o m n M J F J B j O n 7 X o J B R j R y K t i 8 M k g N S w i d k j H r W y p J z E y Q L S L P 3 Q u r j N x I a f s k u g v 1 9 0 Z G Y m N m c W g n 8 4 h m 1 c v F / 7 x + i t F t k H G Z p M g k X X 4 U p c J F 5 e b 3 u y O u G U U x s 4 R Q z W 1 W l 0 6 I J h R t S x V b g r 9 6 8 j r p X N V 9 y x + u a 4 2 7 o o 4 y n M E 5 X I I P N 9 C A J r S g D R Q U P M M r v D n o v D j v z s d y t O Q U O 6 f w B 8 7 n D 3 r z k V 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 V 2 1 3 g O k z v R h Z 9 0 W d k h L r P Z l D p o = " > A A A B 8 n i c b V D L S s N A F L 2 p r 1 p f V Z d u g k V w V R I R d F l 0 0 2 U F + 4 A 2 l M l 0 0 g 6 d z I S Z G 6 G E f o Y b F 4 q 4 9 W v c + T d O 2 i y 0 9 c D A 4 Z x 7 m X N P m A h u 0 P O + n d L G 5 t b 2 T n m 3 s r d / c H h U P T 7 p G J V q y t p U C a V 7 I T F M c M n a y F G w X q I Z i U P B u u H 0 P v e 7 T 0 w b r u Q j z h I W x G Q s e c Q p Q S v 1 B z H B C S U i a 8 6 H 1 Z p X 9 x Z w 1 4 l f k B o U a A 2 r X 4 O R o m n M J F J B j O n 7 X o J B R j R y K t i 8 M k g N S w i d k j H r W y p J z E y Q L S L P 3 Q u r j N x I a f s k u g v 1 9 0 Z G Y m N m c W g n 8 4 h m 1 c v F / 7 x + i t F t k H G Z p M g k X X 4 U p c J F 5 e b 3 u y O u G U U x s 4 R Q z W 1 W l 0 6 I J h R t S x V b g r 9 6 8 j r p X N V 9 y x + u a 4 2 7 o o 4 y n M E 5 X I I P N 9 C A J r S g D R Q U P M M r v D n o v D j v z s d y t O Q U O 6 f w B 8 7 n D 3 r z k V 4 = < / l a t e x i t > e ! < l a t e x i t s h a 1 _ b a s e 6 4 = " C h g K o o d O B F W N d g h X z L H 1 T c 4 E R R 0 = " > A A A B 8 X i c b Z D L S g N B E E V r 4 i v G V 9 S l m 8 E g u A o z I u g y 6 M Z l B P P A Z A g 9 n U r S p B 9 D d 4 8 Q h v y F G x e K u P V v 3 P k 3 d p J Z a O K F h s O t K r r q x g l n x g b B t 1 d Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H T a N S T b F B F V e 6 H R O D n E l s W G Y 5 t h O N R M Q c W / H 4 d l Z v P a E 2 T M k H O 0 k w E m Q o 2 Y B R Y p 3 1 i L 2 s q w Q O y b R X r g T V Y C 5 / F c I c K p C r 3 i t / d f u K p g K l p Z w Y 0 w m D x E Y Z 0 Z Z R j t N S N z W Y E D o m Q + w 4 l E S g i b L 5 x l P / z D l 9 f 6 C 0 e 9 L 6 c / f 3 R E a E M R M R u 0 5 B 7 M g s 1 2 b m f 7 V O a g f X U c Z k k l q U d P H R I O W + V f 7 s f L / P N F L L J w 4 I 1 c z t 6 t M R 0 Y R a F 1 L J h R A u n 7 w K z Y

K o o d O B F W N d g h X z L H 1 T c 4 E R R 0 = " > A A A B 8 X i c b Z D L S g N B E E V r 4 i v G V 9 S l m 8 E g u A o z I u g y 6 M Z l B P P A Z A g 9 n U r S p B 9 D d 4 8 Q h v y F G x e K u P V v 3 P k 3 d p J Z a O K F h s O t K r r q x g l n x g b B t 1 d Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H T a N S T b F B F V e 6 H R O D n E l s W G Y 5 t h O N R M Q c W / H 4 d l Z v P a E 2 T M k H O 0 k w E m Q o 2 Y B R Y p 3 1 i L 2 s q w Q O y b R X r g T V Y C 5 / F c I c K p C r 3 i t / d f u K p g K l p Z w Y 0 w m D x E Y Z 0 Z Z R j t N S N z W Y E D o m Q + w 4 l E S g i b L 5 x l P / z D l 9 f 6 C 0 e 9 L 6 c / f 3 R E a E M R M R u 0 5 B 7 M g s 1 2 b m f 7 V O a g f X U c Z k k l q U d P H R I O W + V f 7 s f L / P N F L L J w 4 I 1 c z t 6 t M R 0 Y R a F 1 L J h R A u n 7 w K z Y

K o o d O B F W N d g h X z L H 1 T c 4 E R R 0 = " > A A A B 8 X i c b Z D L S g N B E E V r 4 i v G V 9 S l m 8 E g u A o z I u g y 6 M Z l B P P A Z A g 9 n U r S p B 9 D d 4 8 Q h v y F G x e K u P V v 3 P k 3 d p J Z a O K F h s O t K r r q x g l n x g b B t 1 d Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H T a N S T b F B F V e 6 H R O D n E l s W G Y 5 t h O N R M Q c W / H 4 d l Z v P a E 2 T M k H O 0 k w E m Q o 2 Y B R Y p 3 1 i L 2 s q w Q O y b R X r g T V Y C 5 / F c I c K p C r 3 i t / d f u K p g K l p Z w Y 0 w m D x E Y Z 0 Z Z R j t N S N z W Y E D o m Q + w 4 l E S g i b L 5 x l P / z D l 9 f 6 C 0 e 9 L 6 c / f 3 R E a E M R M R u 0 5 B 7 M g s 1 2 b m f 7 V O a g f X U c Z k k l q U d P H R I O W + V f 7 s f L / P N F L L J w 4 I 1 c z t 6 t M R 0 Y R a F 1 L J h R A u n 7 w K z Y

K o o d O B F W N d g h X z L H 1 T c 4 E R R 0 = " > A A A B 8 X i c b Z D L S g N B E E V r 4 i v G V 9 S l m 8 E g u A o z I u g y 6 M Z l B P P A Z A g 9 n U r S p B 9 D d 4 8 Q h v y F G x e K u P V v 3 P k 3 d p J Z a O K F h s O t K r r q x g l n x g b B t 1 d Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H T a N S T b F B F V e 6 H R O D n E l s W G Y 5 t h O N R M Q c W / H 4 d l Z v P a E 2 T M k H O 0 k w E m Q o 2 Y B R Y p 3 1 i L 2 s q w Q O y b R X r g T V Y C 5 / F c I c K p C r 3 i t / d f u K p g K l p Z w Y 0 w m D x E Y Z 0 Z Z R j t N S N z W Y E D o m Q + w 4 l E S g i b L 5 x l P / z D l 9 f 6 C 0 e 9 L 6 c / f 3 R E a E M R M R u 0 5 B 7 M g s 1 2 b m f 7 V O a g f X U c Z k k l q U d P H R I O W + V f 7 s f L / P N F L L J w 4 I 1 c z t 6 t M R 0 Y R a F 1 L J h R A u n 7 w K z Y

X t+l = (f θ • • • • • f θ )( X t )

• Multiple State REstimation (MSRE) : We use the forecast operator to produce the next state X t+1 = f θ ( X t ) Then, for the forecast at t + 2, we estimate the state at time t + 1 by plugging the generated observation H( X t+1 ) in the estimation operator e ω :

X t+1 = e ω (Y t-k+2 , ..., Y t , H( X t+1 ))

so that:

X t+2 = f θ ( X t+1

) and then iterate for the following steps.

In both cases, ground truth observations up to Y t+l will be used as targets in the loss term when learning to predict at horizon l.

The SSE method is the classical one in dynamical systems. However, especially when forecasting for long time horizons, it supposes that the estimator e is able to compress all relevant information about the long term evolution of the system into the initial state, which is a difficult task. On the other hand, the MSRE strategy, while less natural from the physical point of view, allows to gather information from the observations space at each step which should be an easier task.

Note that using a multi-step forecasting objective during training could be useful since it imposes additional constraints on the learned functions. This will be explored in the experimental section. When using multi-step forecasting at training time, the error is back-propagated through the l prediction steps and one has to choose a principled way of supervising the training. Moreover, we compare in appendix E different sampling strategies for long sequences training. Finally, following the analysis done in Saxe et al. (2013) , we have observed that orthogonal initialization of the weights of f is also important.

4. EXAMPLES FROM FLUID DYNAMICS : EULER AND NAVIER-STOKES:

As an illustration, we now focus on the problem of predicting the evolution of a two-dimensional fluid mechanics system. The equations governing fluids are among the most studied. One of the challenging aspects of those highly non-linear equations is the chaotic behaviour of turbulent flows. We use, as examples, instances of Euler and Navier-Stokes equations which are briefly introduced below. More information is available in appendix A, and Acheson (1989) is a comprehensive reference textbook. Euler being a simplified version of Navier Stokes, only the latter is presented below.

Navier Stokes equations for incompressible fluids are :

EQUATION

The first equation results from Newton's second law, the second comes from the conservation of density while the third is the incompressibility condition. In those equations, u denotes a twodimensional velocity field, p is a pressure scalar field, ρ is the fluid density and g is the gravitational force. ∇ • u is the divergence of velocity u, u • ∇ is the advection operator u

• ∇ = u x ∂. ∂x + u y ∂.

∂y:

Euler equations are similar to Navier Stokes but do not consider the particular nature of the fluid and ignore the viscosity term ν∇ 2 u in the first equation of ( 6) by taking ν = 0.

For Navier-Stokes, ν is the kinematic viscosity and ∇ 2 is the Lagrangian operator. The importance of viscosity in a fluid, and thus a way to assess its qualitative behaviour, cannot be measured through ν alone. The most commonly used quantity to describe it is the Reynolds number R. This is the ratio of the inertia and the viscosity terms. If R is small, it means that the fluid is very viscous, behaving more like honey ; If R is large, the viscosity term is small and the fluid behaves more like water. A definition of the Reynolds number is provided in appendix A.

Equation equation 6 is not of the same form as equation 1 as we still have the pressure variable p as well as the null divergence constraint. It is, however, possible to show that it follows the general form of equation 1 as there exists an operator P which outputs divergence-free vector fields so that, for Navier Stokes, the third equation is dropped and the first equation takes the following form :

∂u ∂t = P[ -(u • ∇)u + ν∇ 2 u]

This is similar to equation 1 so that we can put the whole system in that form with X = (ρ, u). This means that all the results and developments in previous sections apply for this family of fluid equations. We give more details about this in the appendix B1 

Note that in what follows, we have H(X) = ρ. In other words, we observe the scalar field of densities and the complete state is obtained by concatenating the 2D velocity flow u to ρ.

5. EXPERIMENTS:



5.1. DETAILS:

The architecture of e ω is based on the UNet presented in Ronneberger et al. (2015) which is a convolutional-deconvolutional network with skip connections which takes as input k feature maps corresponding to the k different observations. In all the following experiments, we have chosen k = 3. For the forecasting operator f θ , we use the ResNet architecture He et al. (2016) with Leaky ReLU non-linearities of parameter 0.1 and 6 residual blocks.

Our data is generated from a numerical simulation, using the Mantaflow fluid simulation library Thuerey & Pfaff (2018) . We study three different dynamics : the first one is governed by the Euler equations, the second and the third by the Navier-Stokes equations with R = 5000 and R = 100000. The former corresponds to a viscous fluid with complex dynamics while the latter corresponds to a low viscosity, closer to Euler.

For each setting, we produce 300 training sequences of 500 timesteps each (one timestep being of duration 0.5s), randomly selecting initial conditions for each sequence. We then subsample 5 times in time (two successive states are then separated by 2.5s). In order to tune the hyperparameters of our experiments, we generate a validation set of 200 as well as a test set of 100 sequences generated randomly and independently.

As a normalization strategy, we chose dividing each component of the state by its empirical standard deviation over the dataset. We do not set the mean to zero as the positivity of the density is an important constraint of the problem. All training horizons are set to l = 8. A more thorough exploration of this parameter is done in appendix D.

As for the baselines, we have used (1) a ResNet which takes k = 3 past observations as input to produce a future observation directly (for multi-step forecasting, we just give the produced obser-vations back to the ResNet in an auto-regressive manner), which can be seen as an instance of our framework with e being the identity operator, and (2) a Convolutional LSTM (Shi et al. (2015) ) with 2 layers (we have tried LSTMs with different numbers of layers and hidden dimensions and selected the one with the lowest validation error). In this section, we study the performances on test samples for the different methods described with Euler and Navier-Stokes dynamics. Pre-training (PT) is always done with Euler dynamics, even for the Navier-Stokes experiments.

5.2. FORECASTING:

Table 1 shows results for Euler dynamics. We can observe that :

• JT SSE clearly diverges. This is an overfitting problem that we have not been able to solve. SSE acts as an encoder-decoder architecture: all relevant information about the past observations is encoded into a single initial state, and the forecasting operator then generates a whole sequence of future state estimations. This seems to be a hard task without the additional prior given by the structure of the real physical state (which we have in the PT pre-trained setting). • All other three introduced methods along with the ResNet perform relatively well, even for long training horizons. Our algorithms beat the baselines, even if it is only by a small margin for the ResNet. • As expected, the PT SSE performs well for short to medium forecasting horizons but loses acccuracy in the long run. This is due to the fact that all information is supposed to be stored and rolled forward in a Markovian fashion while the MSRE algorithms, as well as the ResNet and ConvLSTM baselines, work in an auto-regressive fashion over the observations which makes it easier to use information from the past.

Figure 2 shows prediction examples from the different models and largely confirms the observations above. Let us also notice the difference between the SSE and the JT state representations : While the first presents a structure which is very close to the true physical state, the second has a representation which cannot be interpreted as a velocity field but is rather close to the predicted observation itself.

For the remaining part of the paper, we will focus on the PT SSE and the JT MSRE networks which seem to be the most interesting according to the observations above.

Tables 2 and 3 consider the Navier-Stokes equations. As before, we use two algorithms: a jointly trained JT MSRE and a pre-trained estimator PT SSE, in this case pre-trained using Euler equations : Euler dynamics is thus used as structural prior information for the more complex Navier Stokes dynamics. Note that this is clearly an imperfect prior, especially when the viscosity is high (i.e. when R is low). We then have a choice between keeping this imperfect estimator, hoping that the forecasting operator will make up for its shortcomings, or fine-tuning it on observations of Navier Stokes dynamics. We try the two options, the fine-tuning being done by jointly training the whole system, starting with a pre-trained estimator, while dividing the learning rate of the estimator's optimizer algorithm by a factor of 100. Results show that fine-tuning yields clearly better results. Moreover, we can also see that the JT algorithm still gives very good results and that the ResNet is consistently beaten by our methods. In order to further explore the difference in generalization performance between the different algorithms which we found to be efficient, we trained them on smaller datasets. Tables 4 and 5 show test results with halved datasets. We can see that all three algorithms are quite robust to this diminution of available training data which only slightly affects performance.

However, when we make data even scarcer, in some cases as shown in table 6 where R = 5000, the pre-trained network performs more consistently. This is understandable as those dynamics are quite complex (here the fluid is very viscuous), the JT having more unconstrained parameters this explains its inability to generalize in this case while the ResNet does better but still sees its performance very affected by the lack of training of data. We are convinced that this robustness of pre-trained networks is a very important feature and should be studied further as it could prove useful in many real-world settings, especially if combined with more unsupervised algorithms such as JT.

6. RELATED WORK:

The dynamical system view on NN is not new. In the 90s, several papers developed the analogy between recurrent neural networks and dynamical systems. To quote only a few references Pearlmutter (1995) 

7. CONCLUSION:

In this work, we address the problem of forecasting spatio-temporal dynamics governed by differential equations, where information on the system state comes from incomplete observations.

We have presented a general framework separating the task in two problems, state estimation and forecasting, and proposed several instances of this framework in order to accommodate the weakly supervised estimation step. This approach is tested on families of equations governing the dynamics of fluids which are known to be challenging. We have explored empirically some of the properties of the algorithms and evaluated their performance and behavior on high resolution simulated data. We have been able to show that imposing physical priors over the estimator, by pre-training it on simple dynamics then fine-tuning on the target equations, can be beneficial. This is especially true when data is scarce. Jointly training the system without any supervision gives surprisingly good results, especially when lots of data are available.

There are many ideas to explore in order to extend the results. Exploring which physical constraints could be beneficial to the training and to the discovery of physically meaningful models is an important issue. Moreover, we still face difficulties when trying to capture long-term velocity dynamics in the pre-trained setting and this has to be improved if we want to achieve a better performance for the PT SSE model. Our approach also has to be tested on other families of differential equations.

Furthermore, while we have been only considering simulated data, we must obviously aim for data from real-world settings.

As a final note, while we have only considered our physically inspired models for physics motivated datasets, we think that the insights provided by the different settings empirically studied in this work might prove useful for other tasks regarding the forecasting of dynamical systems.

A NAVIER-STOKES AND EULER EQUATIONS:

Let us consider a fluid blob of surface δS, velocity u and density ρ. This blob has a pressure p which exerts a force -∇pδS and is in a gravitational field g so that Newton's second law gives :

ρδS Du Dt = (-∇p + ρg)δS

where Du Dt expresses the acceleration of the blob thus being the temporal derivative of the function t → u(x(t), y(t)) which can also be expressed as :

Du Dt = ∂u ∂t + (u • ∇)u

Moreover, we will also make the incompressibility hypothesis which can be mathematically translated to : ∇ • u = 0 meaning that u is of null divergence.

Finally, by definition, the mass of the blob we are studying is conserved so that :

Dρ Dt = ∂ρ ∂t + (u • ∇)ρ = 0

Those equations are the so called Euler equations and form the system :

           ∂u ∂t + (u • ∇)u = - ∇p ρ + g ∂ρ ∂t + (u • ∇)ρ = 0 ∇ • u = 0

When deriving the Euler equations, we never considered the particular nature of the fluid. This is taken into account by considering an additional force in the previous derivation : viscosity. Even for small values, this additional force is of great importance : For example, while air usually has quite small viscosity, without it one wouldn't be able to explain why planes fly.

For Newtonian fluids2 , viscosity can be taken into account with an additional term ν∇ 2 u where ν = µ ρ is the kinematic viscosity. This gives us the following equation system :

           ∂u ∂t + (u • ∇)u = - ∇p ρ + g + ν∇ 2 u ∂ρ ∂t + (u • ∇)ρ = 0 ∇ • u = 0

Those are the Navier-Stokes equations which are central to fluid dynamics. We can easily see how the Euler equations can be considered as a simplification of this system, found by taking ν = 0.

The importance of viscosity in a fluid, and thus a way to assess its qualitative behaviour, cannot be measured through ν alone as it has to be compared against the importance of advection. The most commonly used quantity to describe it is the Reynolds number. If we take U to be a typical speed and L a typical distance of the studied flow then it can be expressed as :

R = U L ν:

To understand its meaning, let us recall that we have (u • ∇)u ∼ U 2 L and ν∇ 2 u ∼ νU L 2 so that : R ∼ (u • ∇)u ν∇ 2 u which means that R is the ratio of the inertia and the viscosity terms. Thus, if R is small, it means that the fluid is very viscous, thus behaving more like honey ; If R is large, the viscosity term is small and the fluid behaves more like water (which doesn't always mean that every such fluid can be modelled with the Eulerian approximation as viscosity can play a role even when the corresponding term is small).

B LERAY PROJECTION : EULER AND NAVIER STOKES EXPRESSED IN THE FORM OF EQUATION 1:

The two systems equation A and equation A are not of the form equation 1 as we still have the pressure variable p as well as the null divergence constraint. Let us show how we can transform those equations in order to get rid of it.

The Helmholz-Leray decomposition result states that for any vector field a, there exists b and c such that :

a = ∇b + c and ∇ • c = 0

where ∇• is the divergence operator. Moreover, this pair is unique up to an additive constant for b. Thus, we can define a linear operator P by :

P(a) = c

This operator is a continuous linear projector which is the identity for divergence-free vector fields and vanishes for those deriving from a potential.

Let us take a solution of NS and apply P on the first equation of equation A, we have, as u is divergence free from the third equation and as g derives from a potential :

∂u ∂t = -P[(u • ∇)u] + νP(∇ 2 u)

where permuting derivation and P is justified by the continuity of the operator3 .

Thus, if u is solution to equation A, it is also a solution to :

     ∂u ∂t = -P[(u • ∇)u] + νP(∇ 2 u) ∂ρ ∂t = -(u • ∇)ρ

which is of the form of equation 1

Conversely, the solution of the above system is such that :

u t = ∂u ∂t = -P[(u • ∇)u] + νP(∇ 2 u)

which gives, by exchanging P and the integral4 :

u t = P[ -(u • ∇)u + ν∇ 2 u]

so that u is automatically of null divergence by definition of P. The two systems are thus equivalent. a physical meaning is more stable for long term forecasting and benefits more from this additional constraining. However, we still keep l = 8 for the other experiments for computational reasons.

E COMPARISON OF SAMPLING STRATEGIES:

Tables 9 and 10 compare the 3 sampling strategies introduced in section 3, namely systematic teacher forcing, no teacher forcing and scheduled sampling. For the latter, we consider a decreasing sequence ( i ) i , starting from 1 and converging to 0, so that, at time i, we pick Y t-1 with probability i and Y t-1 with probability 1i . This amounts to starting with one-step predictions then gradually increasing the time horizon as the model improves. The ( i ) i sequence has to decay at the same pace as the error of the model and thus has to be chosen carefully depending on the trained model. In this work, we have considered the exponential scheme :

i = α i

Results show that using systematic teacher forcing performs poorly, which is to be expected as it is equivalent to a training horizon of 1, and, surprisingly, not using any teacher forcing works remarkably well, almost as well as scheduled sampling while training much faster which might be used to accelerate training. However, in all the experiments, we used scheduled sampling nonetheless.

F ADDITIONAL FIGURES:

We provide below some example of forecasting images obtained for Navier Stokes equations. The setting of the experiments is introduced in the main text in section 5.

:

Let us note that, while this might just seem like a mere technicality, it still provides us with an important information : forecasting the dynamics of those equations while following equation 1 automatically makes us find a null-divergence velocity vector field.

Some fluids can behave quite differently and this is translated into a different viscosity term

One can use a finite difference approximation to show it for example.

To prove this, we just have to take a sum approximation to the integral and use again the linearity then the continuity of P.

The precise statement is actually that the operators who work forms a dense subset.

Figures and Tables:

x w v 2 I D p Q I B a N o r X a 7 l + H F 6 N y b 9 E p l t + L O R J b B y 6 E M u W q 9 0 l e 3 H 7 M 0 4 g q Z p M Z 0 P D d B P 6 M a B Z N 8 U u y m h i e U j e i A d y w q G n H j Z7 O F J + T U O n 0 S x t o + h W T m / p 7 I a G T M O A p s Z 0 R x a B Z r U / O / W i f F 8 M b P h E p S 5 I r N P w p T S T A m 0 + t J X 2 j O U I 4 t U K a F 3 Z W w I d W U o c 2 o a E P w F k 9 e h s Z l x b P 8 c F W u 3 u Z x F O A Y T u A M P L i G K t x D D er A I I J n e I U 3 R z s v z r v z M W 9 d c f K Z I / g j 5 / M H F X C P 7 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F I 2

x w v 2 I D p Q I B a N o r X a 7 l + H F 6 N y b 9 E p l t + L O R J b B y 6 E M u W q 9 0 l e 3 H 7 M 0 4 g q Z p M Z 0 P D d B P 6 M a B Z N 8 U u y m h i e U j e i A d y w q G n H j Z7 O F J + T U O n 0 S x t o + h W T m / p 7 I a G T M O A p s Z 0 R x a B Z r U / O / W i f F 8 M b P h E p S 5 I r N P w p T S T A m 0 + t J X 2 j O U I 4 t U K a F 3 Z W w I d W U o c 2 o a E P w F k 9 e h s Z l x b P 8 c F W u 3 u Z x F O A Y T u A M P L i G K t x D D er A I I J n e I U 3 R z s v z r v z M W 9 d c f K Z I / g j 5 / M H F X C P 7 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F I 2

x w v 2 I D p Q I B a N o r X a 7 l + H F 6 N y b 9 E p l t + L O R J b B y 6 E M u W q 9 0 l e 3 H 7 M 0 4 g q Z p M Z 0 P D d B P 6 M a B Z N 8 U u y m h i e U j e i A d y w q G n H j Z7 O F J + T U O n 0 S x t o + h W T m / p 7 I a G T M O A p s Z 0 R x a B Z r U / O / W i f F 8 M b P h E p S 5 I r N P w p T S T A m 0 + t J X 2 j O U I 4 t U K a F 3 Z W w I d W U o c 2 o a E P w F k 9 e h s Z l x b P 8 c F W u 3 u Z x F O A Y T u A M P L i G K t x D D er A I I J n e I U 3 R z s v z r v z M W 9 d c f K Z I / g j 5 / M H F X C P 7 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F I 2

x w v 2 I D p Q I B a N o r X a 7 l + H F 6 N y b 9 E p l t + L O R J b B y 6 E M u W q 9 0 l e 3 H 7 M 0 4 g q Z p M Z 0 P D d B P 6 M a B Z N 8 U u y m h i e U j e i A d y w q G n H j Z 7 O F J + T U O n 0 S x t o + h W T m / p 7 I a G T M O A p s Z 0 R x a B Z r U / O / W i f F 8 M b P h E p S 5 I r N P w p T S T A m 0 + t J X 2 j O U I 4 t U K a F 3 Z W w I d W U o c 2 o a E P w F k 9 e h s Z l x b P 8 c F W u 3 u Z x F O A Y T u A M P L i G K t x D D e r A I I J n e I U 3 R z s v z r v z M W 9 d c f K Z I / g j 5 / M H F X C P 7 A = = < / l a t e x i t > Y t

e 6 g B n V g 8 A T P 8 A p v z t h 5 c d 6 d j 3 n r i p P P H M E f O Z 8 / u V + S D Q = = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " S s r N a 5 n e 1 f M 1

t q 6 P j + s l K 7 y e M o w g m c w j m E c A U 1 u I M 6 N I C C h G d 4 h T f P e C / e u / e x a C 1 4 + c w x / J H 3 + Q P R W 5 D / < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C h g

t q 6 P j + s l K 7 y e M o w g m c w j m E c A U 1 u I M 6 N I C C h G d 4 h T f P e C / e u / e x a C 1 4 + c w x / J H 3 + Q P R W 5 D / < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C h g

t q 6 P j + s l K 7 y e M o w g m c w j m E c A U 1 u I M 6 N I C C h G d 4 h T f P e C / e u / e x a C 1 4 + c w x / J H 3 + Q P R W 5 D / < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C h g

Figure 1: Illustration of the two multi-steps prediction strategies -SSE and MSRE -introduced in section 3.2.

PERFORMANCE FOR EULER AND NAVIER-STOKES EQUATIONS Euler equations: test average MSE for two baselines (ResNet and ConvLSTM) and 4 instances of the proposed model for different forecasting horizons. PT and JT are respectively the pre-trained and jointly trained models, SSE and MSRE the two multi-step training strategies introduced in section 3.2.

Table content: <table><tr><td/><td>t 0 + 1</td><td colspan="2">t 0 + 5</td><td>t 0 + 10</td><td>t 0 + 15</td><td colspan="2">t 0 + 20</td><td>t 0 + 25</td><td>t 0 + 30</td></tr><tr><td>ResNet</td><td>0.042</td><td colspan="2">0.106</td><td>0.195</td><td>0.267</td><td>0.324</td><td/><td>0.370</td><td>0.407</td></tr><tr><td>ConvLSTM</td><td>0.16</td><td colspan="2">0.24</td><td>0.35</td><td>0.47</td><td>0.57</td><td/><td>0.70</td><td>0.90</td></tr><tr><td>PT MSRE</td><td>0.010</td><td colspan="2">0.071</td><td>0.102</td><td>0.257</td><td>0.324</td><td/><td>0.377</td><td>0.419</td></tr><tr><td>PT SSE</td><td>0.011</td><td colspan="2">0.059</td><td>0.102</td><td>0.179</td><td>0.234</td><td/><td>0.288</td><td>0.341</td></tr><tr><td>JT MSRE</td><td>0.007</td><td colspan="2">0.044</td><td>0.10</td><td>0.176</td><td>0.235</td><td/><td>0.284</td><td>0.327</td></tr><tr><td>JT SSE</td><td>0.308</td><td colspan="2">0.599</td><td>0.594</td><td>1.27</td><td>5.24</td><td/><td>15.4</td><td>33.5</td></tr><tr><td/><td/><td/><td>t 0 + 1</td><td>t 0 + 5</td><td>t 0 + 10</td><td>t 0 + 15</td><td colspan="2">t 0 + 20</td><td>t 0 + 25</td><td>t 0 + 30</td></tr><tr><td colspan="2">Jointly trained</td><td/><td>0.00097</td><td>0.0066</td><td>0.0196</td><td>0.0376</td><td colspan="2">0.0588</td><td>0.0824</td><td>0.107</td></tr><tr><td colspan="2">Pre-trained estimator</td><td/><td>0.017</td><td>0.108</td><td>0.273</td><td>0.373</td><td>0.446</td><td>0.512</td><td>0.579</td></tr><tr><td colspan="3">Fine-tuned Pre-Trained estimator</td><td>0.0016</td><td>0.0105</td><td>0.0256</td><td>0.0455</td><td colspan="2">0.0742</td><td>0.105</td><td>0.169</td></tr><tr><td colspan="2">ResNet</td><td/><td>0.0070</td><td>0.0181</td><td>0.0423</td><td>0.0720</td><td>0.104</td><td>0.137</td><td>0.171</td></tr></table>

Navier-Stokes equations with R = 5000 : Test average MSE per time-step for a Jointly Trained system, a system with pre-trained estimator and one with a pre-trained estimator fine-tuned on the Navier-Stokes data, for different forecasting horizons.

Table content: <table><tr><td/><td>t 0 + 1</td><td>t 0 + 5</td><td>t 0 + 10</td><td>t 0 + 15</td><td>t 0 + 20</td><td>t 0 + 25</td><td>t 0 + 30</td></tr><tr><td>Jointly trained</td><td>0.0091</td><td>0.047</td><td>0.104</td><td>0.18</td><td>0.24</td><td>0.288</td><td>0.333</td></tr><tr><td>Fine-tuned Pre-Trained estimator</td><td>0.011</td><td>0.049</td><td>0.103</td><td>0.16</td><td>0.23</td><td>0.297</td><td>0.366</td></tr><tr><td>ResNet</td><td>0.0211</td><td>0.0614</td><td>0.129</td><td>0.194</td><td>0.247</td><td>0.291</td><td>0.327</td></tr></table>

Navier Stokes equations with R = 100000 : Test average MSE per time-step for a Jointly Trained system and one with a pre-trained estimator fine-tuned on the Navier-Stokes data, for different forecasting horizons.

Table content: <table/>

Navier Stokes equations with R = 5000 keeping only half the initial dataset for training : Test average MSE per time-step for a Jointly Trained system, a system with an estimator pre-trained on Euler dynamics fine-tuned on the Navier-Stokes data and a ResNet, for different forecasting horizons.

Table content: <table><tr><td colspan="5">PERFORMANCE WITH SMALLER DATASETS</td><td/><td/><td/></tr><tr><td/><td>t 0 + 1</td><td>t 0 + 5</td><td>t 0 + 10</td><td>t 0 + 15</td><td>t 0 + 20</td><td>t 0 + 25</td><td>t 0 + 30</td></tr><tr><td>Jointly trained</td><td>0.0015</td><td>0.0111</td><td>0.0311</td><td>0.056</td><td>0.083</td><td>0.10</td><td>0.14</td></tr><tr><td>Fine-tuned Pre-Trained estimator</td><td>0.0020</td><td>0.010</td><td>0.020</td><td>0.042</td><td>0.096</td><td>0.18</td><td>0.28</td></tr><tr><td>ResNet</td><td>0.0090</td><td>0.028</td><td>0.067</td><td>0.114</td><td>0.163</td><td>0.210</td><td>0.255</td></tr><tr><td/><td>t 0 + 1</td><td>t 0 + 5</td><td>t 0 + 10</td><td>t 0 + 15</td><td>t 0 + 20</td><td>t 0 + 25</td><td>t 0 + 30</td></tr><tr><td>Jointly trained</td><td>0.0091</td><td>0.047</td><td>0.10</td><td>0.18</td><td>0.24</td><td>0.29</td><td>0.33</td></tr><tr><td>Fine-tuned Pre-Trained estimator</td><td>0.011</td><td>0.049</td><td>0.10</td><td>0.16</td><td>0.23</td><td>0.30</td><td>0.37</td></tr><tr><td>ResNet</td><td>0.0290</td><td>0.0843</td><td>0.168</td><td>0.241</td><td>0.299</td><td>0.348</td><td>0.389</td></tr></table>

Navier Stokes equations with R = 100000 keeping only half the initial dataset for training : Test average MSE per time-step for a Jointly Trained system, a system with an estimator pre-trained on Euler dynamics fine-tuned on the Navier-Stokes data and a ResNet, for different forecasting horizons.

Table content: <table><tr><td/><td>t 0 + 1</td><td>t 0 + 5</td><td>t 0 + 10</td><td>t 0 + 15</td><td>t 0 + 20</td><td>t 0 + 25</td><td>t 0 + 30</td></tr><tr><td>Jointly trained</td><td>0.012</td><td>0.114</td><td>0.304</td><td>0.458</td><td>0.568</td><td>0.645</td><td>0.703</td></tr><tr><td>Fine-tuned Pre-Trained estimator</td><td>0.0021</td><td>0.0131</td><td>0.0323</td><td>0.0595</td><td>0.0972</td><td>0.1452</td><td>0.2008</td></tr><tr><td>ResNet</td><td>0.011</td><td>0.037</td><td>0.085</td><td>0.139</td><td>0.193</td><td>0.244</td><td>0.293</td></tr></table>

Navier Stokes equations with R = 5000 keeping only one fourth of the initial dataset for training : Test average MSE per time-step for a Jointly Trained system, a system with an estimator pre-trained on Euler dynamics fine-tuned on the Navier-Stokes data and a ResNet, for different forecasting horizons.

Table content: <table/>

offers a nice review of state of the art on RNNs and their training algorithms including convergence and stability ;Tsoi & Back (1994) review RNN architectures with local recurrent connections together with their interpretation as non linear IIR/FIR filters ;Haykin (2009) introduces RNNs with regard to state estimation in dynamical systems. More recently, the success of ResNetlike architectures has motivated new developments and interpretations of NNs from a dynamical systems perspective. The interpretation of residual blocks as implementing a simple forward Euler numerical scheme for ODE is highlighted by several authors such asChang et al. (2018b);Lu et al. (2018); E (2017). For example,Lu et al. (2018) shows how different ResNet inspired modules can be considered as specific numerical schemes (forward or backward Euler, Runge Kutta, etc..).Chang et al. (2018b) links the depth of ResNet architectures with the size of the discretization step used in ODE. InGomez et al. (2017);Chang et al. (2018a), the connection with the numerical integration schemes is pushed further by exploiting the idea of reversibility in order to build architectures for which all activations can be reconstructed from the next layer, thus alleviating the need for storing the states of a network and allowing to build larger architecture .Ruthotto & Haber (2018)  develop PDE inspired ResNet like models and establish stability results. This line of work concerns networks for static data and targets the development of alternative architectures or training algorithms.Several works have already attempted to learn PDEs from data(Crutchfield & Mcnamara (1987),Alvarez et al. (2013)). More recently, Rudy et al. (2017a) use sparse regression on a dictionary of differential terms to recover the underlying PDE. InRaissi & Karniadakis (2018),Raissi et al. (2017), they propose recovering the coefficients of the differential terms by deriving a GP kernel from a linearized form of the PDE. For dynamic data,Kim et al. (2018) proposes NN models for fluid simulations for computer graphics applications. The objective is to accelerate fluid simulation w.r.t. classical solvers. They learn to generate 2D and 3D fluid velocities (corresponding to states in our examples), using a fully supervised approach. This share some technical similarities with our own work but the approach and objectives are clearly different. A fully supervised NN implementing a multistep numerical scheme is also used inLong et al. (2018) for learning PDEs from simulations.Raissi (2018) develops a NN framework for learning PDEs from data. In this work either the form of the PDE or the variable dependency is supposed to be known and the approach is again fully supervised over the state.

Table content: <table/>

t 0 + 1 t 0 + 5 t 0 + 10 t 0 + 15 t 0 + 20 t 0 + 25 t 0 + 30 Forecasting with Jointly Trained system : Test average MSE per time-step with Scheduled Sampling, Systematic Teacher Forcing and No Teacher Forcing, for different forecasting horizons. t 0 + 1 t 0 + 5 t 0 + 10 t 0 + 15 t 0 + 20 t 0 + 25 t 0 + 30

Table content: <table><tr><td>Scheduled Sampling</td><td colspan="2">0.007 0.044</td><td>0.100</td><td>0.176</td><td>0.235</td><td>0.284</td><td>0.327</td></tr><tr><td>No Teacher Forcing</td><td colspan="2">0.008 0.043</td><td>0.108</td><td>0.175</td><td>0.236</td><td>0.289</td><td>0.335</td></tr><tr><td colspan="3">Systematic Teacher Forcing 0.026 0.215</td><td>0.108</td><td>0.567</td><td>0.663</td><td>0.736</td><td>0.795</td></tr><tr><td>Scheduled Sampling</td><td>0.011</td><td>0.06</td><td>0.102</td><td>0.179</td><td>0.234</td><td>0.288</td><td>0.341</td></tr><tr><td>No Teacher Forcing</td><td colspan="2">0.011 0.058</td><td>0.102</td><td>0.182</td><td>0.24</td><td>0.294</td><td>0.346</td></tr><tr><td colspan="3">Systematic Teacher Forcing 0.009 0.076</td><td>0.225</td><td>0.374</td><td>0.5</td><td>0.605</td><td>0.694</td></tr></table>

Forecasting with pre-trained estimator : Test average MSE per time-step with Sampling, Systematic Teacher Forcing and No Teacher Forcing, for different forecasting horizons.

Table content: <table/>

Peer review (iclr2019):
 I very much like the aim of this work. This is a problem of interest to a wide community, which as far as I*m aware hasn*t yet had much focus from the deep learning community. However, perhaps in part because of this, the paper reads as naive in places. Pages 1-4 are all background saying nothing new, but ignoring the effort made on this problem by other communities. There has been some work done o this problem within statistics, and within the Gaussian process community, to which no reference is made at all by the paper. There are two novelties as far as I can see (these may or may not be novel - but they were novel to me). The first is use of NNs to model the system. The second is the multiple state restimation (MRSE) on page 5. I struggled to get a feeling about how successful these two aspects of the work are. The results section is difficult to follow, and doesn*t compare the method to existing methods and so there is no baseline to say that this is successful or not. Thus I find it hard to judge the execution of the idea. What I really want to know reading a paper like this is should I use this approach? Because there is no comparison to existing methods, it leaves me unsure. Other comments: - Is the title correct? I don*t see how these are PDE guided NNs? You*ve used data from a PDE to train the network and as a test problem. A PDE guided NN would, for me, know something about the dynamics (compare with recently work in the GP community where kernels are derived that lead to GPs that analytically obey simple PDEs). - There is an obvious link to work in the uncertainty quantification community, particularly around the use of multi-fidelity/ multi-level simulation. This paper is likely to be of interest to them and the link could be more explicit. - Page 3, after eq 2 - there is notation used here that is undefined Y_{t-k}^t - The simplifying assumption on page 3 is very strong and unlikely to hold for many systems. But it isn*t clear to me whether this is necessary or not? Presumably if it doesn*t hold then we may still get an approximation that could be useful, but it is just that we lose any guarantee the method will work. - I thought the MSRE idea was interesting. It wasn*t very well explained or motivated, and it was unclear to me whether it works well or not from the results, or whether it is novel to this paper or not. But I*d like to have read more about it. - Is the trick in Section 8.2 original to this paper? If so, it seems a nice idea (I*ve not checked the detail). - Most of section 8.1 strikes me as unnecessary. - There are quite a few typos. In particular, words such as Markovian, Newtonian should be capitalised.