Non-Linear Operator Approximations for Initial Value Problems

1 INTRODUCTION . Predicting the future states using the current conditions is a fundamental problem in machine learning . Such problems fall under the umbrella of a common term , the “ Initial Value Problems ” ( IVPs ) . The basic structure of IVP involves a first-order time-evolution along with non-linear operators . The class of IVPs spans the domain of physics ( modeling gravitational waves ( Lovelace , 2021 ) ) , neuroscience ( Hodgkin-Huxley model ( Zhang et al. , 2020 ) ) , engineering ( fluid dynamics ( Wendt , 2008 ) ) , water waves ( tsunami ( Elbanna et al. , 2021 ) ) , mean field games ( Ruthotto et al. , 2020 ) , to list just a few . Within the current pandemic context , the applications areas like epidemiology ( Kermack–McKendrick model ( Kermack et al. , 1991 ; Diekmann et al. , 2021 ) ) are of tremendous interest . Neural Operators The use of deep learning to solve the IVP like problems for predictions has been exploiting within the framework of convolutional neural networks ( CNNs ) ( Bhatnagar et al. , 2019 ; Guo et al. , 2016 ) , and time-evolution by employing multiple layers ( Khoo et al. , 2020 ) . The multi-layered deep networks with CNNs are suitable to solve problems with a large number of training samples . Moreover , because of the image-regression like structures , such models are restricted to the specifications of the input size . Another research direction aims at solving and modeling the partial differential equations ( PDEs ) versions of the IVPs for a given instance . The works of ( Kochkov et al. , 2021 ) model the IVP solution as NNs for modeling the turbulent flows . Along the same lines , we have physics-informed neural networks ( PINNs ) ( Raissi et al. , 2019 ; Wang et al. , 2021b ) that utilize PDE structure for defining the loss functions . Such models are not applicable within the context of a complete data-driven scenario , or for the setups where the exact PDE structure is not known , for example , modeling the climate , epidemic , or unknown physical and chemical phenomena . Finally , we have the works of Neural Operators that are completely data-driven and input-resolution independent schemes ( Li et al. , 2020b ; c ; a ; Gupta et al. , 2021 ; Bhattacharya et al. , 2020 ; Patel et al. , 2021 ) . Most of these approaches tried to efficiently work with the integral kernel operators , for example , Graph Nyström sampling in ( Li et al. , 2020b ) , convolution approximation in ( Li et al. , 2020a ) , multiwavelets compression in ( Gupta et al. , 2021 ) . Apart from solving non-homogeneous linear differential equations , the PDE operators are mostly non-linear . To tackle the non-linear behavior , these prior works use a multi-cell architecture with non-linearity ( for example , ReLU ) . To work with the IVP like problems , and be data-efficient , we aim to adopt explicitly the non-linear operator ( exponential ) that appears in the IVP solutions . Exponential Operators The exponential of linear transformation has been a subject of research for the last 150 years ( Laguerre , 1898 ) . In the simplest form , the exponential operator appears as a solution of : dydt “ at , yp0q “ y0 as yptq “ e aty0 ( for more general examples , see Table 1 ) . With applications ranging from control systems theory ( converting continuous to discrete systems ) to solving partial differential equations ( Cox & Matthews , 2002 ; Kassam & Trefethen , 2005 ) , the exponential function of operators is a subject of active research . In deep learning , the exponential function to model non-linearity is used in ( Andoni et al. , 2014 ) . Recently , the exponential operators have also been explored in the field of computer vision for generative flows in ( Hoogeboom et al. , 2020 ) . Padé Approximation Although one approach to implementing an exponential operator could be attained through a Taylor series representation , this operator function is prone to errors ( Abramowitz & Stegun , 1965 ) . Scale-and-squaring ( SSQ ) methods are commonly suggested approaches to deal with the errors ( Lawson , 1967 ) . In addition to SSQ , the Padé approximation which represents an analytic function as the ratio of polynomials achieves state-of-the-art accuracy in computing exponential operators ( Fasi & Higham , 2019 ) . Industry standard numerical toolboxes ( for example , MATLAB , SciPy ) use the Padé approximation based approach to compute the matrix exponential expm ( Al-Mohy & Higham , 2009 ) . Matrix exponential via Padé representation requires dense matrix computations ( for example , inverse and higher-order polynomials ) . Such operations are not numerically feasible , in-general , for the inputs with large size . However , the commonly used operators like convolution ( possibly , multi-layered ) have parameters that are fixed beforehand and are much less than the input dimension . A suitable approach , therefore , is a neural architecture based Padé approximation . Our strategy , in this work , is to explicitly embed the exponential operators in the neural operator architecture for dealing with the IVP like datasets . The exponential operators are non-linear , and therefore , this removes the requirement of having multi-cell linear integral operator layers . While with sufficient data in-hand , the proposed approach may work similarly to the existing neural operators with a large number of training parameters . However , this is seldom a feasible scenario for the expensive real-world experiments , or on-going recent issues like COVID19 prediction . Here , the current work is helpful in providing data-efficiency analytics , and is useful in dealing with scarce and noisy datasets ( see Section 3.3 ) . To the advantage of Padé approximation , the exponential of a given operator can be computed with the pre-defined coefficients ( see Section 2.3 ) and a recurrent polynomial mechanism . Our Contributions The main novel contributions of this work are summarized as follows : ( i ) For the IVPs , we propose to embed the exponential operators in the neural operator learning mechanism . ( ii ) By using the Padé approximation , we compute the exponential of the operator using a novel recurrent neural architecture that also eliminates the need for matrix inversion . ( iii ) We theoretically demonstrate that the proposed recurrent scheme , using the Padé coefficients , have bounded gradients with respect to ( w.r.t . ) the model parameters across the recurrent horizon . ( iv ) We demonstrate the data-efficiency on the synthetic 1D datasets of Korteweg-de Vries ( KdV ) and Kuramoto–Sivashinsky ( KS ) equations , where with less parameters we achieve state-of-the-art performance . ( v ) We formulate and investigate the epidemic forecasting as a 2D time-varying neural operator problem , and show that for real-world noisy and scarce data , the proposed model outperforms the best neural operator architectures by 53 % and best non-neural operator schemes by 52 % . 2 OPERATORS FOR INITIAL VALUE PROBLEM . We formalize the partial differential equations ( PDEs ) version of the Initial Value Problem studied in this work in Section 2.1 . Section 2.2 summarizes the multi-resolution analysis using multiwavelets for space-discretization . Section 2.3 describes the proposed use of canonical exponential operators and presents a novel architecture using Padé approximation . 2.1 INITIAL VALUE PROBLEM . The initial value problem ( IVP ) for PDEs can be written in its general form as follows . ut “ Fpt , uq , x P Ω upx , 0q “ u0pxq , x P Ω ( 1 ) where , ut is the first-order time derivative of u , F is a time-varying differential operator ( non-linear in-general ) such that F : R ` Y t0u ˆ B Ñ B with B being a Banach space . Usually , the system in eq . ( 1 ) is required to satisfy a boundary condition such that Bupx , tq “ 0 , x P BΩ @ t in the solution horizon , and BΩ is the boundary of the computational region Ω with B some linear function . Pertaining to our work , the operator map problem for IVP can be formally defined as follows . Operator Problem Given A and U as two Sobolev spaces Hs , p with s ą 0 , p “ 2 , an operator T is such that T : A Ñ U . For a given τ ą 0 and two functions u0pxq and upx , τq , in this work , we take the operator map as T u0pxq “ upx , τq with x P Ω . Table 1 summarizes a few examples of the IVP and their solutions.The exponential operators are ubiquitous in the IVP solutions and , therefore , are important to study . One issue , however , is that the exponential operators are non-linear and unlike convolution like operators , there does not exist a general way to diagonalize them ( Fourier transform diagonalizes convolution operator ) for an efficient representation . Previous work on neural operators ( Li et al. , 2020c ; a ; Gupta et al. , 2021 ) modeled the non-linear operators in one way or another by using multiple canonical integral operators along with non-linearity ( for example , ReLU ) . In this work , we directly produce an exponential operator approximation . First , we discuss an efficient basis ( multiwavelets ) for space discretization of the input/output functions in Section 2.2 . 2.2 MULTI-RESOLUTION ANALYSIS . The multi-resolution analysis ( MRA ) aims at projecting a function to a basis over multiple scales . The wavelet basis ( e.g. , Haar , Daubechies ) are some popular examples . Multiwavelets further this operation by using the family of orthogonal polynomials ( OPs ) , for example , Legendre polynomials for an efficient representation over a finite interval ( Alpert et al. , 2002 ) . The multiwavelets are useful in the sparse representation of the integral operators with smooth kernels . In addition , the multiwavelets also sparsify the exponential functions of the strictly elliptic operators ( Beylkin & Keiser , 1997 ) . However , we do not rely on this assumption in this work . Here , we briefly introduce the MRA and refer the reader to Gupta et al . ( 2021 ) for a detailed picture . Notation We begin by defining the space of finite interval polynomials as Vkn “ tf |f are polynomials of degree ă k defined over interval p2´nl , 2´npl ` 1qq for all l “ 0 , 1 , . . . , 2n ´ 1 , and assumes 0 elsewhereu . The Vkn are contained in each other for subsequent n or , Vk0 Ă Vk1 Ă . . . Ă Vkn´1 Ă Vkn Ă . . . . ( 2 ) 1Time-advection equation with linear operators L , N and non-linear function fp.q . A wide range of problems can be modeled , for example , Korteweg-de Vries , Kuramoto-Sivashinsky , Burgers ’ Equation , Navier-Stokes ( list not exhaustive ) . 2A non-linear integro-differential solution to the time-advection equation using semi-group approach ( Beylkin & Keiser , 1997 ; Pazy , 1983 ; Yoshida , 1980 ) . A slightly general version is discussed in ( Beylkin et al. , 1998 ) . The orthogonal component of these polynomial spaces is termed as multiwavelet space Wkn and are defined such that Vkn à Wkn “ Vkn ` 1 , Vkn K Wkn . ( 3 ) The orthonormal basis of Vk0 are OPs ϕ0 , ϕ1 , . . . , ϕk´1 and we have used appropriately normalized shifted Legendre Polynomials in this work . The basis for Vkn and W k n are ϕ n jlpxq “ 2n { 2ϕjp2nx´ lq and ψnjlpxq “ 2n { 2ψjp2nx´ lq , respectively , for l “ 0 , 1 , . . . , 2n ´ 1 and j “ 0 , 1 , . . . , k ´ 1 . Finally , an important trick for representing the operator T in the multiwavelet basis is called nonstandard ( NS ) form ( Beylkin et al. , 1991 ) . The NS form decouples the interactions of the scales and is useful in obtaining an efficient numerical procedure . Using NS form , the projection of operator T is expanded using a telescopic sum as follows . Tn “ ÿn i “ L ` 1 pQiTQi ` QiTPi´1 ` Pi´1TQiq ` PLTPL , ( 4 ) where , Pn : Hs,2 Ñ Vkn is the projection operator , Tn “ PnTPn , Qn : Hs,2 Ñ Wkn such that Qn “ Pn ´ Pn´1 , and L is the coarsest scale under consideration pL ě 0q . Therefore , the NS form of the operator is a collection of the triplets tAi , Bi , Ciuni “ L ` 1 and PLTPL with Ai “ QiTQi , Bi “ QiTPi´1 and Ci “ Pi´1TQi . In this work , we aim to model Ai , Bi , Ci as the exponential operators to better learn the IVP by explicitly embedding the non-linear operators into the multiwavelet transformation . This is not straightforward due to the non-linearity of exponential functions . We are now in shape to present the main contribution of the current work in the Section 2.3 where we discuss an implementable neural approximation of the exponential operators .

Summary: This paper proposed a recurrent Pade network fro learning non-linear operator approximations for IVP.  The Pade exponential operator uses a recurrent structure with shared parameters to model the non-linearity compared to recent neural operators that rely on using multiple linear operator layers in succession. The paper showed that Pade network does not suffer from the issue of gradient explosion and the boundedness of the gradients can be established