Title: Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees
Venue: NeurIPS 2024
Accepted: yes
Keywords: Physics-informed neural network

Functional differential equation

Functional derivative
Abstract: We propose the first learning scheme for functional differential equations (FDEs).
FDEs play a fundamental role in physics, mathematics, and optimal control.
However, the numerical analysis of FDEs has faced challenges due to its unrealistic computational costs and has been a long standing problem over decades.
Thus, numerical approximations of FDEs have been developed, but they often oversimplify the solutions. 
To tackle these two issues, we propose a hybrid approach combining physics-informed neural networks (PINNs) with the *cylindrical approximation*. 
The cylindrical approximation expands functions and functional derivatives with an orthonormal basis and transforms FDEs into high-dimensional PDEs. 
To validate the reliability of the cylindrical approximation for FDE applications, we prove the convergence theorems of approximated functional derivatives and solutions.
Then, the derived high-dimensional PDEs are numerically solved with PINNs.
Through the capabilities of PINNs, our approach can handle a broader class of functional derivatives more efficiently than conventional discretization-based methods, improving the scalability of the cylindrical approximation.
As a proof of concept, we conduct experiments on two FDEs and demonstrate that our model can successfully achieve typical $L^1$ relative error orders of PINNs $\sim 10^{-3}$.
Overall, our work provides a strong backbone for physicists, mathematicians, and machine learning experts to analyze previously challenging FDEs, thereby democratizing their numerical analysis, which has received limited attention.

Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees

Abstract

We propose the first learning scheme for functional differential equations (FDEs). FDEs play a fundamental role in physics, mathematics, and optimal control. However, the numerical analysis of FDEs has faced challenges due to its unrealistic computational costs and has been a long standing problem over decades. Thus, numerical approximations of FDEs have been developed, but they often oversimplify the solutions. To tackle these two issues, we propose a hybrid approach combining physics-informed neural networks (PINNs) with the cylindrical approximation . The cylindrical approximation expands functions and functional derivatives with an orthonormal basis and transforms FDEs into high-dimensional PDEs. To validate the reliability of the cylindrical approximation for FDE applications, we prove the convergence theorems of approximated functional derivatives and solutions. Then, the derived high-dimensional PDEs are numerically solved with PINNs. Through the capabilities of PINNs, our approach can handle a broader class of functional derivatives more efficiently than conventional discretization-based methods, improving the scalability of the cylindrical approximation. As a proof of concept, we conduct experiments on two FDEs and demonstrate that our model can successfully achieve typical L 1 relative error orders of PINNs ∼ 10 -3 . Overall, our work provides a strong backbone for physicists, mathematicians, and machine learning experts to analyze previously challenging FDEs, thereby democratizing their numerical analysis, which has received limited attention. Code is available at https://github.com/TaikiMiyagawa/FunctionalPINN .

1 Introduction

Functional differential equations (FDEs) appear in a wide variety of research areas [91, 92, 79]. FDEs are partial differential equations (PDEs) involving functional derivatives, where a functional F is a function of an input function θ ( x ) to a real number, i.e., F : θ ↦→ F ([ θ ]) ∈ R , and a functional derivative is defined as the derivative of functional w.r.t. the input function at x , denoted by δF ([ θ ]) /δθ ( x ) . FDEs play a fundamental role in Fokker-Planck systems [27], turbulence theory [67], quantum field theory [71], mean-field games [16], mean-field optimal control [18, 81], and unnormalized optimal transport [32]. Major examples of FDEs include the Hopf functional equation

in fluid mechanics, the Fokker-Planck functional equation in the theory of stochastic processes, and the functional Hamilton-Jacobi equation in optimal control problems in density spaces.

Despite their wide applicability, numerical analyses of FDEs are known to suffer from significant computational complexity; therefore, numerical approximation methods have been developed over decades. They include the functional power series expansion [67], the Reynolds number expansion [67], finite difference methods [18], finite element methods [79], tensor decomposition methods [91, 79], and the cylindrical approximation [7, 29, 34, 88].

However, they tend to oversimplify the solution of FDEs, prioritizing the reduction of computational costs. The functional power series expansion is applicable only to input functions close to the expansion center. Moreover, it has no convergence guarantees in general [67]. The Reynolds number expansion requires the Reynolds number to be close to zero, severely restricting its applicability, because the Reynolds number for turbulent flow can be ≳ 1000 . Discretizationbased methods such as the finite difference and element methods restrict spacetime resolution and/or the class of input functions and functional derivatives [79]. Existing methods relying on the cylindrical approximation, akin to the spec-

tral method for PDEs, include tensor decomposition [91, 92] to reduce computational costs; however, they tend to significantly simplify the class of input functions and functional derivatives. For instance, their expressivity is limited to polynomials or Fourier series of a few degrees.

To address the notorious computational complexity and limited approximation ability, we propose a hybrid approach combining physics-informed neural networks (PINNs) and the cylindrical approximation (Fig. 1). In the first stage, we expand the input function with orthonormal basis functions, thereby transforming a given FDE into a high-dimensional PDE of the expansion coefficients. This approximation is referred to as the cylindrical approximation . We prove the convergence of the approximated functional derivatives and FDE solutions, validating the reliability of the cylindrical approximation for FDE applications, which is our main theoretical contribution. In the second stage, the derived high-dimensional PDE is numerically solved with a PINN, which is known to be a universal approximator tailored to solve high-dimensional PDEs efficiently in a mesh-free manner.

A notable advantage of our approach is that, with the help of PINNs, it reduces the computational complexity by orders of magnitude, compared with previous discretization-based methods. In fact, it requires only O ( m r ) , where m represents the 'class size' of input functions and functional derivatives (e.g., the degree of polynomials), and r ( ≥ 1 ) denotes the order of the functional derivative included in the target FDE (typically 1 or 2 ). This is a notable reduction from the state-of-the-art cylindrical approximation algorithm [91], which requires as large as O ( m 6 ) . Consequently, our approach substantially extends the class of input functions and functional derivatives that can be represented by the cylindrical approximation. For instance, our approach extends the degrees of polynomials or Fourier series used for the approximation from 6 [91] to 1000 , showing unprecedented expressivity.

As a proof of concept, we conduct experiments on two FDEs: the functional transport equation and the Burgers-Hopf equation. The results show that our model accurately approximates not only the solutions but also their functional derivatives, successfully achieving typical L 1 relative error orders of PINNs ∼ 10 -3 [19, 12, 20, 21, 47, 77, 84, 93, 94].

Our contribution is threefold. (1) We propose the first learning scheme for FDEs to address the significant computational complexity and limited approximation ability. Our model exponentially extends the class of input functions and functional derivatives that can be handled accurately and efficiently. (2) We prove the convergence of approximated functional derivatives and FDE solutions, ensuring the cylindrical approximation to be safely applied to FDEs. (3) Our experimental results show that our model accurately approximates not only the FDE solutions but also their functional derivatives.

2 Related Work

FDEs are prevalent across numerous research fields [27, 24, 32, 16, 18, 81, 67, 71]. Research on FDEs has mainly focused on their theoretical aspects and formal solutions, with very few algorithms available to numerically solve general FDEs [18, 79, 91, 104]. In [18], a numerical method specialized for the Hamilton-Jacobi functional equation for optimal control problems in density space is proposed, based on spacetime discretization. Similarly, [79] employs spacetime discretization with tensor decomposition. The state-of-the-art algorithm proposed in [91], the CP-ALS (Canonical Polyadic tensor expansion & Alternating Least Squares) algorithm, uses the cylindrical approximation along with the finite difference method and tensor decomposition, requiring O ( m 6 ) (see App. F.2 for the derivation), whereas our model requires only O ( m r ) ( r is 1 or 2 in most FDEs). Furthermore, our model does not require such discretization, making it mesh-free. See App. B for an additional introduction to FDEs and their approximations.

The cylindrical approximation originates from the theory of stochastic processes [34, 88]. It is reminiscent of the spectral method for PDEs [66] and is a generalization to FDEs. Convergence theorems of the cylindrical approximation are summarized in a recent seminal paper [92]. Note that the cylindrical approximation in this paper (Eq. (2)) is different from the one in [92], tailored for practical use. Consequently, our convergence theorems also differ from those in [92]. See App. C.4.2 for technical details. See App. A for more comparisons with other studies.

3 Proposed Approach

3.1 Step 1: Cylindrical Approximation

We first introduce the cylindrical approximation of functionals, functional derivatives, and FDEs , beginning with the expansion of input functions and culminating in the transformation of FDEs into high-dimensional PDEs. Additionally, we prove the convergence theorems for this approximation. The rigorous mathematical background is reviewed in App. C for interested readers.

Firstly, we define the cylindrical approximation of functionals [7, 29, 34, 88]. Any function θ in a real separable Hilbert space H can be represented uniquely in terms of an orthonormal basis { ϕ k } ∞ k =0 as θ ( x ) = ∑ ∞ k =0 a k ϕ k ( x ) , where a k := ( θ, ϕ k ) H are the coefficients (or spectrum) of θ in terms of { ϕ k } k ≥ 0 , and ( · , · ) H denotes the inner product of H . Substituting this expansion to functional F ([ θ ]) , we can define a multivariable function f ( { a k } ∞ k =0 ) := F ([ ∑ ∞ k =0 a k ϕ k ]) for any functional F : H → R . Truncating k at m -1 ∈ Z ≥ 0 gives the cylindrical approximation of functionals:

where P m is the projection operator s.t. P m θ ( x ) := ∑ m -1 k =0 a k ϕ k ( x ) , and m is referred to as the degree of approximation. See Thm. C.19 and Thm. C.20 for the uniform convergence and convergence rate of this approximation, originally given by [75, 92].

Secondly, we define the cylindrical approximation of functional derivatives . The functional derivative of F w.r.t. θ at x is defined as δF ([ θ ]) δθ ( x ) := lim ϵ → 0 F ([ θ ( y )+ ϵδ ( x -y )]) -F ([ θ ( y )]) ϵ , where δ ( x ) denotes the Dirac delta function. This definition is impractical to simulate on computers with spacetime discretization; thus, we employ the expansion δF ([ θ ]) δθ ( x ) = ∑ ∞ k =0 ( δF ([ θ ]) δθ , ϕ k ) H ϕ k ( x ) . The expansion is possible because δF ([ θ ]) δθ ( x ) itself is a function of x in H and thus can be represented as an orthonormal basis expansion. Note that the expansion coefficients ( δF ([ θ ]) δθ , ϕ k ) H are known to be equal to ∂f ∂a k (see App. C.4.2 for the proof). Hence, truncating θ at m -1 gives the cylindrical approximation of functional derivatives:

Note that Eq. (2) is different from the cylindrical approximation adopted in [91, 92]. They do not apply P m to δF ([ P m θ ]) /δθ ( x ) , and the emerging 'tail term' Σ ∞ k = m ( δF ([ θ ]) /δθ, ϕ k ) ϕ k ( x ) is simply ignored without any rationale.

The first main theoretical contribution of our work is the following convergence theorem of Eq. (2).

Theorem 3.1 (Pointwise convergence of approximated functional derivatives (informal)) . For arbitrary θ ∈ H and orthonormal basis { ϕ 0 , ϕ 1 , . . . } , Eq. (2) converges to δF ([ θ ]) δθ as m →∞ .

The formal statement and proof are given in App. D.1. The convergence rate is the same as ∥ θ -P m θ ∥ if δF ([ θ ]) /δθ ( x ) ∈ span { ϕ 0 , . . . , ϕ m -1 } . A technical discussion when this is not the case is provided in App. D.

Finally, we define the cylindrical approximation of FDEs [92]. In this paper, we consider the abstract evolution equation , a class of FDEs having the following form:

with F ([ θ ] , 0) = F 0 ([ θ ]) , where L ([ θ ]) is a linear functional operator, and F 0 is a given initial condition. The abstract evolution equation is a crucial class of FDEs in physics, mathematics, and engineering, including the Hopf functional equation, Fokker-Planck functional equation, and functional Hamilton-Jacobi equation. The cylindrical approximation of the abstract evolution equation is given by

with f ( a , 0) = f 0 ( a ) , where a := ( a 0 , . . . , a m -1 ) ⊤ , f ( a , t ) := F ([ P m θ ] , t ) , and f 0 ( a ) := F 0 ([ P m θ ]) . The operator L m ([ θ ]) is the cylindrical approximation of L ([ θ ]) . Examples are given in Secs. 3.1.1 & 3.1.2.

The second main theoretical contribution of our work is the following convergence theorem of solutions:

Theorem 3.2 (Convergence of approximated solutions (informal)) . Under the cylindrical approximation (Eq. (2) ), if the FDE depends on functional derivatives only in the form of the inner product ( v, δF ([ θ ]) δθ ) H ( v ∈ H ), then, the solution of the approximated abstract evolution equation ( F ([ P m θ ] , t ) ) converges to the solution of the original one ( F ([ θ ] , t ) ) as m →∞ .

The proof is given in App. E. The convergence is visualized in Fig. 2. Similar theorems for the FDEs with the second or higher-order functional derivatives can be shown in a similar way. The innerproduct assumption in Thm. 3.2 is satisfied by major FDEs, such as the Hopf functional equation, Fokker-Planck functional equation, and functional Hamilton-Jacobi equation.

In the following, we apply the cylindrical approximation to two FDEs: the functional transport equation (FTE) and the Burgers-Hopf equation (BHE).

3.1.1 Application 1: Functional Transport Equation

We first construct a simple FDE, the functional transport equation (FTE), which is a generalization of the transport equation (the continuity equation) [53]. The FTE is provided by

with the initial condition F ([ θ ] , 0) = F 0 ([ θ ]) , where x ∈ [ -1 , 1] , and u ( x ) is a given function. Specifically, we use the initial condition F ([ θ ] , 0) = ρ 0 ∫ dxu ( x ) θ ( x ) with ρ 0 a constant. The exact solution is F ([ θ ] , t ) = F 0 ([ θ -ut ]) = ρ 0 ∫ dxu ( x )( θ ( x ) -u ( x ) t ) , as can be seen by substituting this into Eq. (5). More details and motivations of the FTE are provided in App. E.1.

The cylindrical approximation of the FTE is given by

with the initial condition f ( a , 0) = f 0 ( a ) , where θ ( x ) = ∑ m -1 k =0 a k ϕ k ( x ) , and f 0 ( a ) = F 0 ([ P m θ ]) = ρ 0 Σ m -1 k =0 u k a k with u k := ( u, ϕ k ) L 2 ([ -1 , 1]) = ∫ dxu ( x ) ϕ k ( x ) . We use the Legendre polynomials as the orthonormal basis { ϕ k } k ≥ 0 . The exact solution of Eq. (6) is f ( a , t ) = ρ 0 ∑ m -1 k =0 u k ( a k -u k t ) .

In our experiments in Sec. 4, we consider two types of FTEs: (i) u k := υ 0 for k = 1 (0 otherwise) and (ii) u k := υ 0 for k ≤ 14 (0 otherwise), where υ 0 is a constant. For convenience, we call them the linear and nonlinear initial conditions , respectively. The high-dimensional PDE thus obtained (Eq. (6)) is solved with PINNs (Sec. 3.2).

3.1.2 Application 2: Burgers-Hopf Equation

The second FDE is the Burgers-Hopf equation (BHE), a crucial equation in turbulence theory:

with the initial condition F ([ θ ] , 0) = F 0 ([ θ ]) , where x ∈ [ -1 / 2 , 1 / 2] . Specifically, we use the Gaussian initial condition F 0 ([ θ ]) = -µ ∫ dxθ ( x )+ 1 2 ∫ dx ∫ dx ′ C ( x, x ′ ) θ ( x ) θ ( x ′ ) , where µ is a constant, and C ( x, x ′ ) is the infinite-dimensional covariance matrix. The exact solution is F ([ θ ] , t ) = F 0 ([Θ]) , where Θ([ θ ] , x, t ) := 1 √ 4 πt ∫ ∞ -∞ dx ′ e -1 4 t ( x -x ′ ) 2 θ ( x ′ ) . The derivation is provided in App. D.2.1. Strictly speaking, Eq. (7) is a modification of the original BHE. The modification includes making the BHE dimensionless and neglecting the advection term. For more technical details, see App. E.2.

The cylindrical approximation of the BHE is given by

with the initial condition f ( a , 0) = f 0 ( a ) , where θ ( x ) = ∑ m -1 k =0 a k ϕ k ( x ) , f 0 ( a ) = F 0 ([ P m θ ]) = -¯ µa 0 + 1 2 Σ m -1 k,l =0 ˜ C kl a k a l with ˜ C kl := ∫ dx ∫ dx ′ C ( x, x ′ ) ϕ k ( x ) ϕ l ( x ′ ) . We use the Fourier series as the orthonormal basis: { ϕ k ( x ) } k ≥ 0 = { 1 , √ 2 sin( πkx ) , √ 2 cos( πkx ) } k ≥ 1 . Then, the exact solution under the cylindrical approximation is given by

where m = 2 M ( M ∈ N ). The derivation is given in App. D.2.2.

In our experiments in Sec. 4, we adopt two types of the covariance matrices: (i) ˜ C ij = σ 2 for all i = j ≥ 0 (0 otherwise) and (ii) ˜ C ij = σ 2 for i = j = 0 (0 otherwise), where σ 2 is a constant. Substituting (i) and (ii) into f 0 , we have two types of initial conditions, which we call the delta and constant initial conditions , respectively. Again, the high-dimensional PDE thus obtained (Eq. (8)) is solved with PINNs (Sec. 3.2).

3.2 Step 2: Solving Approximated FDEs with PINNs

We briefly introduce the foundation of PINNs [77]. PINNs are universal approximators and can solve PDEs. Let us consider a PDE ∂f ( t, x ) /∂t = N [ f ] with an initial condition B [ f ] | t =0 = 0 , where t ∈ [0 , 1] , x ∈ [ -1 , 1] . N and B are operators defining the PDE and the initial condition, respectively. The PINN aims to approximate the solution f ( t, x ) . Thus, the inputs to the PINN are t and x , randomly sampled from [0 , 1] and [ -1 , 1] , respectively. Note that ( t = 0 , x ) with x ∈ [ -1 , 1] are also input

to the PINN to compute the boundary loss. The inputs are transformed through linear layers and activation functions. The final output of the PINN is an approximation of f ( t, x ) , denoted by ˆ f ( t, x ) . The loss function is the weighted sum of the residual loss ∥ ∂ ˆ f ( t, x ) /∂t -N [ ˆ f ] ∥ and the boundary loss ∥ B [ ˆ f ] | t =0 ∥ , where ∥ · ∥ is a certain norm. The partial derivatives in the loss function can be computed via automatic differentiation of the PINN's output ˆ f w.r.t. the inputs ( t, x ) . Finally, the weight parameters of the PINN are minimized through backpropagation.

Next, we explain how to apply PINNs to our high-dimensional PDEs: see Fig. 3. For concreteness, consider the FTE (Eq. (6)) with the linear initial condition. The inputs to the PINN are ( t, a 0 , a 1 , . . . , a m -1 ) and (0 , a 0 , a 1 , . . . , a m -1 ) , randomly sampled from finite intervals, and the outputs are ˆ f ( a , t ) and ˆ f ( a , 0) , respectively. Then, ∂ ˆ f ( a , t ) /∂t and ∂ ˆ f ( a , t ) /∂ a are computed via automatic differentiation, and we obtain the residual loss ∥ ∂ ˆ f ( a , t ) /∂t +Σ m -1 k =0 u k ∂ ˆ f ( a , t ) /∂a k ∥ and the boundary loss ∥ ˆ f 0 ( a ) -ρ 0 Σ m -1 k =0 u k a k ∥ , where ˆ f 0 ( a ) := ˆ f ( a , 0) . These losses are minimized via mini-batch optimization.

Computational Complexity The total computational complexity w.r.t. m up to the computation of functional derivatives is given by O ( m ) + O ( m r ) = O ( m r ) , where r ≥ 1 is the order of the functional derivative included in the target FDE (typically 1 or 2 ). The first term O ( m ) comes from the input layer of the PINN. The second term O ( m r ) comes from the computation of functional derivatives under the cylindrical approximation (Eq. (2)). See App. F.4 for more detailed discussions on computational complexity.

This is a notable reduction from discretizationbased methods such as finite difference and element methods, which typically require exponentially large computational complexity w.r.t. the dimension of PDE m . Also, O ( m r ) is significantly smaller than the state-of-the-art cylindrical approximation algorithm, the CP-ALS [91], which requires O ( m 6 ) (the derivation is given in App. F.2). Consequently, given that m represents the 'class size' of input functions and functional derivatives (Eqs. (1) & (2)), our approach significantly extends the range of input functions and functional derivatives that can be represented via the cylindrical approximation. In fact, our approach extends the degrees of polynomials or Fourier series used for the approximation from 6 [91] to 1000 (Sec. 4).

Finally, we note that the selection of basis func-

Table 1: Mean relative and absolute errors of models trained on FTEs under linear (top two) and nonlinear (bottom two) initial conditions. I.C. is short for 'initial condition'. The error bars are the standard deviation over 10 training runs with different random seeds. Note that this table is not for the assessment of the theoretical convergence of the cylindrical approximation (see Fig. 2, App. H.6, and the footnote in Sec. 4.1 instead).

±

×

tions influences computational efficiency. The choice depends on the specific FDE, boundary conditions, symmetry, function spaces, and numerical stability. For further discussions, see App. F.3).

In summary, our proposed approach transforms an FDE into a high-dimensional PDE using cylindrical approximation and then solves it with a PINN, which serves as a universal approximator of the solution (Figs. 1 & 3). It is important to note that our model employs the basic PINN framework, allowing for seamless integration with any techniques developed within the PINN community.

4 Experiment

±

×

As a proof of concept for our approach, we numerically solve the FTE and BHE. 1 These two FDEs are suitable for numerical experiments because their analytic solutions are available, allowing for the computation of absolute and relative errors, major metrics in the numerical analysis of PDEs and FDEs. Note that the analytic solutions for most FDEs are currently unknown due to their mathematical complexity.

Setups. We use a 4-layer PINN with 3 × (linear + sin activation + layer normalization [6]) + last linear layer. The total loss function is the smooth L 1 loss or the sum of L 1 and L ∞ losses. Softmax loss-reweighting is employed. The optimizer is AdamW [59]. The learning rate scheduler is the linear warmup with cosine annealing with warmup [58]. Latin hypercube sampling [64, 39] is used for the training, validation, and test sets. For the BHE, the sampling range is decayed quadratically in terms of k ∈ { 0 , 1 , . . . , m -1 } to stabilize the training. We use L 1 relative and absolute errors, standard performance metrics for numerical analysis of PDEs and FDEs. Absolute error 1 N Σ N i =1 | f ( a i , t i ) -ˆ f ( a i , t i ) | is used instead of relative error 1 N Σ N i =1 | f ( a i , t i ) -ˆ f ( a i , t i ) | / | f ( a i , t i ) | when the analytic solution is close to zero because relative error in such a region blows up by definition, regardless of the model's prediction. υ 0 , ρ 0 , ¯ µ , and σ 2 are set to 1 , 1 , 8 , and 10 , respectively. In App. G, we

±

×

k

provide more detailed setups for reproducibility, including the range of

a

.

4.1 Result : Functional Transport Equation

Tab. 1 shows the main result. 2 Our model achieves typical relative error orders of PINNs ∼ 10 -3 , even when the degree is as large as 1000 , which means our model's capability of representing θ and δF ([ θ ] , t ) /δθ ( x ) as polynomials of degree 1000 . This is a notable improvement from the state-of-the-art algorithm [91], which can handle m ≤ 6 .

Fig. 5 shows the absolute error of the first-order functional derivative estimated at t = 1 and θ = 0 . Again, θ = 0 is not included in the training set. The errors in the top four panels increase at the edges of intervals ( x = ± 1 ) due to Runge's phenomenon [80].

Fig. 4 visualizes the analytic solution, prediction, and absolute error of a model trained on the FTE with degree 100 under the linear initial condition. Note that some of the collocation points used for plotting Fig. 4 are not in the training set, as can be seen from Figs. 11-16 in App. H.3. This aspect highlights the model's ability to extrapolate beyond its training data ( a k ∼ 0 ).

4.2 Result: Burgers-Hopf Equation

Tab. 2 shows the main result. Again, our model successfully achieves ∼ 10 -3 , the typical order of relative error of PINNs. See Fig. 2, App. H.6, and the footnote in Sec. 4.1 for the assessment of the theoretical convergence of the cylindrical approximation.

Fig. 6 shows the relative error of first-order functional derivatives at θ = 0 . Note again that some of the collocation points used for this figure are not included in the training dataset, highlighting the model's ability to extrapolate beyond its training dataset ( a k ∼ 0 ). Additionally, the error is reduced by a factor of 10 -1 by incorporating a loss term corresponding to the identity W ([0] , t ) = 0 (bottom four panels).

Fig. 7 visualizes the analytic solution, prediction, and absolute error of a model trained on the BHE with degree 20 under the delta initial condition. The absolute error w.r.t. a 0 is 10 -4 times smaller than the scale of the solution; i.e., the model learns θ well in the direction of a 0 , which dominates the analytic solution. Conversely, the absolute error w.r.t. a 19 is on par with the scale of the solution. This result is anticipated because the dependence of the solution on a 19 is much smaller than a 0 . This relationship is evident from Eq. (9), which indicates that the higher degree terms decay exponentially in terms of k , l , and t , and the solution is dominated by a k with k ≲ 1 . Therefore, optimizing the model in the direction of a 19 has only a negligible effect on minimizing the loss function.

Finally, many additional experimental results are provided in App. H, including a comparison with the CP-ALS algorithm.

5 Limitations

One limitation of our work is that the spacetime dimension is limited to 1 + 1 ( t and x ) in our experiments. However, generalization to 1 + d dimensions is feasible, albeit with additional computational costs. For d > 1 dimensional spaces, we have several options for expansion bases [79].

Another limitation is that the orders of functional derivatives in FDEs in our experiments are limited to r = 1 . However, extending to r ≥ 2 is straightforward. For instance, the cylindrical approximation of the second-order functional derivative is expressed as δ 2 F ([ θ ] , t ) /δθ ( x ) δθ ( y ) ≈ Σ m -1 k,l =0 ∂ 2 f ( a , t ) /∂a k ∂a l ϕ k ( x ) ϕ l ( y ) , which can be computed via backpropagation twice.

Furthermore, this paper focuses exclusively on the abstract evolution equation. While this includes important FDEs (see Sec. 3), it does not cover certain equations, such as the Schwinger-Dyson equation or the Wetterich equation. Nonetheless, the mathematical foundations regarding the existence and uniqueness of these FDEs remain unestablished, which is beyond the scope of our paper. Once these foundations are defined, applying our model to these equations would be straightforward. More discussions on limitations, including technical ones, are provided in App. F.1.

Acknowledgments and Disclosure of Funding

Taiki Miyagawa, an employee at NEC Corporation, contributed to this paper independently of his company role. He undertook this research as an independent researcher under the permission by the company. Takeru Yokota was supported by the RIKEN Special Doctoral Researchers Program.

References

V. Acary and B. Brogliato. Numerical Methods for Nonsmooth Dynamical Systems: Applications in Mechanics and Electronics . Springer Science & Business Media, 2008.

T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '19, page 2623-2631, New York, NY, USA, 2019. Association for Computing Machinery. License: MIT License.

T. Alankus. The generating functional for the probability density functions of navier-stokes turbulence. Journal of Statistical Physics , 53:1261-1271, 1988.

A. Anandkumar, K. Azizzadenesheli, K. Bhattacharya, N. Kovachki, Z. Li, B. Liu, and A. Stuart. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations , 2019.

N. Aronszajn. Differentiability of Lipschitzian mappings between Banach spaces. Studia Mathematica , 57(2):147-190, 1976.

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

J. C. Baez and S. Sawin. Functional integration on spaces of connections. Journal of Functional Analysis , 150(1):1-26, 1997.

S. Basir and I. Senocak. Critical investigation of failure modes in physics-informed neural networks. In AiAA SCITECH 2022 Forum , page 2353, 2022.

M. Bauer, E. Dupont, A. Brock, D. Rosenbaum, J. R. Schwarz, and H. Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv: 2302.03130 , 2023.

A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: A survey. Journal of Marchine Learning Research , 18:1-43, 2018.

M. Beran. Statistical continuum theories. Transactions of the Society of Rheology , 9(1):339-355, 1965.

J. Blechschmidt and O. G. Ernst. Three ways to solve partial differential equations with neural networks-A review. GAMM-Mitteilungen , 44(2):e202100006, 2021.

N. Bogoliubov. On the theory of superfluidity. Journal of Physics , 11(1):23, 1947.

S. Brugiapaglia, S. Dirksen, H. C. Jung, and H. Rauhut. Sparse recovery in bounded Riesz systems with applications to numerical methods for PDEs. Applied and Computational Harmonic Analysis , 53:231-269, 2021.

M. S. Burlutskaya. On Riesz bases of root functions for a class of functional-differential operators on a graph. Differential Equations , 45:779-788, 2009.

R. Carmona, F. Delarue, et al. Probabilistic theory of mean field games with applications I-II . Springer, 2018.

Y. Chen, B. Hosseini, H. Owhadi, and A. M. Stuart. Solving and learning nonlinear PDEs with Gaussian processes. Journal of Computational Physics , 447:110668, 2021.

Y. T. Chow, W. Li, S. Osher, and W. Yin. Algorithm for Hamilton-Jacobi equations in density space via a generalized Hopf formula. Journal of Scientific Computing , 80:1195-1239, 2019.

S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physics-informed neural networks: Where we are and what's next. Journal of Scientific Computing , 92(3):88, 2022.

S. Das and S. Tesfamariam. State-of-the-art review of design of experiments for physics-informed deep learning. arXiv preprint arXiv:2202.06416 , 2022.

A. Daw, J. Bu, S. Wang, P. Perdikaris, and A. Karpatne. Mitigating propagation failures in physicsinformed neural networks using retain-resample-release (R3) sampling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 7264-7302. PMLR, 23-29 Jul 2023.

P. Di Gianantonio, A. Edalat, and R. Gutin. A language for evaluating derivatives of functionals using automatic differentiation. Electronic Notes in Theoretical Informatics and Computer Science , 3, 2023.

N. Dupuis, L. Canet, A. Eichhorn, W. Metzner, J. Pawlowski, M. Tissier, and N. Wschebor. The nonperturbative functional renormalization group and its applications. Physics Reports , 910:1-114, 2021.

R. Easther, D. Ferrante, G. Guralnik, and D. Petrov. A review of two novel numerical methods in QFT. arXiv preprint hep-lat/0306038 , 2003.

K.-J. Engel, R. Nagel, and S. Brendle. One-parameter semigroups for linear evolution equations , volume 194 of Graduate Texts in Mathematics . Springer, 2000.

L. Fiedler, K. Shah, M. Bussmann, and A. Cangi. Deep dive into machine learning density functional theory for materials science and chemistry. Physical Review Materials , 6(4):040301, 2022.

R. F. Fox. Functional-calculus approach to stochastic differential equations. Physical Review A , 33:467476, Jan 1986.

G. Franzese, G. Corallo, S. Rossi, M. Heinonen, M. Filippone, and P. Michiardi. Continuous-time functional diffusion processes. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

K. Friedrichs and H. Shapiro. Integration over Hilbert space and outer extensions. Proceedings of the National Academy of Sciences , 43(4):336-338, 1957.

N. Fukuda, T. Kinoshita, and T. Kubo. On the finite element method with Riesz bases and its applications to some partial differential equations. In 2013 10th International Conference on Information Technology: New Generations , pages 761-766, 2013.

R. Fukuda, T. Kotani, Y. Suzuki, and S. Yokojima. Density functional theory through Legendre transformation. Progress of Theoretical Physics , 92(4):833-862, 10 1994.

W. Gangbo, W. Li, S. Osher, and M. Puthawala. Unnormalized optimal transport. Journal of Computational Physics , 399(C), dec 2019.

Z. Gao, L. Yan, and T. Zhou. Failure-informed adaptive sampling for pinns. SIAM Journal on Scientific Computing , 45(4):A1971-A1994, 2023.

I. I. Gikhman and A. V. Skorokhod. The theory of stochastic processes I . Classics in Mathematics. Springer Berlin Heidelberg, 2004.

V. Gopakumar, S. Pamela, and D. Samaddar. Loss landscape engineering via data regulation on PINNs. Machine Learning with Applications , 12:100464, 2023.

D. Guidetti, B. Karasözen, and S. Piskarev. Approximation of abstract differential equations. Journal of Mathematical Sciences , 122(2):3013-3054, 2004.

B.-Z. Guo and J.-M. Wang. Riesz basis generation of abstract second-order partial differential equation systems with general non-separated boundary conditions. Numerical Functional Analysis and Optimization , 27(3-4):291-328, 2006.

C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. Del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with NumPy. Nature , 585(7825):357-362, 09 2020. License: BSD 3-Clause "New" or "Revised" License.

J. C. Helton and F. J. Davis. Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems. Reliability Engineering & System Safety , 81(1):23-69, 2003.

D. Hendrycks and K. Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415 , 2016.

J. S. Hesthaven, S. Gottlieb, and D. Gottlieb. Spectral methods for time-dependent problems , volume 21. Cambridge University Press, 2007.

P. Hohenberg and W. Kohn. Inhomogeneous electron gas. Physical review , 136:B864-B871, Nov 1964.

E. Hopf. Statistical hydromechanics and functional calculus. Journal of rational Mechanics and Analysis , 1:87-123, 1952.

Z. Hu, K. Shukla, G. E. Karniadakis, and K. Kawaguchi. Tackling the curse of dimensionality with physics-informed neural networks. arXiv preprint arXiv:2307.12306 , 2023.

Z. Hu, Z. Yang, Y. Wang, G. E. Karniadakis, and K. Kawaguchi. Bias-variance trade-off in physicsinformed neural networks with randomized smoothing for high-dimensional pdes. arXiv preprint arXiv:2311.15283 , 2023.

C. Itzykson and J.-B. Zuber. Quantum field theory . Courier Corporation, 2012.

A. D. Jagtap, K. Kawaguchi, and G. E. Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics , 404:109136, 2020.

M. Jordan, Y. Wang, and A. Zhou. Empirical Gateaux derivatives for causal inference. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 8512-8525. Curran Associates, Inc., 2022.

V. I. Klyatskin. Dynamics of stochastic systems . Elsevier, 2005.

A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems , 34:26548-26560, 2021.

V. Kurdyumov and A. Khromov. Riesz bases formed by root functions of a functional-differential equation with a reflection operator. Differential Equations , 44(2), 2008.

V. P. Kurdyumov and A. P. Khromov. The Riesz bases consisting of eigen and associated functions for a functional differential operator with variable structure. Russian Mathematics , 54:33-45, 2010.

L. D. Landau and E. M. Lifshitz. Fluid Mechanics: Landau and Lifshitz: Course of Theoretical Physics, Volume 6 , volume 6. Elsevier, 2013.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations , 2021.

Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar. Physicsinformed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794 , 2021.

M. Lin. Automatic functional differentiation in JAX. In The Twelfth International Conference on Learning Representations , 2024.

J. Lindenstrauss and D. Preiss. On Fréchet differentiability of Lipschitz maps between Banach spaces. Annals of Mathematics , pages 257-288, 2003.

I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2017.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

L. Lu, P. Jin, and G. E. Karniadakis. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193 , 2019.

L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature machine intelligence , 3(3):218-229, 2021.

L. D. Luigi, A. Cardace, R. Spezialetti, P. Z. Ramirez, S. Salti, and L. di Stefano. Deep learning on implicit neural representations of shapes. In The Eleventh International Conference on Learning Representations , 2023.

P. Mankiewicz. On the differentiability of Lipschitz mappings in Fréchet spaces. Studia Mathematica , 45(1):15-29, 1973.

M. D. McKay, R. J. Beckman, and W. J. Conover. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics , 42(1):55-61, 2000.

R. Melrose. Introduction to functional analysis, 2005.

B. Meuris, S. Qadeer, and P. Stinis. Machine-learning-based spectral methods for partial differential equations. Scientific Reports , 13(1):1739, 2023.

A. S. Monin and A. M. Yaglom. Statistical fluid mechanics, volume II: Mechanics of turbulence , volume 2. Courier Corporation, 2013.

R. D. Neidinger. Introduction to automatic differentiation and MATLAB object-oriented programming. SIAM review , 52(3):545-563, 2010.

R. H. Nichols. Turbulence models and their application to complex flows, 2010.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. License: https://github.com/pytorch/pytorch/blob/master/LICENSE.

M. E. Peskin and D. V. Schroeder. An Introduction to quantum field theory . Addison-Wesley, Reading, USA, 1995.

J. Polchinski. Renormalization and effective lagrangians. Nuclear Physics B , 231(2):269 - 295, 1984.

J. Polonyi and K. Sailer. Effective action and density-functional theory. Physical Review B , 66:155113, Oct 2002.

D. Preiss. Differentiability of Lipschitz functions on Banach spaces. Journal of Functional Analysis , 91(2):312-345, 1990.

P. Prenter. A Weierstrass theorem for real, separable Hilbert spaces. Journal of Approximation Theory , 3(4):341-351, 1970.

R. Rabah, G. M. Sklyar, and A. V. Rezounenko. Generalized Riesz basis property in the analysis of neutral type systems. Comptes Rendus Mathematique , 337(1):19-24, 2003.

M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2019.

S. Ramasinghe and S. Lucey. Beyond periodicity: Towards a unifying framework for activations in coordinate-MLPs. In European Conference on Computer Vision , pages 142-158. Springer, 2022.

A. Rodgers and D. Venturi. Tensor approximation of functional differential equations. arXiv preprint arXiv:2403.04946 , 2024.

C. Runge. Über empirische funktionen und die interpolation zwischen äquidistanten ordinaten. Zeitschrift für Mathematik und Physik , 46:224-243, 1901.

L. Ruthotto, S. J. Osher, W. Li, L. Nurbekyan, and S. W. Fung. A machine learning framework for solving high-dimensional mean field game and mean field control problems. Proceedings of the National Academy of Sciences , 117(17):9183-9193, 2020.

A. Schwenk and J. Polonyi. Towards density functional calculations from nuclear forces. In 32nd International Workshop on Gross Properties of Nuclei and Nuclear Excitation: Probing Nuclei and Nucleons with Electrons and Photons , pages 273-282, 2004.

R. Seiringer. The excitation spectrum for weakly interacting bosons. Communications in Mathematical Mhysics , 306:565-578, 2011.

P. Sharma, L. Evans, M. Tindall, and P. Nithiarasu. Stiff-PDEs and physics-informed neural networks. Archives of Computational Methods in Engineering , 30(5):2929-2958, 2023.

J. Shen, T. Tang, and L. Wang. Spectral methods: Algorithms, analysis and applications , volume 41 of Springer Series in Computational Mathematics . Springer Berlin Heidelberg, 2011.

M. Va˘ ınberg. Variational Methods for the Study of Nonlinear Operators . Holden-Day Series in Mathematical Physics. Holden-Day, 1964.

M. Valiev and G. W. Fernando. Generalized Kohn-Sham density-functional theory via effective action formalism. arXiv preprint arXiv:cond-mat/9702247 , 1997.

J. van Neerven. Stochastic evolution equations. ISEM lecture notes , 2008.

G. Van Rossum and F. L. Drake. Python 3 Reference Manual . CreateSpace, Scotts Valley, CA, 2009.

D. Venturi. Conjugate flow action functionals. Journal of Mathematical Physics , 54(11), 2013.

D. Venturi. The numerical approximation of nonlinear functionals and functional differential equations. Physics Reports , 732:1-102, 2018.

D. Venturi and A. Dektor. Spectral methods for nonlinear functionals and functional differential equations. Research in the Mathematical Sciences , 8(2):27, 2021.

C. Wang, S. Li, D. He, and L. Wang. Is L 2 physics informed loss always suitable for training physics informed neural network? In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

H. Wang, L. Lu, S. Song, and G. Huang. Learning specialized activation functions for physics-informed neural networks. arXiv preprint arXiv:2308.04073 , 2023.

S. Wang, H. Wang, and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science advances , 7(40):eabi8605, 2021.

Y. Wang and C.-Y. Lai. Multi-stage neural networks: Function approximator of machine precision. arXiv preprint arXiv:2307.08934 , 2023.

F. J. Wegner and A. Houghton. Renormalization group equation for critical phenomena. Physical Review A , 8:401-412, Jul 1973.

E. Weinan, J. Han, and A. Jentzen. Algorithms for solving high dimensional PDEs: From nonlinear Monte Carlo to machine learning. Nonlinearity , 35(1):278, 2021.

E. Weinan, J. Han, and Q. Li. A mean-field optimal control formulation of deep learning. arXiv preprint arXiv:1807.01083 , 2018.

C. Wetterich. Exact evolution equation for the effective potential. Physical Letter B , 301:90-94, 1993.

N. Wiener. Nonlinear Problems in Random Theory . MIT Press, 1966.

K. G. Wilson and J. Kogut. The renormalization group and the ϵ expansion. Physical Reports , 12(2):75 199, 1974.

J. Yao, C. Su, Z. Hao, S. Liu, H. Su, and J. Zhu. MultiAdam: Parameter-wise scale-invariant optimizer for multiscale training of physics-informed neural networks. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 39702-39721. PMLR, 23-29 Jul 2023.

T. Yokota. Physics-informed neural networks for solving functional renormalization group on a lattice. Physical Review B , 109:214205, Jun 2024.

T. Yokota and T. Naito. Ab initio construction of the energy density functional for electron systems with the functional-renormalization-group-aided density functional theory. Physical Review Research , 3:L012015, Feb 2021.

Q. Zeng, S. H. Bryngelson, and F. T. Schaefer. Competitive physics informed networks. In ICLR 2022 Workshop on Gamification and Multiagent Solutions , 2022.

A. Zhou, K. Yang, Y. Jiang, K. Burns, W. Xu, S. Sokota, J. Z. Kolter, and C. Finn. Neural functional transformers. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Appendix

Contents

A Supplementary Related Work

Note on [91] & [92]. Most of the numerical results presented in [91] and [92] are derived from simulations based on analytically specified functions and functionals, without involving the numerical integration of FDEs. For instance, Fig. 2 in [92] does not depict the result of numerically solving an FDE. Instead, it illustrates the approximation error of the cylindrical approximation of an analytically given functional (refer to Eq. (127) in [92]). The performance of numerical integration of FDEs using CPU-based algorithms is given in Fig. 38 in [91], which shows the application of the CP-ALS and hierarchical Tucker (HT) methods for cases with m ≤ 6 . Note that the HT is reported to be slower than the CP-ALS [91].

Applications of functionals. Functionals play a fundamental role in stochastic systems [101, 27, 49], Fokker-Planck equations [27], the statistical theory of turbulence [43, 3, 67], the theory of superfluidity [13, 83], quantum field theories [90, 91], mean-field games [16], many-body Schrödinger equations [42], mean-field optimal control [99, 81], and unnormalized optimal transport [32].

Examples of FDEs. Examples of FDEs include the Schwinger-Dyson equation in quantum field theory [46], the Hopf characteristic functional equation in fluid mechanics and random processes [11, 49, 67], probability density functional equations, and effective Fokker-Planck systems [91].

Operator learning. Functionals can be seen as operators that map a function to a scalar; thus, operator learning [4, 54, 60, 61] appears to be a promising approach to learning functionals. However, this method requires simulation or observation data unless PINNs are used simultaneously [55, 95]. In other words, operator learning methods solve inverse problems, while we focus on forward problems, where only the equation to be solved is given.

Other approximation methods for FDEs. A common class of solvers for FDEs is based on truncating power series expansions at a finite order [67]. This includes the functional Taylor expansion, which expands a functional in terms of its argument functions. However, its applicability is limited because solutions obtained from the functional Taylor expansion can only be used for input functions close to the expansion center.

Another type of expansion used in the theory of functional renormalization group is the derivative expansion [23]. It is an expansion in terms of derivatives of the input functions. However, solutions are only feasible for inputs close to constants. For example, in the three-dimensional O ( N ) statistical model, derivative expansions up to the sixth order have been executed [23], but they are limited to calculations in uniform states and cannot handle non-uniform states.

Yet another expansion method uses the Reynolds number to distinguish between laminar and turbulent flow and has been applied to the Hopf equation [67]. However, increasing the truncation order poses a significant challenge. Specifically, calculating each expansion coefficient requires spatial integrals, leading to an exponential increase in computational costs.

Influence functions can be used for approximating Gateaux derivatives. In [48], the proposed approach is based on a finite-difference approximation of Gateaux derivatives, which requires a computational mesh for input function space when solving FDEs. Such an approach is infeasible because the number of mesh points grows exponentially with respect to the size of the input-function space.

High-dimensional PDEs. In our experiments, with the help of PINNs, we numerically solved 1000-dimensional PDEs, which are impossible to handle with discretization-based methods, such as the finite element method. Numerical computation of high-dimensional PDEs is known to suffer from the curse of dimensionality, making PDEs with dimensions d ≥ 40 particularly challenging [98].

However, rapid development in this field, especially in PINNs, has enabled solving much higherdimensional PDEs. For example, a 100,000-dimensional PDE is now tractable [44], which can be combined with our model. Nevertheless, d ∼ O (100) is typically sufficient in practice as long as input functions are regular. See also Fig. 10.

PINNs. PINNs are a type of mesh-free universal approximators of PDE solutions [77]. There are several machine learning-based mesh-free solvers, e.g., [17].

Automatic functional differentiation of higher-order functions. Automatic differentiation of higher-order functions has a long history in theoretical computer science (see [68, 10] and the references therein)." Most studies focus on the mathematical aspects of programming languages, particularly how to implement reliable automatic functional differentiation, which is beyond our scope. We cite two recent papers that explicitly mention functional derivatives. Di et al. [22] develop a language to compute automatic functional derivatives; however, the implementation is not available. Lin [56] provides a JAX implementation of functional derivatives w.r.t. parameterized input functions. However, it supports only local, semilocal, and several nonlocal operators, limiting the functional space. In contrast, our model extends its approximability as m increases.

Density functional theory (DFT). An alternative neural network-based approach to functional analysis utilizes finite element methods for spacetime grid approximation, commonly employed in first-principles computations of density functional theory (DFT) [26]. Examples include a neural network, ˆ F ( { y j := f ( r j ) } j ) , approximating a target functional F ([ f ]) by evaluating f at specific grid points { r j } j . Functional derivatives at each grid point can be computed using automatic differentiation: { δF ([ f ]) δf ( r ) } r ≒ { ∂ ˆ F ( { y j } j ) ∂y i } i . However, the central focus of the machine learning studies for DFT does not include solving PDEs, let alone FDEs.

Neural functional networks. Implicit Neural Representations (INRs) is another strategy to handle functions as the inputs to neural networks [107, 9, 62, 28]. However, this method requires a large number of weight parameters, resulting in substantial computational demands.

B Additional Introduction to FDEs

B.1 Background

Functional differential equations (FDEs) are prevalent across various scientific fields, including the Hopf equation in statistical turbulence theory [43], the Schwinger-Dyson equation in quantum field theory [71], the functional renormalization group equation [97, 102, 72, 100], the Fokker-Planck equation in statistical mechanics [27], and equations for the energy density functional in DFT [73, 82]. The strength of FDEs lies in their comprehensive nature, enabling the derivation of various statistical properties of physical systems. For instance, the Hopf equation yields the characteristic functional, encompassing all information about simultaneous correlations of velocities at different positions, a crucial quantity in turbulence theory. Therefore, a highly accurate, efficient, and universal method for solving FDEs significantly impacts a broad range of scientific research but is yet to be explored.

A common class of solvers for FDEs is based on truncating power series expansions at a finite order [67]. This includes the functional Taylor expansion, which expands a functional in terms of its argument functions. However, its applicability is limited because solutions obtained from the functional Taylor expansion can only be used for input functions close to the expansion center.

Another type of expansion used in the functional renormalization group theory is the derivative expansion [23]. It is an expansion in terms of derivatives of the input functions. However, solutions are only feasible for inputs close to constants. For example, in the three-dimensional O ( N ) statistical model, derivative expansions up to the sixth order have been executed [23], but they are limited to calculations in uniform states and cannot treat non-uniform states.

Yet another expansion method uses the Reynolds number, distinguishing between laminar and turbulent flow, and has been applied to the Hopf equation [67]. However, increasing the truncation order poses a significant challenge. Specifically, calculating each expansion coefficient requires spatial integrals, leading to an exponential increase in computational costs.

An alternative to solving FDEs is the cylindrical approximation [91]. In this method, the input function is expanded using a set of basis functions truncated at a finite degree. The cylindrical approximation transforms an FDE into a high-dimensional PDE, and discretization-based methods are often used together. To address its high computational cost, tensor decomposition methods are also used. Canonical Polyadic (CP) tensor expansion with the Alternating Least Squares (ALS) method [91] is the state-of-the-art algorithm in this class. The reported results to date are limited to cases with six or fewer basis functions. The computational cost related to m is at least O ( m 6 ) , presenting a challenge in increasing the number of bases m . In contrast, our model scales ∼ O ( m ) + O ( m r ) , where r is the order of the functional derivative included in the target FDE. r is typically 1 or 2, and thus the dependence on m is ∼ O ( m ) or O ( m 2 ) .

An alternative to solving FDEs is the cylindrical approximation [91]. In this method, the input function is expanded using a set of basis functions truncated at a finite degree. The cylindrical approximation transforms an FDE into a high-dimensional PDE, often solved using discretizationbased methods. To address the high computational cost, tensor decomposition methods are also used. The CP-ALS method [91] is the state-of-the-art algorithm in this class. The reported results to date are limited to cases with six or fewer basis functions. The computational cost related to m is at least O ( m 6 ) , presenting a challenge in increasing the number of basis functions m . In contrast, our model scales as O ( m r ) , where r is the order of the functional derivative in the target FDE. Typically, r is 1 or 2.

Below, we provide examples from the fields of turbulence, quantum field theory, and density functional theory.

B.2 Turbulence

Turbulence appears everywhere, from natural systems (e.g., river flows and wind currents) to artificial systems (e.g., water flow in pipes and airflow over airplane surfaces). Understanding its properties is important both in natural sciences and in engineering. However, turbulence is a very complex system involving many degrees of freedom, and the only way to theoretically represent the properties of turbulence is through statistical methods. The Hopf equation [43] is an FDE that comprehensively describes the properties of turbulence. For example, the Hopf equation for a fluid following the

Navier-Stokes equations is written as follows:

Here, Φ([ θ ] , t ) is a characteristic functional of a test function θ ( x ) = ( θ 1 ( x ) , θ 2 ( x ) , θ 3 ( x )) , and ν is the kinematic viscosity. The characteristic functional contains all the information about the correlations of velocities at different locations at the same time. Indeed, the moments of velocity can be obtained from the derivatives of the characteristic functional as follows:

where u i ( x , t ) is the i the component of the velocity at ( x , t ) . In particular, the Fourier transform of the second-order cumulant is known as the energy spectrum, which represents the contribution of eddies at various momentum scales to turbulence. The search for a comprehensive method to describe the behavior of the energy spectrum across a wide range of momentum scales continues, and for this reason, developing methods to solve the Hopf equation holds significant importance.

More specifically, let us consider the fluid mechanics of aircraft or pipeline flow. For such systems, the second-order functional derivative of the solution of the Hopf characteristic functional gives the two-point correlation function of the velocity field at arbitrary two positions x and y and an arbitrary time t . The Fourier transform of it w.r.t. x and y is called the energy spectrum, which indicates which scales of motion are most energetic in the fluid flow. The energy spectrum is used to model and predict the behavior of turbulence, e.g., constructing safe and efficient shapes of airplanes or pipelines [69].

B.3 Quantum Field Theory

Quantum mechanics tells us that physical quantities in the microscopic world do not always have deterministic values but fluctuate. In quantum field theory (QFT), which is a branch of quantum mechanics and forms the basis of modern particle physics, particles are described as fluctuating fields spreading throughout spacetime. QFT allows us to describe the properties of elementary particles in a statistical way, i.e., the correlation functions of fields at different points in spacetime. Therefore, developing methods to calculate correlation functions is very important for understanding the properties of elementary particles.

Several FDEs provide information on the correlation function of fields. A well-known example is the Schwinger-Dyson equation [71]. In the statistical model known as the ϕ 4 model, which is described by the following action

the Schwinger-Dyson equation is given as follows:

Here, Z ([ J ]) is a quantity known as the partition function, and by functionally differentiating this quantity w.r.t. J ( x ) , all correlation functions for the field ϕ ( x ) can be obtained. Another method to describe the correlation functions is the functional renormalization group [97, 102, 72, 100]. The renormalization group is a method of analyzing physical systems based on the operation of reducing spacetime resolution. Under such operations, we can define an FDE for the effective action Γ([ ϕ ]) , which is calculated by the Legendre transformation of ln Z ([ J ]) . Γ([ ϕ ]) contains all the information of the correlation functions, similarly to Z ([ J ]) . Γ([ ϕ ]) satisfies the following FDE [100]:

k represents the momentum scale that specifies the resolution at which spacetime is observed, R k ( x ) is a function manually provided to realize the operation of the renormalization group, and G k ([ ϕ ] , x ′ , x ) is the regulated propagator defined as

B.4 Density Functional Theory

Density functional theory (DFT) is widely used in material science, quantum chemistry, and nuclear physics. The properties of materials and molecules are determined by the state of electrons, which follows the Schrödinger equation. Solving the Schrödinger equation becomes challenging especially when the material contains many electrons. Hohenberg and Kohn demonstrated that it is possible to determine the state of electrons by solving a variational equation for the density [42], instead of solving the Schrödinger equation directly. This is a density-based variational equation and has been shown to be easier to solve than the Schrödinger equation.

However, in DFT, exact calculations are usually not possible. The reason is that the Energy Density Functional (EDF), which provides the variational equation for density, cannot be precisely determined. Whether a good approximation for the EDF can be provided or not significantly influences the success of DFT calculations. There has been a lot of research on finding EDFs, including empirical approaches, for a long time. One recent approach is based on FDEs. Specifically, several FDEs are known to describe the evolution of the EDF when the two-body interaction U ( x -x ′ ) , e.g., the Coulomb interaction between electrons, gradually increases [73, 82]. When the interaction is replaced with λU ( x -x ′ ) , and when λ gradually increases, the FDE becomes:

Here, n ( x , τ ) is the density of electrons, and Γ λ ([ n ]) represents an effective action, which is an extension of the EDF [31, 87]. In addition to the coordinates x and x ′ , a virtual dimension, known as the imaginary time τ , is introduced. This FDE is expected to provide a new method for constructing the EDF [73, 82]. For example, the EDF of the three-dimensional electron system is derived from Eqs. (16-17) based on the functional Taylor expansion [105].

C Theoretical Background of Cylindrical Approximation and Convergence

The mathematical background of our theoretical contribution is provided to make our paper selfcontained. This section is based on the recent development of the theory of cylindrical approximation [92] and the classical spectrum theory [41, 85]. The differentiability of functionals is discussed in App. C.3, showing that the solutions of the FDEs in our experiment are differentiable. The equivalence and difference between the functional, Fréchet, and Gâteaux derivative are summarized in App. C.3. They are equivalent in practical settings, and we do not distinguish them in this paper.

C.1 Continuity and Compactness of Functionals

Let X be a Banach space unless otherwise stated. In this paper, a functional on X is defined as a map F from D ( F ) ⊂ X to R , where D ( F ) is the domain of F . Note that R cannot be replaced with Q in all the statements below. We first define pointwise continuity, uniform continuity, compactness, and complete continuity of functionals.

Definition C.1 (Pointwise continuity of F ) . A functional F : X ⊃ D ( F ) → R is continuous at θ ∈ D ( F ) ⊂ X if for any Cauchy sequence { θ n } n ⊂ D ( F ) ,

where || · || X is the norm induced by X .

Definition C.2 (Uniform continuity of F ) . A functional F : X ⊃ D ( F ) → R is uniformly continuous on D ( F ) if

where || · || X is the norm induced by X .

We simply say 'continuous' in the following.

Definition C.3 (Compactness of F ) . A functional F : X ⊃ D ( F ) → R is compact on D ( F ) if F maps any bounded subset of D ( F ) into a pre-compact subset of R .

Recall that A ⊂ R is a pre-compact subset if the closure of A , denoted by ¯ A , is compact.

Definition C.4 (Complete continuity of F ) . A functional F : X ⊃ D ( F ) → R is completely continuous on D ( F ) if F is continuous and compact on D ( F ) .

Based on these concepts, functional derivatives are defined.

C.2 Boundedness, Closedness, Compactness, and Pre-compactness of Metric Space of Functions

Next, we define boundedness, closedness, compactness, and pre-compactness of a metric space of functions.

Definition C.5 (Boundedness of metric space of functions) . Let X be a metric space of functions. K ⊂ X is bounded if ∃ M ∈ R s.t. ∀ θ ∈ K , || θ || X < M .

Definition C.7 (Compactness of metric space of functions) . Let X be a metric space of functions. K ⊂ X is compact if any open cover of K has a finite subcover. Equivalently, K is compact if and only if any sequence in K is a bounded subsequence whose limit is in K .

Definition C.6 (Closedness of metric space of functions) . Let X be a metric space of functions. K ⊂ X is closed if any convergent sequence in K has a limit in K .

Definition C.8 (Pre-compactness of metric space of functions) . Let X be a metric space of functions. K ⊂ X is pre-compact if its closure ¯ K is compact. Equivalently, K is pre-compact if and only if any sequence in K has a convergent subsequence whose limit is in X .

A critical characteristic of pre-compactness is given by the following theorem (a necessary and sufficient condition for the pre-compactness of metric spaces).

Theorem C.9 (Necessary and sufficient condition of pre-compactness of metric space [65]) . A subset E of a real separable Hilbert space H is pre-compact if and only if E is (i) bounded, (ii) closed, and (iii) for any orthonormal basis { ϕ 0 , ϕ 1 , . . . } of H and for any ϵ > 0 , there exists m ∈ N s.t.

where ( · , · ) H is the inner product defined on H .

Eq. (20) is known as the equi-small tail condition. Thm. C.9 characterizes the domain of functions D ( F ) to which the cylindrical approximation can be applied. Note that boundedness is not a necessary nor sufficient condition for the equi-small tail condition.

C.3 Differentiation of Functionals

Next, we define the Gâteaux differential, Fréchet differential, and functional derivative. Higher-order functional derivatives can be defined in a similar way [91].

Definition C.10 (Gâteaux differential) . A functional F : X open ⊃ D ( F ) → R is Gâteaux differentiable at θ ∈ D ( F ) if

exists and is finite for all η ∈ D ( F ) , where dF G η ([ θ ]) is called the Gâteaux differential of F in the direction η .

There are some patterns of differentiability conditions. One of them is:

Theorem C.11 (Gâteaux differentiability of Lipschitz functionals [63, 5, 57]) . Let Banach space X be separable. Then, any Lipschitz functional F : X open ⊃ D ( F ) → R is Gâteaux differentiable outside a Gauss-null set.

Note that a Gauss-null set is a Borel set A ⊂ X s.t. ∀ non-degenerate Gaussian measure µ on X , µ ( A ) is equal to 0. In this theorem, there is no guarantee for non-Lipschitz functionals, e.g., F ([ θ ]) = ∫ dx √ | θ ( x ) | , where θ (0) = 0 .

Under mild conditions, the Gâteaux derivative is defined, based on the Gâteaux differential.

Theorem C.12 (Gâteaux derivative [86]) . If the following two conditions are satisfied, then the Gâteaux differential dF η ([ θ ]) of functional F : X open ⊃ D ( F ) → R at θ ∈ D ( F ) in the direction η ∈ D ( F ) can be represented as a linear operator acting on η , denoted by F ′ ([ θ ]) , s.t.

where F ′ ([ θ ]) : D ( F ) → R is a linear operator, or a linear functional, depending on θ and is called the Gâteaux derivative of F at θ .

dF G η ([ θ ]) exists in some neighborhood of θ 0 ∈ D ( F ) and is continuous w.r.t. θ at θ 0 .

dF G η ([ θ 0 ]) is continuous w.r.t. η at η = η 0 , where || η 0 || X = 0 .

Next, we define the second type of differentials, Fréchet differential.

Definition C.13 (Fréchet differential) . A functional F : X open ⊃ D ( F ) → R is Fréchet differentiable at θ ∈ D ( F ) if dF F η ([ θ ]) ∈ R s.t.

exists and is finite for all η ∈ D ( F ) , where dF F η ([ θ ]) is called the Fréchet differential of F in the direction η .

There are also some patterns of differentiability conditions. One of them is:

Theorem C.14 (Fréchet differentiability of Lipschitz functionals [74]) . Let K be a compact subset of a Hilbert space H . Then, any locally-Lipschitz functional F : K open ⊃ D ( F ) → R is Fréchet differentiable on a dense subset of K .

An example functional that is Gâteaux differentiable but Fréchet non-differentiable is F ([ θ ]) = a 3 k a 2 k + a 2 l , where θ ( x ) = ∑ ∞ i =0 a i ϕ i ( x ) and k, l ∈ N .

Relationship between Gâteaux and Fréchet differential. If F has a continuous Gâteaux derivative on D ( F ) , then F is Fréchet differentiable on D ( F ) , and these two derivatives are equal [86]. The

Gâteaux derivative of the aforementioned example, F ([ θ ]) = a 3 k a 2 k + a 2 l , is not continuous, and thus, the Fréchet differentiability is not guaranteed. In the following, we consider functionals F that are continuously Gâteaux differentiable in D ( F ) ; therefore, we do not care about differentiability and do not distinguish Gâteaux derivatives from Fréchet derivatives. Hereafter, we write dF η ([ θ ]) := dF G η ([ θ ]) = dF F η ([ θ ]) = F ′ ([ θ ]) η

Next, we define the third type of derivatives.

Definition C.15 (Functional derivative) . The functional derivative of a functional F : X open ⊃ D ( F ) → R w.r.t. θ ( x ) is defined as

if it exists and is finite, where δ ( x ) is the Dirac delta.

Strictly speaking, this definition may be informal because θ ( y ) is a function, while δ ( x -y ) is a distribution. The representation theorem below (Thm. C.17) is sometimes regarded as the definition of the functional derivative.

Lem. C.16 and the Riesz representation theorem prove the following relation between the Fréchet derivative and the functional derivative.

Lemma C.16 (Compactness of Fréchet derivative) . Let K be a compact subset of a real separable Hilbert space H . Let F be a continuous functional on H . If the Fréchet derivative F ′ ([ θ ]) exists at θ ∈ K , then it is a compact linear operator.

Theorem C.17 (Representation theorem of Fréchet derivative) . Let K be a compact subset of a real separable Hilbert space H . Let F be a continuous functional on H . If the Fréchet derivative F ′ ([ θ ]) exists at θ ∈ K , then the following unique integral representation of the Fréchet derivative holds:

The representation theorem C.17 is the foundation of the cylindrical approximation of functional derivatives, which is shown below.

C.4 Cylindrical Approximation

C.4.1 Functionals

Let H be a real separable Hilbert space. The cylindrical approximation is based on the fact that any θ ∈ H can be represented uniquely in terms of an orthonormal basis { ϕ 0 , ϕ 2 , . . . } as θ ( x ) = ∑ ∞ k =0 ( θ, ϕ k ) H ϕ k ( x ) .

Thus, we can define

where a k := ( θ, ϕ k ) H . Truncating k ≤ m -1 ( m ∈ N ) gives the cylindrical approximation of functionals:

Definition C.18 (Cylindrical approximation of functionals [29, 7, 34, 88]) . Let P m be the projection operator s.t. P m θ := ∑ m -1 k =0 ( θ, ϕ k ) H ϕ k . Let D m be the finite-dimensional space induced by P m ; i.e., D m := span { ϕ 0 , ϕ 1 , . . . , ϕ m -1 } . The cylindrical approximation of a functional F ([ θ ]) on H is the m -dimensional multivariable function

where a k := ( θ, ϕ k ) H . In short, F ([ θ ]) ∼ f ( a 0 , . . . , a m -1 ) .

The cylindrical approximation of functionals is uniform:

Theorem C.19 (Uniform convergence of cylindrical approximation of functionals [75]) . Let K be a compact subset of a real separable Hilbert space H . Let F be a continuous functional on H . Then,

holds; i.e., F ([ P m θ ]) converges uniformly to F ([ θ ]) on K .

The convergence rate is given by the mean value theorem of functionals:

Theorem C.20 (Convergence rate of cylindrical approximation of functionals [92]) . Let K be a compact and convex subset of a real separable Hilbert space H . Let F be a continuously differentiable functional on K . Then,

̸

|| F ′ ([ η ]) || is the operator norm of the linear operator F ′ ([ η ]) ; i.e., ∥ F ′ ([ θ ]) ∥ := sup η ( =0) ∈ H | F ′ ([ θ ]) η | ∥ η ∥ H . The convergence rate of O ( ∥ θ -P m θ ∥ H ) depends on the basis and is provided in [41] (Chapters 4 & 6) and [85] (Sec. 3.5) for several bases.

C.4.2 Functional Derivatives

The cylindrical approximation of functional derivatives is motivated by the representation theorem C.17, which states that (i) δF ([ θ ]) δθ ( x ) ∈ H and (ii) F ′ ([ θ ]) η = ( δF ([ θ ]) δθ ( x ) , η ) H . Statement (i) means that δF ([ θ ]) δθ ( x ) can be represented in terms of an orthogonal basis as

Statement (ii) means that

where θ ( x ) = ∑ ∞ l =0 a l ϕ l ( x ) with a l = ( θ, ϕ l ) H . Eqs. (30) and (34) gives

Therefore, truncating k ≤ m -1 ( m ∈ N ) gives the cylindrical approximation of functional derivatives:

Definition C.21 (Cylindrical approximation of functional derivatives [91, 92]) . The cylindrical approximation of a functional derivative δF ([ θ ]) δθ ( x ) ∈ H is defined as

If δF ([ P m θ ]) δθ ( x ) ∈ D m , the second term on the left-hand side is equal to zero. ∑ ∞ k = m ( δF ([ P m θ ]) δθ , ϕ k ) H ϕ k ( x ) is called the tail term in this paper.

In short, δF ([ θ ]) δθ ( x ) ∼ ∑ m -1 k =0 ∂f ∂a k ϕ k ( x ) for δF ([ θ ]) δθ ( x ) that satisfies the equi-small tail condition.

Note that this version of the cylindrical approximation of functional derivatives is different from ours (Eq. 2). Specifically, in [91, 92], P m is not applied to δF ([ θ ]) /δθ ( x ) , and the emerging tail term Σ ∞ k = m ( δF ([ θ ]) /δθ, ϕ k ) ϕ k ( x ) is simply ignored without any rationale.

The cylindrical approximation of functional derivatives is uniform:

Theorem C.22 (Uniform convergence of cylindrical approximation of functional derivatives [92]) . Let K be a compact subset of a real separable Hilbert space H . Let F be a continuously differentiable functional on K . Then,

Note that this theorem proves the uniform convergence, while ours (Thm. 3.1) proves the pointwise convergence. The difference comes from that the uniform convergence of the tail term, which is assumed in the above theorem, can be violated when our cylindrical approximation Eq. (2) is used because the tail term is absent in Eq. (2).

Similarly, the cylindrical approximation of Fréchet derivatives is uniform:

Theorem C.23 (Uniform convergence of cylindrical approximation of Fréchet derivatives [92]) . Let K be a compact subset of a real separable Hilbert space H . Let F be a continuously differentiable functional on K . Then,

holds; i.e., F ′ ([ P m θ ]) converges uniformly to F ′ ([ θ ]) .

The convergence rate is given by:

Theorem C.24 (Convergence rate of cylindrical approximation of Fréchet derivatives [92]) . Let K be a compact and convex subset of a real separable Hilbert space H . Let F be a differentiable functional on K with continuous first- and second-order Fréchet derivatives. Then,

̸

̸

In terms of a functional derivative, this is rewritten as

̸

̸

δθδθ ×

̸

̸

where ∥ ∥ ∥ δ 2 F ([ ζ ]) δθδθ ∥ ∥ ∥ := sup ξ,ξ ′ ∈ H,ξ =0 ,ξ ′ =0 ∣ ∣ ∣ δ 2 F ([ ζ ]) δθδθ ξξ ′ ∣ ∣ ∣ ∥ ξ ∥ H ∥ ξ ′ ∥ H := sup ξ,ξ ′ ∈ H,ξ =0 ,ξ ′ =0 ∣ ∣ ∣ ( δ δθ ( δF ([ ζ ]) δθ ,ξ ) H ,ξ ′ ) H ∣ ∣ ∣ ∥ ξ ∥ H ∥ ξ ′ ∥ H ; i.e., we regard δ 2 F ([ ζ ]) as an operator acting on H H .

C.4.3 Pointwise Convergence of Functional Derivatives under Cylindrical Approximation

We prove the pointwise convergence of the functional derivative under the cylindrical approximation. Wealready noted that the cylindrical approximation of functional derivatives (36) uniformly converges. Below, we show that the convergence becomes pointwise if we omit the second term of the r.h.s. of Eq. (36); i.e., we use

as the approximated functional derivative.

Theorem C.25 (Thm. 3.1. Pointwise convergence of cylindrical approximation) . Let K be a compact subset of a real separable Hilbert space H . Let F be a continuously differentiable functional on K . Then,

where P m is the projection onto span { ϕ 0 , . . . , ϕ m -1 } .

Now, the convergence becomes pointwise. Technically, this is because the set K ′ = { δF ([ θ ]) δθ : θ ∈ K } is not guaranteed to satisfy the equi-small tail condition

while its boundedness sup θ ∈ K ∥ δF ([ θ ]) δθ ∥ H < ∞ holds according to Thm. C.16. An example that converges pointwisely but not uniformly is

defined on a compact subset

Anyways, we have to use large m in either case (Eq. (36) or (2)) when we want to approximate a complicated functional derivative, and the degree m that is required for a sufficiently small approximation error depends on the smoothness, or spectral tail, of δF ([ P m θ ]) δθ ( x ) . As discussed in App. C.4.4, while the lack of uniform convergence may affect the convergence of the cylindrical approximation for linear FDEs in general, this is not problematic in our experiment. We use Eq. (2) as the approximated functional derivative in our experiment.

C.4.4 Abstract Evolution Equations

We first provide related theorems to the convergence of equations ( consistency ) [92] and then those to the convergence of solutions ( stability ) [25, 36, 92].

Definitions. Let F ( H ) be a Banach space of functionals from a real separable Hilbert space H into R . We consider an abstract evolution equation [36]

where F is in F ( H ) , and L ([ θ ]) is a linear operator in the set of closed, densely-defined, and continuous linear operators on F ( H ) denoted by C ( F ) . Let D ( L ) denote the domain of operator L . Let F m ( H ) be the Banach space of functionals on H such that F m ( H ) := { F m | F m ([ θ ] , t ) = F ([ P m θ ] , t ) } ; in other words, F m ( H ) is the set of cylindrically approximated functionals. Using F ( H ) and F m ( H ) , we define a continuous linear operator B m : F ( H ) ∋ F ([ θ ] , t ) ↦→ F ([ P m θ ] , t ) ∈

F m ( H ) , which represents the cylindrical approximation of functionals. B m allows us to decompose the right-hand side of the abstract evolution equation:

where L m ([ θ ]) is a linear operator acting on the m -dimensional multivariable function f ( a 0 , . . . , a m -1 , t ) = F m ([ θ ] , t ) (note that L m ([ θ ]) have nothing to do with t ), and R m ([ θ ] , t ) is the functional residual.

Convergence of equations (consistency). Now, we can show that |L ([ θ ]) F ([ θ ]) -L m ([ θ ]) F m ([ θ ]) | = O ( ∥ θ -P m θ ∥ H ) [92].

Theorem C.27 (Consistency of FDEs under cylindrical approximation [92]) . Let H be a real separable Hilbert space. Let F ∈ F ( H ) and L ∈ L ( F ) . If L ([ θ ]) F ([ θ ]) is continuous in θ , then the sequence of operators {L m } is consistent with L on arbitrary compact subset K of H , provided that ∀ θ ∈ K , | R m ([ θ ]) | m →∞ - - - - → 0 .

Definition C.26 (Consistency of sequence of operators) . A sequence of linear operators {L m } ∈ C ( F m ) is consistent with a linear operator L ∈ C ( F ) if ∀ F ∈ D ( L ) , ∃ a sequence F m ∈ D ( L m ) s.t. ∥ F -F m ∥ m →∞ - - - - → 0 and ∥L F -L m F m ∥ m →∞ - - - - → 0 . If ∥L F -L m F m ∥ m →∞ - - - - →O ( m -p ) , {L m } is consistent with L to order p ( > 0) .

Corollary C.28 (Convergence of cylindrical approximation of FDEs [92]) . Let H be a real separable Hilbert space. Let F ∈ F ( H ) and L ∈ L ( F ) . If L ([ θ ]) F ([ θ ]) is continuous in θ , then the sequence of operators {L m } is consistent with L to the same order as ∥ θ -P m θ ∥ H on arbitrary compact, convex subset K of H , provided that ∀ θ ∈ K , | R m ([ θ ]) | = O ( ∥ θ -P m θ ∥ H ) as m → ∞ and that L ([ θ ]) F ([ θ ]) is continuously Fréchet differentiable in K . In short, |L ([ θ ]) F ([ θ ]) -L m ([ θ ]) F m ([ θ ]) | = O ( ∥ θ -P m θ ∥ H ) .

Convergence of solutions (stability). Next, we show that F m ([ θ ] , t ) → F ([ θ ] , t ) if and only if the cylindrical approximation is stable and consistent. Let us consider the approximated abstract evolution equation ∂F m ([ θ ] ,t ) ∂t = L m ([ θ ]) F m ([ θ ] , t ) with F m ([ θ ] , 0) = B m F 0 ([ θ ]) . It is said to be consistent with the original abstract evolution equation if Thm. C.27 holds.

Definition C.29 (Stability of approximated equation) . Suppose that L m of the approximated abstract evolution equation generates a strongly continuous semigroup e t L m . The approximated abstract evolution equation is stable if ∃ M,ω s.t. ∥ e t L m ∥ ≤ Me ωt , where M and ω are independent of m .

Theorem C.30 (Convergence of solutions under cylindrical approximation [25, 36, 92]) . Let K be a compact subset of a real separable Hilbert space H . Suppose that the approximated abstract evolution equation is well-posed, in the sense of an initial value problem, in the time interval [0 , T ] with a finite T . Suppose also that L ([ θ ]) ∈ C ( F ) generates a strongly continuous semigroup in [0 , T ] . Then, the approximated abstract evolution equation is stable and consistent in K if and only if max t ∈ [0 ,T ] max θ ∈ K | F m ([ θ ] , t ) -F ([ θ ] , t ) | m →∞ - - - - → 0 , provided that F m ([ θ ] , 0) m →∞ - - - - → F 0 ([ θ ]) . In short,

F m ([ θ ] , t ) → F ([ θ ] , t ) if and only if the cylindrical approximation is stable and consistent.

For example, the cylindrical approximation of the following initial value problem is stable and consistent [92]: ∂F ([ θ ] ,t ) ∂t = ∫ 2 π 0 θ ( x ) ∂ ∂x δF ([ θ ] ,t ) δθ ( x ) dx with F ([ θ ] , 0) = F 0 ([ θ ]) . To our knowledge, the convergence rate has been unknown so far.

Remark 1. Most of the approximation results for compact subsets of real separable Hilbert spaces hold also in compact subsets of Banach spaces admitting a basis. We refer the readers to Sec. 8 in [92].

Remark 2. Finally, we comment on how the lack of uniform convergence of δF ([ θ ]) δθ -P m δF ([ P m θ ]) δθ affects the cylindrical approximation of linear FDEs. The difference δF ([ θ ]) δθ -P m δF ([ P m θ ]) δθ is manifested in the functional residual R m ([ θ ] , t ) in Eq. (45). The lack of uniform convergence may have a negative effect on the consistency of the cylindrical approximation, given that the convergence of R m ([ θ ] , t ) is required in Thm. C.27. This issue, however, is circumvented in many cases. In fact, let us consider the scenario where functional derivatives in an FDE are expressed in terms of the inner-product ( v, δF ([ θ ]) δθ ) H , which is satisfied by our examples ( v = u in the FTE and v = A a

in Eq. (97)) (see App. E). Importantly, its cylindrical approximation ( v, P m δF ([ P m θ ]) δθ ) H uniformly converge to ( v, δF ([ θ ]) δθ ) H even if P m δF ([ P m θ ]) δθ does not uniformly converge to δF ([ θ ]) δθ .

Lemma C.31 (Uniform convergence of inner products (ours)) . Let K and K ′ be compact subsets of a real separable Hilbert space H . Let F be a continuously differentiable functional on K . Then,

The proof is given in App. D.1. In App. E, we employ this lemma to show the consistency of our FDEs.

In short, Lem. C.31 states that the cylindrical approximation of inner products ( v, P m δF ([ P m θ ]) δθ ) H uniformly converge to ( v, δF ([ θ ]) δθ ) H even if P m δF ([ P m θ ]) δθ does not uniformly converge to δF ([ θ ]) δθ . Because of this mechanism, in App. E, we show that the uniform convergence of R m is ensured, which is one of the assumptions for stability. The full proof of the convergence of solutions is provided in App. E.

D Proofs I

D.1 Theorem: Pointwise Convergence of Functional Derivatives under Cylindrical Approximation

Theorem D.1 (Thm. 3.1. Pointwise convergence of cylindrical approximation) . Let K be a compact subset of a real separable Hilbert space H . Let F be a continuously differentiable functional on K . Then,

where P m is the projection onto span { ϕ 0 , . . . , ϕ m -1 } .

Proof. The triangle inequality gives

The first term on the right-hand side converges to zero uniformly according to Thm. C.22. As for the second term, we first note that δF ([ θ ]) δθ ( x ) is bounded on K in the sense of a function in H , according to Lem. C.16. This, together with Parseval's identity, implies ∥ δF ([ θ ]) δθ ∥ 2 H = ∑ ∞ k =0 | ( δF ([ θ ]) δθ , ϕ k ) H | 2 < ∞ . Therefore, the sequence of the partial sums S m = ∑ m -1 k =0 | ( δF ([ θ ]) δθ , ϕ k ) H | 2 is a convergent sequence and thus is a Cauchy sequence; i.e.,

By taking n →∞ , we can claim that

∀ orthonormal basis { ϕ 0 , ϕ 1 , . . . } , ∀ ϵ ′ > 0 , ∀ θ ∈ K, ∃ M ∈ N

Therefore, the second term on the right-hand side of Ineq. (48) converges pointwisely. The theorem was thus proved.

Convergence rate. The convergence rates of the approximated functional derivative can be derived from Ineq. (48). The first term on the r.h.s. converges at the same rate as || θ -P m θ || (Eq. (39)). The convergence rate of || θ -P m θ || depends on the basis functions and is provided in [41] (Chapters 4 & 6) and [85] (Sec. 3.5) for several bases. The convergence rate of the second term on the r.h.s. depends on the compact subset of functions K ∈ H under consideration, and further assumptions on K are required.

Lemma D.2 (Lem. C.31. Uniform convergence of inner products) . Let K and K ′ be compact subsets of a real separable Hilbert space H . Let F be a continuously differentiable functional on K . Then,

holds; i.e., ( θ ′ , P m δF ([ P m θ ]) δθ ) H converges uniformly to ( θ ′ , δF ([ θ ]) δθ ) H on K and K ′ .

Proof. Using the triangle inequality and the Cauchy-Schwarz inequality, we have

Note that δF ([ θ ]) δθ and θ ′ are bounded on K and K ′ , respectively; thus, C and C ′ s.t. sup θ ∈ K ∥ ∥ ∥ δF ([ θ ]) δθ ∥ ∥ ∥ H ≤ C < ∞ and sup θ ′ ∈ K ′ ∥ θ ′ ∥ H ≤ C ′ < ∞ . Therefore, Ineq. (52) gives

Finally, according to Thm. C.9, the compactness of K ′ means

This, together with Ineq. (53) and Thm. C.22, proves the lemma.

Tail term. In the cylindrical approximation (2), P m projects the 'tail term' ∑ ∞ k = m ( δF ([ θ ]) δθ , ϕ ) H ϕ k ( x ) to zero, unlike the cylindrical approximation (36) adopted in [91, 92]. The tail term vanishes if one considers an inner product of a functional derivative and v ∈ D m . However, it is not always the case that functional derivatives appear in the form of ( δF ([ θ ]) δθ , v ) with v ∈ D m in FDEs. In fact, in the FTE, the functional derivative appears as an inner product with u ( x ) , which can be chosen arbitrarily. The point is that Lem. C.31, which plays a fundamental role in proving the convergence of approximated solutions, guarantees the convergence of the inner product ( δF ([ θ ]) δθ , v ) in a wider variety of situations including v / ∈ D m . In other words, our theorems extend the class of FDEs whose uniform convergence of the approximated solution is theoretically guaranteed.

D.2 Derivation of Exact Solution of Burgers-Hopf Equation

D.2.1 Derivation of Eq. (93)

We show the derivation of Eq. (93). It is based on the functional Taylor expansion

This turns Eq. (92) as follows:

In the momentum space, this is written in the following form

where k 1 , ··· ,n = 2 πl 1 , ··· ,n with l 1 , ··· ,n ∈ Z . The solution of this equation is

Using this result, Eq. (55) gives

where

The Poisson's summation formula transforms the summation w.r.t. l m :

Plugging this into Eq. (60), we arrive at the following expression:

From this result, we conclude that the solution is given by

D.2.2 Derivation of Eq. (104)

We show the derivation of Eq. (104). The cylindrical approximation of Eq. (93) is represented by

The basis (103) satisfies

Using this relation, we evaluate Θ τ ([ P 2 M Θ] , ξ ) as

where a k = ( ϕ k , Θ) L 2 p ([ -1 / 2 , 1 / 2]) .

When the initial condition is given by Eq. (102),

where we have introduced

E Proofs II: Details of Functional Transport Equation and Burgers-Hopf Equation

In this Appendix, we provide the detailed background of the FTE and BHE and prove Thm. 3.2, i.e., the convergence of solutions under the cylindrical approximation (2). The main materials are Lem. C.31 and Thm. C.30. Technical assumptions are summarized in Thm. C.30.

E.1 Functional Transport Equation

x

y

=

g

(

x

)

u

(

x

)

S

A

B

We introduce the functional transport equation (FTE), a generalization of the transport equation (continuity equation) [53]. We consider a two-dimensional fluid system in x -y coordinates (Fig. 8). We assume that the velocity field is given by u ( x ) = (0 , u ( x )) ⊤ . The surface S , defined by y = g ( x ) , separates the space into two distinct areas, y > g ( x ) (area A) and y < g ( x ) (area B).

We calculate the amount of fluid entering the area A from B per unit time:

where ρ ( x, y, t ) denotes the fluid density at ( x, y ) and at time t . Differentiating both sides w.r.t. t and using the transport equation for the fluid density

We refer to this linear FDE as the FTE.

E.1.1 Analytic Solution

Let L 2 ([ -1 , 1]) be a real Hilbert space of functions on [ -1 , 1] with the inner product we have

y

where f, g ∈ L 2 ([ -1 , 1]) . Let F ( L 2 ([ -1 , 1])) be a Banach space of functionals that map L 2 ([ -1 , 1]) to R . The functional transport equation

is a linear FDE in F ( L 2 ([ -1 , 1])) , where F 0 is a given functional. Here, u ∈ L 2 ([ -1 , 1]) is assumed to have a finite norm ∥ u ∥ L 2 ([ -1 , 1]) < ∞ .

The solution of this equation is given by

as can be seen immediately by substituting this into Eq. (70).

E.1.2 Cylindrical Approximation and Convergence

We prove that the solution of the approximated equation converges to that of the non-approximated original equation. Let { ϕ 0 , ϕ 1 , . . . } be an orthonormal basis of L 2 ([ -1 , 1]) and P m the projection onto span { ϕ 0 , . . . , ϕ m -1 } . Following App. C.4.4, we apply B m on both sides and the initial condition, to obtain

where we have introduced

The residual functional R m is defined as

The cylindrical approximation of Eq. (70) is given by

We first prove the consistency of Eq. (75). According to Lem. C.31, 3 R m ([ g ] , t ) uniformly converges to zero on an arbitrary compact subset K in L 2 ([ -1 , 1]) if F ([ g ] , t ) is a continuously differentiable functional on K . Thus, we can use Thm. C.27, which proves the consistency of the cylindrical approximation (75).

We next prove the stability (Def. C.29) of the cylindrical approximation (75) in the L ∞ ( R m ) norm. Suppose that f 0 ( a ) is bounded by a constant c independent of m . 4 From the cylindrical approximation of the solution Eq. (71), f ( a , t ) = F 0 [ P m ( g ( x ) -u ( x ) t )] , we see

Therefore, we obtain

for the operator L m = -∑ m -1 k =0 u k ∂ ∂a k . From this result, the operator norm of e t L m is given by

̸

̸

This shows that ∃ M,ω s.t. ∀ m ∈ N , ∥ e t L m ∥ ≤ Me ωt ( M = 1 and ω = 0 ), and thus the cylindrical approximation is stable in the sense of Def. C.29.

According to Thm. C.27, the fact that the cylindrical approximation (75) is consistent and stable guarantees the convergence F ([ P m g ] , t ) m →∞ - - - - → F ([ g ] , t ) on a compact subset in a finite time interval [0 , T ] if F ([ P m g ] , 0) m →∞ - - - - → F ([ g ] , 0) (this is satisfied by the initial conditions used in our experiment) and if the approximated functional transport equation is well-posed in [0 , T ] .

E.1.3 Initial Conditions

We define the initial condition F ([ g ] , 0) in Eq. (70) as

The exact solution is given by

Under the cylindrical approximation, the solution is

With υ 0 being a constant and { ϕ k } k ≥ 0 being the Legendre polynomials, we consider two types of u ( x ) :

corresponding to two types functional transport equations (70), initial conditions (79), and solutions (81). For convenience, we call them the linear initial condition and the nonlinear initial condition, respectively, though the form of Eq. (70) also changes in accordance with the form of u ( x ) .

E.2 Burgers-Hopf Equation

The BHE, the one-dimensional analog of Eq. (10) in App. B.2, describes the statistical properties of one-dimensional fluids and is a basic tool for studying turbulence, as mentioned in App. B.2. We consider fluid in a one-dimensional box [ -L/ 2 , L/ 2] that evolves in accordance with the Burgers equation

where u ( x, t ) is a velocity field and ν is the kinematic viscosity. Specifically, we focus on the case where the initial value u ( x, 0) is given randomly, thus making u ( x, t ) a random field. Let us introduce the characteristic functional

where P 0 is the probability distribution for the initial velocity field u ( x, 0) , and θ ( x ) is called the test function. It provides statistical properties of the velocity field because the functional derivatives are equal to the moments of the velocity field:

as is already shown in Eq. (11) in App. B.2. The time evolution of Φ([ θ ] , t ) is known to follow the BHE [91]:

The initial condition Φ([ θ ] , 0) is determined by P 0 . For example, in the case of the Gaussian random field, we have

where µ ( x ) is the mean velocity and C ( x, x ′ ) is the infinite-dimensional covariance matrix.

We modify Eqs. (87) and (88) to facilitate numerical computation. First, we replace -iθ ( x ) with θ ( x ) and regard θ ( x ) as a real-valued function to avoid complex numbers. Second, we make Eqs. (87) and (88) dimensionless, a common convention in numerical computation, by introducing

Third, we introduce

to remove the exponential function in Φ([ θ ] , t ) and stabilize numerical computation. Note that the functional derivatives of W ([Θ] , τ ) give the cumulants, not moments, of the velocity field. Fourth, we neglect the advection term (the first term on the right-hand side of Eq. (87)). We derive the analytic solution for the equation in the following.

E.2.1 Analytic Solution

Let us consider L 2 p ([ -1 / 2 , 1 / 2]) , a real Hilbert space of periodic functions on [ -1 / 2 , 1 / 2] with the inner product

where f, g ∈ L 2 p ([ -1 / 2 , 1 / 2]) . Let F ( L 2 p ([ -1 / 2 , 1 / 2])) be a Banach space of functionals that map L 2 p ([ -1 / 2 , 1 / 2]) to R . In F ( L 2 p ([ -1 / 2 , 1 / 2])) , the BHE without the advection term is given by

The exact solution of this equation is given by

where W 0 is a given functional. The proof is a bit technical and is given in App. D.2.1.

E.2.2 Cylindrical Approximation and Convergence

In this section, we prove the convergence of the solution for the BHE under the cylindrical approximation. Let { ϕ 0 , ϕ 1 , . . . } be an orthonormal basis of L 2 p ([ -1 / 2 , 1 / 2]) and P m be the projection onto span { ϕ 0 , . . . , ϕ m -1 } . Following App. C.4.4, we apply B m on both sides and the initial condition, to obtain

where we have introduced

The residual functional is defined as

The cylindrical approximation of Eq. (92) is defined as

We first prove the consistency of Eq. (97). Let K be a compact subset in L 2 p ([ -1 / 2 , 1 / 2]) such that { ∂ 2 Θ ∂ξ 2 : Θ ∈ K } is compact. According to Lem. C.31, R m ([ W ] , τ ) uniformly converges to zero on K if W ([Θ] , τ ) is a continuously differentiable functional on K . Thus, we can use Thm. C.27, which proves the consistency of Eq. (97).

We next prove the stability (Def. C.29) of Eq. (97) in the L ∞ ( R m ) norm. Suppose that w 0 ( a ) is bounded by a constant c independent of m . 5 From the cylindrical approximation of the solution Eq. (71), w ( a , t ) = W 0 ([ P m Θ τ [ P m Θ]]) , we see

where

Therefore, we obtain

for the operator L m = -∑ m -1 k,l =0 A kl a k ∂ ∂a l . From this result, the operator norm of e t L m is evaluated as

̸

This shows that ∃ M,ω s.t. ∀ m ∈ N , e t L m ≤ Me ωt (e.g., M = 1 and ω = 0 ), and thus the cylindrical approximation is stable in the sense of Def. C.29.

Therefore, the cylindrical approximation (97) is consistent and stable, and Thm. C.30 gives the convergence W ([ P m Θ] , τ ) m →∞ - - - - → W ([Θ] , τ ) on a compact subset in finite time interval [0 , T ] if W ([ P m Θ] , 0) m →∞ - - - - → W ([Θ] , 0) and if Eq. (97) is well-posed in [0 , T ] .

E.2.3 Initial Conditions

We use the Gaussian random field as the initial condition:

where µ is the mean velocity and C ( ξ, ξ ′ ) is the covariance matrix. This is equivalent to the initial distribution of the velocity field following the Gaussian distribution (102). We use the Fourier series as the orthonormal basis:

Then, the analytic solution of Eq. (93) under the cylindrical approximation is given by

where m = 2 M and

The derivation is lengthy and is given in App. D.2.2. Note that the higher degree terms decay exponentially in terms of k , l , and τ , and the solution is dominated by a k with k ≲ 1.

We use three types of covariance matrices:

Substituting them into Eq. (102), we have three types of initial conditions: the delta, moderate, and constant initial condition, respectively. They are equivalent to

Eq. (106) is nonsmooth and represents the extremely short-range correlation of the initial velocity field; the velocities at two points in an infinitesimally small neighborhood have no correlation. The spectrum of C ( ξ, ξ ′ ) (Eq. (109)) has an infinite tail. Eq. (108) represents the extremely long-range correlation of the initial velocity; the velocities at any two points have the same correlation σ 2 . The spectrum of C ( ξ, ξ ′ ) (Eq. (111)) decays immediately. Eq. (107) represents a moderate-range correlation of the two above; the velocities at two points have a periodic correlation. The spectrum of C ( ξ, ξ ′ ) (Eq. (110)) decays exponentially. Theoretically, it is said that C ( ξ, ξ ′ ) that has a long tail of spectrum is hard to simulate numerically [91]. In our experiment with PINNs, however, the error of the solution is dominated by the optimization error of PINNs (Sec. 4) and strongly depends on the training setups.

F Supplementary Discussion

F.1 Limitations

Higher spacetime dimensions. In our experiments, the spacetime dimension is limited to 1 + 1 ( t and x ). Generalization to 1 + d dimensions is feasible, albeit with increased computational costs. For spaces with d > 1 dimensional spaces, multiple options exist for expanding θ [79]. Despite the inclusion of additional spacetime dimensions, the computational complexity of our model, up to computing functional derivatives, remains O ( m r ) , where r is the order of the FDE. However, note that m scales exponentially w.r.t. d , which is typically 1 , 2 , 3 , or 4 .

Higher-order functional derivatives. The order of functional derivatives in FDEs in our experiments is limited to r = 1 ; however, extending to r ≥ 2 is straightforward. For example, the cylindrical approximation of the second-order functional derivative is expressed as δ 2 F ([ θ ] , t ) /δθ ( x ) δθ ( y ) ≈ Σ m -1 k,l =0 ∂ 2 f ( a , t ) /∂a k ∂a l ϕ k ( x ) ϕ l ( y ) , which can be computed via backpropagation twice.

To further reduce errors. The relative errors obtained in our experiments are ∼ 10 -3 , which might be unsatisfactory in some applications This is a general problem inherent in PINNs, for which PINNs are sometimes criticized. To further reduce the errors, one can use recently developed techniques for solving high-dimensional PINNs [106, 44, 96, 45, 21, 103, 33]. These methods can be easily equipped with our model; one of the strengths of our model is that it can be integrated with arbitrary techniques developed for PINNs. See also App. F.7 for a better sampling of collocation points.

Challenges toward even higher degrees. Instability of the numerical integration for computing a k given a function θ ( x ) , i.e., a k = ∫ θ ( x ) ϕ k ( x ) dx , is observed for large k s. This instability comes from the intense oscillation of ϕ k for large k ≳ 500 , as we confirmed in our preliminary experiments. This is a general problem in numerical computation, not exclusive to our model. Nevertheless, such higher degrees are unnecessary to approximate smooth functions. Extremely large degrees are required only when one wants to include nonsmooth or divergent functions in the domain of functionals D ( F ) . Training on such functions with the cylindrical approximation is an open problem Dealing with highly oscillatory functions is also a central research interest in numerical methods for non-smooth dynamical systems [1].

Inclusion and diversity of functions What class of functions can the cylindrical approximation represent? The equi-small tail condition (Thm. C.9) characterizes the domain of functions D ( F ) to which the cylindrical approximation can be applied. In most of the convergence theorems in App. C, such as Thm. C.19, the compactness of K ⊂ H is assumed, which can be ensured if Thm. C.9 holds. In other words, the equi-small tail condition is part of a sufficient condition for the convergence of the cylindrical approximation. Surprisingly or not, step functions and ReLU satisfy the equi-small tail condition. In contrast, functions with divergent norms ( | θ | H = ∞ ), which are not typically of interest, cannot be handled in the cylindrical approximation.

Inclusion and diversity of functionals We assume functional differentiability in this paper. Central theorems of differentiability are given in App. C.3, where functional Lipschitzness is assumed. An example of non-Lipschitz functionals is F ([ θ ]) = ∫ dx √ | θ ( x ) | with θ (0) = 0 .

Inclusion and diversity of FDEs In this paper, we consider the abstract evolution equation, a crucial class of linear FDEs. It does not include, for instance, Eq. (13) in quantum field theory or Eq. (14) in functional renormalization group theory. Nevertheless, the cylindrical approximation is applicable to these equations, although the theoretical (non-)convergence of solutions is currently unknown. Finally, note that the approximated functional derivatives converge to the non-approximated ones independently of the specific FDEs.

F.2 Computational Complexity of CP-ALS

In Sec. 1, we mentioned the computational complexity of the state-of-the-art method (CP-ALS) ( O ( m 6 T ) ) We derived it from Eqs. (567) and (568) in [91] in Sec. 7.2.2 in [91]. These equations

require O ( m 5 ) including the summation over the indices e ( = 0 , . . . , m 2 ) and z ( = 0 , . . . , m 2 ), and the Hadamard matrix product over the index k ( = 1 , . . . , m ). They are computed for each q ( = 1 , . . . m ) and timestep t ( = 1 , . . . , T ), leading to O ( m 5 × m × T ) = O ( m 6 T ) in total. Note that Eqs. (567) and (568) in the published version [91] correspond to Eqs. (568) and (569) in the arXiv version ( arXiv:1604.05250v3 ).

F.3 Choice of Bases

We use the Fourier series for periodic systems and the Legendre polynomials for non-periodic systems. Both are common basis functions in numerical computation. Legendre polynomials are part of the Jacobi polynomials family, a typical class of orthogonal bases. Many Jacobi polynomials are numerically unstable and require careful treatment; for example, the Hermite polynomials change scales factorially with degree ( 1 / √ m ! ) and often result in nan . The Laguerre polynomials have similar issues. The Chebyshev polynomials of the first kind and the Legendre polynomials are suitable for numerical computation; they are bounded, defined on a finite interval, and their weight functions are regular (note that the weight function of the Chebyshev polynomials of the first kind, 1 / √ 1 -x 2 , can be easily regularized by changing variables). Moreover, the Chebyshev and Legendre polynomials are known to minimize the L ∞ and L 2 approximation errors, respectively [41]. Nevertheless, we did not observe a significant performance difference between them in curve-fitting experiments, and we use the Legendre polynomials in our experiments.

How to choose basis functions? The choice depends on the FDE, boundary conditions, symmetry, function spaces, and numerical stability. The Fourier series is suitable for periodic functions, but for non-periodic functions, it exhibits the Gibbs phenomenon, causing large numerical errors. Therefore, Legendre polynomials are a good choice for non-periodic functions. The choice of bases is a common concern in numerical analysis, such as in the finite element and spectral methods.

Generalization to other bases: Riesz basis Our approach can be generalized to general nonorthonormal bases, such as Riesz bases [37, 76, 14, 30, 52, 51, 15], where the orthonormality condition ( ϕ k , ϕ l ) H = δ kl is replaced with ( ϕ k , ϕ l ) H = g kl . Here, g kl is a certain matrix, or a metric , whose choice significantly affects the computational costs of the approximated FDEs. Sparse metrics are preferred for efficient computation. See also App. F.4.

F.4 Computational Complexity Revisited

Weshowed the computational complexity of our model up to the computation of functional derivatives is given by O ( m r ) . This does not include the computational complexity of the loss function of PINNs. It could be > O ( m r ) , strongly depending on the form of the FDE and/or the choice of the orthonormal basis. For concreteness, let us consider Eq. (97) (BHE). While the BHE is first-order, the computational complexity of the loss function can be O ( m 2 ) if matrix A kl in Eq. (97) is not diagonal. This is the case if we use the Legendre polynomials instead of the Fourier series.

On the other hand, the choice of the basis function can improve computational complexity. For example, the computational complexity of the second-order FDE can reduce from O ( m 2 ) to O ( m 1 ) when the approximated FDE includes only the diagonal elements of { ∂ 2 f ( a , t ) /∂a k ∂a l } m -1 k,l =0 . Again, whether it is the case or not depends on the form of the FDE and/or the choice of the orthonormal basis.

F.5 Comparison with DFT

In first-principles computations of density functional theory (DFT), an NN-based approach that utilizes finite element methods for spacetime grid approximation is commonly employed. For example, an NN, ˆ F ( { y j := f ( r j ) } j ) , approximating a target functional F ([ f ]) by evaluating f at specific grid points { r j } j . Functional derivatives at each grid point can be computed using automatic differentiation: { δF ([ f ]) δf ( r ) } r ≒ { ∂ ˆ F ( { y j } j ) ∂y i } i . However, the central focus of this area does not include solving PDEs, let alone FDEs. Moreover, this method requires the discretization of spacetime, leading to numerical error of derivatives, while in our approach, such discretization is not necessary, and spacetime differentiation can be performed analytically if the basis functions are analytic.

The finite element methods for spacetime and the cylindrical approximation (basis function expansion) are similar in the sense that they replace spacetime degrees of freedom with other quantities: spacetime grid points and expansion coefficients, respectively. Even when their degrees of freedom for a given problem are comparable, the latter (our approach) is suitable for solving FDEs.

F.6 FDE Has Multiple Meanings

The term "Functional Differential Equation" (FDE) varies in meaning across research areas. In mathematics, an FDE typically refers to a differential equation with a deviating argument, e.g., f ′ ( x ) = xf ( x + 3) . In physics, it denotes a PDE involving functional derivatives, such as the abstract evolution equation described in the main text. Additionally, FDE can imply a functional equation involving functional derivatives, e.g., δF ([ θ ]) δθ ( x ) = A ( x ) F ([ θ ]) , where A ( x ) is a given function. Research on the last interpretation is limited, and the uniqueness, existence, and practical applications remain ambiguous.

F.7 Radial Gaussian Sampling

The region θ ∼ 0 , or a k ∼ 0 k is the central focus when solving the BHE. Thus, sampling as many collocation points from θ ∼ 0 as possible is of crucial importance. To make the training set free from the curse of dimensionality, we can use the following sampling method based on the polar coordinates:

Sample a unit vector v ∈ R d , where d is the dimension of the input space.

Sample a real number (radius) r > 0 from a Gaussian distribution N (0 , σ 2 ) (or a truncated Gaussian distribution).

Obtain a training collocation point r v ∈ R d .

The collocation points thus obtained obviously concentrate around θ ∼ 0 more than the Latin hypercube sampling, which we used in our experiment.

F.8 Curvilinear Patterns in Fig. 4 and Noisy Patterns in Fig. 7 and

The white curves in Fig. 4 represent locations where the predictions and the exact solutions happen to coincide. The noisy patterns that occur for a k , k ≥ 1 in Fig. 7 are because the solution is almost independent of a k with k ⪆ 1 . Thus, optimizing the model in the directions of a k with k ⪆ 1 have only a negligible effect on minimizing the loss function, keeping the random predictions of the model. If we delve deeper, they might be related to the loss landscapes of PINNs [8, 35], which is also an interesting research topic.

F.9 Second-order Functional Derivatives

Fig. 9 shows the estimated second-order derivative of the FTE's solution at θ = 0 and t = 0 under the linear initial condition. The error is as small as ∼ 10 -3 , despite that θ ∼ 0 is not included in the training set due to the curse of dimensionality and that the second-order information is not included in the loss function.

However, such small errors only emerge when the analytic solution is so simple that, e.g., it includes only one coefficient: f ( a , t ) = ρ 0 υ 0 ( a 1 -υ 0 t ) . The errors of the second-order derivatives obtained from other conditions and equations were significant ( ≳ 10 3 in some cases). We could reduce the error by (i) using another sampling method for training sets that does not suffer from the curse of dimensionality (an example is given in App. F.7) and (ii) adding the second-order derivative to the loss function (e.g., differentiate both sides of the BHE w.r.t. a k and use it to the residual loss for PINNs).

G Detailed Experimental Settings

We describe detailed setups of our experiment, including those on the advection-reaction functional equation (ARE) (App. H.4). The experimental settings for the ARE align with those provided in [91].

Datasets. The training, validation, and test sets are sampled with Latin hypercube sampling [64, 39] with range [0 , 1] , [ -1 , 1] , [0 , 0 . 001] , [ -0 . 1 , 0 . 1] , [0 , 0 . 5] , and [ h -0 . 01 , h + 0 . 01] for t (FTE), a k (FTE), t (BHE), a k (BHE), t (ARE), and a k (ARE), respectively, where h := 0 . 698835274542439 . Training mini-batches are randomly sampled every time a mini-batch is created. For the BHE, we decay the sampling range quadratically in terms of k ∈ { 0 , 1 , . . . , m } . This sampling stabilizes the training because most of the loss comes from the region where k and a k are large, which can be seen from the definition of the BHE. That is, all the terms on the right-hand side are proportional to one of a k s, while a k s for large k s are negligible in the analytic solution. Therefore, the collocation points with large a k s with large k s can be seen as noise in training. Note that this decaying sampling does not limit the quality of the solution at all because the solution is the characteristic function and we are interested only in θ ( x ) ≈ 0 , e.g., δW ([ θ ] ,t ) δθ ( x ) | θ =0 = ⟨ u ( x ) ⟩ (see also App. B.2).

Miscellaneous settings. We use a 4-layer PINN with 3 × (linear + sin activation + layer normalization) + last linear layer. Unless otherwise noted, the widths are 1024 for the ARE and FTE and 2048 for the BHE. The batch size is 1024. The activation function is the sin function. The loss function for the FTE with the nonlinear initial condition (main text) and the ARE is the sum of the L 1 and L ∞ losses. The loss function for the others is the smooth L 1 loss. The optimizer is AdamW [59]. The learning rate scheduler is the linear warmup with cosine annealing with warmup [58], which is defined as the following scheduler :

See PyTorch official documentation [70] for the detailed definitions of the parameters. eta_min is set to 0. start_factor is set to 10 -10 . ρ 0 = 1 and υ 0 = 1 are used for the FTE. ¯ µ = 8 and σ 2 = 10 are used for the BHE. The Legendre polynomials and real Fourier series are used for the FTE and BHE, respectively, as the orthonormal basis. For the ARE, the modified Chebyshev polynomials described in [91] are used.

Hyperparameter tuning. The full search space is given Tab. 3. Unless otherwise noted, the number of iterations is 300,000 and 500,000 for the BHE and FTE, respectively. The number of trials for hyperparameter tuning is 50. Optuna [2] is used for hyperparameter tuning with the TPE sampler and median pruner. The number of tuning trials is 50 for all conditions, and the best hyperparameters within the 50 trials are used. See the config file in our repository for more details.

Table 3: Search space of hyperparameters. Learning rates and weight decays are sampled loguniformly. N = 300,000 and 500,000 for the BHE and FTE, respectively. T_0 , T_mult , and milestone are used for the aforementioned scheduler . For the ARE and the main text results of FTE with the nonlinear initial condition, weight decay is fixed to 0 and the search space of learning rates is from 10 -7 to 10 -4 .

3000

3000

30000

How to choose weight parameters for each training. We use validation relative error, i.e., the relative error between the prediction and the analytic solution, as the measure for choosing the best weight parameter during each training trial.

Loss reweighting. The softmax reweighting is used to adaptively balance the residual and boundary loss functions. See the code below and our official code at GitHub ( https://github.com/ TaikiMiyagawa/FunctionalPINN ), where list_tensors is [residual loss, boundary loss] of a mini-batch.

This function generates the loss reweighting factors coeffs = [ λ 1 , λ 2 ], where total loss is defined as λ 1 × residual loss + λ 2 × boundary loss.

Float vs. double. No significant differences in performance measures were observed throughout our training. Accordingly, we used float32 to reduce computational time.

Libraries and GPUs. All the experiments are performed on Python 3.11.8 [89], PyTorch 2.2.0 [70], Numpy 1.26.0 [38], and Optuna 3.5.0 [2]. An NVIDIA A100 GPU is used.

Runtime. The training process takes 2-3 hours, but this duration can be significantly improved because it largely depends on the implementation of components such as data loaders. See App. H.2 for more details on runtime. GPU memory consumption is about 400 MBs for the ARE and 8001500 MBs for the FTE and BHE. We note that previous approaches to numerically solving FDEs rely on CPU processing, where performance varies significantly with the level of parallelization. Consequently, directly comparing the computational speed of our GPU-based approach with these methods is inherently challenging.

H Additional Experimental Results

H.1 P m θ Converges to θ

The convergence rate of functional derivatives in Thm. 3.1 is given by ∥ θ -P m θ ∥ . The convergence of ∥ θ -P m θ ∥ is visualized in Fig. 10 for four functions including non-smooth functions.

H.2 Runtime

Tab. 11 shows training runtime. The runtime remains nearly constant and does not significantly vary from m ∼ 1 to m ∼ 100 . This aspect is of practical significance as it indicates that the computational time of our model remains stable across a broad range of m .

This consistency in runtime may be attributed to the efficiency of the PyTorch library, where operations such as matrix multiplication and automatic differentiation are highly optimized. On the other hand, GPU memory consumption does show variation, ranging from 800 to 1500 MBs as stated App. G, as is naturally expected.

Finally, the previous approaches to numerically solving FDEs rely on CPU processing, where performance significantly varies with the level of parallelization; thus, a direct comparison of computational speed with our GPU-based approach is inherently challenging.

±

H.3 Latin Hypercube Sampling and Curse of Dimensionality

We show the histograms of the L 2 norm of collocation points in the test sets. The sampling method is the Latin hypercube sampling, as mentioned in Sec. 4. Apparently, the collocation points s.t. norms ∼ 0 are not sampled, which would cause the curse of dimensionality in training. In other words, the Latin hypercube sampling cannot completely address the curse of dimensionality in sampling.

Why does extrapolation mean (Sec. 4)? Firstly, Figs. 11-16 show that no collocation points such that ∥ a ∥ ≈ 0 were included in the training sets. Secondly, Figs. 4 & 7 plot the errors of PINN predictions at the collocation points where a = (0 , 0 , . . . , 0 , a k , 0 , . . . , 0) with k = 0 , 1 , 2 , 19 , or 99 . Therefore, the collocation points such that a k ≈ 0 in Figs. 4 & 7 were not included in the training sets, but the errors were as small as the region where a k ̸≈ 0 . We refer to this as "extrapolation."

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

2

.

00

Norm of coefficients

10

0

10

1

10

2

10

3

Count

Norm: min=2.92e-02, mean=1.12e+00, max=1.94e+00

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

Norm of coefficients

10

0

10

1

10

2

10

3

Count

Norm: min=1.33e+00, mean=2.57e+00, max=3.58e+00

4

.

5

5

.

0

5

.

5

6

.

0

6

.

5

7

.

0

Norm of coefficients

10

0

10

1

10

2

10

3

Count

Norm: min=4.58e+00, mean=5.77e+00, max=6.89e+00

10

-

3

10

-

2

10

-

1

Norm of coefficients

10

1

10

2

10

3

Count

Norm: min=2.92e-05, mean=5.00e-02, max=1.00e-01

10

-

3

10

-

2

10

-

1

Norm of coefficients

10

1

10

2

10

3

Count

Norm: min=2.12e-05, mean=5.00e-02, max=1.00e-01

10

-

3

10

-

2

10

-

1

Norm of coefficients

10

1

10

2

10

3

Count

Norm: min=2.75e-05, mean=5.00e-02, max=1.00e-01

H.4 Comparison with Baseline: Advection-reaction Functional Equation (ARE)

We consider the advection-reaction functional equation (ARE).

We follow the experimental setting described in [91].

In [91], the reported time-dependent relative error of the CP-ALS and HT fluctuates from ∼ 10 -8 to ∼ 10 -1 , and the temporally-averaged relative error is estimated to be ≲ 10 -2 (the bottom two panels in Fig. 38 in [91]), where we roughly approximated the time-dependent error curves as an exponential function of the form y = 10 -6+10 t .

Our preliminary results are shown in Tab. 12. We observe that the learned functional is almost constant in time, a failure mode of PINNs, causing large errors. We anticipate the result can improve with further hyperparameter optimization [50].

±

×

±

×

H.5 Relative and Absolute Errors When W ([0] , t ) = 0 Is Included In Loss

We show the histograms of the relative and absolute errors for the models trained on the BHE with the loss term corresponding to the identity W ([0] , t ) = 0 . They are almost the same as those for the models trained without W ([0] , t ) = 0 , but the errors of the first-order derivative are 10 -1 times smaller, as shown in Sec. 4.

10

-

8

10

-

6

10

-

4

10

-

2

10

0

Relative error

10

0

10

1

10

2

10

3

10

4

Count

10

-

8

10

-

6

10

-

4

10

-

2

10

0

Absolute error

10

0

10

1

10

2

10

3

10

4

Count

10

-

8

10

-

6

10

-

4

10

-

2

10

0

Relative error

10

0

10

1

10

2

10

3

10

4

Count

10

-

8

10

-

6

10

-

4

10

-

2

10

0

Absolute error

10

0

10

1

10

2

10

3

10

4

Count

10

-

8

10

-

6

10

-

4

10

-

2

10

0

Relative error

10

0

10

1

10

2

10

3

10

4

Count

10

-

8

10

-

6

10

-

4

10

-

2

10

0

Absolute error

10

0

10

1

10

2

10

3

10

4

Count

10

-

8

10

-

6

10

-

4

10

-

2

10

0

Relative error

10

0

10

1

10

2

10

3

10

4

Count

10

-

8

10

-

6

10

-

4

10

-

2

10

0

Absolute error

10

0

10

1

10

2

10

3

10

4

Count

H.6 Cross-degree Evaluation

In this section, we attempt to empirically investigate the theoretical convergence of the cylindrical approximation. The convergence is illustrated in Fig. 2, where PINN's training is not included, while our focus here is the convergence of the solutions estimated via PINN's training. According to the convergence theorem of the cylindrical approximation and the universal approximability of PINNs, the models trained on D m = span { ϕ 0 , . . . , ϕ m -1 } with a large m have a high capability of expression, and they can cover D n with n ≤ m . On the other hand, the models trained on D m with a small m have a high capability of expression only for small m s, and they cannot cover D n for n ≥ m . In theory, this phenomenon can be observed by performing a cross-degree evaluation , in which we use training data sampled from D m and then evaluate the trained model on D m ′ , where m may not equal to m ′ .

The results are shown in Tabs. 13-22 below. Contrary to the aforementioned intuition, most of the cross-degree evaluations seem to contradict the convergence theorem of FDE solutions under the cylindrical approximation. Possible reasons are:

This is because of the curse of dimensionality, i.e., the optimization error of PINNs in high dimensions. Higher-dimensional PDEs require a much larger number of iterations to train PINNs; however, our experiments are performed with a fixed number of iterations (500,000 and 300,000 for the FTE and the BHE, respectively). Changing the number of iterations depending on the degree and applying other techniques for training PINNs [106, 44, 96, 45, 21, 103, 33] would decrease the optimization error, and we can focus on the approximation error and thus will observe the decay of error w.r.t. increasing m .

In addition, note that FDEs in our experiments, except for the FTE with the nonlinear initialization, are dominated by a 0 and/or a 1 only. Therefore, they show a tiny effect on relative and absolute errors by definition. Nevertheless, the FTE with the nonlinear initialization (Tabs. 15 and 16) does not show the theoretical convergence either, which means the optimization error of PINNs, rather than the cylindrical approximation error, dominates the errors anyways.

One can see a strong dependence of the errors on the experimental setups for PINNs. The absolute errors of the FTE under the nonlinear initial condition are reduced by a factor of 2 by simply changing the width of the PINN and the number of training iterations (1024 to 2048 and 5 × 10 5 to 8 × 10 5 , respectively): from

(20 . 3384 ± 1 . 0449) × 10 -3 for m = 10 , and

(2 . 46499 ± 0 . 26492) × 10 -3 for m = 4 ,

(98 . 8854 ± 2 . 7068) × 10 -3 for m = 20 ,

(1 . 43591 ± 0 . 1832) × 10 -3 for m = 4 ,

(55 . 2715 ± 2 . 3179) × 10 -3 for m = 20 ,

(11 . 0609 ± 0 . 3671) × 10 -3 for m = 10 , and

where the error bars are the standard deviation over 10 runs. Note that the hyperparameters are different from those used in the main text. Moreover, as shown in Fig. 6, simply adding a regularization term ∥ W ([0] , t ) ∥ to the loss function reduces relative errors by an order of magnitude.

Therefore, to observe the convergence of the cylindrical approximation after PINN's training, careful hyperparameter tuning and optimization are needed. Note that optimizing PINNs for high-dimensional PDEs is an active field of research that has been developing (see Apps. A and F) and is of independent interest.

to

H.6.1 Relative and Absolute Errors of Functional Transport Equation

±

±

±

±

±

±

±

±

±

±

±

±

H.6.2 Relative and Absolute Errors of Burgers-Hopf Equation

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

H.7 Loss Curves and Learning Rate Scheduling

Loss curves and learning rates are provided in Figs. 21-29.

H.8 Additional Figures: Functional Transport Equation

H.8.1 Worst Coefficients and Best Coefficients

We provide the worst-/best-case relative/absolute error. The mean and minimum of errors in the test set do not strongly depend on training random seeds, while the maximum of errors intensely fluctuates. This observation tells us that the worst-case relative and absolute errors are not stable measures of performance.

±

±

±

±

±

±

±

×

±

H.8.2 First-order Functional Derivative: Linear Initial Condition & Relative Error

The relative errors of the first-order functional derivative are provided. The models are trained under the linear initial condition.

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

10

-

4

10

-

3

Relative error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

6

10

-

5

10

-

4

10

-

3

Relative error

tau=1

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

10

-

5

10

-

4

10

-

3

10

-

2

Relative error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

10

-

2

Relative error

tau=1

x

x

x

x

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

10

-

3

10

-

2

Relative error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

10

-

2

10

-

1

Relative error

tau=1

x

x

H.8.3 First-order Functional Derivative: Nonlinear Initial Condition & Relative Error

The relative errors of the first-order functional derivative are provided. The models are trained under the nonlinear initial condition.

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

1

2

3

4

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

10

-

4

10

-

3

10

-

2

10

-

1

Relative error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

1

2

3

4

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

10

-

2

10

-

1

10

0

Relative error

tau=1

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

5

10

15

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

10

-

5

10

-

4

10

-

3

10

-

2

10

-

1

10

0

Relative error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

5

10

15

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

2

10

-

1

10

0

Relative error

tau=1

x

x

x

x

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

5

10

15

20

25

30

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

10

-

4

10

-

3

10

-

2

10

-

1

Relative error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

5

10

15

20

25

30

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

1

10

0

Relative error

tau=1

x

x

H.8.4 First-order Functional Derivative: Linear Initial Condition & Absolute Error

The absolute errors of the first-order functional derivative are provided. The models are trained under the linear initial condition.

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

.

00000

0

.

00005

0

.

00010

0

.

00015

0

.

00020

Absolute error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

00000

0

.

00005

0

.

00010

0

.

00015

0

.

00020

0

.

00025

0

.

00030

Absolute error

tau=1

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

.

00000

0

.

00025

0

.

00050

0

.

00075

0

.

00100

0

.

00125

0

.

00150

Absolute error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

0000

0

.

0005

0

.

0010

0

.

0015

Absolute error

tau=1

x

x

x

x

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

Absolute error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

000

0

.

005

0

.

010

0

.

015

0

.

020

Absolute error

tau=1

x

x

H.8.5 First-order Functional Derivative: Nonlinear Initial Condition & Absolute Error

The absolute errors of the first-order functional derivative are provided. The models are trained under the nonlinear initial condition.

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

1

2

3

4

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

.

00000

0

.

00025

0

.

00050

0

.

00075

0

.

00100

0

.

00125

0

.

00150

Absolute error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

1

2

3

4

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

000

0

.

005

0

.

010

0

.

015

0

.

020

0

.

025

Absolute error

tau=1

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

5

10

15

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

.

000

0

.

005

0

.

010

0

.

015

0

.

020

0

.

025

Absolute error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

5

10

15

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Absolute error

tau=1

x

x

x

x

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

5

10

15

20

25

30

First-order derivative

Prediction (tau=0)

Analytic (tau=0)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

x

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

0

.

12

0

.

14

Absolute error

tau=0

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

5

10

15

20

25

30

First-order derivative

Prediction (tau=1)

Analytic (tau=1)

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

1

2

3

4

5

6

Absolute error

tau=1

x

x

H.8.6 Time-dependent Relative and Absolute Errors at θ = 0 : Linear Initial Condition

The time-dependent relative and absolute errors at θ = 0 are provided. The models are trained under the linear initial condition.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

10

-

5

10

-

4

10

-

3

Relative error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

0

2

4

6

Absolute error

×

10

-

5

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

-

0

.

6

-

0

.

4

-

0

.

2

0

.

0

J([theta], t)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

10

-

4

10

-

3

10

-

2

Relative error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

0

.

0000

0

.

0002

0

.

0004

Absolute error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

-

0

.

6

-

0

.

4

-

0

.

2

0

.

0

J([theta], t)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

10

-

4

10

-

2

Relative error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

0

.

0000

0

.

0005

0

.

0010

Absolute error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

-

0

.

6

-

0

.

4

-

0

.

2

0

.

0

J([theta], t)

Analytic solution (input function=0)

Prediction (input function=0)

H.8.7 Time-dependent Relative and Absolute Errors at θ = 0 : Nonlinear Initial Condition

The time-dependent relative and absolute errors at θ = 0 are provided. The models are trained under the nonlinear initial condition.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

10

-

5

10

-

3

Relative error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

0

.

00000

0

.

00025

0

.

00050

0

.

00075

Absolute error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

-

2

-

1

0

J([theta], t)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

10

-

4

10

-

3

Relative error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

0

.

00

0

.

01

0

.

02

0

.

03

Absolute error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

-

6

-

4

-

2

0

J([theta], t)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

10

-

4

10

-

3

10

-

2

Relative error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

t

0

.

0

0

.

1

0

.

2

0

.

3

Absolute error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

10

-

5

0

J([theta], t)

Analytic solution (input function=0)

Prediction (input function=0)

t

H.8.8 Time-averaged Pointwise Relative Error: Linear Initial Condition

The time-averaged pointwise relative errors are provided. The models are trained under the linear initial condition.

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

10

-

4

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

10

-

4

10

-

3

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

4

6

×

10

-

5

2

×

10

-

4

3

×

10

-

4

4

×

10

-

4

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

4

4

×

10

-

5

6

×

10

-

5

2

×

10

-

4

3

×

10

-

4

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

10

-

3

4

×

10

-

4

6

×

10

-

4

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

10

-

4

10

-

3

10

-

2

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

2

×

10

-

4

3

×

10

-

4

4

×

10

-

4

6

×

10

-

4

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

3

×

10

-

4

4

×

10

-

4

6

×

10

-

4

Time-averaged relative error

a2

a2

a3

a19

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

10

-

3

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

10

-

3

10

-

2

10

-

1

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a2

10

-

3

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a99

10

-

3

Time-averaged relative error

H.8.9 Time-averaged Pointwise Relative Error: Nonlinear Initial Condition

The time-averaged pointwise relative errors are provided. The models are trained under the nonlinear initial condition.

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

10

-

3

10

-

2

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

10

-

3

10

-

2

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

10

-

2

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

10

-

2

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

10

-

3

10

-

2

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

10

-

3

10

-

2

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

10

-

2

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

10

-

3

10

-

2

Time-averaged relative error

a2

a2

a3

a9

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

10

-

2

10

-

1

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

10

-

2

10

-

1

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a2

10

-

2

10

-

1

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a19

5

×

10

-

3

6

×

10

-

3

Time-averaged relative error

H.8.10 Time-averaged Pointwise Absolute Error: Linear Initial Condition

The time-averaged pointwise absolute errors are provided. The models are trained under the linear initial condition.

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

1

2

3

4

5

6

Time-averaged absolute error

×

10

-

5

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

0

.

00000

0

.

00025

0

.

00050

0

.

00075

0

.

00100

0

.

00125

0

.

00150

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a2

1

2

3

4

5

Time-averaged absolute error

×

10

-

5

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a3

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

Time-averaged absolute error

×

10

-

5

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

0

.

00010

0

.

00012

0

.

00014

0

.

00016

0

.

00018

0

.

00020

0

.

00022

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

0

.

0000

0

.

0005

0

.

0010

0

.

0015

0

.

0020

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

00008

0

.

00010

0

.

00012

0

.

00014

0

.

00016

0

.

00018

0

.

00020

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

00010

0

.

00012

0

.

00014

0

.

00016

0

.

00018

0

.

00020

0

.

00022

Time-averaged absolute error

a2

a19

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

0

.

000

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a2

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a99

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Time-averaged absolute error

H.8.11 Time-averaged Pointwise Absolute Error: Nonlinear Initial Condition

The time-averaged pointwise absolute errors are provided. The models are trained under the nonlinear initial condition.

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

0

.

0000

0

.

0005

0

.

0010

0

.

0015

0

.

0020

0

.

0025

0

.

0030

0

.

0035

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

0

.

0000

0

.

0005

0

.

0010

0

.

0015

0

.

0020

0

.

0025

0

.

0030

0

.

0035

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a2

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a3

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

0

.

014

0

.

016

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

0

.

014

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

0

.

014

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

0

.

014

0

.

016

Time-averaged absolute error

a2

a9

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a0

0

.

02

0

.

03

0

.

04

0

.

05

0

.

06

0

.

07

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a1

0

.

02

0

.

03

0

.

04

0

.

05

0

.

06

0

.

07

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a2

0

.

02

0

.

03

0

.

04

0

.

05

0

.

06

0

.

07

Time-averaged absolute error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a19

0

.

034

0

.

036

0

.

038

0

.

040

0

.

042

Time-averaged absolute error

H.9 Additional Figures: Burgers-Hopf Equation

H.9.1 Worst Coefficients and Best Coefficients

We provide the worst-/best-case relative/absolute error. The mean and minimum of errors in the test set do not strongly depend on training random seeds, while the maximum of errors intensely fluctuates. This observation tells us that the worst-case relative and absolute errors are not stable measures of performance.

±

±

±

H.9.2 First-order Functional Derivative: Delta Initial Condition & Relative Error

The relative errors of the first-order functional derivative are provided. The models are trained under the delta initial condition.

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0000

0

.

0002

0

.

0004

0

.

0006

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

0000

0

.

0002

0

.

0004

0

.

0006

Relative error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

02

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

-

7

.

97

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

00025

0

.

00050

0

.

00075

0

.

00100

0

.

00125

0

.

00150

0

.

00175

0

.

00200

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

02

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

-

7

.

97

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

00025

0

.

00050

0

.

00075

0

.

00100

0

.

00125

0

.

00150

0

.

00175

0

.

00200

Relative error

tau=0.001

x

x

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

03

-

8

.

02

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

-

7

.

97

-

7

.

96

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

0

.

0005

0

.

0000

0

.

0005

0

.

0010

0

.

0015

0

.

0020

0

.

0025

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

03

-

8

.

02

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

-

7

.

97

-

7

.

96

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

0

.

0005

0

.

0000

0

.

0005

0

.

0010

0

.

0015

0

.

0020

0

.

0025

Relative error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

04

-

8

.

02

-

8

.

00

-

7

.

98

-

7

.

96

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

00025

0

.

00050

0

.

00075

0

.

00100

0

.

00125

0

.

00150

0

.

00175

0

.

00200

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

04

-

8

.

02

-

8

.

00

-

7

.

98

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

00025

0

.

00050

0

.

00075

0

.

00100

0

.

00125

0

.

00150

0

.

00175

Relative error

tau=0.001

x

x

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

0075

-

8

.

0050

-

8

.

0025

-

8

.

0000

-

7

.

9975

-

7

.

9950

-

7

.

9925

-

7

.

9900

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0000

0

.

0001

0

.

0002

0

.

0003

0

.

0004

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

0075

-

8

.

0050

-

8

.

0025

-

8

.

0000

-

7

.

9975

-

7

.

9950

-

7

.

9925

-

7

.

9900

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

0000

0

.

0001

0

.

0002

0

.

0003

0

.

0004

Relative error

tau=0.001

x

x

H.9.3 First-order Functional Derivative: Constant Initial Condition & Relative Error

The relative errors of the first-order functional derivative are provided. The models are trained under the constant initial condition.

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

008

-

8

.

006

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

-

7

.

996

-

7

.

994

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0000

0

.

0001

0

.

0002

0

.

0003

0

.

0004

0

.

0005

0

.

0006

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

006

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

-

7

.

996

-

7

.

994

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

0000

0

.

0001

0

.

0002

0

.

0003

0

.

0004

0

.

0005

0

.

0006

Relative error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

0015

-

8

.

0010

-

8

.

0005

-

8

.

0000

-

7

.

9995

-

7

.

9990

-

7

.

9985

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

0015

-

8

.

0010

-

8

.

0005

-

8

.

0000

-

7

.

9995

-

7

.

9990

-

7

.

9985

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Relative error

tau=0.001

x

x

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

002

-

8

.

001

-

8

.

000

-

7

.

999

-

7

.

998

-

7

.

997

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

002

-

8

.

001

-

8

.

000

-

7

.

999

-

7

.

998

-

7

.

997

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Relative error

tau=0.001

x

x

H.9.4 First-order Functional Derivative: Moderate Initial Condition & Relative Error

The relative errors of the first-order functional derivative are provided. The models are trained under the moderate initial condition.

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

10

-

8

.

05

-

8

.

00

-

7

.

95

-

7

.

90

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

001

0

.

002

0

.

003

0

.

004

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

10

-

8

.

05

-

8

.

00

-

7

.

95

-

7

.

90

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

001

0

.

002

0

.

003

0

.

004

Relative error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

04

-

8

.

02

-

8

.

00

-

7

.

98

-

7

.

96

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0000

0

.

0005

0

.

0010

0

.

0015

0

.

0020

0

.

0025

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

04

-

8

.

02

-

8

.

00

-

7

.

98

-

7

.

96

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

0000

0

.

0005

0

.

0010

0

.

0015

0

.

0020

0

.

0025

Relative error

tau=0.001

x

x

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

03

-

8

.

02

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

00000

0

.

00025

0

.

00050

0

.

00075

0

.

00100

0

.

00125

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

03

-

8

.

02

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

0

.

0012

Relative error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

015

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

-

7

.

985

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

0

.

0012

0

.

0014

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

015

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

-

7

.

985

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

0

.

0012

0

.

0014

Relative error

tau=0.001

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Relative error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

0002

0

.

0004

0

.

0006

0

.

0008

Relative error

tau=0.001

x

x

H.9.5 First-order Functional Derivative: Delta Initial Condition & Absolute Error

The absolute errors of the first-order functional derivative are provided. The models are trained under the delta initial condition.

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

Absolute error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

-

7

.

985

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

0

.

014

0

.

016

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

-

7

.

985

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

0

.

014

0

.

016

Absolute error

tau=0.001

x

x

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

-

7

.

985

-

7

.

980

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

000

0

.

005

0

.

010

0

.

015

0

.

020

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

-

7

.

985

-

7

.

980

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

000

0

.

005

0

.

010

0

.

015

0

.

020

Absolute error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

015

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

0

.

014

0

.

016

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

015

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

0

.

012

0

.

014

Absolute error

tau=0.001

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

001

-

8

.

000

-

7

.

999

-

7

.

998

-

7

.

997

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

0000

0

.

0005

0

.

0010

0

.

0015

0

.

0020

0

.

0025

0

.

0030

0

.

0035

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

001

-

8

.

000

-

7

.

999

-

7

.

998

-

7

.

997

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

000

0

.

001

0

.

002

0

.

003

Absolute error

tau=0.001

x

x

H.9.6 First-order Functional Derivative: Constant Initial Condition & Absolute Error

The absolute errors of the first-order functional derivative are provided. The models are trained under the constant initial condition.

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

004

-

8

.

003

-

8

.

002

-

8

.

001

-

8

.

000

-

7

.

999

-

7

.

998

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

Absolute error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

-

7

.

996

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

-

7

.

996

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

Absolute error

tau=0.001

x

x

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

-

7

.

996

-

7

.

994

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

-

7

.

996

-

7

.

994

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

000

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

Absolute error

tau=0.001

x

x

H.9.7 First-order Functional Derivative: Moderate Initial Condition & Absolute Error

The absolute errors of the first-order functional derivative are provided. The models are trained under the moderate initial condition.

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

03

-

8

.

02

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

005

0

.

010

0

.

015

0

.

020

0

.

025

0

.

030

0

.

035

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

03

-

8

.

02

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

005

0

.

010

0

.

015

0

.

020

0

.

025

0

.

030

Absolute error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

01

-

8

.

00

-

7

.

99

-

7

.

98

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

000

0

.

005

0

.

010

0

.

015

0

.

020

Absolute error

tau=0.000

0

.

4

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

015

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

-

7

.

990

-

7

.

985

-

7

.

980

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

0

.

4

0

.

2

0

.

0

0

.

2

0

.

4

0

.

000

0

.

005

0

.

010

0

.

015

0

.

020

Absolute error

tau=0.001

-

x

x

x

x

-

-

-

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

000

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

0100

-

8

.

0075

-

8

.

0050

-

8

.

0025

-

8

.

0000

-

7

.

9975

-

7

.

9950

-

7

.

9925

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

000

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

Absolute error

tau=0.001

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

000

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

010

-

8

.

005

-

8

.

000

-

7

.

995

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

000

0

.

002

0

.

004

0

.

006

0

.

008

0

.

010

Absolute error

tau=0.001

x

x

x

x

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

-

8

.

006

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

-

7

.

996

-

7

.

994

First-order derivative

Prediction (tau=0.000)

Analytic (tau=0.000)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

x

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

Absolute error

tau=0.000

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

-

8

.

006

-

8

.

004

-

8

.

002

-

8

.

000

-

7

.

998

-

7

.

996

-

7

.

994

First-order derivative

Prediction (tau=0.001)

Analytic (tau=0.001)

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

Absolute error

tau=0.001

x

x

H.9.8 Time-dependent Relative and Absolute Errors at θ = 0 : Delta Initial Condition

The time-dependent relative and absolute errors at θ = 0 are provided. The models are trained under the delta initial condition.

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

-

0

.

0001

0

.

0000

0

.

0001

0

.

0002

W([theta],tau)

Analytic solution (input function=0)

Prediction (input function=0)

Tau

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

Absolute error

×

10

-

5

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

-

0

.

0002

-

0

.

0001

0

.

0000

0

.

0001

0

.

0002

W([theta],tau)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

1

.

6

1

.

8

2

.

0

2

.

2

2

.

4

Absolute error

×

10

-

5

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

-

0

.

0002

-

0

.

0001

0

.

0000

0

.

0001

0

.

0002

W([theta],tau)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

2

.

3

2

.

4

2

.

5

2

.

6

Absolute error

×

10

-

5

H.9.9 Time-dependent Relative and Absolute Errors at θ = 0 : Constant Initial Condition

The time-dependent relative and absolute errors at θ = 0 are provided. The models are trained under the constant initial condition.

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

-

5

0

5

W([theta],tau)

×

10

-

6

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

0

1

2

3

4

Absolute error

×

10

-

6

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

W([theta],tau)

×

10

-

6

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

1

.

0

1

.

5

2

.

0

Absolute error

×

10

-

6

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

W([theta],tau)

×

10

-

6

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

0

1

2

3

Absolute error

×

10

-

6

H.9.10 Time-dependent Relative and Absolute Errors at θ = 0 : Moderate Initial Condition

The time-dependent relative and absolute errors at θ = 0 are provided. The models are trained under the moderate initial condition.

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

-

0

.

00010

-

0

.

00005

0

.

00000

0

.

00005

0

.

00010

W([theta],tau)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

1

2

3

Absolute error

×

10

-

5

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

-

0

.

0002

-

0

.

0001

0

.

0000

0

.

0001

0

.

0002

W([theta],tau)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

1

.

5

2

.

0

2

.

5

3

.

0

Absolute error

×

10

-

5

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

-

0

.

00010

-

0

.

00005

0

.

00000

0

.

00005

0

.

00010

W([theta],tau)

Analytic solution (input function=0)

Prediction (input function=0)

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

0008

0

.

0010

Tau

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

Absolute error

×

10

-

5

H.9.11 Time-averaged Pointwise Absolute Error: Delta Initial Condition

The time-averaged pointwise absolute errors are provided. The models are trained under the delta initial condition.

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

00000

0

.

00005

0

.

00010

0

.

00015

0

.

00020

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

Time-averaged absolute error

×

10

-

5

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

Time-averaged absolute error

×

10

-

5

-

0

.

0006

-

0

.

0004

-

0

.

0002

0

.

0000

0

.

0002

0

.

0004

0

.

0006

1

.

2

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

Time-averaged absolute error

×

10

-

5

10

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

00002

0

.

00004

0

.

00006

0

.

00008

0

.

00010

0

.

00012

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Time-averaged absolute error

×

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Time-averaged absolute error

×

10

-

5

-

2

-

1

0

1

2

5

1

.

6

1

.

8

2

.

0

2

.

2

2

.

4

Time-averaged absolute error

×

10

-

5

-

5

a2

a2

a3

a19

×

10

-

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

00000

0

.

00005

0

.

00010

0

.

00015

0

.

00020

0

.

00025

0

.

00030

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Time-averaged absolute error

×

10

-

5

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a2

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Time-averaged absolute error

×

10

-

5

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a99

10

-

6

2

.

3

2

.

4

2

.

5

2

.

6

Time-averaged absolute error

×

10

-

5

×

H.9.12 Time-averaged Pointwise Absolute Error: Constant Initial Condition

The time-averaged pointwise absolute errors are provided. The models are trained under the constant initial condition.

-

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

0000

0

.

0001

0

.

0002

0

.

0003

0

.

0004

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

2

4

6

Time-averaged absolute error

×

10

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

0

2

4

6

Time-averaged absolute error

×

10

-

6

-

0

.

0006

-

0

.

0004

-

0

.

0002

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

1

2

3

4

Time-averaged absolute error

×

10

-

6

6

10

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

0000

0

.

0001

0

.

0002

0

.

0003

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Time-averaged absolute error

×

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

0

.

5

1

.

0

1

.

5

2

.

0

Time-averaged absolute error

×

10

-

6

-

2

-

1

0

1

2

10

-

5

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

2

.

00

2

.

25

Time-averaged absolute error

×

10

-

6

-

6

a2

a2

a3

a19

×

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

00000

0

.

00005

0

.

00010

0

.

00015

0

.

00020

0

.

00025

0

.

00030

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

1

2

3

Time-averaged absolute error

×

10

-

6

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a2

0

1

2

3

Time-averaged absolute error

×

10

-

6

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a99

10

-

6

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

Time-averaged absolute error

×

10

-

6

×

H.9.13 Time-averaged Pointwise Absolute Error: Moderate Initial Condition

The time-averaged pointwise absolute errors are provided. The models are trained under the moderate initial condition.

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

00000

0

.

00005

0

.

00010

0

.

00015

0

.

00020

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

1

2

3

4

5

Time-averaged absolute error

×

10

-

5

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

0

1

2

3

4

5

Time-averaged absolute error

×

10

-

5

-

0

.

0006

-

0

.

0004

-

0

.

0002

0

.

0000

0

.

0002

0

.

0004

0

.

0006

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

Time-averaged absolute error

×

10

-

5

10

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

00000

0

.

00005

0

.

00010

0

.

00015

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

1

2

3

Time-averaged absolute error

×

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

1

2

3

Time-averaged absolute error

×

10

-

5

-

2

-

1

0

1

2

10

-

5

1

.

5

2

.

0

2

.

5

3

.

0

Time-averaged absolute error

×

10

-

5

-

5

a2

a2

a3

a19

×

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

0

.

00000

0

.

00005

0

.

00010

0

.

00015

Time-averaged absolute error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

Time-averaged absolute error

×

10

-

5

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a2

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

Time-averaged absolute error

×

10

-

5

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a99

10

-

6

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

Time-averaged absolute error

×

10

-

5

×

H.9.14 Time-averaged Pointwise Relative Error: Delta Initial Condition

The time-averaged pointwise relative errors are provided. The models are trained under the delta initial condition.

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

0

.

0006

-

0

.

0004

-

0

.

0002

0

.

0000

0

.

0002

0

.

0004

0

.

0006

10

1

10

2

10

3

10

4

10

5

Time-averaged relative error

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

2

-

1

0

1

2

10

-

5

10

7

Time-averaged relative error

a2

a2

a3

a19

×

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a2

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a99

×

10

-

6

2

.

3

×

10

7

2

.

4

×

10

7

2

.

5

×

10

7

2

.

6

×

10

7

Time-averaged relative error

H.9.15 Time-averaged Pointwise Relative Error: Constant Initial Condition

The time-averaged pointwise relative errors are provided. The models are trained under the constant initial condition.

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

6

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

10

-

7

10

-

6

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

10

-

9

10

-

8

10

-

7

10

-

6

10

-

5

Time-averaged relative error

-

0

.

0006

-

0

.

0004

-

0

.

0002

0

.

0000

0

.

0002

0

.

0004

0

.

0006

10

-

8

10

-

7

10

-

6

Time-averaged relative error

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

6

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

10

-

6

4

×

10

-

7

6

×

10

-

7

2

×

10

-

6

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

10

-

6

4

×

10

-

7

6

×

10

-

7

2

×

10

-

6

Time-averaged relative error

-

2

-

1

0

1

2

5

10

-

6

2

×

10

-

6

Time-averaged relative error

a2

a2

a3

a19

×

10

-

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

2

×

10

-

6

3

×

10

-

6

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a2

10

-

9

10

-

8

10

-

7

10

-

6

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a99

10

-

6

2

×

10

-

6

3

×

10

-

6

Time-averaged relative error

×

H.9.16 Time-averaged Pointwise Relative Error: Moderate Initial Condition

The time-averaged pointwise relative errors are provided. The models are trained under the moderate initial condition.

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

6

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

10

-

2

10

0

10

2

10

4

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

0

.

0006

-

0

.

0004

-

0

.

0002

0

.

0000

0

.

0002

0

.

0004

0

.

0006

10

1

10

2

10

3

10

4

10

5

Time-averaged relative error

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

6

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

2

-

1

0

1

2

5

10

7

Time-averaged relative error

a2

a2

a3

a19

×

10

-

-

0

.

100

-

0

.

075

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

a0

10

-

5

10

-

4

10

-

3

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a1

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

a2

10

-

1

10

0

10

1

10

2

10

3

10

4

Time-averaged relative error

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

a99

10

-

6

10

7

1

.

2

×

10

7

1

.

4

×

10

7

1

.

6

×

10

7

1

.

8

×

10

7

Time-averaged relative error

×

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our contribution is summarized at the end of Introduction. How much the results can be expected to generalize to other settings is also discussed in App. F.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in Sec. 5 and App. F.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of assumptions and complete proofs for each theoretical result is given in Sec. 3, Apps. C, D, and E. In particular, App. C is dedicated to the mathematical background for non-experts of functional analysis, making our paper as self-contained as possible.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the information for reproduction is given in Sec. 4 and App. G. We also provide our code in the supplementary materials. See also our GitHub https://github. com/TaikiMiyagawa/FunctionalPINN .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is given in the supplementary materials and GitHub ( https:// github.com/TaikiMiyagawa/FunctionalPINN ).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the information for reproduction is given in Sec. 4 and App. G. We also provide our code in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars (standard deviations) are reported in Sec. 4 and App. H.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources are reported in App. G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We carefully read the NeurIPS Code of Ethics and checked our work complies it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of functional differential equations, and there is no societal impact which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We appropriately cite relevant assets. The license and terms of use are mentioned in our code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We carefully read and follow the NeurIPS Code and Data Submission Guidelines.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper considers solving FDEs (Functional Differential Equations) using neural networks. The difficulty of solving FDEs compared to PDEs (Partial Differential Equations) lies in the fact that the input space is infinite-dimensional. In this case, the author employs Cylindrical Approximation to reduce the infinite-dimensional space to a finite-dimensional space. Then reduce the problem to a PDE case and use PINN to solve it.

STRENGTHS:
Consider the challenges of utilizing neural networks in infinite-dimensional spaces. The author's use of Cylindrical Approximation as a method to reduce the infinite-dimensional space to a finite-dimensional one is commendable. The results presented in the paper are interesting and well-written.

WEAKNESSES:
I did not find the main issue of the paper addressed. My concern is that the theoretical part of this paper is somewhat weak. The theorems primarily present asymptotic results, i.e., asking for \(m \to \infty\). In my knowledge, there are some papers considering the functional approximation such as: 
1. T. Chen and H. Chen. Approximations of continuous functionals by neural networks with application to dynamic systems. IEEE Transactions on Neural Networks, 4(6):910–918, 1993.
 2. Y. Yang and Y. Xiang. Approximation of Functionals by Neural Network without Curse of Dimensionality.
 3. L. Song et al. Approximation of smooth functionals using deep ReLU networks.

Except for the first paper, the other two papers provide the approximation error of functionals. In my opinion, if you have Cylindrical Approximation rates and PINN approximation rates, the total order of approximation may also be obtainable. Are there any difficulties in obtaining the approximation rate?

QUESTIONS:
Mentioned in the Weakness.

LIMITATIONS:
All right.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-26

comments: SUMMARY:
The power of PINNs is leveraged to solved high-dimensional PDEs which are obtained from FDEs through the cylindrical approximation. FDEs are computationally expensive to learn, and PDEs are more well-studied in the context of learning. This is a novel work in making FDEs more accessible for computation since they have a wide impact across many technical disciplines.

STRENGTHS:
1. This is a novel way to circumvent longstanging numerical issues concerning computing FDEs. The idea is easy to follow, at a high level. 
2. Computational complexity and expressive power are significantly improved compared to the current SOTA using cylindrical approximations that are shown to converge.
3. Notation and steps are clearly presented.

WEAKNESSES:
The authors presented their weaknesses/limitations in a section.
They could try to expand their suites of experiments to the applications outlined in the appendix e.g. Navier-Stokes modeling, etc.

QUESTIONS:
I don't have questions

LIMITATIONS:
The limitations are clearly presented and have been addressed, when possible, reasonably.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, they used cylindrical approximation to transform the functional differential equation (FDE) into a higher-dimensional PDE in order to solve it using PINN.\n\nSTRENGTHS:\nThis is a relatively new and intriguing topic in the field of functional differential equations (FDEs), aiming to solve them using PINN as an innovative approach. To this end, specific examples of FDEs such as FTE and BHE were provided, accompanied by a clear mathematical explanation of what cylindrical approximation entails.\n\nWEAKNESSES:\nIt seems that significant improvements are needed in the experimental section. Particularly, Figure 4 and Figure 7 suffer from poor readability, with unclear x-axis and y-axis labels that make it difficult to discern their intended purpose. Moreover, to demonstrate the advantages of the proposed method effectively, it is essential to compare it with existing numerical analysis methods such as CP-ALS. Such comparisons would clarify how the proposed method reduces computational complexity, as mentioned on line 224 of the manuscript, and illustrate how the order varies with respect to $m$ ($m^6$ vs $m^r$) using the experimental results. Therefore, the experimental section appears to be lacking overall.\n\nFurthermore, cylindrical approximation has already been studied in the field of numerical analysis. Clarifying what novelty exists in applying this approach to convert FDEs into high-dimensional PDEs and then using the well-established PINN method would be beneficial. If the author claims that using deep learning to solve FDEs is novel, it is crucial to provide clear comparisons with state-of-the-art numerical methods in terms of experimental results, computation time, computational cost, accuracy, and other relevant metrics. This comparison would ultimately demonstrate how the proposed approach stands out.\n\nQUESTIONS:\n* I'm curious about the range of \\(\\boldsymbol{a}\\) in the experiments. To train with PINN, \\(\\boldsymbol{a}\\)'s range needs to be fixed beforehand for sampling. How was this managed?\n\n* I'm interested in why r is typically \"typically 1 or 2,\" as mentioned in line 229 of the manuscript. A detailed explanation would be helpful.\n\n* The abbreviation BHE in the caption of Figure 2 first appears in line 162. It should be mentioned earlier in the text.\n\n* Aside from FTE and BHE, I'm curious if there are more complex or challenging applications related to actual physical phenomena.\n\n* In Table 1, it seems that the relative error increases as the degree increases. Is this trend correct? I would appreciate a more detailed explanation and analysis.\n\n* Figures 11-16 are labeled as extrapolation. Could you clarify what this means?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}