Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees
Authors: Taiki Miyagawa
Takeru Yokota
Venue: NeurIPS 2024 poster (2024)
Keywords: Physics-informed neural network
Functional differential equation
Functional derivative
Primary area: machine_learning_for_physical_sciences
TL;DR: We propose the first learning scheme to solve functional differential equations, built on the physics-informed neural network.

Abstract: We propose the first learning scheme for functional differential equations (FDEs).
FDEs play a fundamental role in physics, mathematics, and optimal control.
However, the numerical analysis of FDEs has faced challenges due to its unrealistic computational costs and has been a long standing problem over decades.
Thus, numerical approximations of FDEs have been developed, but they often oversimplify the solutions. 
To tackle these two issues, we propose a hybrid approach combining physics-informed neural networks (PINNs) with the *cylindrical approximation*. 
The cylindrical approximation expands functions and functional derivatives with an orthonormal basis and transforms FDEs into high-dimensional PDEs. 
To validate the reliability of the cylindrical approximation for FDE applications, we prove the convergence theorems of approximated functional derivatives and solutions.
Then, the derived high-dimensional PDEs are numerically solved with PINNs.
Through the capabilities of PINNs, our approach can handle a broader class of functional derivatives more efficiently than conventional discretization-based methods, improving the scalability of the cylindrical approximation.
As a proof of concept, we conduct experiments on two FDEs and demonstrate that our model can successfully achieve typical $L^1$ relative error orders of PINNs $\sim 10^{-3}$.
Overall, our work provides a strong backbone for physicists, mathematicians, and machine learning experts to analyze previously challenging FDEs, thereby democratizing their numerical analysis, which has received limited attention.

Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guarantees
Taiki Miyagawa* Independent Researcher
chopin. grande. valse. brillatnte@gmail.cmmmm
Takeru Yokota*
Interdisciplinary Theoretical and Mathematical Sciences Program (iTHEMS), RIKEN
Center for Quantum Computing, RIKEN takeru. yokkeru.ookotttiken....
Abstract
We propose the first learning scheme for functional differential equations (FDEs). FDEs play a fundamental role in physics, mathematics, and optimal control. How- ever, the numerical analysis of FDEs has faced challenges due to its unrealistic computational costs and has been a long standing problem over decades. Thus, nu- merical approximations of FDEs have been developed, but they often oversimplify the solutions. To tackle these two issues, we propose hybrid approach combining physics- -nssessinfoormed neural networks (PINNs) with the cylindrical approximation. The cylindrical approximation expands functions and functional derivatives with orthonormal basis and transforms FDEs into dimenmsiiii PDEs. To validate the reliability of the cylindrical approximation for FDE applications, we prove the convergence theorems of approximated functional derivatives and solutions. Then, the derived high-dimensional PDEs are numerically solved with PINNs. Through the capabilities PINNs, our approach can handle a broader class of functional derivatives more efficiently than conventional discretization- based meth- ods, improving the scalability of the cylindrical approximation. As a proof concept, conduct experiments two FDEs and demonstrate that our model can successfully achieve typical L relative error orders of PINNs 10-3 Overall, work provides strong backbone for physicists, mathematicians, and machine learning experts to analyze previously challenging FDEs, thereby democratizing their numerical analysis, which has received limited attention. Code is available at hhtpps:/ https:///gitubbe wwt://aaiiiiiiaaaaaaaaaaaaaaaa www.aaaa/nnnnnnnnaaaaaaaaa
1 Introduction
Functional differential equations (FDEs) appear wide variety of research areas 91 92
FDEs are partial differential equations (PDEs) involving functional derivatives, where functional F is a function of an input function 0(x) to a real number, i.e.,
+ F(])) E R, and functional derivative is defined the derivative of functional w.r.t. the input function at ær, denoted by (F)()))/6((2))) FDEs play fundamental role in Fokker-Plancle systems [27). turbulence theory
[671. quantum field theory (7I). mean--ield games [16]. mean--fccd optimal control [18 81] and
unnormalized optimal transport [32). Major examples of FDEs include the Hopf functional equation
These authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

in fluid mechanics, the Paakker-Planck functional equation in the theory of stochastic processes, and the functional Hamilton-Jacob equation in optimal control problems in density spaces.
Despite their wide applicability, numerical analyses of FDEs are known to suffer from significant computational complexity; therefore, numerical approximation methods have been developed over decades. They include the functional power series expansion [67] the Reynolds number expansion
667. finite difference methods [88, finite element methods
77 tensor decomposition methods
91
and the cylindrical approximation 77 29 34 88
However, they tend to oversimplify the solution of FDEs, prioritizing the reduction of computa- tional costs. The functional power series expan- sion is applicable only to input functions close to the expansion center. Moreover, it has no conver- gence guarantees in general [67]. The Reynolds number expansion requires the Reynolds num- ber to be close to zero, severely restricting its applicability, because the Reynolds number for
Fundiional fifonnttial Equation
Physics-informed Neural Networks
aF((OL() (00))((01)
8z
PLALO fiA -aa.
Cylindrical Approximation
F((0] 7 -aa-1 f)
offweelartion
am8 6ite E (x(x) Madi
Epat Day
Partill Difirential Equation
41 (,.(
Fandimaa Delistive
turbulent flow can be Z 1000. Discretization- based methods such as the finite difference and element methods restrict spacetime resolution and/or the class of input functions and functional derivatives [79]. Existing methods relying on
Figure I: Overall architecture. An FDE is simpli- fied to a high-dimensional PDE via the cylindrical approximation. The PDE is solved with a PINN The approximated functional derivative can be effi- ciently computed with automatic differentiation.
the cylindrical approximation, akin to the spec-
tral method for PDEs, include tensor decomposition [91 92 to reduce computational costs; however, they tend to significantly simplify the class of input functions and functional derivatives. For instance, their expressivity is limited to polynomials or Fourier series of few degrees.
To address the notorious computational complexity and limited approximation ability, we propose hybrid approach combining physics informed neural networks (PINNs) and the cylindrical approxi- mation (Fig. [T]. In the first stage, we expand the input function with orthonormal basis functions, thereby transtorming a given FDE into a high-dimensional PDE of the expansion coefficients. This approximation is referred to as the cylindrical approximation. We prove the convergence of the approximated functional derivatives and FDE solutions, validating the reliability of the cylindrical approximation for FDE applications, which is our main theoretical contribution. In the second stage, the derived high-dimensional PDE is numerically solved with PINN, which is known to be universal approximator tailored to solve high-dimensional PDEs efficiently in a mesh-free manner.
A notable advantage of our approach is that, with the help of PINNs, it reduces the computational complexity by orders of magnituee, compared with previous discretization-basse requires only O(m"), ), where m represents the "class size" of input functions and functional derivatives (c.g.. the degree of polynomials), and r (2 1) denotes the order of the functional derivative included in the target FDE (typically 1 or 2). This is notable reduction from the state-of-the-ar cylindrical approximation algorithm (91), which requires as large as (m))). Consequently, our approach substantially extends the class of input functions and functional derivatives that can be represented by the cylindrical approximation. For instance, our approach extends the degrees of polynomials or Fourier series used for the approximation from 6 91 to 1000, showing unprecedented expressivity.
methods. In fact,
As proof of concept, we conduct experiments on two FDEs: the functional transport equation and the Burgers-Hopl equation. The results show that our model accurately approximates not only the solutions but also their functional derivatives, successfully achieving typical L' relative error orders
of PINNs ~ 10
20
93 94].
Our contribution is threefold. (1) We propose the first learning scheme for FDEs to address the significant computational complexity and limited approximation ability. Our model exponentially extends the class of input functions and functional derivatives that can be handled accurately and efficiently. (2) We prove the convergence of approximated functional derivatives and FDE solutions, ensuring the cylindrical approximation be safely applied to FDEs. (3) Our experimental results show that our model accurately approximates not only the FDE solutions but also their functional derivatives.
2

---

2
Related Work
Research on FDEs has mainly focused on their theoretical aspects and formal solutions, with very few algorithms available to numerically solve general FDEs [8 79 9 [04]. In [88), numerical method specialized for the Hamilton-Jacobi functional equation for optimal control problems in density space is proposed, based on spacetime discretization. Similarly, [79] employs spacetime discretization with tensor decomposition. The state-of-the-art algorithm proposed in [9I], the CP-ALS (Canonical Polyadic tensor expansion & Alternating Least Squares) algorithm, uses the cylindrical approximation along with the finite difference method and tensor decomposition, requiring (m (see App. F.2 for the derivation), whereas our model requires only ((m") (r is 1 or 2 in most FDEs). Furthermore, model does not require such discretization, making it mesh-free See App. for additional introduction to FDEs and their approximations.
FDEs are prevalent across numerous research fields [27
81 67
The cylindrical approximation originates from the theory of stochastic processes 34 It is reminiscent of the spectral method for PDEs (66) and is a generalization to FDEs, Convergence theorems of the cylindrical approximation are summarized in recent seminal paper [92] Note that
cylindrical approximation this paper (Eq. (2) is different from the one 922 tailored for
practical use. Consequently, our convergence theorems also differ from those in [92 See App. C.4.2
for technical details. See App. A for more comparisons with other studies.
3 Proposed Approach
3.1 Step 1: Cylindrical Approximation
We first introduce the cylindrical approximation of functionals, functional derivatives, and FDEs, beginning with the expansion of input functions and culminating the transformation of FDEs into high-dimensional PDEs. Additionally, we prove the convergence theorems for this approximation. The rigorous mathematical background is reviewed App. [fr interested readers.
Any function 0 in a real separable Hilbert space H can be represented uniquely terms of an orthonormal basis { ) as ((x) De of {ox k200 and (---) и denotes the inner product of H. Substituting this expansion to functional F((O)), we can define a multivariable function f((a ) := F k= F(E
Firstly, we define the cylindrical approximation of functionals 7 34 88
k= (x), where a₃ := (0, σk)H H are the coefficients (or spectrum) of 0 in terms
(gok)) for any functional
F H R. Truncating at m e Zzo gives the cylindrical approximation of functionals:
f(1ax 1 = F()
where Pm. is the projection operator s.t. Pr, (xx) = En= k=0 ag ag (x), and m referred to as the degree of approximation. See Thm. C.19 and Thm. C.20 the uniform convergence and convergence rate of this approximation, originally given by 92I.
m-1
Secondly, we define the cylindrical approximation fumctional derivatives. The functional derivative of F w.r.t. 0 at is defined as
(F(D) lim p((l)) where 8(x) denotes the Dirac c-0
50(x)
delta function. This definition is impractical to simulate on computers with spacetime discretization; thus, employ the expansion ar([o))
E (F((()) ( (6)) The expansion is possible
50(z)
because 6F((0)) itself is a function of in and thus can be represented as an orthonormal basis
60(x)
expansion. Note that the expansion coefficients ( (fF(())) Φx) н are known to be equal to 8f Day (see
App. C.4.2 for the proof). Hence, truncating 0 at m - gives the cylindrical approximation of functional derivatives:
8F 8F((Pmm))
())
m-1
0f
Pm 50(x))
Σ
db (x)
Dap
k=
Note that Eq. 22 is different from the cylindrical approximation adopted in 91 922. They do not
apply P., 87 [[Pm 0)) (0000))/)/0) (xx), and the emerging "tail term" E (8)( dh (x)))())) is simply ignored without any rationale.
The first main theoretical contribution of our work is the following convergence theorem of Eq. 2
3

---

Theorem 3.1 (Pointwise convergence of approximated functional derivatives (informal)). For arbi- trary E H and orthonormal basis {du, ..... Eq. (2) converges 5F(
The formal statement and proof are given in App. D.1 The convergence rate the same as ||0 P. 0
if /F((0))50(a) span (o. ..... 1m- A technical discussion when this is not the case is
provided in App. D
Finally, we define the cylindrical approximation of FDEs [9]. In this paper, we consider the abstruct evolution equation, class of FDEs having the fol- lowing form:
10
aF( -(())F((0)))
(3)
Relative arror
with F( = (l)()). where C(()) is a linear functional operator, and Fu is given initial condi- tion. The abstract evolution equation is a crucial class FDEs in physics, mathematics, and enginecring, in- cluding the Hopf functional equation, Bekker-Plannee functional equation, and functional ComnittonJJachbe equation. The cylindrical approximation of the ab- stract evolution equation is given by
10 10
200
Degree 600
800
1000
Figure 2: Cylindrical approximation of FDE's solution. The L relative error, defined as N (_==.)((0o.)) F( [[]
Of(a, fm ((0))(((a)))) 1)
(4)
diminishes with increasing The Burgers-Hopp equation with the delta initial condition (See. (.1.23 is considered. Note that PINN's training is not included.
with ()a,0) = So(a), where a
(o.....amm11.. T), f(a,t) = F([Pmo), t), and
fola) == Fo( ()) The operator L. ([O)) is the cylindrical approximation of [((O)). Examples are given in Sees. 3.1.I & 3.1.2
The second main theoretical contribution of our work is the following convergence theorem of solutions:
Theorem 3.2 (Convergence of approximated solutions (informal)). Under the cylindrical approxi- mation (Eg. ). if the FDE depends on functional derivatives only in the form of the inner prod- uct (v. (F(()) )a (v H), then, the solution of the approximated abstract evolution equation (F([ 0, t)) converges to the solution of the original one (F((), t)) as m + cc.
The proof is given App. E| The convergence visualized in Fig. 2 Similar theorems for the FDEs with the second or orgher-order functional derivatives can be shown in a similar way. The inner- product assumption Thm. 3.2 is satisfied by major FDEs, such as the Hopf functional equation, Fokker-Planck functional equation, and functional Hamilton-Jacobi equation.
In the following, we apply the cylindrical approximation to two FDEs: the functional transport equation (FTE) and the Hoop equation (BHE).
3.1.1 Application 1: Functional Transport Equation
We first construct simple FDE, the functional transport equation (FTE), which is generalization of the transport equation (the continuity equation) [53]. The FTE is provided by
a [((0).)) = -((0)))-- |dcu(()) u(x) 8F(()),)
-f
(5)
at
50(x)
with the initial condition F([0), 0) Fo((0)) where x 1-1, 1], and u(x) is a given function. Pa
Specifically, we use the initial condition F((0).0) f (()) with Po a constant. The exact solution is F F(00).) 4) Fo( ut)) = Po f de u(x)) (0(x) - u(x))), , as can be seen by substituting this into Eq. (5). More details and motivations of the FTE provided in App. E.
The cylindrical approximation of the FTE is given by
0 f(a,t) 0r
m
à f(a t)
uk aak
(6)
k=0

---

with the initial condition f(a,0) = fo(a), where ((x) Ek=o (x)x()), and fo(a) = ()( po2 ugar with ug := (u, (a)))2()) (((11111) (z We use the Leg- endre polynomials as the orthonormal basis for h220 The exact solution of Eq. 6 is f(a. t)
Po Ee=o uz(ax ug()) -4g))
In our experiments in Sec. 4 we consider two types of FTEs: (i) uy := up for (0 otherwise) and (ii) Ug ;= Up for k < 14 (0 otherwise), where up is constant. For convenience, call them the linear and nonlinear initial conditions, respectively. The high-dimensional PDE thus obtained (Eq. (6) is solved with PINNs (Sec. 3.2
3.1.2 Application 2: Burgers-Hopp Hopf Equation
The second FDE is the Burgers- equation (BHE), a crucial equation in turbulence theory:
aF(
2² (F))
dz((T)
01
Ox2 50(z)
with the initial condition =) where E (-122, 1/2]. Specifically, we use the Gaus-
sian initial condition Fo( dx)(()++ +2 de f de'C((x x' (=))))(())') where f is constant,
and C(x,, x²) is the dimensinal covariance matrix. The exact solution F( where (((0) xx t) : 1 dz'e à (a-=') 0(z).. The derivation is provided in App. D.2.1 Strictly speaking. Eq. is modification of the original BHE. The modification includes making the BHE dimensionless and neglecting the advection term. For more technical details, see App. E.2
(()).
The cylindrical approximation of the BHE is given by
i σt ((a ()aa))=)
m- 11-11m 1mm1
Ook(x)) Ox
ô s(a
Σ Σ
da
(i())a Dai"
(8)
K=0
with the initial condition l(a, 0) So(a). where 0(x) En= (x)x(x), so(a) Fo((P.))))
-Man + = Cuaaaa with CR == f dx Ozz x' (z) We use the Fourier series as the orthonormal basis: {sx (2)) = {1, (2sin(Tkr)) v2 cos (Tkx)) Then, the exact solution
under the cylindrical approximation given by
S(a,1) 1) = -pa00 DM-1 2 4x 4x (1² C2 22
+ +e Ann Am (N2 +(+++)))>* 222.
214 122 + 4m (1++)) ( 124+1,2000000012
+e 1n - ((++1)+(++))) C25+11211 2,0e+1,2+1122kk 102 (2) /2,
where = 2M (M E N). The derivation is given in App. D.2.2
In our experiments
Sec. 4 we adopt two types of the covariance matrices: (i) C.3 = o² for all i = 2 0 (0 otherwise) and (ii) Cij = o² for i =j = 0 (0 otherwise), where x² is constant. Substituting (i) and (ii) into fo- we have two types of initial conditions, which we call the delta and constant initial conditions, respectively. Again, the high-dimensional PDE thus obtained (Eq. (8) is
solved with PINNs (Sec. 3.2).
3.2 Step 2:
Solving Approximated FDEs with PINNS
We briefly introduce the foundation of PINNs
77]. PINNS are universal approximators and can solve PDEs. Let us consider PDE 2t Nifl with an initial condition Bifl = 0, where t E [0. 1], E [-1, WV and B are operators defining the PDE and the initial condition, respectively. The PINN aims to approximate the solution
Inputs
Automatic Differentiation
Backprop.
of(a.)) dt
Residual Loss
1f(a,1) at
m-
3f 2k
Neural
f(at)
Networke
Boundary Loss
((), Thus, the inputs to the PINN are and randomly sam- pled from [0, 1] and [-1, 1, 1]. respectively. Note that (t 0, with x E [-1, 1] are also input
or(a,))
Lif(a
Po
144))
Figure 3: PINN's architecture.
5

---

to the PINN to compute the boundary loss. The inputs are transformed through linear layers and activation functions. The final output of the PINN is an approximation of f(t, denoted by f(t, z)). The loss function is the weighted sum of the residual loss (I0 - -VJill and the boundary loss |[ where || is certain norm. The partial derivatives in the loss function can be computed via automatic differentiation of the PINN's output f w.r.t the inputs (t) Finally, the weight parameters of the PINN are minimized through backpropagation.
Next, we explain how to apply PINNs to our high-dimensional PDEs: see Fig. For concrete- ness, consider the FTE (Eq. (6)) with the linear initial condition. The inputs to the PINN are (t, do, a1. am--1 and (0, co, @1...., 0m-1), randomly sampled from finite intervals, and the outputs are f(a,1) 1) and [(a O), respectively. Then, @f(a, 1)/Ot and 0f(a, 1)/8a are computed via automatic differentiation, and we obtain the residual loss [[0] (a, t)/Ot+ === = u t)/daz and the boundary loss llfo(a) po -o ugaxll. where fo(a) = f(a, 0). These losses are minimized via mini-bated optimization.
Computational Complexity The total com- putational complexity w.r.t. m up the com- putation of functional derivatives is given by (m)) + (mm)) = O(m"). where r 2 is the order of the functional derivative included in the target FDE (typically 1 or 2). The first term (m) comes from the input layer of the PINN. The second term O(m") comes from the compu- tation of functional derivatives under the cylin- drical approximation (Eq. ) See App. [F.4 for more detailed discussions computational complexity.
Table I: Mean relative and absolute errors of models trained on FTEs under linear (top two) and nonlinear (bottom two) initial conditions. I.C. is short for "imitial condition". The error bars the standard deviation over 10 training runs with different random seeds. Note that this table is not for the assessment of the theoretical conver- gence of the cylindrical approximation (see Fig. 2 App. H.6 and the footnote in Sec. 4.1 instead).
DEGREF
RELATIVE ERROR (LINEAR I.C.)
This is a notable reduction from discretization- based methods such as finite difference and el- ement methods, which typically require expo- nentially large computational complexity w.r.t.
4
(1.26820 0.31421) x 10-3
20
(2.01716 0.21742) x 10-3
100
(6.21740 + 0.33412) x 10-3
the dimension of PDE m. Also, O(m") is sig- nificantly smaller than the state-of-the-art cylin- drical approximation algorithm, the CP-ALS (91, which requires (mm)) (the derivation is given in App. F.2) Consequently, given that m represents the "class size" of input functions and functional derivatives (Eqs. & ( our approach significantly extends the range of in- put functions and functional derivatives that can be represented via the cylindrical approxima- tion. In fact, our approach extends the degrees of polynomials or Fourier series used for the approximation from 6 19 to 1000 (Sec.
DEGREE
ABSOLUTE ERROR (LINEAR I.C.)
4
(1.32203 0.44061) x 10-3
20
(2,29632 1 0.16449) X 10 10-3
100
(1.23312 0.18931) X 10-3
DEGREE
RELATIVE ERROR (NONLINEAR I.C.)
4
(1.79295 + 0,28535) x 10-3
100
(7.63769 0.90872) x 10-3
1000
(8,27096 + 1.19378) x 10-3
DEGREE
ABSOLUTE ERROR (NONLINEAR L.C.)
4
(2.37627
0.15278) x 10-4
100
(1.84506 057665 x 10-3
1000
(1.76470 0.36885) x 10
Finally, we note that the selection of basis fune- tions influences computational efficiency. The choice depends on the specific FDE, boundary conditions, symmetry, function spaces, and numerical stability. For further discussions, see App. F3).
In summary, our proposed approach transforms an FDE into a high nigh-dimensional PDE using cylindrical approximation and then solves it with PINN, which serves as universal approximator of the solution (Figs. & 3) It is important to note that our model employs the basic PINN framework, allowing for seamless integration with any techniques developed within the PINN community.
4 Experiment

---

Table Mean relative (top) and absolute (bottom) errors. The models are trained on the BHE. The eтror bars are the standard deviation over 10 training runs with different random seeds. Note that this table not for the assessment the theoretical convergence the cylindrical approximation (see Fig. 2 App. H.6. and the footnote in Sec. 4.1 instead).
INITIAL CONDITIONS
DEGREE
DELTA
CONSTANT
4
(2.93905 + 0.17403)
10"4
(12.1782 8.54758) 10
(2.20842 + 0.28531) X 10
(6.14352 = 1.28641) x 10
100
(2.41667 + 0.25264) X 10-4
(5.50375 + 1.75507) x 10
INITIAL CONDITIONS
DEGREE
DELTA
CONSTANT
(1.66451 + 0.47547) x 10
(1.62849 = 0.66896)
10
20
(1.34640 0.12105) x 10
(1.12371 = 0.32367)
10
100
(1.60980 + 0.08952) x 10-5
(1.05558 = 0.16673) x 10 -5
As proof of concept for our approach, we merically solve the FTE and BHE These two FDEs are suitable for numerical experiments because their analytic solutions are available, allowing for the computation of absolute and
relative errors, major metrics in the numerical analysis of PDEs and FDEs. Note that the ana- lytic solutions for most FDEs are currently un- known due their mathematical complexity.
02
24
-44
-042
00
69
07
Setups. We use a 4-layer PINN with 3x (lin-
24
+ sin activation + layer normalization [6]
+ last linear layer. The total loss function is
the smooth 7,- loss the of T.- and 1
losses. Softmax loss-reweighting is employed. The optimizer is AdamW [59]. The learning rate scheduler is the linear warmup with co- sine annealing with warmup [58). Latin hyper-
cube sampling 644 39 is used for the train-
ing, validation, and test sets. For the BHE, the sampling range is decayed quadratically in terms of k E 10.1,.... lize the training. We use I,- relative and ab- solute errors, standard performance metrics for numerical analysis of PDEs and FDEs. Ab- solute error 1 I5 4.) i(a,, is used instead of relative error N [
cc
10
1} to stabi-
Figure 4: Analytic solution (top four panels), prediction (second four panels), and absolute error (bottom four panels) of FTE with degree 100 under linear initial condition. The horizontal axes represent ag for k = 0, 2, 99, with all the other coefficients set to O. Our model successfully learns the FTE.
(a,, 4.1 (); 4)) when the analytic solution is close zero because relative error in such region blows by definition, regardless of the model's 's prediction. vo. Po. μ. and x² to 1, 1, 8, and 10, respectively. In App. G we provide more detailed setups for reproducibility, including the range of ag.
!Code: http:://ghthoooooaaaaaaaaa/eeaat /tw.

---

Firs-arde derivaivee Firk-rrds dervatt
Prodition (t=0)
Absolute entor
+ t=0
Analyie (t-0)
8,00
0,002
0.000
1.0
1.00
x
10
-+--
Predition (Im1)
Absolute erar
0.000
t=1
0.5
Analydic (t=1)
8.
010
0.5
1.0
0.000
-1.00
-0.00
0.00
0.50
1.00
1.00
-0.50
0.00
050
1.00
First-rrce dervative
30
Prodiction (t-0)
Absulute enor
10
t=0
Anaytic (t=0)
10
10
10
10
1,00
0.00
0.50
1.00
X 0.00
0,50
1.00
Firsk-rrde derivative
30
Prefidion (t=1)
t=1
Aratytic (t-1)
20
Absolute
16
16
-100
-0.50
0.50
1.00
-1.00
-0.50
x
0.00
0.30
1.00
Figure 5: Absolute error of first-order functional derivative of FTE with degree 100 (top) and 1000 (bottom) under linear (top) and nonlinear (bottom) initial conditions. The error bars represent the standard deviation over 10 runs with different random seeds.
B-7.6
0.000
+ t=00000
7:9
Analytic
Reative ertor
30,0000
-7999
6:00
6,000c
-.00
8.00
8.000
-04
0.4
0.2
00
inn-ordd derivddive W
4-790
-738
-7.7
t=0.001
Prediction Analytic
Relative error
0.0000
0.001
7.99
80000
8.0
-.02
88.09
-044
-02
0.0
t=0.000
Prediction Analytic
Rdative crtor
0m09
00000
7,990
79995
00002
00001
8000
00X0
-0.4
-02
40
-04
0.2
00
4070000
-79992
0=0.001
Prediction
Analytic
Rolative error
00000
0.001
79900
0.0000
8--0000
0.002
42005
0.000
80030
00000
-0.4
0.0
0.2
0.4
0.4
0.2
0.0
0.2
014
Figure 6: Relative error of first-order functional derivative of BHE with degree 100 under delta initial condition. The error bars represent the standard deviation over 10 runs. The top/bottom four panels show the results with/without the loss function ||[[[), t)]l. respectively. With this loss function, the error reduces by 10 10

---

4.1 Result Functional Transport Equation
Tab. I shows the main result Our model achieves typical relative error orders of PINNs ~ 10 10-3, even when the degree is large as 1000, which means our model's capability of representing 0 and 8F((0), (z))0(z) as polynomials of degree 1000. This is a notable improvement from the state-of-the-ar algorithm (TT. which handle S 6.
Fig. visualizes the analytic solution, prediction, and absolute error of a model trained on the FTE with degree 100 under the linear initial condition. Note that some of the collocation points used for
plotting Fig. 4 are not in the training set, as can be seen from Figs. 16 in App. H.3 This aspect
highlights the model's 's ability to extrapolate beyond its training data (ak 0).
Fig. 5 shows the absolute error of the first-order functional derivative estimated at and Again, 0 = 0 is not included in the training set. The errors in the top four panels increase at the edges of intervals (x = 1) due to Runge's phenomenon [8)]
4.2 Result: Hurgers-Hopf Equation
Tab. Tab. shows the main result. Again, our model
successfully achieves 10-3 the typical order of relative етгоr of PINNs. See Fig. 2 App. H.6
and the footnote in Sec. 4.1 for the assessment
of the theoretical convergence of the cylindrical approximation.
10
-100
9.00
0.300
0.5
01
0000
0001
0.02
v10
Myri Ti
Fig, shows the relative error of first-order func- tional derivatives at 9 0. Note again that some of the collocation points used for this figure are not included in the training dataset, highlight- ing the model's ability to extrapolate beyond its training dataset (ax ~ 0). Additionally, the error is reduced by a factor of 10-1 1 by incorpo- rating loss term corresponding W ([(0),) 1) = (bottom four panels).
0.00
-10
xto
8:355
x00
2.50
100
-260 1.30
-0.00 2000 3:00
the identity
L770
Lot
1 m
Fig. 7 visualizes the analytic solution, predic- tion, and absolute error a model trained on the BHE with degree 20 under the delta initial con- dition. The absolute error w.r.t. ag is 10-4 times smaller than the scale of the solution; i.e., the model learns f well in the direction of ag, which dominates the analytic solution. Conversely, the absolute error w.r.t. 019 is on par with the scale of the solution. This result is anticipated because the dependence of the solution on 019 is much smaller than ag- This relationship is evident from Eq. (9). which indicates that the higher de- gree terms decay exponentially in terms of k;, and f, and the solution is dominated by 0Ą with S 1. Therefore, optimizing the model in the direction of a1 has only negligible effect minimizing the loss function.
1uae -030 2000
0061
-2
-0.0 RoDe
tp
0000
00.00
,,000 2001
D004
10
178
1m5
1
Figure 7: Analytic solution (top four panels), prediction (second four panels), and absolute error (last four panels) of BHE with degree 20 under delta initial condition. The horizontal axes are ag (k = 0, 1,2, or 19). The other coefficients are kept O. Our model successfully learns the BHE.
Finally, many additional experimental results are provided in App. including a comparison with the CP-ALS algorithm.
NNote that Tabs. i are not for the assessment of the theoretical convergence of the cylindrical approxima- tion, unlike Fig. 2 because the analytic solutions used for error computation vary across each row, depending on the degree. See App. H.6 instead, where additionally perform cross-degree evaluation.

---

Limitations
One limitation of our work is that the spacetime dimension is limited + 1 (t and in our experi- ments. However, generalization to + d dimensions is feasible, albeit with additional computational costs. For d dimensional spaces, we have several options for expansion bases 79]
Another limitation is that the orders of functional derivatives in FDEs in our experiments are lim- 2 2 is straightforward. For instance, the cylindrical approximation of the second-order functional derivative is expressed as 8 F(O), ()/50(1))00()) R 22 f(a, f) (1) ())), which can be computed via backpropagation twice.
ited to r 1. However, extending
Furthermore, this paper focuses exclusively on the abstract evolution equation. While this includes important FDEs (see Sec. B). it does not cover certain equations, such as the Schwinger- Dyson equation or the Wetterich equation. Nonetheless, the mathematical foundations regarding the existence and uniqueness of these FDEs remain unestablished, which is beyond the scope of our paper. Once these foundations are defined, applying our model these equations would be straightforward. More discussions limitations, including technical ones, are provided in App. F.1
Acknowledgments and Disclosure of Funding
Taiki Miyagawa, an employee at NEC Corporation, contributed to this paper independently of his company role. He undertook this research independent researcher under the permission by the company. Takeru Yokota was supported by the RIKEN Special Doctoral Researchers Program.
10

---

References
[1] V. Acary and B. Brogliato. Numerical Methods for Nonsmooth Dynamical Systems: Applications in Mechanics and Electronics. Springer Science & Business Media, 2008.
[2] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter opti- mization framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '19, page 2623-2631, New York, NY, USA, 2019. Association for Computing Machinery. License: MIT License.
[3] T. Alankus. The generating functional for the probability density functions of navier-stokes turbulence. Journal of Statistical Physics, 13:121111 1988.
[4] A. Anandkumar, K. Arizzadcneshcli, K. Bhattacharya, N. Kovachki, L, B. Liu, and A. Stuart, Neural operator: Graph kemel network for partial differential equations. In /CLR 2020 Workshop Integration of Deep Neural Models and Differential Equations, 2019.
[5] N. Aronszajn. Differentiability of Lipschitzian mappings between Banach spaces. Studia Mathematica, 57(2):147-19) 190, 1976.
[6] J. L. J., R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.064550 2016.
[7] J. C. Bacz and S. Sawin. Functional integration spaces of connections. Journal of Functional Analysis, 1501)); 1997.
[8] S. Basir and I. Senocak. Critical investigation of failure modes in physics-informed neural networks. In AiAA SCITECH 2022 Forum, page 2353, 2022.
[9] M. Bauer, F. Dupont, A. Brock, D. Rosenbaum, J. R. Schwarz, and H. Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv: 2302.03130, 2023.
[10] A. G. Baydin, A. Pearlmutter, A. A. Radul, and M. Siskind. Automatic differentiation in machine lcarning: A survey, Journal of Marchine Learning Research, 18:1 1311,3,,77 2018.
[11] M. Beran, Statistical continuum theories. Transactions of the Society of Rheology, 9(1::339-335 1965.
[12] J. Blechschmidt and O, G. Ersst Three ways solve partial differential equations with neural networks
A review. GAMM- Mitteiluuiin,, 44()):202100006, 2021.
[13] N. Bogoliuhov. On the theory of superfluidity. Journal of Physics, 11(1):23, 1947.
[14] Brugiapaglia, S. Dirksen, H. C. Jung, and H. Rauhut. Sparse recovery in bounded Riesz systems
with applications to numerical methods for PDEs. Applied and Computational Harmonic Analysis, 33:231-2693 2021.
[15] M. Burlutskaya, On Riesz bases of root functions for class of dunftionalldiffeeeentaal operators on a graph. Differential Equations, 45:779-788, 2009.
[16] R. Carmona, F. Delarue, et al. Probabilistic theory of mean feln games with applications 1-Il. Springer, 2018.
[17] Y. Chen, B. Hosseini. H. Owhadi, and A. M. Stuart. Solving and learning nonlinear PDEs with Gaussian processes, Journal of Computational Physics, 447:1 10668, 2021.
[18] Y. T. Chow, W. Li, S. Osher, and W. Yin. Algorithm for Hamilton-Jacee equations in density space via a generalized Hopf formula. Journal of Scientific Computing, 80:1 191-1299999 2019.
[19] S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physies-informed Computing, 92(3):88, 2022.
informed neural networks: Where we are and what's next. Journal of Scientific
[20] S. Das and S. Tesfamariam. State-of-the-art review of design of experiments for physics-informed deep lcarning, arXiv preprint arXiv:2202.064/ 2022.
[21]] A. Daw, J. Bu, S. Wang, P. Perdikaris, and A. Karpatne. Mitigating propagation failures in physics- informed neural networks using www.--reeemeeeeeeeaaaa (R3) sampling. Krause, E. Brunskill, K. Cho. B. Engelhardtt S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7264-7302. PMLR, 23-29 Jul 2023.

---

[22] P. Di Gianantonio, A. Edalat, and R. Gutin. A language for evaluating derivatives of functionals using automatic differentiation. Electronic Notes in Theoretical Informatics and Computer Science. 2023.
[23] N. Dupuis, L. Canet, A. Eichhorn, W. Metzner, J. Pawlowski, M. Tissier, and N. Wschebor: The nonperturbative functional renormalization group and its applications. Physics Reports, 910:1-114, 2021.
[24] R. Easther, D. Ferrante, G. Guralnik, and D. Petrov. A review of two novel numerical methods in QFT. arXiv preprint hep-lat/0306038, 2003.
[25] K.-J. Engel. R. Nagel, and S. Brendle. One-paruneter semigroups for linear evolation equations, volume 194 of Graduate Texts in Mathematics. Springer, 2000.
[26] Fiedler, K. Shah, M. Bussmann, and A. Cangi. Deep dive into machine learning density functional theory for materials science and chemistry. Physical Review Materials, 6(4):040301, 2022.
[27] Fox. Functional-calculus approach stochastic differential equations. Physical Review A, 33:467- 476, Jan 1986.
[28] G. Franzese, G. Corallo, S. Rossi, M. Heinonen, M. Filippone, and Michiardi. Continuous-tinn functional diffusion processes. In Thirty-seventt Conference on Neural Information Processing Systems, 2023.
[29] K. Friedrichs and H. Shapiro. Integration over Hilbert space and outer extensions. Proceedings of the National Academy of Sciences, 43(4):336-33 338. 1957.
[30] Fukuda, T. Kinoshita, and T. Kubo. On the finite element method with Riesz bases and applications some partial differential equations. In 2013 10th International Conference on Information Technology: New Generations, pages 761-766, 2013.
[31] R. Fukuda, T. Kotani, Y. Suzuki, and Yokojima. Density functional theory through Legendre transfor- mation. Progress of Theoretical Physics, 92(4):833-862, 10 1994.
[32] W. Gangbo, W. Li, S. Osher, and M. Puthawala. Unnormalized optimal transport. Journal of Computa- tional Physics, (9), dee 2019.
[33] Z. Gao, I.. Yan, and T. Zhou. Failure-informed adaptive sampling for pinns. SIAM Journal on Scientific Computing, 45(4): A1711-A1994 2023.
[34] I. I. Gikhman and A. V. Skorokhod. The theory of stochastic processes I. Classics in Mathematics. Springer Berlin Heidelberg, 2004.
[35] V. Gopakumar, S. Pamela, and D. Samaddar. Loss landscape engineering via data regulation on PINNs. Machine Learning with Applications, 12: 10:40444, 2023.
[36] D. Guidetti, B. Karasözen, and S. Piskarev. Approximation of abstract differential equations. Journal of Mathematical Sciences, 122(2):3013 30543 2004.
[37] B. Guo and J.-M. Wang. Riesz basis generation of abstract second-order partial differential equa- tion systems with general separated boundary conditions. Numerical Functional Analysis and Optimization, 27(3-4):2913222 2006.
[38] C. R. Harris, K. J. Millman, S. J. van der Walt. R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg. N. Smith, R. Kern, M. Picus, S. Hoyer. M. H. van Kerkwijk, M. Brett, A. Haldane, J. Del Río, Wiebe, P. Peterson, P. Gérard Marchant, K. Sheppard, Reddy, W. Weekesser, Abbasi, Gohlke, and T. E, Oliphant. Array programming with NumPy, Nature, 585(7825):3577366) 362, 09 2020. License: BSD 3 3-Clause "New" or "Revised" License.
C.
[39] J. C. Helton and F. Davis. Latin hypercube sampling and the propagation of uncertainty analyses of complex systems. Reliability Engineering & System Safety, 8 81(1):23-65 2003.
[40] Hendrycks and K. Gimpel. Gaussian Error Linear Units (GELUs), arXiv preprint arXiv:1606.08415. 2016.
[41] J.S. Hesthaven, Gottlieb, and Gottlieb. Spectral methods for time-dependent problems, volume 21. Cambridge University Press, 2007.
[42] P. Hohenberg and W. Kohn. Inhomogeneous electron gas. Physical review, 136:B864-BS3 Nov 1964.
[43] E. Hopf. Statistical hydromechanics and functional calculus. Journal of rational Mechanics and Analysis. 1:87-123. 1952.
12

---

[44] Z. Hu, K. Shukla, G. E. Karniadakis, and K. Kawaguchi. Tackling the curse of dimensionality with physics-informed neural networks. arXiv preprint arXiv:2307.12306, 2023.
[45] Z. Hu, Z. Yang. Y. Wang. G. E. Karniadakis, and K. Kawaguchi. variane trade-oft in physics- informed neural networks with randomized smoothing for high-dimensional pdes, arXiv preprint arXiv:231 152283, 2023.
[46] C. Itzykson and J.-B. Zuber. Quantum field theory. Courier Corporation, 2012.
[47] A. D. Jagtap, K. Kawaguchi, and E. Karmiadakis. Adaptive activation functions accelerate convergence in deep and physics-iffrmed neural networks. Journal of Computational Physics, 404:1091 1913119,166000 2020.
[48] M. Jordan, Y. Wang, and A. Zhou. Empirical Gateaux derivatives for causal inference. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 8512 22225,, Curran Associates, Inc., 2022.
[49] V. L. Klyatskin. Dynamics stochastic systems. Elsevier, 2005.
[50] A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548-26560, 2021.
[51] Kurdyumov A. Khromov. Riesz bases formed by root functions a functional-diifereee equation with a reflection operator. Differrnnial Equations, 44(2), 2008.
[52] Kurdyumov and A. P. Khromov. The Riesz bases consisting of cigen and associated functions for a functional differential operator with variable structure. Russian Mathematics, 54:33-45, 2010.
[53] L. D. Landau and F. M. Lifshitz. Fluid Mechanics: Landau and Lifshitz: Course of Theoretical Physics, Volume 6, volume 6. Elsevier, 2013.
[54] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
[55] Li. H. Zheng, N. Kovachki. Jin. H. Chen, Liu, Azizzadenesheli, and A. Anandkumar. Physies- informed neural operator learning partial differential equations. arXiv preprint arXiv:211.003994, 2021.
[56] M. Lin. Automatic functional differentiation in JAX. In The Twelfth International Conference on Learning Representations, 2024.
[57] J. Lindenstrauss and D. Preiss. On Fréchet differentiability of Lipschitz maps between Banach spaces. Anmals of Mathematics, pages 257-288, 2003.
[58] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
[59] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[60] Lu, P. Jin, and G. E. Karniadakis. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint 1rXiv:1910.03192 2019.
[61] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation thcorem of operators. Nature machine intelligence, 2(3):218-223 2021.
[62] I. D. Luigi, A. Cardace, R. Spezialetti, P. Z. Ramirez, Salti, and L. di Stefano. Deep learning implicit neural representations shapes. In The Eleventh International Conference on Learning Representations, 2023.
[63] P. Mankiewicz. On the differentiability of Lipschitz mappings in Fréchet spaces. Studia Mathematica, 45(1):15-29, 1973.
[64] M. D. McKay, R. J. Beckman, and W. J. Conover. A comparison of three methods for selecting values of input variables in the analysis of output from computer code. Technometrics, 42(1):55 61, 2000.
13

---

[65] R. Melrose. Introduction to functional analysis, 2005.
[66] B. Meuris, S. Qadeer, and P. Stinis. Machine-leariing-basee spectral methods for partial differential equations. Scientific Reports, 13(1):1739, 2023.
[67] A. S. Monin and A. M. Yaglom. Statistical fluid mechanics, volume II: Mechanics turbulence, volume 2. Courier Corporation, 2013.
[68] D. Neidinger. Introduction to automatic diffcrentiation and MATLAB object-oriented programming.
SIAM review, 52(3):545 563, 2010.
[69] R. H. Nichols. Turbulence models and their application to complex flows, 2010.
[70] A. Paszke, S. Gross, F. Massa, Lerer, Bradbury. G. Chanan, T. Killeen, Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019.
License:
[71] M. E. Peskin and V. Schroeder. An Introduction to quantum field theory. Addison- Saaiison-Weeley... Reading, USA, 1995.
[72] J. Polchinski. Renormalization and effective lagrangians. Nuclear Physics B, 231())2669 295, 1984.
[73] J. Polonyi and K. Sailer. Effective action and funnity-fu theory. Physical Review B, 66: 66:155513: 13, Oct 2002.
[74) D. Preiss. Differentiability of Lipschitz functions on Banach spaces. Journal of Functional Analysis, 91(2):122-345, 1990.
[75] P. Prenter. A Weierstrass theorem for real, separable Hilbert spaces. Journal of Approximation Theory. 3(4):341-351, 1970.
[76] R. Rabah, G. M. Sklyar, and A. V. Rezounenko. Generalized Riesz basis property in the analysis of neutral type systems. Comptes Rendus Mathematique, 337(1):19 2211992222222 2003.
[77] M. Raissi, Perdikaris, and E. Karniadakis. Physics-informed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[78] S. Ramasinghe and S. Lucey. Beyond periodicity: Towards a unifying framework for activations in Coordinate-MI.Ps. In Eurppean Conference on Computer Vision, pages 142-158. Springer, 2022.
[79] A. Rodgers and D. Venturi. Tensor approximation of functional differential equations. arXiv preprint 26X1v:2403.04946, 2024.
[80] C. Runge. Über empirische funktionen und die interpolation zwischen äquidistanten ordinaten. Zeitschrift für Mathematik und Physik, 46:224 243, 1901.
[81] Ruthotto, S. J. Osher, W. Li, L. Nurbekyan, and S. Fung. machine learning framework for solving high-ddieesionnl mean field game and mean field control problems. Proceedings of the National Academy of Sciences, 1 117(7))1183999 199939333999,,991 2020.
[82] A. Schwenk and J. Polonyi. Towards density functional calculations from nuclear forces. In 32nd International Workshop on Gross Properties of Nuclei and Nuclear Excitation: Probing Nuclei and Nucleons with Electrons and Photons, pages 273-282, 2004.
[83] R. Seiringer. The excitation spectrum for weakly interacting bosons. Communications in Mathematical Mhysics, 306:565-578, 2011.
[84] Sharma, I.. Evans, M. Tindall, and Nithiarasu. Stiff-PDEs and physics-informed neural networks. Archives of Computational Methods in Engineering, 30(5))292922999 29588, 2023.
[85] J. Shen, T. Tang, and L. Wang. Spectral methods: Algorithmss analysis and applications, volume 41 of Springer Series in Computational Mathematics. Springer Berlin Heidelberg, 2011.
[86] M. Vainberg. Variational Methods for the Study of Nonlinear Operators. Holden- Day Series in Mathe- matical Physics. Holden- -ay, 1964.
14

---

[87] M. Valiev and G. W. Fernando. Generalized Kohn-Sham density-functional theory via effective action formalism, arXiv preprint urXi::comddmmu//7722244 1997.
[88] J. van Neerven. Stochastic evolution equations. ISEM lecture notes, 2008.
[89] G. Van Rossum and F. L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009.
[90] D. Venturi. Conjugate flow action functionals. Journal of Mathematical Physics, 54(11), 2013.
[91] D. Venturi. The numerical approximation of nonlinear functionals and functional differential equations. Physics Reports, 732:1 102, 2018.
[92] Venturi and A. Dektor. Spectral methods for nonlinear functionals and functional differential equations. Research in the Mathematical Sciences, 8(2):27, 2021.
[93] Wang. S. Li, D. He, and Wang. Is L physics informed loss always suitable for training physics informed neural network? In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[94] H. Wang, L. Lu, S. Song, and G. Huang. Learning specialized activation functions for physics-informed neural networks. arXiv preprint arXiv:2308, 04073, 2023.
[95] S. Wang. H. Wang. and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science advances, 7(40)=ca 2021.
[96] Wang and C.-Y. Lai. Multi-stage neural networks: Function approximator of machine precision. arXiv preprint ar arXiv:2307.08900 2023.
[97] F. J. Wegner and A. Houghton. Renormalization group equation for critical phenomena. Physical Review A, 8:401-412, Jul 1973.
[98] E. Weinan, J. Han, and A. Jentzen. Algorithms for solving high dimensional PDEs: From nonlinear Monte Carlo to machine learning, Nonlinearity, 35(1):278, 2021.
[99] Weinan, Han, and Q. Li. A mean-field optimal control formulation of deep learning. arXiv preprint arXiv:1807.0108) 2018.
[100] C. Wetterich. Exact evolution equation for the effective potential. Physical Letter B, 301:90094, 1993.
[101] N. Wiener. Nonlinear Problems in Random Theory. MIT Press, 1966.
[102] K. G. Wilson and J. Kogut. The renormalization group and the expansion. Physical Reports, 12(2):75- 199, 1974.
[103] J. Yao, C. Su, Z. Hao, S. Liu, H. Su, and J. Zhu. MultiAdam: Parameter-wise scale-invariant optimizer for multiscale training of physics-informed neural networks.
Krause, E. Brunskill, Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages PMLR, 23 29 Jul 2023.
[104] T. Yokota. Physics-ifformed neural networks for solving functional renormalization group lattice. Physical Review B, 109:214205, Jun 2024.
[105] T. Yokota and T. Naito. Ab initio construction of the energy density functional for electron systems with the 3:1012015, Feb 2021.
density functional theory. Physical Review Research,
[106] Q Zeng. S. H. Bryngelson, and F. T. Schaefer. Competitive physics informed networks. In ICLR 2022 Workshop on Gamification and Multiagent Solutions, 2022.
[107] A. Zhou, K. Yang. Y. Jiang, K. Burns, W. Xu, S. Sokota. Z. Kolter, and C. Finn. Neural functional transformers. In seirth- Conference on Neural Information Processing Systems, 2023.
15

---

Appendix
Contents
Introduction
2
Related Work
3
Proposed Approach
3.1
Step
Cylindrical Approximation
3.1.1
Application I: Functional Transport Equation
3.1.2
Application 2: Burgers-Hopf Equation
3
3.2
Step 2: Solving Approximated FDEs with PINNS
4
Experiment
4.1
Result: Functional Transport Equation
4.2
Result: Burgers Hopf Equation
Limitations
10
Supplementary Related Work
18
Additional Introduction to FIDE
20
B.
Background
20
B.2
Turbulence!
................................
20
B.3
Quantum Field Theory
21
B.4
Density Functional Theory
22
Theoretical Background of Cylindrical Approximation and Convergence
23
Continuity and Compactness of Functionals
23
C.2
Boundedness, Closedness, Compactness, and Pre coepatepsssssss of Metric Space of
Functionsl
23
C,3
Differentiation of Functionals
24
C.4
Cylindrical Approximation
25
C.4.1
Functionals
25
C.4.2
Functional Derivatives
26
C.4.3
Pointwise Convergence of Functional Derivatives under Cylindrical Approxi-
mationl
28
C.4.4
Abstract Evolution Equations
28
D Proofs 1
31
D.I
Theorem: Pointwise Convergence of Functional Derivatives under Cylindrical Ap- proximation
31
D.2
Derivation of Exact Solution of Burgers Hurgers-HHoff Equation
32
D.2.1
Derivation of Eq. (93)
32
D.2.2
Derivation of Eq. (104)
33
Proofs II: Details of Functional Transport Equation and Burgers- Burgers-Hopf Equation Functional Transport Equation
35
E.1
35
E.I.1
Analytic Solution
35
E.1.2
Cylindrical Approximation and Convergence
36
E.1.3
Initial Conditions
37
E.2
Burgers-Hooe Equation
37
E.2.1
Analytic Solution
38
E.2.2
Cylindrical Approximation and Convergence
38
E.2.3
Initial Conditions
40
Supplementary Discussion
41
Limitations
41
16

---

F2
Computational Complexity of CP-ALS
41
Choice of Bases
42
F.4
Computational Complexity Revisited
42
F.5
Comparison with DFT
F.6
FDE Has Multiple Meanings
43
E7
Radial Gaussian Sampling
43
F8
Curvilinear Patterns in Fig. and Noisy Patterns in Fig. and
43
Second-order Functional Derivalives
43
Detailed Experimental Settings
45
Additional Experimental Results
49
H.1
P. Converges to
49
H.2
Runtime
49
H.3
Latin Hypercube Sampling and Curse Dimensionality
51
H.4
Comparison with Baseline: Advection-reattin reaction Functional Equation (ARE)
54
H.5
Relative and Absolute Errors When W(0,0) () Is Included Loss
55
H.6
Cross-degree Evaluation
59
H.6.1
Relative and Absolute Errors of Functional Transport Equation
60
H.6.2
Relative and Absolute Errors Burgers-Hopf Equation
62
H.7
Loss Curves and Learning Rate Scheduling
64
H.8
Additional Figures: Functional Transport Equation
67
H.8.1
Worst Coefficients and Best Coefficients
67
H.8.2
First-order Functional Derivative: Linear Initial Condition & Relative Error
69
H.8.3
First-order Functional Derivative: Nonlincar Initial Condition & Relative Error
71
H88.4
First-order Functional Derivative: Linear Initial Condition & Absolute Error
73
H.8.5
First-order Functional Derivative: Nonlinear Initial Condition & Absolute Error
75
H.8.6
Time-dependent Relative and Absolute Errors f Linear Initial Condition
77
H.8.7
"eme-dependent Relative and Absolute Errors at 0 0: Nonlinear Initial
Condition
H.8.8
Time-averaged Pointwise Relative Error: Linear Initial Condition
83
H.8.9
Time-averaged Pointwise Relative Error: Nonlinear Initial Condition
85
H.8.10
Time-averaged Pointwise Absolute Error: Linear Initial Condition
87
H.8.11
Time-averaged Pointwise Absolute Error: Nonlinear Initial Condition
89
H.9
Additional Figures: Targers-Hopi Equation Equuiaaioo
91
H.9.1
Worst Coefficients and Best Coefficients
91
H.9.2
First-order Functional Derivative: Delta Initial Condition & Relative Error
93
H.9.3
First-order Functional Derivative: Constant Initial Condition & Relative Error
96
H.9.4
First-order Functional Derivative: Moderate Initial Condition & Relative Error
98
1.9.5
First-order Functional Derivative: Delta Initial Condition Absolute Error
101
H.9.6
First-order Functional Derivative: Constant Initial Condition & Absolute Error
04
1.9.7
First-order Functional Derivative: Moderate Initial Condition Absolute Error
06
H.9.8
Time -ime-dependent Relative and Absolute Errors 0 0: Delta Initial Condition
009
H.9.9
Time-depended Relative and Absolute Errors at = 0: Constant Initial
Condition
H.9.10
Time-dependen Relative and Absolute Errors at 0 0: Moderate Initial Condition
115
H.9.11
Tvme-averaged Pointwise Absolute Error: Delta Initial Condition
118
H.9.12
Time-averaged Pointwise Absolute Error: Constant Initial Condition
120
H.9.13
Time-averaged Pointwise Absolute Error: Moderate Initial Condition
122
H.9.14
Time-averaged Pointwise Relative Error: Delta Initial Condition
124
1.9.15
Time-averaged Pointwise Relative Error: Constant Initial Condition
126
H.9.16
Time-averaged Pointwise Relative Error: Moderate Initial Condition
128
17

---

A Supplementary Related Work
Note on 91 & [92]. Most of the numerical results presented in [9]] and [9] are derived from simulations based on analytically specified functions and functionals, without involving the numerical integration of FDEs. For instance, Fig. 2 in [92] does not depict the result of numerically solving an FDE. Instead, it illustrates the approximation error of the cylindrical approximation of analytically given functional (refer to Eq. (127) in (92). The performance of numerical integration FDEs using CPU-based algorithms is given in Fig. 38 in [9I], which shows the application of the CP-ALS and hierarchical Tucker (HT) methods for cases with m S 6. Note that the HT is reported to be slower than the CP-ALS (9T.
Applications of functionals. Functionals play a fundamental role in stochastic systems [1OT] 49, Fokker-Planck equations [27]. the statistical theory of turbulence [43 3 67]. the theory of superfluidity [13 [8)] quantum field theories [90 91], mean-field games [16]. meany-body Schrödinger
equations mean mean-field optimal control 99 and unnormalized optimal transport [32].
Examples of FDEs. Examples of FDEs include the Schwinger- -ysonnDyonnn equation in quantum field theory 46). the Hopf characteristic functional equation in fluid mechanics and random processes probability density functional equations, and effective Fokker-Plande systems [9T].
Operator learning. Functionals can be seen as operators that map a function to scalar; thus, operator learning [ 54 60) 61] appears to be promising approach to learning functionals. However, this method requires simulation or observation data unless PINNs are used simultaneously [55] 95). In other words, operator learning methods solve inverse problems, while we focus on forward problems, where only the equation to be solved given.
Other approximation methods for FDEs. A common class of solvers for FDEs is based on truncating power series expansions at a finite order (67). This includes the functional Taylor expansion, which expands functional in terms of its argument functions. However, its applicability is limited because solutions obtained from the functional Taylor expansion can only be used for input functions close to the expansion center.
Another type of expansion used in the theory of functional renormalization group is the derivative expansion [23]. It is expansion in terms of derivatives of the input functions. However, solutions are only feasible for inputs close to constants. For example, in the dime-diimeiional O(N) statistical model, derivative expansions up the sixth order have been executed 233, but they are limited to calculations in uniform states and cannot handle non-uniform states.
Yet another expansion method uses the Reynolds number to distinguish between laminar and turbulent flow and has been applied the Hopf equation (67. However, increasing the truncation order poses significant challenge. Specifically, calculating each expansion coefficient requires spatial integrals, leading to an exponential increase in computational costs.
Influence functions can be used for approximating Gateaux derivatives. In 48 the proposed approach based on a -ifiteddiffeeenc approximation of Gateaux derivatives, which requires a computational mesh for input function space when solving FDEs. Such an approach is infeasible because the number of mesh points grows exponentially with respect to the size of the input-function space.
ligge-iimeniionll PDEs. In our experiments, with the help of PINNs, we numerically solved 9000-dimensional PDEs, which are impossible to handle with discretization-basse methods, such as the finite element method. Numerical computation of high-dimensional PDEs is known suffer from the curse of dimensionality, making PDEs with dimensions d 2 40 particularly challenging 988)
However, rapid development in this field, especially in PINNs, has enabled solving much higher- dimensional PDEs. For example, a 100.000-dimensionaal PDE is now tractable [44), which can be combined with our model. Nevertheless, d ~ ((100) is typically sufficient in practice long as input functions are regular. See also Fig. 10
PINNs. PINNs are a type of mesh-free universal approximators of PDE solutions 77 There are several machine learning-bass based mesh-free solvers, e.g.. [17].
18

---

Automatic functional differentiation of higher-order functions. Automatic differentiation of higher-order functions has long history in theoretical computer science (see 668 [0] and the references therein)." Most studies focus on the mathematical aspects of programming languages, particularly how implement reliable automatic functional differentiation, which beyond our
scope. We cite two recent papers that explicitly mention functional derivatives. Di et al. [22] develop a language to compute automatic functional derivatives; however, the implementation is not available. Lin (56) provides JAX implementation of functional derivatives w.r.t parameterized input functions. However, supports only local, semilocal, and several nonlocal operators, limiting the functional increases.
space. In contrast, our model extends its approximability
Density functional theory (DFT), An alternative neural network-based approach to functional analysis utilizes finite element methods for spacetime grid approximation, commonly employed in first-principlss computations density functional theory (DFT) [26]. Examples include neural network, F(1y; = frr))}), approximating a target functional F((f) by evaluating f at specific grid points {ry}s- Functional derivatives each grid point can be computed using automatic
differentiation: ()((r) Jr (f }:s- However, the central focus of the machine learning studies for DFT does not include solving PDEs, alone FDEs.
5y:
Neural functional networks. Implicit Neural Representations (INRs) is another strategy to handle functions as the inputs to neural networks [07
62 28 However, this method requires large number of weight parameters, resulting in substantial computational demands.
19

---

B Additional Introduction to FDEs
B.1 Background
Functional differential equations (FDEs) are prevalent across various scientific fields, including the Hopf equation in statistical turbulence theory 43 the Schwinger-Dyson equation in quantum field theory [7], the functional renormalization group equation [97 T02 72 1OO], the Tokker-Planck equation in statistical mechanics [27], and equations for the energy density functional in DFT [73 [2]
The strength of FDEs lies in their comprchensive nature, enabling the derivation of various statistical properties of physical systems. For instance. the Hopf equation yields the characteristic functional, encompassing all information about simultancous correlations of velocities at different positions, a crucial quantity in turbulence theory. Therefore, highly accurate, efficient, and universal method for solving FDEs significantly impacts a broad range of scientific research but is yet to be explored.
A common class of solvers for FDEs is based on truncating power series expansions at finite order [67]. This includes the functional Taylor expansion, which expands a functional terms of its argument functions. However, its applicability is limited because solutions obtained from the functional Taylor expansion can only used for input functions close to the expansion center.
Another type of expansion used in the functional renormalization group theory is the derivative expansion 23 is an expansion in terms of derivatives of the input functions. However, solutions are only feasible for inputs close to constants. For example, in the three-dimensional O(N) statistical model, derivative expansions up to the sixth order have been executed 23 but they are limited to calculations in uniform states and cannot treat non-uniform states.
Yet another expansion method uses the Reynolds number, distinguishing between laminar and turbulent flow, and has been applied to the Hopf equation [67]. However, increasing the truncation order poses significant challenge. Specifically, calculating each expansion coefficient requires spatial integrals, leading to an exponential increase in computational costs.
An alternative solving FDEs is the cylindrical approximation [9]]. In this method, the input function is expanded using a set of basis functions truncated at a finite degree. The cylindrical approximation transforms FDE into nigh-dimenniioal PDE, and discretization-batt are often used together. To address its high computational cost, tensor decomposition methods are also used. Canonical Polyadic (CP) tensor expansion with the Alternating Least Squares (ALS) method [91] is the state-of-the-art algorithm in this class. The reported results to date are limited to cases with six fewer basis functions. The computational cost related m is least O(m)), presenting a challenge in increasing the number of bases m. In contrast, our model scales ~ ((m) + O(m"), where the order of the functional derivative included in the target FDE, r is typically 2, and thus the dependence on m is O (m) or Om²)).
methods
An alternative solving FDEs is the cylindrical approximation 9 In this method, the input function is expanded using a set of basis functions truncated at a finite degree. The cylindrical approximation transforms FDE into sigh-dimmnsional PDE, often solved using discretization- based methods. To address the high computational cost, tensor decomposition methods are also used. The CP-ALS method (91) is the state-of-the-ard algorithm in this class. The reported results to date are limited to cases with six or fewer basis functions. The computational cost related to m is at least (m))) presenting challenge in increasing the number of basis functions m. In contrast, our model scales as O(m"). where r is the order of the functional derivative in the target FDE. Typically, r is or 2.
Below, we provide examples from the fields of turbulence, quantum field theory, and density functional theory.
B.2 Turbulence
Turbulence appears everywhere, from natural systems (e.g., river flows and wind currents) to artificial systems (e.g., water flow in pipes and airflow over airplane surfaces). Understanding its properties is important both in natural sciences and in engineering. However, turbulence a very complex system involving many degrees of freedom, and the only way to theoretically represent the properties of turbulence is through statistical methods. The Hopf equation [4] is an FDE that comprehensively describes the properties of turbulence. For example, the Hopf equation for a fluid following the
20

---

Navier-Stokes equations is written as follows:
0 (+))))
Σ k=1
3
/ (x)
; 2 3=1
3
8 (
2 (0)(
(10)
Øt
dz, (0)((0000,)))))))))))))))),)) (0))0
(x)
Here, (((0), 1) is characteristic functional of test function (ax) (01 (a:), (2(2),, (2))) and is the kinematic viscosity. The characteristic functional contains all the information about the correlations of velocities at different locations at the same time. Indeed, the moments of velocity can be obtained from the derivatives of the characteristic functional as follows:
('))
(us, (I11 t) Wi, (Ins t)) m (-i)" 80,, (201) 50.,, (I e=
(11)
where (2, t) is the ithe component of the velocity at (æ, t). In particular, the Fourier transform of the secend-order cumulant is known as the energy spectrum, which represents the contribution of eddies at various momentum scales to turbulence. The search for comprehensive method to describe the behavior the energy spectrum across a wide range of momentum scales continues, and for this reason, developing methods solve the Hopf equation holds significant importance.
More specifically, let us consider the fluid mechanics aircraft or pipeline flow. For such systems, the second-order functional derivative of the solution of the Hopf characteristic functional gives the two-point correlation function of the velocity field at arbitrary two positions æ and y and an arbitrary time f. The Fourier transform of it w.r.t and y is called the energy spectrum, which indicates which scales of motion are most energetic the fluid flow. The energy spectrum is used to model and predict the behavior of turbulence, e.g., constructing safe and efficient shapes of airplanes or pipelines 69
B.3 Quantum Field Theory
Quantum mechanics tells us that physical quantities in the microscopic world do not always have deterministic values but fluctuate. In quantum field theory (QFT), which is branch of quantum mechanics and forms the basis of modern particle physics, particles are described as fluctuating fields spreading throughout spacetime. QFT allows us to describe the properties of elementary particles in statistical way, i.e., the correlation functions of fields at different points in spacetime. Therefore, developing methods to calculate correlation functions very important for understanding the properties of elementary particles.
Several FDEs provide information the correlation function of fields. A well-known example is the Seawinger-Dysonn equation (71). In the statistical model known as the de model, which is described by the following action
S(()) / dx (50(m) (-v2 m²)
(-V²+m²) o(x) + A 4! }
(12)
the Schwinger-Dyse Dyson equation is given as follows:
8Z((J)) + 82((1) À ((])
(()x) 3! 8)(2)3 iJ (x) 0.
(13)
(xx))
Here, ([J]) is a quantity known as the partition function, and by functionally differentiating this quantity w.r.t. J(x). all correlation functions for the field ((x) can be obtained. Another method describe the correlation functions the functional renormalization group (97 102 72 100]. The
renormalization group is method of analyzing physical systems based on the operation of reducing spacetime resolution. Under such operations, we can define an FDE for the effective action ((Ø)),
which is calculated by the Legendre transformation of In ([J)). [((Ø) contains all the information of the correlation functions, similarly to Z(J]). r(()) satisfies the following FDE [100]:
dgTx((o)) f dx dx' x' Ga)G& ([o) x',
(14)
represents the momentum scale that specifies the resolution at which spacetime is observed, Rx(x)) (x) is function manually provided to realize the operation of the renormalization group, and Gg ([o), x', x) is the regulated propagator defined as
f dx (06()))6((x))
((o) + R5(x 2')) G2(lo), x²) = 6(x - x²). "to"-)).
(15)
21

---

B.4 Density Functional Theory
Density functional theory (DFT) is widely used in material science, quantum chemistry, and nuclear physics. The properties of materials and molecules are determined by the state of electrons, which follows the Schrödinger equation. Solving the Schrödinger equation becomes challenging especially when the material contains many electrons. Hohenberg and Kohn demonstrated that it is possible to determine the state of electrons by solving variational equation for the density (42, instead of solving the Schrödinger equation directly. This is density-based variational equation and has been shown to easier to solve than the Schrödinger equation.
However, in DFT, exact calculations are usually not possible. The reason is that the Energy Density Functional (EDF), which provides the variational equation for density, cannot be precisely determined. Whether good approximation for the EDF can be provided or not significantly influences the success DFT calculations. There has been a lot of research on finding EDFs, including empirical approaches, for a long time. One recent approach is based on FDEs. Specifically, several FDEs are known to describe the evolution of the EDF when the two-body interaction U( x²), e.g., the Coulomb interaction between electrons, gradually increases 73 82) When the interaction is replaced with AU x²), and when À gradually increases, the FDE becomes:
(ml) = 1 / dr f da f da'' ddx'U( x') [m(2, G2([n), x, ",,,,r"))) n1x T)o(ar x²)],
(16)
f dx' f dr' (
(n(x,,))nkl ,") G2([n), x', , ,0,""" ,") 8(x x" x")0(T= - T").
(17)
Here, n(x 7) is the density of electrons, and ([xl) represents effective action, which is an extension of the EDF [3887] In addition to the coordinates æ and x', a virtual dimension, known as the imaginary time T, is introduced. This FDE is expected to provide a new method for constructing the EDF 77 82) For example, the EDF of the three-dimensional electron system is derived from based on the functional Taylor expansion 105
Eqs. (T6
22

---

C Theoretical Background of Cylindrical Approximation and Convergence
The mathematical background of our theoretical contribution is provided make our paper self- contained. This section is based on the recent development of the theory of cylindrical approximation [92 and the classical spectrum theory 41 85]. The differentiability of functionals discussed in App. C.3
showing that the solutions of the FDEs in our experiment are differentiable. The equivalence and difference between the functional, Fréchet, and Gâteaux derivative are summarized in App.
C33 They are equivalent in practical settings, and we do not distinguish them in this paper.
C.1
Continuity and Compactness of Functionals
Let X be Banach space unless otherwise stated. In this paper, functional X defined map F from D(F) C X to R. where D(F) is the domain of F. Note that R cannot be replaced with Q in all the statements below. We first define pointwise continuity, uniform continuity, compactness, and complete continuity of functionals.
Definition C.I (Pointwise continuity of F). A functional F X D(F) + continuous at 0 E D(F) C X if for any Cauchy sequence Jon C D(F),
lim
|10,
0||x = 0 = lim [(((,)) - F((O)) = 0,
(18)
n 10000
n0000
where || ||x is the norm induced by X.
Definition C.2 (Uniform continuity of F). A functional F X D(F) - R is uniformly continuous on D(F) if
VE 0, 26 > 0 s.t. v01,0z 02 E ((F) satisfyingl10, 0211x S 6. [F([).)) F((02)) holds,
(19)
where || ||x is the norm induced by X.
We simply say "continuous" in the following.
Definition C.3 (Compactness of F). A functional F: X 3 D(F) - I is compact on D(F) if F maps any bounded subset D(F) into a pre-compact subset IR.
Recall that A C IR is pre-compact subset if the closure of A, denoted by A, is compact.
Definition C.4 (Complete continuity of F). A functional F X J D(F) + is completely continuous on D(F) if F is continuous and compact (FF).
Based on these concepts, functional derivatives are defined.
C.2 Boundedness, Closedness, Compactness, and Pre-compactness of Metric Space of Functions
Next, we define boundedness, closedness, compactness, and pre-compactness of a metric space of functions.
Definition C.5 (Boundedness metric space of functions). Let X be metric space of functions.
K X bounded if 3M R s.t. V0 K, ||0||x x < M.
Definition C.6 (Closedness of metric space of functions). Let X be metric space of functions. K X closed any convergent sequence in K has a limit K.
Definition C.7 (Compactness of metric space of functions). Let X be metric space of functions. K C X is compact if any open cover of has finite subcover. Equivalently, K is compact if and only if any sequence in K is bounded subsequence whose limit is in K.
Definition C.8 (Pre-compactness of metric space of functions). Let X be metric space of functions. K X pre-compact if its closure K is compact. Equivalently, K pre-compact if and only if any sequence in K has convergent subsequence whose limit is in X.
critical characteristic of pre-compactness given by the following theorem (a necessary and sufficient condition for the pre-compactness of metric spaces).
23

---

Theorem C.9 (Necessary and sufficient condition of pre- cre-paamet of metric space 665 A subset E of a real separable Hilbert space H is pre-compact if and only if E is bounded. (ii) closed. and (iii) for any orthonormal basis {0o: .1..... of H and for any > 0, there exists m E N s.I.
00
E E, Σ 1(0,0)) SE,
(20)
k=
where (c,-) is the inner product defined on H.
Eq. 22 is known as the equi-small tail condition. Thm. C.9 characterizes the domain of functions D(F) to which the cylindrical approximation can be applied. Note that boundedness is not necessary nor sufficient condition for the equi-small tail condition.
C.3 Differentiation of Functionals
Next, we define the Gâteaux differential, Fréchet differential, and functional derivative. Higher-order functional derivatives can be defined similar way (91.
Definition C.10 (Gâteaux differential). functional F open D(F) + is Gâteaux differen- tiable 0 E D(F) if
= lim F( + +n)) F((O) c-
(10)
(21)
exists and is finite for all 17 D(F), where dF ([0) called the Gâteaux differential of F in the direction 1).
There are some patterns differentiability conditions. One of them is:
Theorem C.11 (Gâteaux differentiability of Lipschitz functionals [63 57)). Let Banach space
open
X be separable. Then, any fipschitz F X D(F) R is Gâteaux differentiable outside a Gauss-null set.
Note that Gauss-null set is Borel set A X s.t. A non-degenerate Gaussian X, ((A) is equal to O. In this theorem, there no guarantee for non-Lipschitz functionals, e.g., F((O)) = J dr / [0(]]] where 0(0) 0.
μ
Under mild conditions, the Gâteaux derivative is defined, based the Gâteaux differential.
Theorem C.12 (Gâteaux derivative (86)). If the following two conditions are satisfied, then the
Gâteaux differential ((0)) of functional F X open D(F) + RK at E D(F) in the direction n E ((F) can be represented as linear operator acting on 7) denoted by F' F'((O), S.I.
([0))(())))))))))))(()) ((()))=((())))
(22)
where F'((O): D(F) + R is linear operator, or a linear functional, depending on and is called the Gâteaux derivative of F at 0.
1. (1A) exists in some neighborhood of 0u E D(F) and is continuous w.r.t. at Bu-
2. ((oo)) continuous w.r.f. 1 at 7) 17o, where | [ 0.
Next, we define the second type of differentials, Fréchet differential.
Definition C.13 (Fréchet differential). A functional F X 5 D(F) + IR is Fréchet differentiable
open
E (FF) if dFF ((0) s.t.
F( + en)) F((0) d.F.F ((0)
lim c+
(23)
exists and is finite for all y) E D(F), where dFF ((0) is called the Fréchet differential of F in the direction 7)
There are also some patterns of differentiability conditions. One of them is:

---

Theorem C.14 (Fréchet differentiability of Lipschitz functionals (74). Let K be a compact subset
open
of a Hilbert space H. Then, any locally-Lisscittz functional F K 3 D(F) + R is Fréchet differentiable on dense subset of K.
An example functional that is Gâteaux differentiable but Fréchet non-differentiablee is F(()) where (()) = Ei=o (2) and k, E N.
a
af
Relationship between Gâteaux and Fréchet differential. If F has continuous Gâteaux derivative on D(F), then F is Fréchet differentiable on DFF,, and these two derivatives are equal [86]. The Gâteaux derivative of the aforementioned example, F( a + not continuous, and thus, the Fréchet differentiability is not guaranteed. In the following, we consider functionals F' that are continuously Gâteaux differentiable D(F); therefore, do not care about differentiability and do not distinguish Gâteaux derivatives from Fréchet derivatives. Hereafter, we write d,F, == dF ([0) = d.F, ([0) = )²
Next, we define the third type of derivatives.
Definition C.15 (Functional derivative). The functional derivative of functional F X 5
D(F) - R w.r.t. ((x) is defined as
8F( lim (0(x)
F((O()) + cδ(x - v)1) ((O())))
(24)
c-0
it exists and finite, where (()) is the Dirac delta.
Strietly speaking, this definition may be informal because 0(y) is function, while 8(x y) is distribution. The representation theorem below (Thm. C.17) sometimes regarded as the definition of the functional derivative.
Lem. and the Riesz representation theorem prove the following relation between the Fréchet derivative and the functional derivative.
Lemma C.16 (Compactness of Fréchet derivative). Let K a compact subset of a real separable Hilbert space H. Let F be continuous functional H. If the Fréchet derivative F" F[O[)) exists at E K, then it a compact linear operator.
Theorem C.17 (Representation theorem of Fréchet derivative). Let K be a compact subset of a real separable Hilbert space H. Let F' be a continuous functional on H. If the Fréchet derivative F (() exists E K, then the following unique integral representation of the Fréchet derivative holds:
(8F(O))
Vn H, F((
80 77
(25)
where 8F(0)) 60(x) E H.
The representation theorem C.17 the foundation of the cylindrical approximation of functional derivatives, which shown below.
C.4
Cylindrical Approximation
C.4.1 Functionals
Let H be real separable Hilbert space. The cylindrical approximation is based on the fact that any 0 E can be represented uniquely in terms of an orthonormal basis (do, 02,...) 0(x)
(0, Φh (z)
Thus, we can define
8
f(ag, a1,. (1....):= = F() acde),,
(26)
R=0
where aą : (.. Truncating k m (m E N) gives the cylindrical approximation of functionals:
25

---

Definition C.18 (Cylindrical approximation of functionals 22
Let Pm be the projection operator s.t. Pm0 := _x=0 (0, Φ 11 HR Let Dm be the fiiteedimenniionll space induced by Pm:
7
34
88
i.e. Dm = span (o, 01,..., (0....,0mm111) The cylindrical approximation a functioonl F(O0) on H is
}.
the m-ddimensional multivariable function
f(a: 01....,, 0m-1) F((PmO)) F(
ag oz (x)))
(27)
where ag = (0, Ōk)H. In short, F((0) flau. ..... amm
The cylindrical approximation of functionals is uniform:
Theorem C.19 (Uniform convergence of cylindrical approximation functionals (51). Let K be compact subset of a real separable Hilbert space H. Let F be a continuous functional on H. Then,
YE > 0, 3M E N s.t. Vm 2 M, 40 E K, [
<E
(28)
holds; i.e, F() converges uniformly to F((0) on K.
The convergence rate is given by the mean value theorem of functionals:
Theorem C.20 (Convergence rate of cylindrical approximation of functionals (2]). Let K be compact and convex subset of a real separable Hilbert space H. Let F be continuously differentiable functional on K. Then,
VO E K, [((()) - F([PmO)) < sup | ( 1 0 Pmoll
(29)
[[F'(( is the operator norm of the linear operator F" ((nnl): [[)) = suPo()0)en [F((0))
||ll
The convergence rate of [ Pmol)H) depends on the basis and is provided in [41] (Chapters & 6) and [55 (Sec. 3.5) for several bases.
C.4.2
Functional Derivatives
The cylindrical approximation of functional derivatives is motivated by the representation theorem
C.17
which states that *F(()) H and (ii) F'( (50(z) ...H. Statement (i) means that
60(z)
can be represented in terms of an orthogonal basis as
OF((O))
OF(()
(30)
50(x)
50(x)
Statement (ii) means that
flao.... ak + e....) .)...)- ffao....... ak....)
lim
(31)
Dak
+00
1
lim
F(
0ioi + EΦxh) - F(
a(lil)
C: Eq. 26
(32)
1=0
1=0
F((
ator1) 1o
(:: Eqs. 21
(33)
1=0
(F(
ΦK)H
Eq. 25
(34)
dΦR
where 0(z) = Eizo (1))(2) with a (6, Φ1)
Eqs.
30 and 34
gives
5F FF(())
20
0f dx (x).
(35)
50(x)
dap
kmm
Therefore, truncating & S m (m E N) gives the cylindrical approximation of functional derivatives:
26

---

Definition C.21 (Cylindrical approximation of functional derivatives approximation of functional derivative
91 92 ) The cylindrical
(F(()) H is defined as
m)
m )F((Pm))) 80(x)
0f
(F([Pm)) 50
Σ Φ (x)
Σ
Øk ox(x). H
dak
h=0
If
6F((Pmm 01)
Dm the second
term the left-hand side is equal
88(2)
00
(FF([P 01)
06 (x((x) is called the tail term in this paper.
k=m
68
In short,
(F(() 68(x)
Σk=
1-- 0f dh (x) for SF(() that satisfies the equi-small tail condition. 60(x)
Bog
Note that this version of the cylindrical approximation of functional derivatives is different from ours (Eq. 2 Specifically, in 191 (92. Pm,. is not applied {F(())) /50(z), and the emerging tail term K Kemmi ((F(())) Φu on (xr is simply ignored without any rationale.
The cylindrical approximation of functional derivatives is uniform:
Theorem C.22 (Uniform convergence of cylindrical approximation of functional derivatives 92 Let K be a compact subset of real separable Hilbert space H. Let F be continuously differentiable functional on K. Then,
8F((O)
8F((Pm))
VE > O, 3M E. Vm 2 M, 4A E K,
<6
(36)
50
50
H
holds; i.e. PPm0) converges uniformly 8F(()) 80
80
Note that this theorem proves the uniform convergence, while (Thm. 3.1 proves the pointwise convergence. The difference comes from that the uniform convergence of the tail term, which is assumed in the above theorem, because the tail term is absent in Eq. 2
be violated when our cylindrical approximation Eq. is used
Similarly, the cylindrical approximation of Fréchet derivatives is uniform:
Theorem C.23 (Uniform convergence of cylindrical approximation of Fréchet derivatives 992 Let K be a compact subset of a real separable Hilbert space H. Let be a continuously differentiable functional on K. Then,
Ve 0, 3M IN s.t. Vm 2 M, 40 K, [[F ((Pm))))
(37)
holds; i.e., F" ((P( f)) converges uniformly to (())).).
The convergence rate given by:
Theorem C.24 (Convergence rate of cylindrical approximation of Fréchet derivatives 92 Let K be a compact and convex subset of real separable Hilbert space H. Let F be differentiable functional on K with continuous first- and second-order Fréchet derivatives. Then,
4O E K, |F [[0)) - F' ((PmO))l & sup ||F" - Pm H yEK
(38)
where [FF((() =
sup C.
In terms of a functional derivative, this is rewritten as
8F((O)
8F([Pm O])
8FF((()) 5050
)) |10 PmO0lH
sup CEK
(39)
4P gg* 50.00 x
SPitC (
where F((()
60
8080
sup SuDp.gieH.c4ooo444
8 F(C))
i.c., regard
operator acting H H.
6080
27

---

C.4.3 Pointwise Convergence of Functional Derivatives under Cylindrical Approximation
We prove the pointwise convergence of the functional derivative under the cylindrical approximation. We already noted that the cylindrical approximation of functional derivatives (36) uniformly converges. Below, we show that the convergence becomes pointwise if we omit the second term of the r.h.s. of Eq. (36); i.e., we use
( P m (0)x)
m2-1
0f
2h (I) Dak
(Eq.
as the approximated functional derivative.
Theorem C.25 (Thm. 3.1 Pointwise convergence of cylindrical approximation). Let K be compact subset of a real separable Hilbert space H. Let F be a continuously differentiable functional on K. Then,
8F((D)
SF((Pm))
Northonormal basis (0o, (o,.,.,...., Ve > O, 4A J0EK, K, EMENsL. Vm 2 M,
Pm
<
58
50
(40)
where Pm is the projection onto span @u. pm-17
Now, the convergence becomes pointwise. Technically, this is because the set K' K} } is not guaranteed to satisfy the equi-small tail condition
6F((0)
68
(oF(())
2
orthonormal basis {0o, $1....., Ve' > 0, 3M E E s.t. Vm 2 M, VOEK,
Σ
<
<E
60
k=m
H
(41)
while its boundedness suPsek SF(())
1H do holds according to Thm. C.16
An example that
converges pointwisely but not uniformly is
00 e tk-
(0 00 < 1/2 (oo = π 7/2)
F((O))
0x = (0, Φx)H
(42)
defined on compact subset
K 0 aa (0, (0,,d)))) ax - 2(k + 1) for h: 0, 1,
(43)
Anyways, we have to use large in either case (Eq. (36) or (2) when we want to approximate a complicated functional derivative, and the degree m that is required for a sufficiently small approximation error depends on the smoothness, or spectral tail, of (F((P.,)) As discussed in
60(x))
App. C.4.4, while the lack of uniform convergence may affect the convergence of the cylindrical approximation for linear FDEs in general, this is not problematic in our experiment, We use Eq. as the approximated functional derivative in our experiment.
C.4.4 Abstract Evolution Equations
We first provide related theorems to the convergence equations (consistency) 92 and then those to the convergence of solutions (stability) (25] 36 92].
Definitions. Let F(H) be a Banach space of functionals from real separable Hilbert space H into
R. We consider an abstract evolution equation 36)
aF(()),)
[(1)) with F([0),0) = (o))))
(44)
31
where F is in F(H), and [((0)) is a linear operator in the set of closed, censely-defined and continuous linear operators on F(H) denoted by C(F). Let ((C) denote the domain of operator C. Let Fm F. (H) be the Banach space of functionals on H such that F.. (H) := {F... F... (10), ()
(). in other words, (H) the set of cylindrically approximated functionals. Using F F(H) and F. (H), we define continuous linear operator Bm F(H) e F((), + F(IP 0),t) e
28

---

(H), which represents the cylindrical approximation of functionals. Bm allows us to decompose the right-hamt side of the abstract evolution equation:
Bm (((((()))((0))t = Lm (l)) + Rm ((0), t),
(45)
where (()) is linear operator acting on the m-dimensional multivariable function 35m-1,5 t) = Fm (note that ([O) have nothing to do with t), and Rm ([0, t)
the functional residual.
Convergence of equations (consistency). Now, we fm (m
can
show that ((()))
O(1 Pm.0 (H) 922.
Definition C.26 (Consistency of sequence of operators), A sequence of linear operators {[me] E C(Fm. consistent with a linear operator C E C(F) if E D(C). 3 sequence Fm D (m) s.t. || Fmll _=* 0 and || - -m Fm m= O. If CF - L. Full m=+00, O(m P), {fm is consistent with L to order p(> 0).
Theorem C.27 (Consistency of FDEs under cylindrical approximation 92 Let H be a real
separable Hilbert space. Let F E F(H) and L E L(F). If ((()) is continuous in 6, then the sequence of operators {Cm} J is consistent with L on arbitrary compact subset K of H, provided that 40 E K. [mm (10)) == O.
Corollary C.28 (Convergence of cylindrical approximation of FDEs (22 ). Let H be a real separable
Hilbert space. Let F E F(H) and C E L(F). If ((()))((O)) is continuous in 0, then the sequence of operators {Lm} is consistent with to the same order as ||0 P' 0 и on arbitrary
compact, convex subset K of H. provided that 40 E K. (()) O mn 0|H0) as + oo and that ((()))((()) is continuously Fréchet differentiable in K. In short, [(())
f. (()) ([0)) O p (m))
Convergence of solutions (stability). Next, we show that Fm ( - F((O).) t) if and only if the cylindrical approximation stable and consistent. Let us consider the approximated abstract evolution equation ((0).t = Lm ([0)) Fm ([A), t) with F. F..(()) ([0) 0) (((o)) is said to be St
consistent with the original abstract evolution equation if Thm. C.27 holds.
Definition C.29 (Stability of approximated equation). Suppose that L... of the approximated abstract evolution equation generates a strongly continuous semigroup e'Cm. The approximated abstract evolution equation is stable if E,, w s.t. | || S Me where M and w are independent of m.
Theorem C.30 (Convergence of solutions under cylindrical approximation (25 36 921). Let K
be a compact subset of a real separable Hilberi spuce H. Suppose that the approximated abstract evolution equation well-pose well-posed, in the sense of an initial value problem, in the time interval [0, T] with a finite T. Suppose also that C((6]) E C(F) generates strongly continuous semigroup in [0, Then, the approximated abstract evolution equation is stable and consistent in K if and only if max max Fm (10), t) -F([O,,)) m=+30, O, provided that Fm [10, 0) m-+00, (o(()). In short, tEJU,100EK ((),(()))) 1) + F(()).) 1) if and only if the cylindrical approximation is stable and consistent.
For example, the cylindrical approximation of the following initial value problem is stable and consistent 92
a 6F((9).t) dx with F( 0) Fo(()). To our knowledge, the (((()))) = - (xx) ax 6B(ar)
St
convergence rate has been unknown so far.
Remark 1. Most of the approximation results for compact subsets of real separable Hilbert spaces hold also in compact subsets of Banach spaces admitting basis. We refer the readers to Sec. 8 in 922
Remark 2.
Finally, we comment on how the lack of uniform convergence of (F((0) P,, SF( )) 69
affects the cylindrical approximation of linear FDEs. The difference 6F( P... 6F((P.)) 86
is manifested in the functional residual ([)) t) in Eq. (45) The lack of uniform convergence may have a negative effect on the consistency of the cylindrical approximation, given that the convergence of R,, (10).t) t) is required in Thm. C.27 This issue, however, is circumvented in many cases. In fact, let us consider the scenario where functional derivatives in an FDE are expressed in terms of the inner-product (u, sF([O]] H, which is satisfied by our examples (u u in the FTE and Aa
32
29

---

in Eq. 197 (see App.
Importantly, its cylindrical approximation (u, P.. 5F( 91) oe
uniformly
converge to (v, 80
(
H even Pm (),().,))) does not uniformly converge to 66
(F((0)
69
Lemma C.31 (Uniform convergence of inner products (ours)). Let K and K' be compact subsets of
real separable Hilbert space H. Lel F be a continuously differentiable funcciional K. Then,
OF((O))
-(80, P
( 0)
Vc O, 3M EMENAR s.t. Vm M, A0 E K, 4O' EK',
8'
80
80
(46)
holds; ie., (0', Pm
(F((P.))) 68 H
converges uniformly to 0r
(F(()) 88
on and K'.
H
The proof is given in App. FDEs.
D.I In App.
B
employ this lemma show the consisteney of our
In short, Lem. C.31 states that the cylindrical approximation of inner products (u, Pm 5F((P 91)
uniformly converge to (v, <F((d)) н even if Pm (F(((mm)) does not uniformly converge to 60
(F(())
40
Because of this mechanism, App. we show that the uniform convergence of Rm is ensured,
which is one of the assumptions for stability. The full proof of the convergence of solutions is provided in App. E
30

---

D Proofs
D.I Theorem: Pointwise Convergence of Functional Derivatives under Cylindrical Approximation
Theorem D.1 (Thm. 3.1 Pointwise convergence of cylindrical approximation). Let K be compact subset of a real separable Hilbert space Let F be a continuously differentiable functional on K. Then,
orthonormal basis {Øu, .1....): VE > O, AO K, 3M EN (F((O)
)F((Pm))
s.t. vm 2 M,
Pm
<,
(47)
60
80
where Pm is the projection onto span do. .... mm-
Proof.
The triangle inequality gives
OF((O)
8F([Pm0]
8F((O)
8F((Pm))
8F((0))
Pm
I
P
+
(1
Pm)
50
80
50
59
80
H
11
(F(())
(F((Pm))
8F
8F((O))
9
(48)
50
80
80
The first term on the right-hand side converges to zero uniformly according to Thm. C.22 As for the second term, we first note that 6F((0)) is bounded on K in the sense of function in H, according to
(0(x)
Lem. C.16 This, together with Parseval's identity, implies || ( 11 k= K( (F((O))) $x)
00. Therefore, the sequence of the partial sums Sm = m 1( sequence and thus is Cauchy sequence; i.e.
(F(())
Φx H 1 is convergent
60
Vé' > 0, EM E N, s.t. Vm, M, Sm - Snl < <
(49)
By taking n - oc, can claim that
Y orthonormal basis {0o, ..... Ve' > O, E E K,
EMEN
5F((O))
s.L. vm 2 M,
<é.
(50)
88
F
Therefore, the second term the side of Ineq. was thus proved.
488
converges pointwisely. The theorem
Convergence rate. The convergence rates the approximated functional derivative can be derived from Ineq. (48) The first term on the r.h.s. converges at the same rate as |10 P... (Eq. (39) ). The
convergence rate of |00 @l depends on the basis functions and provided 41 (Chapters 6) and 85 (Sec. 3.5) for several bases. The convergence rate of the second term on the r.h.s. depends
on the compact subset functions K H under consideration, and further assumptions on K are required.
Lemma D.2 (Lem. C.31 Uniform convergence of inner products). Let K and K' be compact subsets of a real separable Hilbert space H. Let F be a continuously differentiable functional on K. Then,
Vc 0, EM EN s.t. Vm 2 M, 4Q E K, 4O EK )F((Pm))
0 )F(())
8 P. 80
<c
(51)
80
holds; i.e., (0), P. SF(P...))
converges uniformly to (0r, FF(
K and K'.
58
H
59
H
31

---

Proof. Using the triangle inequality and the Cauchy-Schwarc inequality, we have
(F(
(F((Pm)])
6'
0
Pm
80
50
H
H
8F((0))
(
#1)
8F((8))
0', Pm
+
0' 0,, Pm.)
50
50
50
H
OF(()))
SF((Pm))
0))
(1 )6' 8F((())
|00 II
Pm
+
50
80
60
(F(()))
(F 0))
)F((8)
110'1
100 (b) Hl
(52)
50
80
88
Note that
SP((O)
and 0' are bounded on K and K', respectively; thus, C and C" s.t.
80
suPack
5F([0))
S C < Do and SUP 110'l I C oc. Therefore, Ineq. 52 gives
(0: OF((()
(0', Pm
(F((Pm)))
(F((O)
SF((Pm))
C
+
100', 0k)
80
50
80
50
Rmm
(53)
Finally, according to Thm.
C.9 the compactness of K" means
VV' 0, ME N s.t. V0' K', Vm M,
100 dx))) <E.
(54)
k=m
This, together with Ineq. 53 and Thm. C.22 proves the lemma.
Tail term. In the cylindrical approximation Ekem (5F(()) (dx(()) to zero, unlike the cylindrical approximation (36) adopted in 191 92
Pm projects the "tail
term"
The tail term vanishes if one considers an inner product of functional derivative and Dm-
However, is not always the case that functional derivatives appear in the form of (F((8) .) with
58
E Dm in FDEs. In fact, in the FTE, the functional derivative appears as an inner product with u(x),, which can be chosen arbitrarily. The point is that Lem. C.31 which plays a fundamental role in proving the convergence of approximated solutions, guarantees the convergence of the inner product (()) v) in wider variety of situations including D In other words, theorems extend the class of FDEs whose uniform convergence of the approximated solution is theoretically guaranteed.
D.2 Derivation of Exact Solution of Burgers-Hopl Equation
D.2.1 Derivation Eq. 93
We show the derivation of Eq.
93 is based on the functional Taylor expansion
w([0),
dEi
de,
1...s.......))(().--e.... O(En).
(55)
This turns Eq. 92 as follows:
02
02
w (1) (I,,,....,
+ +
(T.....6..........
(56)
057
DE2
In the momentum space, this is written in the following form
① wr(n) (T, lk.....,kyy = (kz + + k (i(n) (T,1,,, .i,...,,
(57)
0r
(T, En) E..
Σ
sk + in (T, kn);
(58)
kn
32

---

where k1...... = 2011 a with 11..... n E Z. The solution of this equation is
W (T, ..,..., kn) e -4.+....43 Wh() (0, .i,..., .......,,)))).
(59)
Using this result, Eq. 55
gives
((
W(() (T, k1 ki,..,, kn) k:)) ((nkin)
n
==
k1
kn
n
dEp
den w(() (0, I1. F1.,,,..,, II
k2th 7 Ti 260 ((-km)))
71!
km
1E
den (()) (0, En)
II
de (5
Σ
Nyy tri Emn)
m=0
m=1
Rim
den ((n) ,1. Sn
II
dg (
Σ
T +2 (5m
(60)
where
6(k) / dge the O(S)
(61)
The Poisson's summation formula transforms the summation w.r.t. lmn
dye 2xiqu
+ -in)
47 (5m-tin- -0)²
(62)
ATT
Plugging this into Eq. (60), we arrive at the following expression:
w((0),r)= = Σ { de
/ den Wi,,wo
En)
II
(1/
de
O
Σ
e
(sm
+
5m
-9)
AnT
n=
m=1
g
(63)
From this result, we conclude that the solution is given by
W([O), r) = (o((e-(()))),
(-((0),5)
Σ
de'
+ (5-5' -e()))))))))))))))
(Eq. 93
ATT
9=-00
D.2.2 Derivation of Eq. 104
We show the derivation of Eq. 104). The cylindrical approximation of Eq.
93
is represented by
1/2
W W[P22. (o(le,((P2M
(.((
Σ
dg' + (8-6'-0)² P())
/AAT
1/2
q=-oc
The basis (03) satisfies
(k: = 0)
(x + E')
si +)(E+ /2 cos((Tk(E + +())
(k: odd)
(k: nonzero even)
1
(k = 0)
V2 sin + 1)5) cos(T + 1)(') + /2 cos( + 1)5) sin(= + 1)E')
(k odd)
V2 cos cos()) (inkn( sin (TkE')
(k nonzero even)
cos(T - 1)5)) sin 1) (E)+ cos(Tkg")ou(e sin(
:dd
(64)
(k :
33

---

Using this relation, we evaluate θ (r((2v) as (,((2))))
1
1/2
dg'e
(5'-)))²
2M-1
apoi(5' +
VAT
1/2
M
de
coos (2 (02z (22x2T7())
4TT
k=
g/a
de'e
[cos + 1) (d2=)=12 sin 1)22))2++2(())) 02k+1
4TT
M
4x
Φ22 (E)azs te -x= 22 (5)a2x+)
(65)
k=
where ag (0x, θ) (2((-1/2.1/3]
When the initial condition is given by Eq. 102
1/2
1 1/2
1/2
((PP2M)),
(E)
dg
(FC05:51000(21000000000010000 J-1/2
2
11/2
2
Taa +
1
M-1M-1
Σ Σ
1/2
dp
1/2
de' ((E((E)))
2
-1/2
k=0
1=0
1/2
(22+1 (e - )2((5))a2 +e 4=7 12
(2)(E)a2
M-1M-1
Σ
(12
C2 22 (x 12 (2,
=- - +
2 k= 1=0
+e 4n ((k+1) C20+1,2102211 + (0++))+ 010 (Eq. (104))
where we have introduced
1/2
1/2
Čk
dg dC(C(s,,)))(())(()))
-1/22
-1/2
34

---

E Proofs II: Details of Functional Transport Equation and Burgers-Hopf Equation
In this Appendix, we provide the detailed background of the FTE and BHE and prove Thm. 3.2
i.., the convergence of solutions under the cylindrical approximation 2 The main materials
Lem. C.31 and Thm. C.30 Technical assumptions are summarized in Thm. C.30
E.1 Functional Transport Equation
y
A
S
y = g(x)
u(x)
X
B
Figure 8: Two-dimensional fluid. Let u=x) (0, =(x)) denote velocity field. The surface S is defined by = g(z), which separates the space into two areas, A and B.
We introduce the functional transport equation (FTE), generalization of the transport equation
(continuity equation) [53]. We consider two-dimensional fluid system in z-y coordinates (Fig (8. We assume that the velocity field is given by u(x) = (0, u(x)) The surface S, defined by = g(x), separates the space into two distinct areas, g(x) (area A) and y g(x) (area B).
We calculate the amount of fluid entering the area A from per unit time:
F(gg).t) - ff dx (()) g(()))
(66)
where p(r, y: 1) denotes the fluid density at (xr, y) and at time I. Differentiating both sides w.r.t. and using the equationn for the fluid density
2
or p(z, v) u((x) By
plz, v.t),
(67)
have
0
F((gl.)) = / dx u(x) 1F(()),
(68)
Ot
5g(x)
We refer to this linear FDE as the FTE.
E.1.1
Analytic Solution
Let (1-1,,,1) be real Hilbert space of functions on [-1, 1] with the inner product
(f. (f:)) == S" dx dzf())g(z),
(69)
35

---

where f.ge E 1)) Let ((1)((-1,1)) be Banach space of functionals that map (2-1,1,
R. The functional transport equation
0 ((()).) = [ da ((r
8F((g).0) F((s).0) = Fo((s)): 5g(z)
(70)
Ot
a linear FDE F(L2((-1,1)), where Fo is given functional. Here, u E (3 1J) is assumed to have finite norm [[-I))((-)1.)) < co.
The solution of this equation is given by
F((g). () Foloo), 9e(x) := g(x) u(x)),,
(71)
as can be seen immediately by substituting this into Eq. (70).
E.1.2 Cylindrical Approximation and Convergence
We prove that the solution of the approximated equation converges to that of the non-approximated original equation. Let {do, $1....} be an orthonormal basis of (12 and Pm. the projection onto span (on .... Pm-1)- Following App. C.4.4. we apply Bm on both sides and the initial condition, to obtain
0 Ōt sa.
-- km0
Σ
S(a + Rm)()), f(a.0) So(a),
(72)
Dax
where we have introduced
faa ((a.))==((PM))..)) F((Pmg)))) 1), () Po((Pmgl): ax == (Øx. (0x.9)2((-11))))) == (Øu. u) (2([-111))
(73)
The residual functional Rm is defined
+ u, u, 6F((Pm9).t)
Rm (1g), == 24 The cylindrical approximation of Eq.
(74)
8g
5g
121(-1,1)
12([-1,1))
70
is given by
m
i uk Dak
Đr f(a,t)
f(a.t), f(a.0)= fo(a).
(75)
R=0
We first prove the consistency of Eq. 75
According to Lem. C.31
Rm (gg)) t) uniformly converges
to zero on an arbitrary compact subset K ( 1J) if F F(g)). 2) a continuously differentiable
functional on K. Thus, we can use Thm. C.27 which proves the consistency of the cylindrical
approximation 775).
We next prove the stability (Def. C.29 of the cylindrical approximation (7) in the - norm. Suppose that fo(a) is bounded by constant independent of m From the cylindrical approximation of the solution Eq. (7I). [(a, () Pm (g(x) u(z)()), we see
sup lf(a = sup
- u(z)t)]] sup Ab(Pmg())) = sup [f((a)].
(76)
Therefore, we obtain
[I (Rm)) (= ||e.cm (= = || (. S S, Vm E N,
(77)
'In this case, K' Lem. C.31 is set to a singleton K' {u). Obviously, this is compact because ( po.
"Strictly speaking, if as are defined on an infinite interval, this assumption boundedness not valid the FDEs used in our experiments because || p + 00 as aд + oc. However, the range au is usually set to finite interval in numerical experiments, and thus 2 is also finite: i.e., the assumption of boundedness holds.
36

---

for the operator 2. = -Kk=0 0ux"
a.
From this result, the operator norm of et is given by
||eLE.n fo fo11
I[(())
||e sup
sup
(78)
foro
I1
fay0
[ (Rm))
This shows that 3M, w s.t. Vm E N, ||eeEm < Me (M- = 1 and w 0), and thus the cylindrical approximation is stable the sense Def.
C.29
According to Thm. C.27 the fact that the cylindrical approximation
75 is consistent and stable guarantees the convergence F([Pmg).t) ==-00, F( on a compact subset in finite time interval [0, 7' if F[[Pm.)) 0) -0 [[o)), 0) (this is satisfied by the initial conditions used in our experiment) and if the approximated functional transport equation well-posed [0, 7].
E.1.3 Initial Conditions
We define the initial condition F(g)).) 0) in Eq. 770p as
F F(lg). 0) Pa [ dz uz))()(()).
(79)
The exact solution is given by
(( =po [" du uu())) -(()))))(()
(80)
Under the cylindrical approximation, the solution is
F([Pmg))tt f(a, P un(an -ut))
(81)
With up being constant and f being the Legendre polynomials, consider two types of u(z):
vo for k = 1, otherwise 0,
(82)
0g Up for k < 14, otherwise
(83)
corresponding to two types functional transport equations (70), initial conditions 79). and solutions (8). For convenience, call them the linear initial condition and the nonlinear initial condition, respectively, though the form of Eq. (70) also changes in accordance with the form of u(z).
E.2
Burgers-Hopf Equation
The BHE, the one- mee-dimnsiooal analog of Eq. (1O) in App. B.2, describes the statistical properties of mie-dimensional fluids and is basic tool for studying turbulence, as mentioned in App.|B.. We consider fluid in one-dimensional box [-L/2, L/2] that evolves in accordance with the Burgers equation
Qu(z, t) Øl
@u(x,t) t) Or
02 Pulax, t)
+ u(xz
(84)
Ox.2
where u(x, t) is velocity field and v is the kinematic viscosity. Specifically, we focus on the case where the initial value u(z, is given randomly, thus making u(x, f) a random field. Let introduce the characteristic functional
[ dx u(x 1)((z))
1/22
Φ ([0)). (M(
E
exp $
(85)
where Po is the probability distribution for the initial velocity field u(z 0), and 0(x) is called the test function. It provides statistical properties of the velocity field because the functional derivatives are equal the moments of the velocity field:
8 (10)
[u(21,t) .- u(xn, t)] = (-i)" 50(x1) ... 60(x)) 8=0
(86)
[~Po
37

---

as is already shown in Eq.
11 in App.
B.2
The time evolution of ( t) is known to follow the
BHE 91:
(0),))
[ dx (x:)
L/2
i a (00(10)))
8 (0(()).)
+ v
(87)
Ot
-L/2
2 dr ())))((())
Or.2 50(x)
The initial condition ([0), is determined by Po. For example, in the case the Gaussian random field, we have
((0)) = exp (1 ( dxp(2)0(x)) L/2
L/2
- L/2
L/2
L/2
dr dx' 2(
(88)
-L/
where μ((z) is the mean velocity and C(x,x') is the dimfesiodimmessionlll covariance matrix.
We modify Eqs. (87) and (88 to facilitate numerical computation. First, we replace (i0(x) with ((x)
and regard (zr as real-valued function to avoid complex numbers. Second, we make Eqs. (87) and (88) dimensionless, a common convention in numerical computation, by introducing
& T= T:= T
I?2
I,2 - + = T O(LE).
(89)
Third, we introduce
W ((()),)) = (().
(90)
to remove the exponential function in ((0). and stabilize numerical computation. Note that functional derivatives of W(O)),7 7) give the cumulants, not moments, of the velocity field. Fourth, we neglect the advection term (the first term on the right-hand side of Eq. (87). ). We derive the analytic solution for the equation in the following.
E.2.1 Analytic Solution
Let us consider [ a real Hilbert space of periodic functions on [-1/2, 1/2] with the inner product
(f. (f00)2((-1/21//))) = /" dz)())g(z)
1/2
(91)
where /.g E [3((-1/2, Let F(12(((112211/2) be Banach space of functionals that map 1/((-1/2)1/2 to R. In F(L ( 1/2,1))),, the BHE without the advection term is given by
aw (W(()),) ÔT
8 (W),))
(E
W(O)), (o
(92)
0g
80(E)
The exact solution of this equation is given by
w([O),r) r) = (l()))-(())))
(,([)),5) = /ATT
Σ g=-00 -
de'o + (s-e' -())))))))))
(93)
where Wo is given functional. The proof is bit technical and is given in App.
D.2.1
E.2.2 Cylindrical Approximation and Convergence
In this section, we prove the convergence of the solution for the BHE under the cylindrical approxi- mation. Let (do. $1.... ...} be orthonormal basis of ( 1/2)) and Pm be the projection onto span (do. ..., 1m. Following App. C.4.4 C.4.4 we apply Bn, both sides and the initial condition, to obtain
a w(a, T)
Σ k.
m-1
Anaa da
w(a, r) + Ru, ( r), w(a, 0) wo(a),
(94)
0r
38

---

where we have introduced
w(a, (((Pm.)),7) wo(a, 7) Wo([Pm))), ap -Ox,, 1)(((-1/2111))))
Aui = (p: O [2((-1/2,1/2)
(95)
The residual functional is defined as Rm ([+).)) = (0 oW([+).7)
180
8W (WPmm.))
0 Pm
0E2 60
80
12(1-1/2,1/2)
- 1/2.1/211
(96)
The cylindrical approximation of Eq.
92 is defined
θr w a. T)
Au da
w(a,T), w(a, 0) = wo(a).
(97)
k 1,1=0
We first prove the consistency of Eq. 997
Let K be a compact subset in 42(111/2,1/2) such that E K} is compact. According Lem. C.31 Pm (W],7) r) uniformly converges zero on K ([O), is a continuously differentiable functional on K. Thus, we can use Thm. C.27 which
@8 o5
proves the consistency of Eq.
We next prove the stability (Def. C.29 of Eq. (97) the L (K") norm. Suppose that wa(a) is
bounded by constant independent of m From the cylindrical approximation of the solution
Eq. 711 w(a, t) [o (Pm @]), we see
sup [w(a = sup mcRm
sup [W]([Pm < sup [Wo([Pm = sup [wo (()(a)) a
m
aes,
agk
(98)
where
Sy a dj
m-1
Σ
0C
1/2
df
1/2
de'e- (e-6'-a)) (((()))(()) +(==E-0)(((())(()))
as. a E m
k=0
V
-x-
11/2
-1/2
C
(99)
Therefore, we obtain
[|w(T)ll==(mm) = 11e w0 1.0 1m (Rm)
S |l || S c, Vm E N,
(100)
for the operator fm Ex. Agaa Ta:
a_
From this result, the operator norm of elCm evaluated
|ellC wollsw < 1.
|
sup
(101)
|1 l1 (R
This shows that 3M, w s.t. Vm E N, et approximation is stable in the sense of Def.
Me (e.g., M and 3 = 0), and thus the cylindrical
C.29
Therefore, the cylindrical approximation
97
is consistent and stable, and Thm. C.30 gives the
convergence ((Pm.)),, 7), m= W([))
on a compact subset in finite time interval [0, T]
W(P,,, 1=-+00 W(O)),0) 0) and if Eq. 97 is well-posed in [0, T7.
"Strictly speaking, if as are defined on an infinite interval, this assumption boundedness not valid the FDEs used in our experiments because [|wwlllz>> (2m) - 20 as a + co. However, the range of ou is usually set to finite interval in numerical experiments, and thus ||wlll (Rm) is also finite; i.e., the assumption of boundedness holds.
39

---

E.2.3 Initial Conditions
We use the Gaussian random field as the initial condition:
[ [ 1/2
de S 2
1/2 1/2
Wo( = - -1/2
dE -
dE(C(E.)))0()))())
(102)
where z is the mean velocity and C(E, <") is the covariance matrix. This is equivalent to the initial distribution of the velocity field following the Gaussian distribution (102). We use the Fourier series the orthonormal basis:
{
1
(k 0)
(x
(inin 1)5)
(k: odd)
(103)
(Tkk)
(k: nonzero even)
Then, the analytic solution of Eq.
(93) under the cylindrical approximation given by
W((P2M
1 M- M-1
pao + Σ km0
( - (12 (0 22.22110222111
te 4n ((++1)++++))))) 0228112002841 + - (x+t)) 02410)00000011111100000000
(104)
where 2M and
1/2
1/2
ag (Øx, 1330-1/22//) Czj
dg
dECCIE.E)oi()))(()) -1/2
(105)
-1/2
The derivation is lengthy and is given in App. [.2.2] exponentially in terms of k. 1, and r, and the solution is dominated by with l 1.
Note that the higher degree terms decay
We three types of covariance matrices:
C(E,E") a² 825 -(),
(106)
C(E,E") ṭ² k/10 Φn (E)
Σ
1/10 o)
(107)
==
1=0
C(g,E") o².
(108)
Substituting them into Eq. constant initial condition, respectively. They are equivalent to
[02). have three types of initial conditions: the delta, moderate, and
Ög = o² for all i 2 0 (0 otherwise),
(109)
Ö₁ 2e 1/110 1/10 for S 99 (0 otherwise),
(110)
Ö2 = の² for i.j = 0 (0 otherwise).
(111)
Eq. (06) is nonsmooth and represents the extremely short-range correlation of the initial velocity field; the velocities at two points an infinitesimally small neighborhood have no correlation. The spectrum of C(E, E') (Eq. (109) has infinite tail. Eq. (08) represents the extremely long-range correlation of the initial velocity; the velocities at any two points have the same correlation a². The spectrum of C(E,E) E') (Eq. (IIT)) decays immediately. Eq. (I07) represents a moderate-range correlation of the two above: the velocities at two points have periodic correlation. The spectrum of C(E,E") (Eq. (110)) decays exponentially. Theoretically. it is said that C(E, E') that has a long tail of spectrum is hard to simulate numerically [9I]. In our experiment with PINNs, however, the error of the solution dominated by the optimization error of PINNs (Sec. and strongly depends on the training setups.
40

---

F Supplementary Discussion
F.1 Limitations
Higher spacetime dimensions. In our experiments, the spacetime dimension is limited to 1 + (t and z). Generalization to 1 + d dimensions is feasible, albeit with increased computational costs. For spaces with d > 1 dimensional spaces, multiple options exist for expanding 0 [79]. Despite the inclusion of additional spacetime dimensions, the computational complexity of our model, up to computing functional derivatives, remains O(m"), where is the order of the FDE. However, note that m scales exponentially w.r.t. d, which is typically 1, 2, 3, or
Higher-order functional derivatives. The order of functional derivatives in FDEs in our ex- periments is limited to = 1; however, extending to > 2 is straightforward. For ex- ample, the cylindrical approximation of the second-order functional derivative is expressed as 8 t x = (a which can be computed via back- propagation twice.
To further reduce errors. The relative errors obtained in our experiments are ~ 10-3, which might be unsatisfactory in some applications This is general problem inherent in PINNs, for which PINNs are sometimes criticized. To further reduce the errors, one can use recently developed techniques for solving high-dimensional PINNs [106] 44
These methods be easily equipped with our model; one of the strengths of our model is that it can be integrated with arbitrary techniques developed for PINNs. See also App. F. for better sampling of collocation points.
96 45
21 103 3X]
Challenges toward even higher degrees. Instability of the numerical integration for computing as given a function ((x)), i.e., 0₁ = = from the intense oscillation of Φb for large k Z 500, as we confirmed in our preliminary experiments. This is a general problem in numerical computation, not exclusive to our model. Nevertheless, such higher degrees are unnecessary to approximate smooth functions. Extremely large degrees are required only when one wants to include nonsmooth or divergent functions the domain of functionals D(F). Training on such functions with the cylindrical approximation is open problem Dealing with highly oscillatory functions is also central research interest in numerical methods for non-smooth dynamical systems [I]
observed for large ks. This instability comes
Inclusion and diversity of functions What class of functions can the cylindrical approximation represent? The equi-small tail condition (Thm. C.9) characterizes the domain of functions (EF) to
which the cylindrical approximation can be applied. In most of the convergence theorems in App. C
such as Thm. C.19 the compactness of K C H is assumed, which can be ensured if Thm. C.9 holds. In other words, the squi-small tail condition is part of a sufficient condition for the convergence of
the cylindrical approximation. Surprisingly or not, step functions and ReLU satisfy the equi-small tail condition. In contrast, functions with divergent norms (10),r co), which are not typically of interest, cannot handled in the cylindrical approximation.
Inclusion and diversity of functionals We assume functional differentiability in this paper. Central theorems of differentiability are given in App. example non-Lipschitz functionals F((0)
C.3 where functional Lipschitzness assumed. An dar / with 0(0) = O.
Inclusion and diversity of FDEs In this paper, we consider the abstract evolution equation, crucial class of linear FDEs. It does not include. for instance, Eq. (3) in quantum field theory or (1) in functional renormalization group theory. Nevertheless, the cylindrical approximation applicable to these equations, although the theoretical (non- Jconvergence of solutions is currently unknown. Finally, note that the approximated functional derivatives converge to the non-approximated ones independently of the specific FDEs.
F.2 Computational Complexity of CP-ALS
In Sec. T we mentioned the computational complexity of the state-of-the-art method (CP-ALS) (O TT) We derived it from Eqs. (567) and (568) in 191] in Sec. 7.2.2 in [91]. These equations
41

---

require ((m²) including the summation over the indices (= O,..., m²) and 2 (= 0.... m²), and the Hadamard matrix product over the index i (= ...... m). They are computed for each (=1,.. m) and timestep / (= 1, ..... 7), leading to Om x x T) ( in total. Note that Eqs. (567) and (568) in the published version [91] correspond to Eqs. (568) and (569) in the arXiv
....
version (arXiv: 05250v3).
E3 Choice of Bases
We use the Fourier series for periodic systems and the Legendre polynomials for non-periodic systems. Both are common basis functions in numerical computation. Legendre polynomials part of the Jacobi polynomials family, a typical class of orthogonal bases. Many Jacobi polynomials are numerically unstable and require careful treatment; for example, the Hermite polynomials change scales factorially with degree (1 \/mm)) and often result in nan. The Laguerre polynomials have similar issues. The Chebyshev polynomials of the first kind and the Legendre polynomials are suitable for numerical computation; they bounded, defined
finite interval, and their weight functions regular (note that the weight function of the Chebyshev polynomials of the first kind, 1/v1 x², can be easily regularized by changing variables). Moreover, the Chebyshev and Legendre polynomials known to minimize the L and L approximation errors, respectively [41]. Nevertheless, did not observe a significant performance difference between them in curve-fitting experiments, and we the Legendre polynomials in our experiments.
How to choose basis functions? The choice depends on the FDE, boundary conditions, symmetry, function spaces, and numerical stability. The Fourier series is suitable for periodic functions, but for non-periodic functions, it exhibits the Gibbs phenomenon, causing large numerical errors. Therefore, Legendre polynomials are good choice for non-pen non-peeio functions. The choice of bases a common concern in numerical analysis, such as in the finite element and spectral methods.
Generalization to other bases: Riesz basis
Our approach can be generalized to general non-
52 51 15). where the orthonormality condition (6k, ΦI) - = бдt is replaced with (Øx, Φ1) и = Ist- Here, Iki is a certain matrix, or a metric,
orthonormal bases, such Riesz bases 33
76
14
30
whose choice significantly affects the computational costs of the approximated FDEs. Sparse metrics are preferred for efficient computation. See also App. F.4
E4 Computational Complexity Revisited
We showed the computational complexity of our model up to the computation of functional derivatives is given by Omm"). ) This does not include the computational complexity of the loss function of PINNs. It could be > O(m"), strongly depending on the form of the FDE and/or the choice of the orthonormal basis. For concreteness, let us consider Eq. (97) (BHE). While the BHE is first-order, the computational complexity of the loss function can be (m?) matrix Agi in Eq. (97) is not diagonal. This is the case if we use the Legendre polynomials instead of the Fourier series.
On the other hand, the choice of the basis function can improve computational complexity. For example, the computational complexity of the second-order FDE can reduce from O(m²) to ((m²) when the approximated FDE includes only the diagonal elements of (0? f(a t) 10/0aa0a 1m-1 3 k Again, whether it is the case or not depends on the form of the FDE and/or the choice of the orthonormal basis
E,5 Comparison with DFT
In first-principles computations of density functional theory (DFT), an NN-based approach that utilizes finite element methods for spacetime grid approximation commonly employed. For example, an NN, F((y; (r,)),), approximating a target functional F ((f)) by evaluating f at specific grid points {r,}y. Functional derivatives at each grid point can be computed using automatic
differentiation: [ }r = { (f(1))))) }.. However, the central focus of this area does not include solving PDEs, let alone FDEs. Moreover, this method requires the discretization of spacetime, leading to numerical error of derivatives, while in our approach, such discretization is not necessary, and spacetime differentiation be performed analytically if the basis functions are analytic.
42

---

The finite element methods for spacetime and the cylindrical approximation (basis function expansion) are similar the sense that they replace spacetime degrees of freedom with other quantities: spacetime grid points and expansion coefficients, respectively. Even when their degrees of freedom for a given problem are comparable, the latter (our approach) suitable for solving FDEs.
F.6
FDE Has Multiple Meanings
The tern "Functional Differential Equation" (FDE) varies in meaning across research areas. In mathematics, an FDE typically refers to differential equation with deviating argument, e-g.. fx) zf(x + 3). In physics. it denotes PDE involving functional derivatives, such as the abstract evolution equation described in the main text. Additionally, FDE can imply functional equation involving functional derivatives, e.g., 68(z)
5F((0) A(x) F(()). where A(x) is given function.
Research on the last interpretation is limited, and the uniqueness, existence, and practical applications remain ambiguous.
F.7 Radial Gaussian Sampling
The region 0 ~ O, or ~ O2 is the central focus when solving the BHE. Thus, sampling as many collocation points from 0 ~ possible is of crucial importance. To make the training free from the curse of dimensionality, we can use the following sampling method based on the polar coordinates:
1. Sample unit vector v E RR, where d is the dimension of the input space.
2. Sample a real number (radius) r > 0 from Gaussian distribution (00, o²) (or a truncated Gaussian distribution).
3. Obtain training collocation point r'v E Rd.
The collocation points thus obtained obviously concentrate around ~ more than the Latin hypercube sampling, which used in experiment.
F88 Curvilinear Patterns in Fig. 4 and Noisy Patterns in Fig. and
The white curves Fig. 4 represent locations where the predictions and the exact solutions happen to coincide. The noisy patterns that occur for ax. k 2 in Fig. a because the solution is almost independent of ag with ž 1. Thus, optimizing the model in the directions of αz with À / have only negligible effect minimizing the loss function, keeping the random predictions of the model. we delve deeper, they might be related to the loss landscapes of PINNs S 35). which is also an interesting research topic.
F.9
Second-order Functional Derivatives
Fig. shows the estimated second-order derivative of the FTE's solution at 0 = 0 and under the linear initial condition. The error is small as 10-3 despite that 0 ~ 0 is not included in the training set due to curse of dimensionality and that the second-order information not included in the loss function.
However, such small errors only emerge when the analytic solution is so simple that, e.g., it includes only one coefficient: fa, t) (0 vot). The errors of the second-order derivatives obtained from other conditions and equations were significant (2 103 in some cases). We could reduce the error by (i) using another sampling method for training sets that does not suffer from the curse of dimensionality (an example given App. F.7) and (ii) adding the second-order derivative the loss function (e.g., differentiate both sides of the BHE w.r.t. and use it to the residual loss for PINNs).
43

---

Analytic solution
Prediction
K
Relative error
10-
Absolute error
10
0
10
10
Figure 9: Second-order derivative at 0 = 0 & 1. The model is trained on the FTE with degree
100 under the linear initial condition. The errors are as small as 10 -3
44

---

G Detailed Experimental Settings
We describe detailed setups of our experiment, including those on the advection-reaction functional equation (ARE) (App. H.4). The experimental settings for the ARE align with those provided in 991
Datasets. The training, validation, and test sets are sampled with Latin hypercube sampling 64 39
with range [0, 1]. [-1, 1], [o, 0.001]. [-0.1, 0.1]. [0, 0.5], and [h 0.01, h + 0.01] for t (FTE), (FTE), (BHE), dj (BHE), t (ARE), and a₁ (ARE), respectively, where == 1.69883527454424433 Training mini-batches are randomly sampled every time mini-batch is created. For the BHE, we decay the sampling range quadratically in terms of h E {0. 1.....,,. .. This sampling stabilizes the training because most of the loss comes from the region where A and Og are large, which can be seen from the definition of the BHE. That is, all the terms on the right- side are proportional to one of ags, while 0& for large ks are negligible in the analytic solution. Therefore, the collocation points with large 8 with large ks can be seen as noise in training. Note that this decaying sampling does not limit the quality of the solution at all because the solution is the characteristic function and we
are interested only in (xx) O, e.g., 66 (n)
(W())))) o=
(u(z)) (see also App. B.2
Miscellaneous settings. We use 4-yee PINN with 3x (linear + sin activation + layer normaliza- tion) + last linear layer. Unless otherwise noted, the widths are 1024 for the ARE and FTE and 2048 for the BHE The batch size is 1024. The activation function the sin function. The loss function for the FTE with the nonlinear initial condition (main text) and the ARE is the sum of the L and _ losses. The loss function for the others is the smooth L loss. The optimizer is Adam W [59]. The learning rate scheduler is the linear warmup with cosine annealing with warmup [58]. which is defined as the following scheduler:
scheduleri torch.opting 1r www.:ssehhcccerccceeeee torchh.ptin..rr.shhhheeeeeeLeee LinearRR
optimizer, wtt.:optt....sccceleeeeeeeeeeeee htt.::/hw.....aaa.........aaaaaa oassinicinnaaannlaamammmttttttt scheduler2 torch. torch.optim........
total_iterss (tgalllittersmmlesttnn)))
optimizer, T_0=T_0, T_mul=T__mllt sta_min=eta_min) scheduler optimizer,
Segpeettalee..eeeeettiaalalaCCCC scheduler2],
silestonns==miieestoon
See PyTorch official documentation 770 for the detailed definitions of the parameters. eta eta_min min is set to 0. start_factor is set to 10- Po and Uh = 1 are used for the FTE. jí 8 and 0² = 10 are used for the BHE. The Legendre polynomials and real Fourier series are used for the FTE and BHE, respectively, as the orthonormal basis. For the ARE, the modified Chebyshev polynomials described in [9T] are used.
Hyperparameter tuning. The full search space is given Tab. Unless otherwise noted, the number of iterations is 300,000 and 500,000 for the BHE and FTE, respectively. The number of trials for hyperparameter tuning is 50. Optuna (2) is used for hyperparameter tuning with the TPE sampler and median pruner. The number of tuning trials is 50 for all conditions, and the best hyperparameters within the 50 trials are used. See the config file in repository for more details.
Table 3: Search space of hyperparameters. Learning rates and weight decays are sampled log- uniformly. N 330,000 and 500,000 for the BHE and FTE, respectively. T_0, T f_mult, and milestone are used for the aforementioned scheduler. For the ARE and the main text results of FTE with the nonlinear initial condition, weight decay is fixed to 0 and the search space of learning rates from 10-7 7 to 10-4
HYPERPARAMETERS
SEARCH SPACE
LEARNING RATE WEIGHT DECAY T_O T_MULT MILESTONE
FROM 10 TO 10-3 FROM 10-7 TO 10 (OTHERS)
N/5, N/2, OR N OR 2 O. N/10, OR N/100
45

---

Remaining hyperparameters.
We show the hyperparameters used in our experiments in Tabs. 10
Table 4: Hyperparameters: FTE with linear initial condition.
DEGREE
HYPERPARAMETERS
4
20
100
LEARNING RATE
8.675
10
1.878 X 10
2.839
10 5
WEIGHT DECAY
4.534 10
6.184 x 10-7
2.008 10
T_0
250000
500000
500000
T_MULT
MILESTONE
0
5000
0
Table 5: Hyperparameters: FTE with nonlinear initial condition (main text).
DEGREE
HYPERPARAMETERS
4
100
1000
LEARNING RATE
8.456
10
1.042 x 10
1.130
10
WEIGHT DECAY
0
T_0
500000
500000
250000
T_MULT
1
1 5000
MILESTONE
5000
Table 6: Hyperparameters: FTE with nonlinear initial condition (Appendix).
DEGREE
HYPERPARAMETERS
20
LEARNING RATE
4.837
10
5.810 X 10 10
6,076
10
WEIGHT DECAY
8.637
10-5
2.877 x 10
2.068
10-7
T_0
250000
500000
500000
T_MULT
2
2 50000
MILESTONE
0
50000
46

---

Table 7: Hyperparameters: BHE with delta initial condition.
DEGREE
HYPERPARAMETERS
4
20
100
LEARNING RATE
6.680 x 10 -5
1.372 x 10 4
1.204 x 10
WEIGHT DECAY
4.429
10-6
6.644 x 10
4.893
10-3
T_0
300000
300000
150000
T_MULT
2
1
MILESTONE
0
3000
Table 8: Hyperparameterss: BHE with constant initial condition.
DEGREE
HYPERPARAMETERS
4
20
100
LEARNING RATE
5.131 X 10
9.989 X 10 10-5
8.637 x 10 5
WEIGHT DECAY
1.83
10-5
9.353 x 10-7
8.378
10-5
T_0
300000
150000
300000
T_MULT
2
1 30000
MILESTONE
3000
3000
Table 9: Hyperparameters: BHE with moderate initial condition.
DEGREE
HYPERFARAMETERS
4
20
100
LEARNING RATE
2.248
10
3.254 X 10 10-5
1.209 x 10
WEIGHT DECAY
3.669
10-5
1.771 x 10-6 6
1.423
10-6
T_0
300000
300000
300000
T_MULT
1
MILESTONE
3000
0
0
Table 10: Hyperparameters: ARE.
DEGREF
HYPERPARAMETERS
LEARNING RATE
6.3105 x 10
WEIGHT DECAY
0
T_0
250000
MULT
MILESTONE
5000
47

---

How to choose weight parameters for each training. We use validation relative error, i.e., the relative enror between the prediction and the analytic solution, as the measure for choosing the best weight parameter during each training trial.
Loss reweighting. The softmax reweighting is used to adaptively balance the residual and bound- ary loss functions. See the code below and our official code at GitHub (https: t/ubbb. thub. com/
Baalaliiaggaaa//uccciiool/l//. where list list_ tensors [residual loss, boundary loss] of mini-batch.
@torch.c (oarch.no_gra()) def softmax_ c
list_tensoct List [Tensor], "*"
temperature: float 0.25) Tensor:
# Args
list_tensors: List of scalar loss Tensors.
temperature: Temperature parameter. Default 0.25
# Returns coeffs: Tensor with shape [len(listttensors) ,].
ts =
(list_tensors)
ts /= porch.max(ts) EPSILON generates scale-invarian weight
coeffs = torch. smat (ts / temperature, dim=0)
return coeffs
This function generates the loss reweighting factors coeffs [11 A2), where total loss is defined as A x residual loss + X2 x boundary loss.
Float vs. double. No significant differences in performance measures were observed throughout our training. Accordingly, we used float32 to reduce computational time.
Libraries and GPUs. All the experiments are performed on Python 3.118 189 PyTorch 2.2.0 70
Numpy 1.26.0 38 and Optuna 3.5.0 2 An NVIDIA A100 GPU used.
Runtime. The training process takes 2-3 hours, but this duration can be significantly improved because it largely depends on the implementation of components such as data loaders. See App. H..2 for more details runtime. GPU memory consumption is about 400 MBs for the ARE and 800 1500 MBs for the FTE and BHE. We note that previous approaches to numerically solving FDEs rely on CPU processing, where performance varies significantly with the level of parallelization. Consequently, directly comparing the computational speed of our GPU-based approach with these methods is inherently challenging.
Figure
F((0) is approximated by [[PP1000]) for convenience.
48

---

H Additional Experimental Results
H.1 Pn. Converges to 0
The convergence rate of functional derivatives in Thm. 3.1 is given by 110 - Pm O|I The convergence of |00 0l is visualized Fig. [or four functions including non-smooth functions.
gelu: L1 loss 0.001479
sinc: L1 loss=0.036089
3
1.00
Ground truth
2-
Expansion
0.50
0.25
0.00
0-
0.25
-3
2
relu: L1 loss 0.019053
laplacian: L1 loss 0.018665
3-
1.0
0.6
0.4
0.2
gelu: L1 loss 0.000...00 000004
sinc: L1 loss 0.000000
1.00
0.75
0.50
2-
0.25
0.00
0-
0.25
3
relu: L1 loss 0.000322
laplacian: L1 loss 0.000322
1.0
0.6
0.2
Figure 10: Cylindrical approximation of (zx) (red) by (m)((x) (blue) with degree 10 (top four panels) and 100 (bottom four panels). I absolute error N >N 10(x;) Pm.((::) decreases as
1
increases. N 10a I; [-M,, т] are linearly spaced. GeLU 40), sinc, ReLU, and Laplacian
functions 78 are used. The basis is the Legendre polynomials.
H.2 Runtime
Tab. 1I shows training runtime. The runtime remains nearly constant and does significantly vary from m ~ 1 to m ~ 100. This aspect is of practical significance as it indicates that the computational time of model remains stable across broad range of m.
This consistency in runtime may be attributed to the efficiency the PyTorch library, where operations such matrix multiplication and automatic differentiation highly optimized. On the other hand, GPU memory consumption does show variation, ranging from 800 to 1500 MBs as stated App. G as naturally expected.
Finally, the previous approaches numerically solving FDEs rely on CPU processing, where performance significantly varies with the level of parallelization; thus, direct comparison of computational speed with our GPU-based approach is inherently challenging.
49

---

Table 11: Degree vs. runtime. "x:yz" denotes "x hours yz minutes." A "RUN" refers to training process with 500k, 300k, and 500k iterations for the FTE, BHE, and ARE, respectively.
FTE WITH 500K
ITER ATIONS
DEGREE #RUNS
4
20
100
11
8
RUNTIME 2:18 + 0:04
2:09 + 0:10
2:33 0:02
BHE WITH 300K ITERATIONS
DEGREE
4
20
100
#RNNS
RUNTIME 2:36 1 0:02
2:29 1 0:06
2:28 1 0:00
ARE WITH 500K ITERATIONS
DEGREE
#RUNS
10
RUNTIME
3:05 + 0:04
50

---

H.3 Latin Hypereube Sampling and Curse of Dimensionality
We show the histograms of the L norm of collocation points the sets. The sampling method is the Latin hypercube sampling, as mentioned in Sec. H Apparently. the collocation points s.t. norms are not sampled, which would cause the curse of dimensionality training. other words, the Latin hypercube sampling cannot completely address the curse of dimensionality in sampling.
Why does extrapolation mean (Sec. Firstly, Figs. 16 show that no collocation points such that ||a]| = O were included in the training sets. Secondly, Figs. & 7 plot the errors of PINN predictions the collocation points where Therefore, the collocation points such that σą R 0 in Figs. 4 were not included in the training sets, but the errors were as small as the region where ag 56 We refer to this as "extrapolation."
(0, 0, O, ag. O.
with i: 0, 1, 2, 19, or 99.
Norm: min min-2292e-02, mean 1000-11.11 1.12e+ max 1o0k=1.94e+00
103
10a
Count
10a
10"
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Norm of coefficients
Figure 11: Histogram of L norms of 4-dimensional coefficients in test set for the FTE.
Norm: min=1.3 100 mean-2.57e+00, max=3.58e-00
10a
Count
10a
10a
1.5
2.0
2.5 Norm of coefficients
3.0
3.5
Figure 12: Histogram of 7,2 norms 20-dimensional coefficients test for the FTE.
51

---

Norm: min=4.58e +. 00, mean=5.77e++00. max- 00
10s
10a
Count
10a
109
4.5
5.0
5.5
6.0
6.5
7.0
Norm of coefficients
Figure 13: Histogram of T.? norms of 100-dimensional coefficients in test set for the FTF.
Norm: min-2.92e--55 mean mean=5.00e-1 5.00e-02, max 1.00e-01
10s
Count
10a
10a
10-3
10-2
10-1
Norm of coefficients
Figure 14: Histogram of I norms of 4-dimensional coefficients test set for the BHE.
52

---

Norm: min-2.12e-05. lean=5.00e-00 max=1.00e
10a
Count
10a
10
10-3
10-2
10-1
Norm of coefficients
Figure 15: Histogram of T.2 norms of 20-dimensional coefficients in test set for the BHE.
Norm: miin-2.55e-05 mean mean=5.00e-00 5.00e-02, max 1.00e-01
103
Count
10a
10a
10-3
10-2
10-1
Norm of coefficients
Figure 16: Histogram of L norms 100-dimensional coefficients in test set for the BHE,
53

---

H.4 Comparison with Baseline: Advection-reaction Functional Equation (ARE)
We consider the advection-reaction functional equation (ARE).
(F( Ōt J. Jo (z)(x) xx) dz (0(x)
Ø (F(().t)
(112)
We follow the experimental setting described in
In [91], the reported time-dependent relative error of the CP-ALS and HT fluctuates from ~ 10-8 10-1, and the temporally-averaged relative error is estimated to be 10-2 (the bottom two panels in Fig. 38 in [9I], where we roughly approximated the time-dependent error curves as an exponential function of the form y 10 60-64
Our preliminary results are shown in Tab. 12 We observe that the learned functional almost constant in time, a failure mode of PINNs, causing large errors. We anticipate the result can improve with further hyperparameter optimization [50].
Table 12: Relative and absolute errors. The models are trained on the ARE. The error bars are the standard deviation over 10 training runs with different random seeds. Relative and absolute error represent the averaged relative and absolute error over all collocation points and all runs. Best relative and absolute error represent the relative and absolute error of the best collocation point averaged over all runs. The worst relative and absolute errors represent the averaged relative and absolute errors of the worst collocation point averaged over all runs.
DEGREE
RELATIVE ERROR
ABSOLUTE ERROR
6
(0.954940 + 2.86337) x 10-1
(6. (6.17040 + 1.83985) x 10-2
DEGREE
BEST RELATIVE ERROR
BEST ABSOLUTE ERROR
6
(0.12016 + 2.49220) x 10-1
(0.58046 + 1.73938) 10
DEGRER
WORST RELATIVV ERROR
WORST ABSOLUTE ERROR
(9.29019 2.85457) 10
(8.49163 + 2.0718) 10
54

---

H.S Relative and Absolute Errors When ([))) = 0 Is Included In Loss
We show the histograms of the relative and absolute errors for the models trained on the BHE with the loss term corresponding to the identity ([O))tt t) = O. They are almost the same as those for the models trained without W[[0), t) 0, but the errors of the first-order derivative are 10-1 times smaller, as shown in Sec.
104
103
Count
10a
10a
109
10
10
10
10-2
10
Relative error
10a
103
Count
10a
10a
109
10
10-6
10
10-2
10
Absolute error
Figure 17: Histogram of relative and absolute error. The model is trained on the BHE of degree 4 with the delta initial condition. A single random seed is used for the training.
55

---

104
103
Count
10a
10a
109
10-8
10-6 6
10
10-2
10"
Relative error
104
10a
Count
10a
10a
10
10-8 8
10-6 6
10
10-2
10"
Absolute
Figure 18: Histogram of relative and absolute error. The model is trained the BHE of degree 100 with the delta initial condition. single random seed is used for the training.
56

---

104
103
Count
10a
10a
10
10-8
10-6
10-4
10
10"
Relative error
104
10a
Count
10a
10a
10a
10-8
10-6
10 Absolute error
10-2
10
Figure 19: Histogram of relative and absolute error. The model is trained the BHE of degree with the constant initial condition. single random seed is used for the training.
57

---

104
103
Count
10a
10a
10
10-8
10-6
10-4
10
10"
Relative error
104
10a
Count
10a
10a
10a
10 10-8
10
10 10-4
10-2 2
10"
Absolute error
Figure 20: Histogram of relative and absolute error. The model is trained the BHE of degree 100 with the constant initial condition. A single random seed is used for the training.
58

---

H.6 Cross-degree Evaluation
In this section, we attempt to empirically investigate the theoretical convergence of the cylindrical approximation. The convergence is illustrated in Fig. 2 where PINN's training is not included, while our focus here is the convergence of the solutions estimated via PINN's training. According to the convergence theorem of the cylindrical approximation and the universal approximability of PINNs, the models trained on Dm span (cp. 0m-1) with large m have a high capability of expression, and they can cover Dn with n with small m have high capability of expression only for small ms, and they cannot cover D, for In theory, this phenomenon can be observed by performing cross-degree evaluation, in which we use training data sampled from Dm and then evaluate the trained model on Dm', where m may not equal to m'.
On the other hand, the models trained on D.
The results are shown in Tabs. 13 22 below. Contrary to the aforementioned intuition, most of the cross-degree evaluations seem to contradict the convergence theorem of FDE solutions under the cylindrical approximation. Possible reasons are:
1. This is because of the curse of dimensionality, i.e., the optimization enror of PINNs in high dimensions. Higher-dimensional PDEs require a much larger number of iterations to train PINNs; however, our experiments are performed with fixed number of iterations (500,000 and 300,000 for the FTE and the BHE, respectively). Changing the number of iterations depending on the degree and applying other techniques for training PINNS [106] 44 96 45 21 103 33] would decrease the optimization error, and we can focus on the approximation error and thus will observe the decay of error w.r.t. increasing
2. In addition, note that FDEs in our experiments, except for the FTE with the nonlinear initialization, are dominated by ag and/or di only. Therefore, they show a tiny effect relative and absolute errors by definition. Nevertheless, the FTE with the nonlinear initialization (Tabs. T3 and [6) does not show the theoretical convergence either, which means the optimization error of PINNs, rather than the cylindrical approximation error, dominates the errors anyways.
3. One can see strong dependence of the errors on the experimental setups for PINNs. The absolute eπrors of the FTE under the nonlinear initial condition are reduced by factor of 2 by simply changing the width of the PINN and the number of training iterations (1024 to 2048 and 5 x 10 to 8 x 105, respectively): from
(2.46499 0.26492) x 10-3 for m = 4,
(20.3384 = 1.0449) x 10 for m 10, and 10-3 for 20,
(98.8854 = 2.7068)
to
(1.43591 + 0.1832) x 10-3 for m = 4,
(11.0609 = 0.3671) x 10 for m 10, and
(55.2715 2.3179)
10 for 20.
where the eтror bars are the standard deviation over 10 runs. Note that the hyperparameters are different from those used in the main text. Moreover, as shown in Fig. 66) simply adding regularization term I ([0), f)ll to the loss function reduces relative errors by an order of magnitude.
Therefore, to observe the convergence of the cylindrical approximation after PINN's training, careful hyperparameter tuning and optimization are needed. Note that optimizing PINNs for high-dimensional PDEs is active field of research that has been developing (see Apps. and F and is of independent interest.
59

---

H.6.1 Relative and Absolute Errors of Functional Transport Equation
Table 13: Mean relative error 10 (functional transport equation with linear initialization) on test set. The model is trained on the functional transport equation with the linear initialization condition. The error bars represent the standard error of mean with sample size corresponding to different training seeds.
TRAINING SET DEGREE
TEST SET DEGREE
4
20
100
1. 1.26821 + 0. 09936
2.56697 + 0.20431
14.6402 + 2.1739
20
0.559035 + 0.026630
2.01716 + 0.06875
5.82531 + 0.19-199
100
0.483585 + 0.010872
1.88511 + 0.05292
6.21740 + 0.10591
Table 14: Mean absolute error x x10 (functional transport equation with linear initialization) on test set. The model is trained on the functional transport equation with the linear initialization condition. The error bars represent the standard error of mean with sample size corresponding to different training seeds.
Training SET DEGREE
TEST SET DEGREE
4
20
100
1.32203 + 0.13933
3.06281 + 0.28009
15.2530 = 0.7296
20
1.34579 + 0.14329
2.29632 + 0.05205
13.4665 1 0.7117
100
1.32709 + 0.14107
2.27854 + 0.05028
12.3312 + 0.5986
Table 15: Mean relative error x 1 (functional transport equation with nonlinear initialization) on test set. The model is trained on the functional transport equation with the nonlinear initialization condition. The error bars represent the standard error of mean with sample size corresponding to different training seeds.
TRAINING SET DEGREE
TEST SET DEGREE
10
20
1.09131
0.029455
2.06847
0.06531
7.51237
0.25359
10
1.14029
1 0.03166
3.59902 1 0.05139
8.61269 1 0.10351
20
1.11229 1 0.02789
3.62669 1 0.06044
11.7414 1: 0.1112
60

---

Table 16: Mean absolute error x10 (functional transport equation with nonlinear initialization) on test set. The model is trained on the functional transport equation with the nonlinear initialization condition. Note that the hyperparameters are different from those used in the main text. The error bars represent the standard error of mean with sample size 10 corresponding to different training seeds.
TRAINING DEGREE
TEST SET DEGREE
4
10
20
4
2.46499 + 0.08377
9.00259 + 0.21308
47.5449 + 0.6410
10
2.46885 1 0.08409
20.3384 1 0.3304
69.6191 1 0.7526
20
2.48688 1 0.084366
20.5866 1 0.3346
98.8854 1 0.8560
61

---

H.6.2 Relative and Absolute Errors of Burgers-Hopf Equation
Table 17:
Mean relative error x x10 (BHE with delta initialization) on test set. The model is trained on the BHE with the delta initialization condition. The error bars represent the standard error of mean with sample size 10 corresponding different training seeds.
TRAINING DEGREE
TEST SET DEGREE
20
100
4
2.93905 + 0.05503
2.97886 + 0.03187
2.99835 + 0.03851
20
2.24827 + 0.07876
2.20842 + 0.09022
2.23898 + 0.05530
4.11688 0.31740
2.98177 + 0.20264
2.11667 + 0.07989
Table 18: Mean absolute error x 10 (BHE with delta initialization) on test set. The model is trained the BHE with the delta initialization condition. The error bars represent the standard error of mean with sample size 10 corresponding to different training seeds.
TRAINING SET DEGREE
TEST SET DEGREE
20
100
1.66451
0.15036
1.35700
0.03771
1.62045
0.02767
20
1.66216 + 0.15005
1.34640 + 0.03828
1.60869 + 0.02892
100
1.66510 + 0.15030
1.34980 + 0.03779
1.60980 = 0.02831
Table 19: Mean relative error x00 (BHE with constant initialization) on test set. The model is trained on the BHE with the constant initialization condition. The error bars represent the standard error of mean with sample size 10 corresponding different training seeds.
Training DEGREE
TEST SET DEGREE
20
100
4
12. +: 2.70998
8.32131 + 0.79992
8.44530 +: 1.4148
20
9.07441 + 1.50640
6.14352 + 0.40680
5.99939 = 0.68357
100
8.26214
1.17503
5.75086 0.35933
5.50375
0.55500
62

---

Table 20: Mean absolute x x00 (BHE with constant initialization) test set. The model is trained on the BHE with the constant initialization condition. The error bars represent the standard error of mean with sample size 10 corresponding different training seeds.
TRAINING SET DEGREE
TEST SET DEGREE
20
100
4
1.62849 + 0.21154
1.12380 + 0.10234
1.05562 = 0.05272
20
1.62892
0.21162
1.12371 = 0.10235
1.05556 = 0.05272
100
1.62931
+ 0.21148
1.12369 + 0.10231
1.05558 0.05273
Table 21: Mean absolute error x 10 (BHE with moderate initialization) on test set. The model is trained the BHE with the moderate initialization condition. The error bars represent the standard етror of mean with sample size 10 corresponding to different training seeds.
TRAINING SET DEGREE
TEST SET DEGREE
20
100
1.32699
0.06872
1.60725
0.03521
1.84851
0.01689
20
1.24102 1 0.04793
1.63232 1 0. 0.02236
1.92692 1 0.00862
100
1.18115 1 0.04836
1.55581 1: 0.02152
1.82844 1: 0.00684
Table 22: Mean absolute error 10 (BHE with moderate initialization) on test set. The model trained on the BHE with the moderate initialization condition. The error bars represent the standard error of mean with sample size 10 corresponding different training seeds.
TRAINING SET DEGREE
TEST DEGREF
20
100
4
3.91187 + 008667
5.49339 +: 0.03494
6.73537 + 0.01891
20
3.93392
0.11011
5.63933 0.03427
6.87334 0.01884
100
3.90431
0.1087
5.60266 + 0.03403
48241440.0118
63

---

H.7 Loss Curves and Learning Rate Scheduling
Loss curves and learning rates are provided in Figs. 21
29
1.2e-5
1.1e-5
1e-5
9e-6
8
7e-6
6e-6
5e-6
4e-6
3e-6
2e-6
1e-6
0
100k
200k
300k
400k
500k
Figure 21: Learning rate scheduling. FTE with nonlinear initial condition with degree 1000.
10
0.1
0.01
1e-3
100k
200k
300k
400k
500k
Figure 22: Training loss. FTE with nonlinear initial condition with degree 1000.
10
1
0.1
0.01
1e-3
1e-4
100k
200k
300k
400k
500k
Figure 23: Validation loss.
FTE with nonlinear initial condition with degree 1000.
64

---

1.2e-4
M
1.1e-4
1e-4
9e-5
8e-5
7e-5
6e-5
5e-5
4e-5
3e-5
2e-5
1e-5
100k
200k
300k
Figure 24: Learning rate scheduling. BHE with delta initial condition with degree 100.
1e+4
100
1
0.01
1e-4
1e-6
1e-8
1e-10
100k
200k
300k
Figure 25: Training loss. BHE with delta initial condition with degree 100.
1e+4
100
1
0.01
1e-4
1e-6
1e-8
1e-10
O
100k
200k
300k
Figure 26: Validation loss. BHE with delta initial condition with degree 100.
65

---

6e-5
5e-5
4e-5
3e-5
2e-5
1e-5
0
0
100k
200k
300k
400k
500k
Figure 27: Learning rate scheduling, ARE with degree 6.
1et3
100
10
1
0.1
0.01
1e-3
0
100k
200k
300k
400k
500k
Figure 28:
Training loss. ARE with degree 6.
1e+3
100
10
0.1
0.01
1e-3
1e-4
1e-5
0
100k
200k
300k
400k
500k
Figure 29:
Validation loss. ARE with degree 6.
66

---

H.8
Additional Figures: Functional Transport Equation
H.8.1 Worst Coefficients and Best Coefficients
We provide the worst-/best-case relative/absolute error. The mean and minimum of errors in the test set do not strongly depend on training random seeds, while the maximum of errors intensely fluctuates. This observation tells us that the worst-case relative and absolute errors are not stable measures of performance.
Table 23: Relative error at the collocation point with the smallest relative error. The models are trained on the functional transport equation.
INITIAL CONDITION
INITIAL CONDITION
DEGREE
LINEAR
DEGREE
NONLINEAR
0.00000 + 0.00000
4
0.00000 + 0.00000
20
0.00000 + 0.00000
10
0.00000 + 0.00000
100
0.00000 = 0.00000
20
0.00000 + 0.00000
Table 24: Absolute error the collocation point whose absolute error is smallest. The models are trained on the functional transport equation.
INITIAL CONDITION
INITIAL CONDITION
DEGREE
LINEAR
DEGREE
NONLINEAR
4
0.00000
0.00000
0.00000 0.00000
20
0.00000 + 0.00000
10
0.00000 + 0.00000
100
0,00000 + 0.00000
20
0,00000 + 0.00000
Table 25: Relative error at the collocation point whose relative error is largest. The models are trained on the functional transport equation.
INITIAL CONDITION
INITIAL CONDITION
DEGREE
LINEAR
DEGREE
NONLINEAR
4
51.4467 + 22.3805
2.60833 + 1.03956
20
23.0638
11.7964
10 8.1 1617665- 3.50015
100
52.0792 + 19.0932
20
13.9782 1 7.87448
67

---

Table 26: Absolute error at the collocation point whose absolute error is largest. The models trained on the functional transport equation.
INITIAL CONDITION
INITIAL CONDITION
DEGREE
LINEAR
DEGREE
NONLINEAR
4
(7.91653 2.14557 x 10-3
0.19718 + 0.01640
20
(8.39847 1 1.54967) x 10
10
1.07628 + 0.06364
100
(34.2531 6,2608) 10
20
2.33652 = 0.08511
68

---

H.8.2 First-order Functional Derivative: Linear Initial Condition & Relative Error
The relative errors of the first-order functional derivative are provided. The models are trained under the linear initial condition.
Prod otinn (ax-1) Asabtic (tarr0)
20
-1.3
Prod 1oam13 Asayyi
-1.3
107
Figure 30: Degree 4.
Prod coon (au-4) Asabiic (tanr))
10
0.75
-2.00
-2.00
0.55
Prefetion (a Asayii (mxi1)
10"
-1.2
Figure 31: Degree 20.
69

---

Fied eilos (24-4))
taamd
Asabtic (tern))
Iabr wrte
0.3
-1.30
-1.75
-0.3
-100
0.00
0.50
1.50
0.75
1.00
-075
-0.0
-0.50
1100
0.55
0.55
1.00
Prediction
taam1
0.5
0.3
-1.30
-1.00
0.55
8.75
Figure 32: Degree 100.
70

---

H.8.3 First-order Functional Derivative: Nonlinear Initial Condition & Relative Error
The relative errors of the first-order functional derivative are provided. The models are trained under the nonlinear initial condition.
Protictin (seme))
Anaydi: [tar:]]
Protiction (sur-)) Ausyel
33
Figure 33: Degree 4.
- Prodiction (sem))) Anabicc (tau)91
107
8.70
2.00
-355
(sam)1)
Figure 34: Degree 10.
71

---

Prediction ((aa))) - Anabdic (taurt)
taumd
10-
10-4
-1.23
0.20
8.70
-225
Prefition (semm))
Asabiic
107
-355
Figure 35: Degree 20.
72

---

H.8.4 First-order Functional Derivative: Linear Initial Condition & Absolute Error
The absolute errors of the first-order functional derivative are provided. The models are trained under the linear initial condition.
Prod ction (ax-1) Asabtic (taund)
0.23005
Modhreeva
-1.3
0.0006
Prod ciinn (oxm)3 Asayyi
1100000
0.23055
0.0001
0.2065
-1.3
Figure 36: Degree 4.
Prod coinn Bax-t) Asabtic (taarO)
02200
124.-
0.2000
0.2068
3.0
0.25
Prdd (3
Asayiic (1xi.))
110013
80000
-1.
810
10.50
Figure 37: Degree 20.
73

---

Poediction (24-4))
Asabtic (terrd)
10.00
0.36-
0.304
0.3
0.000
-122
0.2
0.50
1,00
2.50
4.700
0.00
1.20
6.72
Prod otinn Baxe1)
Asabyic (man:))
10310
10:00
-2.75
3.55
0.50
1.00
Figure 38: Degree 100.
74

---

H.8.5 First-order Functional Derivative: Nonlinear Initial Condition & Absolute Error
The absolute errors of the first-order functional derivative are provided. The models are trained under the nonlinear initial condition.
Protiction (seme))
000030
Analyii [tar)]]
600090
000003
Protiction (sur-1)
0.005
0.000
0.003
0.000
Figure 39: Degree 4.
- Prodiction (sem)))
Anabiic (taur91
00025
120-0
0003)
0115
00000
01000
-0.00
0.35
1.00
57
-0.35
8.50
0.75
(2am)1)
3.5
2.5
0.55
Figure 40: Degree 10.
75

---

Prod (bam1))
0.14
0.12
5.10
0000
0.94
DU00
1000
-0.55
8.55
Pred
1aa-1
-0.25
-0.0
0.35
-0.00
8.50
Figure 41: Degree 20.
76

---

H.8.6 Time-dependent Relative and Absolute Errors at O: Linear Initial Condition
The time-dependent relative and absolute errors at 0 = O are provided. The models are trained under the linear initial condition.
Relative error
10
10
10-9
0.0
0.2
0.4
0.6
0.8
1.0
x 10-5
Absolute error
0.0
0.2
0.6
0.8
1.0
0.0
Analytic solution (input function=0)
J[[theta], t)
Prediction (input function=0)
0.2
0.4
-0.6
0.0
0.2
0.4
0.6
0.8
1.0
Figure 42: Degree
77

---

10-2
Relative error
10-3
0.0
0.2
0.4
0.6
0.8
1.0
Absolute error
0.0004
0.0002
0.0000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
Analytic solution (input function-0)
[[thhtta], t)
Prediction (input function 0)
10.4
-0.6
0.0
0.2
0.4
0.6
0.8
1.0
Figure 43: Degree 20.
78

---

10-2
Relative error
10-4
0.0
0.2
0.4
0.6
0.8
1.0
0.0010
Absolute error
0.0005
0.0000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
Analytic solution (input function-0)
[[thhtta], t)
Prediction (input function 0)
10.4
-0.6
0.0
0.2
0.4
0.6
0.8
1.0
Figure 44: Degree 100.
79

---

H.8.7 Time-dependent Relative and Absolute Errors at O: Nonlinear Initial Condition
The time-dependent relative and absolute errors at 0 = O are provided. The models are trained under the nonlinear initial condition.
Relative error
10
10-5
0.0
0.2
0.4
0.6
0.8
1.0
0.00075
Absolute error
0.00050
0.00025
0.00000
0.0
0.2
0.8
Analytic solution (input function 0)
Prediction (input function 0)
[(ttheta], t)
-2
0.0
0.2
0.4
0.6
0.8
1.0
t
Figure 45: Degree
80

---

Relative error
10-3
0.0
0.2
0.4
0.6
0.8
1.0
Absolute
0.02
0.01
0.00
0.0
0.2
0.4
0.6
0.8
1.0
t
04
Analytic solution (input function=0)
[((theta], t)
Prediction (input function 0)
-4
6
0.0
0.2
0.4
0.6
0.8
1.0
t
Figure 46: Degree 10.
81

---

10-2
Relative error
10-3
10
0.0
0.2
0.4
0.6
0.8
1.0
0.3
Absolute error
0.2
0.1
0.0
0.0
0.2
0.4
0.6
0.8
1.0
t
04
Analytic solution (input function-0)
[[theta]] t)
Prediction (input function 0)
10
0.0
0.2
0.4
0.6
0.8
1.0
t
Figure 47: Degree 20.
82

---

H.8.8 Time-averaged Pointwise Relative Error: Linear Initial Condition
The time-averaged pointwise relative errors are provided. The models are trained under the linear initial condition.
miko erer
3.2
1020
-9.2
11/0
9:23
10.00
00111
38"1
Figure 48: Degree 4.
3.5
8.50
0.50
355
X19
Figure 49: Degree 20.
83

---

30.44
-.50
-0.25
0.25
3.50
2.50
13.55
0.75
5.5
0.5
-1..0
8.55
300
Figure 50: Degree 100.
84

---

H.8.9 Time-averaged Pointwise Relative Error: Nonlinear Initial Condition
The time-avergeen pointwise relative errors are provided. The models are trained under the nonlinear initial condition.
10
38
LA 1 LEN
Traewsedd
Figure 51: Degree 4.
03-1
20"7
20
1.53
-2.00
0.55
1.00
22nd
Figure 52: Degree 10.
85

---

1.00
-1.55
-225
0.50
8.22
1.1
0.73
-0.00
0.5
Yer
-1.00
8.00
-.555
-8.0
-000
0.55
310
Figure 53: Degree 20.
86

---

H.8.10 Time-averaged Pointwise Absolute Error: Linear Initial Condition
The time-averaged pointwise absolute errors are provided. The models are trained under the linear initial condition.
(0003))
LARV190
C000600
EUR600
Figure 54: Degree 4.
Tht wwroodrddd... tr
80001E
00010
810010
0.23022
3.000
010000
3.0000
x10
Figure 55: Degree 20.
87

---

1..122
0U008
0.33
00000
0.530
2.20
00.75
1.23
3.500
-2.05
-6.00
0.75
10
abeddo 0317
0,0000
00060
-0.35
0.35
0.75
0.70
0.55
190
Figure 56: Degree 100.
88

---

H.8.11 Time-averaged Pointwise Absolute Error: Nonlinear Initial Condition
The time-averaged pointwise absolute errors are provided. The models are trained under the nonlinear initial condition.
(UK0)
2ho 600
(UX03
00000
0/006
00005
0.000
00003
0.70
11.00
11.33
1.00
1.00
3.10
0.70
0000
0000
13.
Figure 57: Degree 4.
0005
1.01
602
8.03
3.000
8.08
0.009
8.00
0000
0.334
0102
00.022
-1.25
1..01
1.00
6.00
11.20
(00)
11.01
0.75
8.02
0.12
0003
8.000
0.038
0.030
0.00
0.534
0,522
Figure 58: Degree 10.
89

---

0.17
TE wwagee Beice 4114
1100
100
2.00
-0.5
-0.25
1.0
0.50
-.00
-0.00
100
0.21
0.72
82
033122
0.303
DU0
610
0.33
1.00
0.50
1100
-1.00
0.55
0.55
8.75
310
Figure 59: Degree 20.
90

---

H.9
Additional Figures: Surgers-Hopf Equation
H.9.1 Worst Coefficients and Best Coefficients
We provide the worst-/best-case relative/absolute error. The mean and minimum of errors in the test set do not strongly depend on training random seeds, while the maximum of errors intensely fluctuates. This observation tells us that the worst-case relative and absolute errors are not stable measures of performance.
Table 27: Relative error the collocation point whose relative error smallest. The models trained on the BHE. The error bars are the standard deviation over 10 training runs with different random seeds.
INITIAL CONDITION
DEGREE
DELTA
MODERATE
CONSTANT
4
(7.52793 + 7.45055) x 10 10
(1.03507 + 1.15271) x 10
(6.89188 + 9.74890) x 10 10
20
(7.74760 3.91492) x 10-10
(1.04685 = 0.68460)
10 8
(5.03755 + 4.77551)
10-10
100
(10.6824 4.8721) x 10011
(2.21203 + 1.65951)
x 10
(2.03155 + 2.45872) x 100-1
Table 28: Absolute error at the collocation point whose absolute error is smallest. The models are trained on the BHE, The error bars are the standard deviation over 10 training runs with different random seeds.
INITIAL CONDITION
DEGRBE
DELTA
MODERATE
CONSTANT
4
(2.24063 1.79890) x 10-10
(2.87494 2.50055)
10
(1.84662 + 2.89692) 10-10
20
(3.64107 + 1.88671) x 10-10
(1.64646 + 1.14713) x 10-9
(1.73323 + 1.31036) x 10-10
100
(3.87652 2.24305) x 10-10
(4.2245
10-9
(0.98167 + 1.1928) x 100
Table 29: Relative error at the collocation point whose relative error is largest. The models are trained on the BHE The error bars are the standard deviation over 10 training runs with different random seeds.
INITIAL CONDITION
DEGREF
DELTA
MODERATE
CONSTANT
9.34454 1.45301
13.4754 11.0877
4.32330 5.31288
20
3.60415 1 0.86643
7.64168 + 2.60517
0.404178 + 0.207159
100
7.63891 + 2.85191
1.03030 + 0.19994
0.168201 + 0.176840
91

---

Table 30: Absolute error at the collocation point whose absolute error is largest. The models are trained on the BHE, The error bars are the standard deviation over training runs with different random seeds.
INITIAL CONDITION
DEGREE
DELTA
MODERATE
CONSTANT
4
(2.16137 0.40053) x 10
(2.89219 + 0.18509)
10
(2.77233 10 10-
20
(1.16468 0.16788) x +.134744 x 10 3
(2.51673 1.15930) x 10
100
(2.41825 0.66709) x 10
(4.64874 1 0.04121) x 10-3
(2.55274 1 0.48728) X 10-4
92

---

H.9.2 First-order Functional Derivative: Delta Initial Condition & Relative Error
The relative errors of the first-order functional derivative are provided. The models are trained under the delta initial condition.
190
Prodiaton 1o (00)
-7.995
-88 000
-8.003
$10
Predeticn (1xa-3
-7 7.95
2.000
Figure 60: Degree 4.
-7.7
0.000
-7.35
Audgyee Drased E30y
0.20175
0.03150
0.2000
-8.9 02
0,200h
-7
Rediriaa (1 201)
0.0000
-7.58
Aualyti (1am1133))
0.3077
(1:0110
-2.22
0.2005
-18
Figure 61: Degree 20.
93

---

Peod (ba=MU0R)) Asabyic (rauid 500
11.125
-7.90
77.35
+3 0000
13
0,000
Pred (000)
0.9122
7.87
03:000
010010
0.000
Figure 62: Degree 100.
Analyii (au-)1200)
0.01555
7.35
0.2010
-.02
0.31155
azmne
a.aom
Figure 63: Degree 4. W([0), r) = is included in the loss function.
94

---

-7.000
Prodetion Iaur 020)
1.10115
100 110
-7.030
Scalke ohi
-7.99 -7.990-
-5.000
600
0000
-7 5325
0.000
Prodiction (laam D01)
0.0034
1am000001
Rdilkc tre
10:00
1
5.00
c00m
0.000
Figure 64: Degree 100. W(0), = 0 is included in the loss function.
95

---

H.9.3 First-order Functional Derivative: Constant Initial Condition & Relative Error
The relative errors of the first-order functional derivative are provided. The models are trained under the constant initial condition.
Peed clum Oa4-0100
Anabiic 0.20002000
7288
-000
60003
EURIL
500
-000
01000
-194
Pred colan Cas 0.000
Avalytic
010005
1190
6000
0J0022
2094
-5080
01066
Figure 65: Degree 4.
Pled Medicion 6600)
-5 2012
-7.00
0000
2017
Figure 66: Degree 20.
96

---

Podeltee Boa-i (80)
Anayric
9991
Producto Beamd 600 E.0e
990
Rddgim U18
Figure 67: Degree 100.
97

---

H.9.4 First-order Functional Derivative: Moderate Initial Condition & Relative Error
The relative errors of the first-order functional derivative are provided. The models are trained under the moderate initial condition.
-7.90
Prodetio (laae) D30
Du 200
-7.39
Rolline Cro
0.001
Pradictor Tour D011 0000
0.004
6:33
Roddle GrEA
002
Figure 68: Degree 4.
-3.0
IUICC
10000
0:066
0.000
7.00
-0.5
00000
Figure 69: Degree 20.
98

---

Predicion (24..500)
1ac.0.000
-7.8
0.0070
640025
-0.33
Proditioon (60+8000)
taa+0:001
0000
Rddbe
Figure 70: Degree 100.
Pradiction (tu 6.000
Arabdi (ct -0.000
11.00
100-00000
0.0014
Proditionn (0)
77090
11000
3.000
140 0.001
Figure 71: Degree 4. W([0), r) = i included in the loss function.
99

---

Predetie [ex (30) Analysc (00-
2000
-7.993
-5.00
Prodection Haard D01) 6000
11a 2000
Rdlaii Ure
+5 110
Figure 72: Degree 100. W(D), = 0 is included in the loss function.
100

---

H.9.5 First-order Functional Derivative: Delta Initial Condition & Absolute Error
The absolute errors of the first-order functional derivative are provided. The models are trained under the delta initial condition.
-7 9M2
Prodecion [laze] E20
bu 2.00
Aralysic 64KC)
01000
-:8 9M7
E000
Prodiction (.22
$,000
2.01
51001
0.000
EU00
Figure 73: Degree 4.
Prodaator tea 000)
2.00
-7.000
0.310
-7.90G
0.310
0,000
00002
350
Prodectoo
11:112
01.200
0.900
0,002
Figure 74: Degree 20.
101

---

Peed mifon Coe E000:
1.0
EUKG
60
Fredi ttio Coc-tront
E1100
E0000
Figure 75: Degree 100.
0.300
0.310
0.006
310
(1200
5000
0.002
11.314
1aa-2 CC0
0.312
11.000
$6.003
11.000
[taa 125
Ar Aralyic (00-00001
Figure 76: Degree 4. W([0), r) = i is included in the loss function.
102

---

0.3005
12s 1000
BLace
tuans
0.000
-5.001
0000
0.000
-14
Mookas
Analysc 000000000
100
Figure 77: Degree 100. W(D), = is included in the loss function.
103

---

H.9.6 First-order Functional Derivative: Constant Initial Condition & Absolute Error
The absolute errors of the first-order functional derivative are provided. The models are trained under the constant initial condition.
9M
-:8 007
E000
9M4
Analytic Laaam 2013
1.000
0.000
-V.996
0/001
Prepect en (Sumn 2013 Asalytic Itea 4211
6100
1aa-1 CC0
Figure 78: Degree 4.
Peed ction 60000
7..08
EUK0
Fred Cin--
5.006
-7. 990
5.00
0.000
01000
51000
Figure 79: Degree 20.
104

---

51000
51000
51007
994
6000
-1.4
0.000
1,800
0,000
2000
80
Figure 80: Degree 100.
105

---

H.9.7 First-order Functional Derivative: Moderate Initial Condition & Absolute Error
The absolute errors of the first-order functional derivative are provided. The models are trained under the moderate initial condition.
0.535
-7%
Prodictin (eaed Avolytic (1000))00)
0.033
2.031
8:00
0.003
Prefitton (rawd 0E1)
0.033
3:000
0.00
Figure 81: Degree 4.
-7.58
Prediction (ta. 603)
Aaskit at2"
0.015
3.90
0.000
7.500
Prefition
-7005
-7. 090
3:000
2.003
0.000
Figure 82: Degree 20.
106

---

Peed eiion (24. 60000
124. 060
Aualiti
E106
01000
-C.4
6.
Pred coion Oue 00000
1ea-0000
01001
EUK02
0.000
Figure 83: Degree 100.
Aned
EUMA
Analystc (teu
5100
1eamm 100
-E.4
0.112
eslan (2t B000))
200
Avalytic (Sased 201)
124 3 000
Figure 84: Degree 4. W([0), 7) = 0 is included in the loss function.
107

---

Amabi (1am00300)
Piedition D005
11800
104 2000
-7.0%
5.066
1:8.0 3000
6001
0000
-1.4
Anabitic (sanr
Pred colm Cax-01002
01000
10 200
51000
Figure 85: Degree 100. W(O),,) 7) = is included in the loss function.
108

---

H.9.8 Time-dependent Relative and Absolute Errors at 0: Delta Initial Condition
The time-dependent relative and absolute errors at 0 = 0 are provided. The models are trained under the delta initial condition.
0.0002
Analytic solution (input function=0)
Prediction (input function=0)
0.0001
W[[theta]
0.0000
-0.0001
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
x10-5
Absolute error
2.0
1.8
1.6
1.4
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
Figure 86: Degree 4.
109

---

0.0002
Analytic solution (input function-0)
Prediction (input function=0)
W([theta] tau)
0.0001
0.0000
0.0001
-0.0002
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
x 10
2.4
Absolute error
2.2
2.0
1.6
0.0000
0.0002
0.0004
0.0006
0.000S
0.0010
Tau
Figure 87: Degree 20.
110

---

Analytic solution (input function=0)
0.0002
Prediction (input function=0)
W([thetal, tau)
0.0001
0.0000
0.0001
0.0002
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
x 10
2.6
Absolute error
2.5
0.0000
0.0002
0.0004
0.0006
0.000S
0.0010
Tau
Figure 88: Degree 100.

---

H.9.9 Time-dependent Relative and Absolute Errors at O: Constant Initial Condition
The time-dependent relative and absolute errors at 0 = O are provided. The models are trained under the constant initial condition.
10
Analytic solution (input function=0)
Prediction (input function=0)
5
W[[theta] ttau)
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
10
Absolute error
3
2
0.0000
0.0002
0.0004
0.0006
0.000S
0.0010
Tau
Figure 89: Degree 4.
112

---

x 10
5.0
Analytic solution (input function=0) 0)
Prediction (input function=0)
2.5
W([theaa],
0.0
2.5
-5.0
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
x10-6
2.0
Absolute error
1.5
1.0
0.0000
0.0002
0.0004
0.0006
0.000S
0.0010
Tau
Figure 90: Degree 20.
113

---

x 10
Analytic solution (input function=0) 0)
1.0
Prediction (input function= 0)
W([thtaa], ttau)
0.5
0.0
-0.5
-1.0
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
x10-6
Absolute error
2
0.0000
0.0002
0.0004
0.0006
0.000S
0.0010
Tau
Figure 91: Degree 100.
114

---

H.9.10 Time-dependent Relative and Absolute Errors at 0 = 0: Moderate Initial Condition
The time-dependent relative and absolute errors at 0 = 0 are provided. The models are trained under the moderate initial condition.
Analytic solution (input function=0)
0.00010
Prediction (input function=0)
W([theta] tau)
0.00005
0.00000
-0.00005
-0.00010
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
x 10-6
Absolute error
2
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
Figure 92: Degree 4.
115

---

0.0002
Analytic solution (input function-0)
Prediction (input function=0)
W([thetal, tau)
0.0001
0.0000
-0.0001
-0.0002
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
x 10
3.0
Absolute error
2.5
2.0
1.5
0.0000
0.0002
0.0004
0.0006
0.000S
0.0010
Tau
Figure 93: Degree 20.
116

---

Analytic solution (input function-0)
0.00010
Prediction (input function=0)
W([thetal,.t tau)
0.00005
0.00000
-0.00005
-0.00000
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
x 10
1.8
1.6
Absolute error
1.4
1.2
1.0
0.8
0.0000
0.0002
0.0004
0.0006
0.0008
0.0010
Tau
Figure 94: Degree 100.
117

---

H19.1 Time-averaged Pointwise Absolute Error: Delta Initial Condition
The aveeravee pointwise absolute errors are provided. The models are trained under the delta initial condition.
This sinage -ieeen PI1E
13
000005
21
X33-
22
13
Figure 95: Degree 4.
Tite-oo cergyr dacken FIM
000099
13
000007
Teee2 wap 3000 20h
Figure 96: Degree 20.
118

---

[UK03)
Tree SAMOON SPAINTY
[UKK3)
000000
Hogh
000000
-2.33
-00ms
-Ere3
01833
0.270
-0000
-0.311
0.300
0.001
3.02
D006
0.001
15
2.50
3.50
1.50
0.20
100
5.20
Figure 97: Degree 100.
119

---

H.9.12 Time-averaged Pointwise Absolute Error: Constant Initial Condition
The time-averagee pointwise absolute enrors are provided. The models are trained under the constant initial condition.
01063
01007
01000
1006
01000
30
-110m
Figure 98: Degree 4.
Tiee Seroo duckee 201
10002
1000L
01000
110
2.35
03
X29
Figure 99: Degree 20.
120

---

0/00055
000000
100001
-2.30
-007
-0300
-ER03
0.23
0.270
0000
-0.01
0.001
3.02
40
42.
Thi 3i00e 2012
6100
15
2.00
3.50
1.50
0.20
32
100
5.20
Figure 100: Degree 100.
121

---

H.9.13 Time-averaged Pointwise Absolute Error: Moderate Initial Condition
The time-averaged pointwise absolute errors are provided. The models are trained under the moderate initial condition.
[UKKE)
600005
000000
-0
21.
X33-1
83
Figure 101: Degree 4.
118
1100L5
UIKEU
k33
111
3000 201
X22
Figure 102: Degree 20.
122

---

Tree SAMOga -houttraa
000600
000003
00009)
-2.20
-2.07S
-0.90
-EX03
1086
0302
0.270
-0002
-0.301
11.00
0.001
3.02
10
41.
-0000
E006
01001
55
4300
3.0
3.50
100
X.20
Figure 103: Degree 100.
123

---

H.9.14 Time-averaged Pointwise Relative Error: Delta Initial Condition
The time-averaged pointwise relative errors are provided. The models are trained under the delta initial condition.
10-1
Hulk
Figure 104: Degree 4.
307
301
2.7
Figure 105: Degree 20.
124

---

312
Teroeerap reinte 2017
33.-
-C.100
-007
DU00
0022
00000
00775
0.000
-1.003
-0.001
4.31
DUK2
40
2.000000
-000
-100
6/06
1001
40.00
-050
-355
0.75
10
390
X.20
Figure 106: Degree 100.
125

---

H.9.15 Time-averaged Pointwise Relative Error: Constant Initial Condition
The time-averaged pointwise relative errors are provided. The models are trained under the constant initial condition.
22
30
Figure 107: Degree 4.
18a
2xx
00
X29
Figure 108: Degree 20.
126

---

18"4
-C.100
-0UTT
-2.225
13.000
0.225
2.075
0.100
000
D0K2
00
2.000
3.50
0.75
23
399
5.20
Figure 109: Degree 100.
127

---

H.9.66 Time-averaged Pointwise Relative Error: Moderate Initial Condition
The time- aieeaaeergged pointwise relative errors are provided. The models are trained under the moderate initial condition.
307
Figure 110: Degree 4.
6772
2011
18
Figure 111: Degree 20.
128

---

Tre-snrrggh reaale 4117
-0005
6005
1.6 007
-100
-1001
6/00
8.01
48.70
-0.50
-255
0.75
12
X.20
Figure 112: Degree 100.
129

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contribution is summarized at the end of Introduction. How much the results can be expected to generalize to other settings is also discussed App.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Sec. and App.
F.I
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
130

---

Justification: The full set of assumptions and complete proofs for each theoretical result is given in Sec. [ Apps. C and In particular, App. Cis dedicated to the mathematical background for non-experts of functional analysis, making our paper as self-contained as possible.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
All the information for reproduction is given in Sec. and App. G We also provide our code in the supplementary materials. See also our GitHub http:: //gg://b//tub
Justification:
com http://aiiMMygggggeee//ccooooooo
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
131

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our code is given in the supplementary materials and GitHub (https:// gi github. http://wwwiiiaaaaaaa/uuuuiiiaaaa
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
mublic//gudee//Ddeeelbniisssss www.aaammmiioooooooooooooooeeiie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //niss.c/ppll ptpp://cuceclliggudddeeeeee guides/ ProodeSubmissiooooll...o
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All the information for reproduction is given in Sec. and App. provide our code in the supplementary materials.
G
We also
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Error bars (standard deviations) are reported Sec. and App.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
132

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are reported in App. G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification:
We carefully read the NeurIPS Code Ethics and checked our work complies
it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of functional differential equations, and there is no socictal impact which we feel must be specifically highlighted here.
Guidelines:
133

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We appropriately cite relevant assets. The license and terms of use mentioned our code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
134

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We carefully read and follow the NeurlPS Code and Data Submission Guide-
lines.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
135

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
136


Reviews:
[{"review_id": "NQE8YKxbUF", "reviewer": "Reviewer_5Xf5", "summary": "In this paper, they used cylindrical approximation to transform the functional differential equation (FDE) into a higher-dimensional PDE in order to solve it using PINN.", "questions": "* I'm curious about the range of \\(\\boldsymbol{a}\\) in the experiments. To train with PINN, \\(\\boldsymbol{a}\\)'s range needs to be fixed beforehand for sampling. How was this managed?\n\n* I'm interested in why r is typically \"typically 1 or 2,\" as mentioned in line 229 of the manuscript. A detailed explanation would be helpful.\n\n* The abbreviation BHE in the caption of Figure 2 first appears in line 162. It should be mentioned earlier in the text.\n\n* Aside from FTE and BHE, I'm curious if there are more complex or challenging applications related to actual physical phenomena.\n\n* In Table 1, it seems that the relative error increases as the degree increases. Is this trend correct? I would appreciate a more detailed explanation and analysis.\n\n* Figures 11-16 are labeled as extrapolation. Could you clarify what this means?", "limitations": "See above.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "This is a relatively new and intriguing topic in the field of functional differential equations (FDEs), aiming to solve them using PINN as an innovative approach. To this end, specific examples of FDEs such as FTE and BHE were provided, accompanied by a clear mathematical explanation of what cylindrical approximation entails.", "weaknesses": "It seems that significant improvements are needed in the experimental section. Particularly, Figure 4 and Figure 7 suffer from poor readability, with unclear x-axis and y-axis labels that make it difficult to discern their intended purpose. Moreover, to demonstrate the advantages of the proposed method effectively, it is essential to compare it with existing numerical analysis methods such as CP-ALS. Such comparisons would clarify how the proposed method reduces computational complexity, as mentioned on line 224 of the manuscript, and illustrate how the order varies with respect to $m$ ($m^6$ vs $m^r$) using the experimental results. Therefore, the experimental section appears to be lacking overall.\n\nFurthermore, cylindrical approximation has already been studied in the field of numerical analysis. Clarifying what novelty exists in applying this approach to convert FDEs into high-dimensional PDEs and then using the well-established PINN method would be beneficial. If the author claims that using deep learning to solve FDEs is novel, it is crucial to provide clear comparisons with state-of-the-art numerical methods in terms of experimental results, computation time, computational cost, accuracy, and other relevant metrics. This comparison would ultimately demonstrate how the proposed approach stands out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate Reviewer 5Xf5 for their time and invaluable comments.\nWe will incorporate all the suggestions into our manuscript.\n\n> I'm curious about the range of $\\boldsymbol{a}$ in the experiments.\n\nIt is provided in lines 1456-1458.\n\n> Aside from FTE and BHE, I'm curious if there are more complex or challenging applications related to actual physical phenomena.\n\nApp. B provides an additional introduction to FDEs. For example, the Navier-Stokes-Hopf equation (Eq. (10) in line 805) comprehensively describes the statistical properties of turbulence of realistic 3D fluids, involving complicated differentiation and functional differentiation operations. The Schwinger-Dyson equation (Eq. (13) in line 833) solely provides properties of quantum particles, while solving it is even more challenging because it can involve, for example, third-order functional derivatives.\n\n> I'm interested in why r is \"typically 1 or 2,\" as mentioned in line 229 of the manuscript. A detailed explanation would be helpful.\n\nMajor examples of FDEs include the Hopf functional equation, the Fokker-Planck functional equation, and the functional Hamilton-Jacobi equation (lines 22-30 & Sec. 2).\nAll of them have $r=1$ or $2$. \nOther less-known FDEs, such as Eq. (16-17) in line 859, also have $r=2$.\n\nA notable example with $r=3$ is the Schwinger-Dyson equation (Eq. (13)).\nTo our knowledge, there are no other major significant examples with $r > 2$, although arbitrary toy FDEs can be constructed.\n\n> In Table 1, it seems that the relative error increases as the degree increases. Is this trend correct? I would appreciate a more detailed explanation and analysis.\n\nAs noted in the captions of Tables 1 & 2, these tables are not intended to assess the theoretical convergence of the cylindrical approximation; therefore, a decrease in relative error is not necessarily expected. This is because different exact solutions (ground truth functionals) are used for different rows, as is stated in footnote 1 on page 7. \n\n- Firstly, these tables serve as a proof of concept for our proposed approach, demonstrating its capability to learn FDEs effectively.\n- Secondly, the expected decrease of relative error for large $m$ *without* training PINNs is illustrated in Figure 2.\n- Thirdly, the expected decrease of relative error for large $m$ *with* training PINNs is analyzed in App. H.6, where a cross-degree evaluation is performed. \n\n> Figures 11-16 are labeled as extrapolation. Could you clarify what this means?\n\n- Firstly, figures 11-16 shows that no collocation points such that $\\| \\boldsymbol{a} \\| \\approx 0$ were included in the training sets.\n- Secondly, Figures 4 & 7 plot the absolute errors of PINN predictions at the collocation points where $\\boldsymbol{a} = (0, 0, \\dots, 0, a_k, 0, \\dots, 0)$ with $k = 0, 1, 2, 19,$ or $99$.\n- Therefore, the collocation points such that $a_k \\approx 0$ in Figures 4 & 7 were not included in the training sets, but the errors were as small as those in the region where $a_k \\not\\approx 0$. We refer to this as \"extrapolation.\"\n- Additionally, we have revised confusing sentences in lines 296-300 and 312-316 for clarity.\n\n> cylindrical approximation has already been studied ... Clarifying what novelty exists in applying this approach ... and then using the well-established PINN method would be beneficial. \n\n- Theoretical Contribution Compared with Previous Studies on the Cylindrical Approximation:\n    - For details, please see lines 97-101 and, for technical details, lines 1016-1019 in App. C.4.2.\n    - In summary, we established convergence theorems of functionals and FDE solutions under a modified cylindrical approximation tailored for practical use (i.e., the cylindrical approximation of functionals without the \"tail term\", as mentioned in lines 125-127).\n\n- Empirical Contribution, Especially with the Use of PINNs:\n    - In functional analysis, there are several approaches to reduce infinite dimensional function spaces to finite spaces to cut computational costs. Researchers have sought efficient and scalable methods. \n    - Among them, we found that the cylindrical approximation, combined with PINNs, can offer significant scalability, which may seem obvious in hindsight. \n    - Previous approaches using the cylindrical approximation have solely focused on tensor decomposition and finite difference methods (lines 90-92), which limit scalability. Thus, their experiments were confined to the input functions such as polynomials with degrees no higher than $10$. In contrast, our model can treat degrees up to $m \\sim 1000$, demonstrating unprecedented expressivity.\n\n> it is essential to compare it with existing numerical analysis methods such as CP-ALS.\n\n> it is crucial to provide clear comparisons with state-of-the-art numerical methods.\n\nFor the runtime comparison with the CP-ALS, please see lines 1523-1526 in App. G (Detailed Experimental Settings: Runtime).\n\nFor the empirical comparison of computational complexities, note that previous approaches suffer from the severe curse of dimensionality, making it *impossible* to perform experiments with as large as $m \\sim 50$ or higher. Besides, the implementations of their complex algorithms are unavailable, further making complicating performance comparisons.\n\nFor the error comparison, see lines 337-338 and App. H.4, where we compare the CP-ALS with our model in solving the advection-reaction equation. In summary, the error of the CP-ALS is $\\lesssim 10^{-2}$, while ours is $\\sim 10^{-1}$. Please note that the following:\n- 1. We further tuned hyperparameters after submission, resulting in even smaller errors for our model. Please see also lines 1327-1332 in App. F.1 (\"To further reduce errors\").\n- 2. Table 12 contains typos: RELATIVE ERROR, BEST RELATIVE ERROR, and WORST RELATIVE ERROR should be read as WORST RELATIVE ERROR, RELATIVE ERROR, and BEST RELATIVE ERROR, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R7wSMWRRWa", "reviewer": "Reviewer_LXGL", "summary": "The power of PINNs is leveraged to solved high-dimensional PDEs which are obtained from FDEs through the cylindrical approximation. FDEs are computationally expensive to learn, and PDEs are more well-studied in the context of learning. This is a novel work in making FDEs more accessible for computation since they have a wide impact across many technical disciplines.", "questions": "I don't have questions", "limitations": "The limitations are clearly presented and have been addressed, when possible, reasonably.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This is a novel way to circumvent longstanging numerical issues concerning computing FDEs. The idea is easy to follow, at a high level. \n2. Computational complexity and expressive power are significantly improved compared to the current SOTA using cylindrical approximations that are shown to converge.\n3. Notation and steps are clearly presented.", "weaknesses": "The authors presented their weaknesses/limitations in a section.\nThey could try to expand their suites of experiments to the applications outlined in the appendix e.g. Navier-Stokes modeling, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate Reviewer LXGL for their time and recognition of our paper's strength.\n\n> The authors presented their weaknesses/limitations in a section. They could try to expand their suites of experiments to the applications outlined in the appendix e.g. Navier-Stokes modeling, etc.\n\nThis is indeed our next project, and we are currently conducting the relevant analyses and experiments. We recognize that this work will require a separate paper, likely better suited for physics journals, because the numerical analysis of higher-order FDEs ($r \\geq 2$) is particularly important in physics (see Sec. 1 and App. B).\n\nPlease let us know if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S3TWuE17IM", "reviewer": "Reviewer_33cu", "summary": "This paper considers solving FDEs (Functional Differential Equations) using neural networks. The difficulty of solving FDEs compared to PDEs (Partial Differential Equations) lies in the fact that the input space is infinite-dimensional. In this case, the author employs Cylindrical Approximation to reduce the infinite-dimensional space to a finite-dimensional space. Then reduce the problem to a PDE case and use PINN to solve it.", "questions": "Mentioned in the Weakness.", "limitations": "All right.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Consider the challenges of utilizing neural networks in infinite-dimensional spaces. The author's use of Cylindrical Approximation as a method to reduce the infinite-dimensional space to a finite-dimensional one is commendable. The results presented in the paper are interesting and well-written.", "weaknesses": "I did not find the main issue of the paper addressed. My concern is that the theoretical part of this paper is somewhat weak. The theorems primarily present asymptotic results, i.e., asking for \\(m \\to \\infty\\). In my knowledge, there are some papers considering the functional approximation such as: \n1. T. Chen and H. Chen. Approximations of continuous functionals by neural networks with application to dynamic systems. IEEE Transactions on Neural Networks, 4(6):910–918, 1993.\n 2. Y. Yang and Y. Xiang. Approximation of Functionals by Neural Network without Curse of Dimensionality.\n 3. L. Song et al. Approximation of smooth functionals using deep ReLU networks.\n\nExcept for the first paper, the other two papers provide the approximation error of functionals. In my opinion, if you have Cylindrical Approximation rates and PINN approximation rates, the total order of approximation may also be obtainable. Are there any difficulties in obtaining the approximation rate?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 33cu for their insightful comments and recognition of our paper's strength.\nThank you for pointing out the related papers; we enjoyed reading them and will include them in the Related Work section.\nPlease note that, just in case, their focus is functional approximation, not functional derivatives or solving FDEs.\n\n> In my opinion, if you have Cylindrical Approximation rates and PINN approximation rates, the total order of approximation may also be obtainable. Are there any difficulties in obtaining the approximation rate?\n\nThis is a very interesting question, which we have been discussing in our current research. \nOur response is as follows:\n\n- \"the total order of approximation may also be obtainable\"\n    - Yes.\n    - The total approximation error rate can be derived by combining the cylindrical approximation error (e.g., lines 1140-1145) with the PINN approximation error; specifically, $|F(\\theta) - \\hat{f}(\\boldsymbol{a})| \\leq | F(\\theta) - f(\\boldsymbol{a}) | + | f(\\boldsymbol{a}) - \\hat{f}(\\boldsymbol{a}) |$, where $F(\\theta)$ is the target functional with input $\\theta$, $f(\\boldsymbol{a})$ is its cylindrical approximation, and $\\hat{f}(\\boldsymbol{a})$ is the prediction of a PINN after training. Here, $| F(\\theta) - f(\\boldsymbol{a}) |$ represents the cylindrical approximation error we evaluated in our work, and $| f(\\boldsymbol{a}) - \\hat{f}(\\boldsymbol{a}) |$ is the PINN approximation error. In light of the recent advances in the analysis of PINN approximability, at first glance, [ref1] appears a promising starting point because of its minimal assumptions on the form of PDEs.\n    \n- \"Are there any difficulties in obtaining the approximation rate?\"\n    - Yes.\n    - The challenge is that it may be difficult to observe theoretical convergence of approximation error in experiment: if we successfully derive the combined cylindrical + PINN approximation error, it may be theoretically interesting but practically less relevant, because the optimization error of PINNs often overshadows the approximation error in experiments, as noted in our manuscript (e.g., App. H.6).\n\nTherefore, we conclude that a careful and detailed discussion on this point is required, which warrants a separate paper.\n\n[ref1] Ryck & Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper considers neural network based approach for solving functional differential equations, and establish a convergence result based on cylindrical approximation. The method is a natural extension of neural network based PDE solvers. Overall the reviewers find the approach interesting and most concerns are addressed during the discussion period, even though more empirical tests are desired. After carefully reading the manuscript and discussion, the meta reviewer would recommend acceptance of the manuscript.
Decision: Accept (poster)