Data-driven discovery ofGreen’s functions

Nicolas Boullé
University College
University of Oxford

A thesis submitted for the degree ofDoctor of Philosophy
Trinity 2022

Abstract

Discovering hidden partial differential equations (PDEs) and operatorsfrom data is an important topic at the frontier between machine learningand numerical analysis. Theoretical results and deep learning algorithms
are introduced to learn Green’s functions associated with linear partial
differential equations and rigorously justify PDE learning techniques.
A theoretically rigorous algorithm is derived to obtain a learning rate,
which characterizes the amount of training data needed to approximately
learn Green’s functions associated with elliptic PDEs. The construction
connects the fields of PDE learning and numerical linear algebra by extending the randomized singular value decomposition to non-standard
Gaussian vectors and Hilbert–Schmidt operators, and exploiting the lowrank hierarchical structure of Green’s functions using hierarchical matrices.
Rational neural networks (NNs) are introduced and consist of neural networks with trainable rational activation functions. The highly compositional structure of these networks, combined with rational approximation
theory, implies that rational functions have higher approximation power
than standard activation functions. In addition, rational NNs may havepoles and take arbitrarily large values, which is ideal for approximating
functions with singularities such as Green’s functions.
Finally, theoretical results on Green’s functions and rational NNs are combined to design a human-understandable deep learning method for discovering Green’s functions from data. This approach complements stateof-the-art PDE learning techniques, as a wide range of physics can be
captured from the learned Green’s functions such as dominant modes,
symmetries, and singularity locations.

Acknowledgements

I would first like to thank my supervisors Patrick Farrell, Marie Rognes,
and Alex Townsend for their guidance and suggestions. Their passion
and excitement for the field of numerical analysis, as well as their highacademic standards, have been a constant source of inspiration and motivation during my DPhil.
I would also like to thank my confirmation examiners, Christoph Reisinger
and Justin Sirignano, for their comments and suggestions, as well as Andrew Stuart and Jared Tanner for accepting to be my thesis examiners.
This thesis benefited from discussions with great collaborators, including Efstathios Charalampidis, Vassilios Dallas, Christopher Earls, Ada
Ellingsrud, Panayotis Kevrekidis, Seick Kim, Yuji Nakatsukasa, Alberto
Paganini, Debasmita Samaddar, Tianyi Shi, and Jonasz Slomka.
I am grateful to Simula Research Laboratory for co-funding my DPhilalong with University College, the Oxford-Radcliffe scholarship, and the
InFoMM CDT.
I thank my friends and colleagues from Oxford and Cornell, Boris Andrews, Francis Aznaran, Pablo Brubeck, Dan Fortunato, Marc Gilles,
Gonzalo Gonzalez de Diego, Andrew Horning, Fabian Laakmann, Maike
Meier, John Papadopoulos, Alex Puiu, Tianyi Shi, and Heather Wilber,
who made this DPhil always enjoyable and fun with great academic andnon-academic conversations.
Finally, I am grateful to my family and Tina for their continuous supportand encouragements throughout the years.

Contents
1 Introduction
1.1 Deep learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1
2

1.2

Physics-informed machine learning . . . . . . . . . . . . . . . . . . .

3

1.3

Green’s functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

1.4
1.5

Low-rank approximation . . . . . . . . . . . . . . . . . . . . . . . . .
Randomized singular value decomposition . . . . . . . . . . . . . . .

8
9

1.6

Hilbert–Schmidt operators . . . . . . . . . . . . . . . . . . . . . . . .

10

1.7

Quasimatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

1.8
1.9

Gaussian processes . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12
13

2 Learning elliptic PDEs with randomized linear algebra
2.1

2.2

2.3

15

Low-rank approximation of Hilbert–Schmidt operators . . . . . . . .

18

2.1.1
2.1.2

Three caveats that make the generalization non-trivial . . . .
Deterministic error bound . . . . . . . . . . . . . . . . . . . .

19
20

2.1.3

Probability distribution of Ω1 . . . . . . . . . . . . . . . . . .

21

2.1.4

Quality of the covariance kernel . . . . . . . . . . . . . . . . .

23

2.1.5
2.1.6

Probabilistic error bounds . . . . . . . . . . . . . . . . . . . .
Randomized SVD algorithm for HS operators . . . . . . . . .

24
27

Recovering the Green’s function from input-output pairs . . . . . . .

30

2.2.1

Recovering the Green’s function on admissible domains . . . .

31

2.2.2
2.2.3

Ignoring the Green’s function on non-admissible domains . . .
Hierarchical admissible partition of domain . . . . . . . . . . .

34
35

2.2.4

Recovering the Green’s function on the entire domain . . . . .

36Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

39

2.3.1
2.3.2

Fast and stable reconstruction of hierarchical matrices . . . .
Extension to other partial differential operators . . . . . . . .

39
40

2.3.3

Connection with neural networks . . . . . . . . . . . . . . . .

41i

3 A generalization of the randomized singular value decomposition

43

3.1

Theoretical bounds for non-standard covariance matrices . . . . . . .

44

3.2
3.3

Randomized SVD for Hilbert–Schmidt operators . . . . . . . . . . . .
Covariance kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . .

49
50

3.3.1

Sample random functions from a Gaussian process . . . . . . .

51

3.3.2

Influence of the kernel’s eigenvalues and Mercer’s representation 52

3.3.3
3.3.4

Jacobi covariance kernel . . . . . . . . . . . . . . . . . . . . .
Smoothness of functions sampled from a GP with Jacobi kernel

52
53

Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . .

56

3.4.1

Covariance matrix with prior knowledge . . . . . . . . . . . .

56

3.4.2

Randomized SVD for Hilbert–Schmidt operators . . . . . . . .

59

3.4

4 Rational neural networks

62

4.1

Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63

4.2

Theoretical results on rational neural networks . . . . . . . . . . . . .

64

4.2.1
4.2.2

65
74

4.3

Approximation of ReLU networks by rational neural networks
Approximation of functions by rational networks . . . . . . . .

Experiments using rational neural networks. . . . . . . . . . . . . .

80

4.3.1

Approximation of functions . . . . . . . . . . . . . . . . . . .

81

4.3.2

Generative adversarial networks . . . . . . . . . . . . . . . . .

85

5 Data-driven discovery of Green’s functions with deep learning
5.1

5.2

5.3

89

Learning Green’s functions . . . . . . . . . . . . . . . . . . . . . . . .

90

5.1.1

Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

91

5.1.2 Theoretical justification . . . . . . . . . . . . . . . . . . . . .
Deep learning method . . . . . . . . . . . . . . . . . . . . . . . . . .

92
92

5.2.1

Generating the training data . . . . . . . . . . . . . . . . . . .

94

5.2.2

Rational neural networks . . . . . . . . . . . . . . . . . . . . .

95

5.2.3
5.2.4

Loss function . . . . . . . . . . . . . . . . . . . . . . . . . . .
Optimization algorithm . . . . . . . . . . . . . . . . . . . . . .

97
98

5.2.5

Measuring the results . . . . . . . . . . . . . . . . . . . . . . .

99

Robustness of the method . . . . . . . . . . . . . . . . . . . . . . . . 100
5.3.1
5.3.2

Influence of the activation function on the accuracy . . . . . . 100
Number of training pairs and spatial measurements . . . . . . 101

5.3.3

Noise perturbation . . . . . . . . . . . . . . . . . . . . . . . . 103

5.3.4

Location of the measurements . . . . . . . . . . . . . . . . . . 103

5.3.5

Missing measurements data . . . . . . . . . . . . . . . . . . . 104
ii

5.4

5.5

Human-understandable features . . . . . . . . . . . . . . . . . . . . . 104
5.4.1

Linear constraints and symmetries

5.4.2
5.4.3

Eigenvalue decomposition . . . . . . . . . . . . . . . . . . . . 107
Singular value decomposition . . . . . . . . . . . . . . . . . . 110

5.4.4

Schrödinger equation with double-well potential . . . . . . . . 112

5.4.5

Singularity location and type . . . . . . . . . . . . . . . . . . 112

Viscous shock and multiphysics examples . . . . . . . . . . . . . . . . 114
5.5.1 Viscous shock . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.5.2

5.6

. . . . . . . . . . . . . . . . 117

Differential operators in two dimensions . . . . . . . . . . . . 117
System of differential equations . . . . . . . . . . . . . . . . . 120

Nonlinear and vector-valued equations . . . . . . . . . . . . . . . . . 121
5.7.1

5.8

Advection-diffusion operator . . . . . . . . . . . . . . . . . . . 117

Two-dimensional operators and systems
5.6.1
5.6.2

5.7

. . . . . . . . . . . . . . . 106

Linearized models of nonlinear operators . . . . . . . . . . . . 121

5.7.2 Lid-driven cavity problem . . . . . . . . . . . . . . . . . . . . 123
Time-dependent equations . . . . . . . . . . . . . . . . . . . . . . . . 126

Conclusions

129

Bibliography

132

iii

Chapter 1
Introduction
This thesis aims at understanding whether partial differential equations (PDEs) canbe discovered from data by connecting standard mathematical fields, such as numerical linear algebra, probability, and PDE analysis, with modern deep learningtechniques. We focus on learning Green’s functions associated with linear PDEs frompairs of forcing functions and solutions. Theoretical bounds exploiting the regularityof the problem are derived and a practical deep learning algorithm is proposed.
Chapter 2 derives a theoretically-rigorous scheme for learning Green’s functions
associated with elliptic PDEs in three dimensions, given input-output pairs. A learning rate is obtained, giving a bound on the number of training pairs needed to learn
a Green’s function to within a prescribed accuracy with high probability. Along theway, the randomized singular value decomposition (SVD) is extended from matrices
to Hilbert–Schmidt (HS) operators, and a quantity is introduced to measure the quality of the training forcing terms to learn Green’s functions. The randomized SVD isa popular and effective algorithm for computing a near-best rank k approximation ofa matrix using matrix-vector products with standard Gaussian vectors.
Chapter 3 extends the randomized SVD to multivariate Gaussian vectors, allowing one to incorporate prior knowledge of the matrix into the algorithm. This enables
us to explore the continuous analogue of the randomized SVD for HS operators using
operator-function products with functions drawn from a Gaussian process (GP). A
new covariance kernel for GPs, based on weighted Jacobi polynomials, is constructed
to rapidly sample the GP and control the smoothness of the randomly generatedfunctions. Numerical examples on matrices and HS operators demonstrate the applicability of the algorithm.
Chapter 4 considers neural networks with rational activation functions. The choiceof the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds in1

terms of network complexity and prove that rational neural networks approximate
smooth functions more efficiently than networks with Rectified Linear Unit (ReLU)
activation functions with exponentially smaller depth. The flexibility and smoothness of rational activation functions make them an attractive alternative to ReLU, asdemonstrated by numerical experiments.
Chapter 5 develops a data-driven approach for learning Green’s functions using
deep learning. By collecting physical system responses under excitations drawn froma Gaussian process, we train rational neural networks to learn Green’s functions ofhidden linear PDEs. These functions reveal human-understandable properties andfeatures, such as linear conservation laws and symmetries, along with shock andsingularity locations, boundary effects, and dominant modes. The technique is illustrated on several examples and allows us to capture a range of physics, including
advection-diffusion, viscous shocks, and Stokes flow in a lid-driven cavity.

1.1

Deep learning

Deep learning has become an important topic across many domains of science due to
its recent successes in image recognition, speech recognition, and drug discovery [89,114, 118, 138]. Deep learning techniques are based on objects called artificial neuralnetworks (NNs), which apply a succession of mathematical transformations on an
input variable x to output a variable y, where N (x) = y and N denotes the neuralnetwork. An example of simple data fitting task is to assign labels 0 or 1 to points
in R2 , where, in this case, x ∈ R2 and y ∈ {0, 1} [88]. A large number of NN

architectures, characterized by the type of mathematical operations used, have been
proposed over the past decades for performing different tasks, such as convolutional
NNs for classifying images [114, 119], recurrent and long short-term memory neuralnetworks for speech recognition [78, 90, 196], and generative adversarial network to
generate realistic images [76, 103].
We consider one of the most standard types of deep learning model called feedforward neural networks or multilayer perceptrons [75, Chapt. 6]. Let L ≥ 1 be an
integer and n1 , nL be the respective dimension of the input and output data. A feed-

forward network N : Rn1 → RnL , mapping from Rn1 to RnL , with L layers consists ofa composition of L − 1 functions f1 , . . . , fL−1 of the form
N (x) = fL−1 ◦ · · · ◦ f1 (x),2x ∈ Rn1 .

At a given layer 1 ≤ i ≤ L − 1, the nonlinear transformation fi : Rni → Rni+1

determines the output of the neural network. The layers 2 ≤ i ≤ L − 1 are called thehidden layers of the network and their dimensionality determines the width of the
network, while the number of layers is referred to as the depth [75, Chapt. 6]. For
1 ≤ i ≤ L − 1, we choose the function fi to be of the formfi : x 7→ σ(Wi x + bi ),x ∈ Rni ,where Wi ∈ Rni ×ni+1 is a matrix called the weight matrix, bi ∈ Rni+1 is a bias vector,and σ is a nonlinear function called the activation function (also called activationunit). The weight matrices and bias vectors are trainable parameters of the network,and their coefficients are usually obtained using a gradient-based optimization algorithm, such as stochastic gradient descent, applied to a training dataset containing
examples of inputs and expected associated outputs of the network [75, Chapt. 6.2].
In this thesis, we will measure the network complexity using its total number of parameters (i.e., size) and number of layers (depth), which are standard measures intheoretical deep learning [8].
In Chapter 4, we will consider NNs with rational activation functions and derivetheoretical results that quantify the size needed to approximate smooth functions
within a prescribed accuracy. We will establish a connection between standard approximation theory for rational functions and the highly compositional structure ofneural networks to show that rational neural networks require fewer parameters than
ReLU networks to approximate smooth functions. We expect that the smoothnessof rational neural networks away from their poles, together with their potential singularities, make them an interesting alternative to standard activation functions forphysics-informed machine learning applications.

1.2

Physics-informed machine learning

Over the past decades, there has been spectacular progress in numerical techniques
for solving PDEs, such as finite element methods, finite differences, and spectral
methods [102]. However, solving inverse problems to identify parameters of a model orlearn a physical model from real-world data remains highly challenging due to missing
and noisy data [11, 209]. Hence, such problems are often ill-posed and require a datadriven approach. Recently, the fields of numerical analysis and machine learning havesuccessfully converged towards physics-informed machine learning, which integrates
partial data and prior knowledge on governing physical laws to solve inverse problems
3

using neural networks [102]. The flexibility of the networks, due to the large potential
choices of architectures, along with their generalization ability in the presence of big
data, either generated by numerical simulations or acquired via experiments, makes
them ideal for such tasks. On the other hand, the selection of a specific architecture
is a challenging task and is difficult to justify mathematically due to the complexityof the models.
One example of problems that can be tackled by deep learning is to solve a PDE
by training a NN on initial and boundary training data. Two popular approaches are
physics-informed neural networks (PINNs) [184] and the deep Galerkin method [203],
which, in their original formulation, aim to solve PDEs of the form∂u
+ L(u) = 0,∂t

x ∈ D ⊂ Rd ,t ∈ [0, T ],(1.1)

where the partial differential operator L is potentially nonlinear. The left-hand sideof Equation (1.1) is denoted by f (x, t), i.e., f := ut + L(u). These techniques are

attractive because they are mesh-free as they do not require a spatial discretizationof the domain and can be applied in high dimensions. The PINN approach consistsof approximating the solution u to Equation (1.1) by a neural network. This results

in a physics-informed neural network f , which can be evaluated using chain ruleand automatic differentiation [14, 184]. The loss function is expressed as a sum of asupervised loss of data measurements at the boundary and an unsupervised loss of
PDE [102, 184]:
Loss = wdata Ldata + wPDE LPDE ,where wdata and wPDE are weights balancing the two terms and Ldata , LPDE are definedas

Ldata =

1

N
data
X

Ndata i=1

bdr
bdr 2
|u(xbdri , ti ) − ui | ,LPDE =1N
PDE
X

NPDE j=1|f (xdom, tdom
)|2 .
j
j

bdr
Here, {(xbdr
i , ti )} are points sampled at the initial and boundary locations, while

the points {(xdom, tdom)} are sampled on the entire domain, and LPDE is the averagejj, tdom)}. These methods haveof the squared residual of the PDE evaluated at {(xdomjjbeen generalized since their introductions to tackle a wide range of PDEs such as
integro-differential equations [136], fractional PDEs [169], and stochastic PDEs [244],
and have been applied to problems in fluid mechanics [185], geophysics [124], andmaterials science [202].

4

This thesis focuses on another aspect of physics-informed machine learning called
PDE learning, whose aim is to discover, or learn, a mathematical model from data.
We consider stationary PDEs of the form:
L(u) = f,where L is a partial differential operator, f is called the forcing term, and u the associated solution of the PDE. The approaches that dominate the PDE learning literature
focus on the “forward” problem and aim to discover properties of the differential
operator L. As an example, sparsity-promoting techniques [36, 195, 246] consist ofbuilding a library of states u and its spatio-temporal derivatives ut , ux , uxx , uy , . . . to
identify parameters (or coefficients) and discover the main contributing terms in L.

Another method aims to find a symbolic expression for L and identify its dominantcoefficients by solving a regression problem [224, 225]. Finally, one can also project
the operator L onto a low-dimensional subspace to build a reduced-order model andto significantly speed up standard numerical solvers [177, 178].
An alternative approach, which we will consider, is to study the “inverse” problem
and directly approximate the PDE solution operator, L−1 : f 7→ u, by an artificial

neural network N from training pairs of forcing terms and solutions {fj , uj }N
j=1 [71,
112, 126, 127, 128, 135, 233]. The network N takes a forcing term f evaluated at aNffinite number of sensors {yi }i=1
and a point x in the domain of L−1 (f ) and outputs

a real number approximating the solution u to the PDE L(u) = f evaluated at x:⊤ 
N f (y1 ) · · · f (yNf ) , x ≈ u(x).

The NN is then trained by minimizing the following loss function using stochastic
gradient descent algorithms:
Nf
N X
Nu X

X
1
Loss =
N fk (y1 ) · · ·N Nu Nf k=1 i=1 j=1

2
⊤ 
fk (yNf ) , xi − uk (xi ) ,uwhere {xi }N
i=1 are spatial points at which the solutions are measured. Unlike coefficient discovery techniques, this approach provides a fast solver for PDEs, which may

outperform state-of-the-art numerical solvers [127]. However, the physical interpretation of the learned solution operator remains highly challenging due to the mathematical complexity of the neural network that approximates it. Several black-box
deep learning techniques are proposed to approximate the solution operator, whichmaps forcing terms f to observations of the associated system’s responses u such5that L(u) = f . These methods are based on the concept of neural operators [112],

which generalize neural networks to learn maps between infinite-dimensional function

spaces, and mainly differ in their choice of the neural network architecture that isused to approximate the solution map. For example, Fourier neural operator [127]uses a Fourier transform at each layer, while DeepONet [135] contains a concatenationof ‘trunk’ and ‘branch’ networks to enforce additional structure.
On the theoretical side, most of the research has focused on the approximation
theory of infinite-dimensional operators by NNs, such as the generalization of the universal approximation theorem [46] to shallow and deep NNs [38, 135] as well as error
estimates for Fourier neural operators and DeepONets with respect to the network
width and depth [111, 112, 116]. Other approaches aim to approximate the matrixof the discretized Green’s functions associated with elliptic PDEs from matrix-vector
multiplications by exploiting sparsity patterns or hierarchical structure of the matrix [130, 198]. In addition, [49] derived convergence rates for learning linear selfadjoint operators based on the assumption that the target operator is diagonal in thebasis of the Gaussian prior.
In this thesis, we focus on learning linear partial differential operators L for which

the solution operator can be written as an integral operator,−1

L (f )(x) =G(x, y)f (y) dy = u(x),Dwhose kernel G is known as the Green’s function. Our approach contrasts with prior
works because we aim to approximate the Green’s function instead of the integral
operator. As we will see in Chapters 2 and 5, imposing a prior structure on thesolution operator offers theoretical and practical advantages over recent PDE learning techniques. First, standard mathematical techniques from elliptic PDE theory
and numerical analysis can be exploited to derive rigorous results that quantify theamount of training data needed to learn the solution operator to within a prescribedaccuracy. These types of results are notoriously challenging to obtain for deep learning algorithms due to the high nonlinearity of neural network architectures and thecomplexity of the optimization procedure. Secondly, unlike black-box deep learningtechniques, it is possible to extract physical features of the original PDE from theassociated Green’s function, which is a well-understood mathematical object.

6

1.3

Green’s functions

Throughout this thesis, we consider linear boundary value problems defined on abounded domain D ⊂ Rd , with d ≥ 1, of the form:
Lu = f, in D,
u = 0,

on ∂D,

where L is a linear partial differential operator and f : D → R is a given forcing function. A typical example of such problems is the Poisson equation in one dimension:
−

d2 u
= f,
dx2

x ∈ (0, 1),u(0) = u(1) = 0.

(1.3)

Equation (1.3) can be solved for any forcing function f by introducing a kernel G :[0, 1] × [0, 1] → R so that the solution u can be expressed as the following integral [64,79, 190],1u(x) =G(x, y)f (y) dy,0x ∈ [0, 1].

(1.4)

The function G is called the Green’s function and is a solution to the equation
LG(x, y) = δ(x − y), where δ is the Dirac delta function and x, y ∈ [0, 1].
Green’s functions are useful because they are independent of the forcing terms andonly characterize the partial differential operators and boundary conditions. Once theGreen’s function has been determined, then the solution to Equation (1.3) with any
forcing term can be obtained by computing the integral in Equation (1.4), which isnumerically easier than solving the original PDE and imposing the appropriate boundary conditions [190]. Additionally, several properties of the PDE can be recoveredfrom the Green’s function, such as symmetries or eigenvalues.
Traditional methods for finding Green’s functions can be summarized as deriving
analytical formulas, computing eigenvalue expansions, or numerically solving a singular PDE [64, 190]. This is difficult when the geometry of the domain is complex orwhen the PDE has variable coefficients. Moreover, it requires knowledge of the partial
differential operator, which may not be accessible in real applications [102]. Other
works study properties of Green’s functions and provide theoretical results such as
decay bounds along the diagonal of the domain [42, 80, 91, 100] or low-rank structure
on separable domains [16, 28, 63].
In this thesis, we aim to approximate Green’s functions from pairs of forcing
terms and system’s responses {(fj , uj )}N
j=1 by exploiting their low-rank structure on

well-separated domains [16], and combining it with randomized numerical linear algebra [86].
7

1.4

Low-rank approximation

Let A be an m × n real matrix with m ≥ n and k ≤ n be an integer. The best rank

k approximation to A in the Frobenius norm is the m × n real matrix Ak , which issolution to the following minimization problem:min

Ak ∈Rm×n∥A − Ak ∥F

subject to rank(Ak ) ≤ k,where ∥ · ∥F denotes the Frobenius norm defined as ∥A∥F =

(1.5)

p
Tr(AA∗ ). The Eckart–

Young theorem [58] states that (1.5) has a unique solution given by the truncationof the singular value decomposition of A to the kth term. The SVD of an m × nreal matrix A, with m ≥ n, is a factorization of the form A = UΣV∗ , where U isan m × m orthogonal matrix of left singular vectors, Σ is an m × n diagonal matrix
with entries σ1 (A) ≥ · · · ≥ σn (A) ≥ 0, and V is an n × n orthogonal matrix of rightsingular vectors [74]. Then,min

Ak ∈Rm×nrank(Ak )≤k∥A − Ak ∥F =

where
Ak =

k
X

n
X

σj (A)2

j=k+1

!1/2

,

σj (A)uj vj∗ .

j=1Here, uj and vj denote the jth column of U and V, respectively.
This result can be generalized to functions [200, 216] and, in particular, Green’s
functions of the form G : D1 × D2 → R, where D1 , D2 ⊂ Rd . As an example, if

D1 = [a, b] and D2 = [c, d] are two real intervals, and G is square-integrable, then it
can be written as the following infinite series, which converges in the L2 (D1 × D2 )sense to G,

G(x, y) =∞X

σj uj (x)vj (y),

j=1
σj >0

x ∈ D1 , y ∈ D2 ,where {uj }j≥1 and {vj }j≥1 form an orthonormal basis of L2 (D1 ) and L2 (D2 ), andσ1 ≥ σ2 ≥ · · · ≥ 0 are called the singular values of G. This series is referred to as
the SVD of G. Similar to matrices, the best rank k approximant to G is obtained bytruncating its SVD after k terms to obtain a separable approximation
Gk (x, y) =kXσj uj (x)vj (y),j=18x ∈ D1 , y ∈ D2 .

By the Eckart–Young theorem, Gk is solution to the following minimization problem:
!1/2
k
∞
X
X
min
∥G −
fj gj ∥L2 (D1 ×D2 ) = ∥G − Gk ∥L2 (D1 ×D2 ) =σj2.
2fj ∈L (D1 )gj ∈L2 (D2 )

j=1

j=k+1

Let 0 < ϵ < 1. If there exists an integer k > 0 and a separable expression satisfying
∥G −

k
X
j=1

fj gj ∥L2 (D1 ×D2 ) ≤ ϵ∥G∥L2 (D1 ×D2 ) ,fj ∈ L2 (D1 ), gj ∈ L2 (D2 ),then we say that G has numerical rank smaller than k. We remark that one can easily
obtain a bound on the tail of the singular values of G by applying the Eckart–Youngtheorem as follows,
!1/2
∞
X
σj2
=
j=k+1

min
2

fj ∈L (D1 )gj ∈L2 (D2 )

∥G −

k
X
j=1

fj gj ∥L2 (D1 ×D2 ) ≤ ϵ∥G∥L2 (D1 ×D2 ) .

When k = O(logδ (1/ϵ)) for some small δ ∈ N as ϵ → 0, then we say that G has
exponentially decaying singular values on D1 × D2 .

1.5

Randomized singular value decomposition

Computing the SVD of a matrix is a fundamental linear algebra task in machine learning [173], statistics [240], and signal processing [7, 227]. As we saw in Section 1.4,
the SVD plays a central role in numerical linear algebra because truncating it after
k terms provides the best rank k approximation to A in the spectral and Frobenius
norms [58, 151]. Since computing the SVD of a large matrix can be computationallyinfeasible, there are various principal component analysis (PCA) [2, 92, 174] algorithms that perform dimensionality reduction by computing near-best rank k matrix
approximations from matrix-vector products [86, 145, 159, 164, 238]. The randomized
SVD uses matrix-vector products with random test vectors and is one of the mostpopular algorithms for constructing a low-rank approximation to A [86, 145]. Whilethe error analysis performed in [86] for the randomized SVD uses standard Gaussian
random vectors, other random embedding techniques have been considered such as
random permutations [5], sparse sign matrices [44, 147, 162, 226], and subsampled
randomized trigonometric transforms (SRTTs) [4, 5, 172, 241] to mitigate the computational cost of Gaussian vectors in practical applications. Throughout this thesis,
we will focus on Gaussian vectors because they yield a more precise error analysis(cf. [145, Sec. 8.3]).
9

First, one performs the matrix-vector products y1 = Ax1 , . . . , yk+p = Axk+p ,where x1 , . . . , xk+p are standard Gaussian random vectors with identically and independently distributed entries and p ≥ 1 is an oversampling parameter. Then, one
computes the economized QR factorization y1 · · · yk+p = QR, before forming
the rank ≤ k + p approximant QQ∗ A. Note that if A is symmetric, one can form
QQ∗ A by computing Q(AQ)∗ via matrix-vector products involving A; otherwise it

requires the adjoint A∗ . The quality of the rank ≤ k + p approximant QQ∗ A is
characterized by the following bound for u, t ≥ 1 [86, Thm. 10.7],
s
!v
√
uX
u n
3k
k+p
∗
2
t
∥A − QQ A∥F ≤ 1 + t
σj (A) + ut
σk+1 (A),
(1.6)
p+1
p
+
1
j=k+1
2

with failureqprobability at most 2t−p + e−u . The squared tail of the singular values
Pn
2of A, i.e.,
j=k+1 σj (A), gives the best rank k approximation error to A in the

Frobenius norm. This result shows that the randomized SVD can compute a nearbest low-rank approximation to A with high probability. In Chapters 2 and 3, we
will generalize this result to random vectors sampled from a multivariate normal
distribution with any covariance matrix, and Hilbert–Schmidt operators.

1.6

Hilbert–Schmidt operators

Hilbert–Schmidt operators generalize the notion of matrices acting on vectors to infinite dimensions with linear operators acting on functions [93, Ch. 4]. First, let
D1 , D2 ⊂ Rd be two domains with d ≥ 1. For 1 ≤ p ≤ ∞, we denote by Lp (D1 ) the
space of measurable functions defined on the domain D1 with finite Lp norm, where∥f ∥Lp (D1 ) =



p

D1

|f (x)| dx

1/p

if p < ∞,∥f ∥L∞ (D1 ) = inf {C > 0, |f (x)| ≤ C for almost every x ∈ D1 } .
Since the space of square-integrable functions, L2 (D1 ), is a separable Hilbert space,it admits a complete orthonormal basis {ej }∞
j=1 .
2
2
A linear operator F : L (D1 ) → L (D2 ) is an HS operator [93, Def. 4.4.2] if it
has finite HS norm, ∥F ∥HS , defined as∥F ∥HS :=∞X
j=1∥F ej ∥2L2 (D2 )

10

!1/2

< ∞.

This norm does not depend on the choice of the basis [93, Thm. 4.4.1]. The archetypical example of an HS operator is an integral operator F : L2 (D1 ) → L2 (D2 ) defined

asG(x, y)f (y) dy,(F f )(x) =D1

f ∈ L2 (D1 ), x ∈ D2 ,where G ∈ L (D2 × D1 ) is the kernel of F and ∥F ∥HS = ∥G∥L2 (D2 ×D1 ) . The adjoint
2

operator F ∗ : L2 (D2 ) → L2 (D1 ) is defined as∗(F g)(y) =G(x, y)g(x) dx,
D2g ∈ L2 (D2 ), y ∈ D1 .

Since HS operators are compact operators, they have an SVD [93, Thm. 4.3.1].
That is, that for any f ∈ L2 (D1 ) we have
Ff =

∞
X
j=1

σj ⟨q1j , f ⟩q2j ,(1.7)

where the equality holds in the L2 (D2 ) sense. Here, σ1 ≥ σ2 ≥ · · · ≥ 0 denotethe square roots of the eigenvalues of the self-adjoint operator F ∗ F , {q1j } are theorthonormal eigenvectors of F ∗ F , and {q2j } are the orthonormal eigenvectors of

F F ∗ . We refer to {(σj , q1j , q2j )}∞j=1 as the singular system of F . When the HS
operator is an integral operator, we refer to its singular values as the singular valuesof the underlying kernel.
Moreover, one finds that ∥F ∥2HS =P∞

2
j=1 σj , which shows that the HS norm is an

infinite dimensional analogue of the Frobenius matrix norm ∥ · ∥F . In the same waythat truncating the SVD after k terms gives the best rank k matrix approximation,
truncating Equation (1.7) gives the best rank k approximation in the HS norm. Thatis, [93, Thm. 4.4.7]
minuj ∈L2 (D1 ),vj ∈L2 (D2 )

k
X
∥F −
⟨uj , ·⟩vj ∥HS = ∥F − Fk ∥HS =
j=1

∞
X

j=k+1

σj2

!1/2

,

where the operator Fk is defined as
Fk f =kX
j=1

σj ⟨q1j , f ⟩q2j ,f ∈ L2 (D1 ).

This result is known as the Eckart–Young–Mirsky theorem [58, 151]. We will exploit
this theorem in Chapter 2 to extend the randomized SVD to HS operators and learn
Green’s functions.
11

1.7

Quasimatrices

Quasimatrices are an infinite dimensional analogue of tall-skinny matrices [218]. Let
D1 , D2 ⊆ Rd be two domains with d ≥ 1, we say that Ω is a D1 × k quasimatrix, ifΩ is a matrix with k columns where each column is a function in L2 (D1 ). That is,Ω = ω1 | · · · | ωk ,

ωj ∈ L2 (D1 ).

Quasimatrices are useful to define analogues of matrix operations for HS operators [48,208, 218, 221]. For example, if F : L2 (D1 ) → L2 (D2 ) is an HS operator, then we

write F Ω to denote the quasimatrix obtained by applying F to each column of Ω.

Moreover, we write Ω∗ Ω and ΩΩ∗ to mean the following:⟨ω1 , ω1 ⟩ · · · ⟨ω1 , ωk ⟩
k
X


..
∗
...
Ω∗ Ω =  ...
ωj (x)ωj (y),
,
ΩΩ
=

.
j=1
⟨ωk , ω1 ⟩ · · · ⟨ωk , ωk ⟩where ⟨·, ·⟩ is the L2 (D1 ) inner-product. Many operations for rectangular matrices

in linear algebra can be generalized to quasimatrices such as the SVD, QR, LU, andCholesky factorizations [218].
Throughout this thesis, the HS operator denoted by ΩΩ∗ F : L2 (D1 ) → L2 (D2 )P
is given by ΩΩ∗ F f = kj=1 ⟨ωj , F f ⟩ωj . Moreover, if Ω has full column rank thenPΩ F := Ω(Ω∗ Ω)† Ω∗ F is the orthogonal projection of the range of F onto thecolumn space of Ω. Here, (Ω∗ Ω)† is the pseudo-inverse of Ω∗ Ω. This notation isconvenient to state the generalization of the randomized SVD in infinite dimensions.

1.8

Gaussian processes

A Gaussian process is an infinite dimensional analogue of a multivariate Gaussian
distribution and a function drawn from a GP is analogous to a randomly generated
vector. If K : D × D → R is a continuous symmetric positive semi-definite kernel,

where D ⊆ Rd is a domain, then a GP is a stochastic process {Xt , t ∈ D} such that forevery finite set of indices t1 , . . . , tn ∈ D the vector of random variables (Xt1 , . . . , Xtn )is a multivariate Gaussian distribution with mean (0, . . . , 0) and covariance Kij =K(ti , tj ) for 1 ≤ i, j ≤ n. We denote a GP with mean (0, . . . , 0) and covariance kernel
K by GP(0, K).

Since K is a continuous symmetric positive semi-definite kernel, it has nonnegative
eigenvalues λ1 ≥ λ2 ≥ · · · ≥ 0 and there is an orthonormal basis of eigenfunctions12

2{ψj }∞
j=1 of L (D) such that [93, Thm. 4.6.5]:

K(x, y) =∞X

K(x, y)ψj (y) dy = λj ψj (x),λj ψj (x)ψj (y),D

j=1x, y ∈ D,(1.8)

where the infinite sum is absolutely and uniformly convergent [148]. Note that the
eigenvalues of K are the ones of the integral operator with kernel K. In addition,P
we define the trace of the covariance kernel K by Tr(K) := ∞j=1 λj < ∞. The

eigendecomposition of K gives an algorithm for sampling functions from GP(0, K).
In particular, if

ω=

∞
X
p
λj cj ψj ,j=1

where the coefficients {cj }∞j=1 are independent and identically distributed (i.i.d.) stan-

dard Gaussian random variables and the series converges in mean-square and uniformly, then ω ∼ GP(0, K). This is known as the Karhunen–Loève theorem [101, 133].
We also have [93, Thm. 7.2.5]

∞
∞
h
i X
X
 2
2
2
E ∥ω∥L2 (D) =λj E cj ∥ψj ∥L2 (D) =λj =K(y, y) dy < ∞,
j=1

j=1

D

where the last equality is analogous to the fact that the trace of a matrix is equal
to the sum of its eigenvalues. In this thesis, we restrict our attention to GPs withpositive definite covariance kernels so that the eigenvalues of K are strictly positive.

1.9

Contribution

The material of Chapter 2 to Chapter 5 is based on the following four papers withcollaborators:• Learning elliptic PDEs with randomized linear algebra
Nicolas Boullé and Alex Townsend
Foundations of Computational Mathematics, 2022
• A generalization of the randomized singular value decomposition
Nicolas Boullé and Alex Townsend
International Conference on Learning Representations, 2022
• Rational neural networks
Nicolas Boullé, Yuji Nakatsukasa, and Alex Townsend
Neural Information Processing Systems, 202013• Data-driven discovery of Green’s functions with human-understandable
deep learning
Nicolas Boullé, Christopher J. Earls, and Alex Townsend
Scientific Reports, 2022
My co-authors had advisory roles; I proved the main theoretical results, performed
the numerical experiments, and was the lead author in writing the papers.

14

Chapter 2
Learning elliptic PDEs withrandomized linear algebra∗
Can one learn a differential operator from pairs of solutions and righthand sides?
If so, how many pairs are required? These two questions have received significant
research attention [65, 127, 134, 170]. From data, one hopes to eventually learn
physical laws of nature or conservation laws that elude scientists in the biological sciences [243], computational fluid dynamics [185], and computational physics [180]. The
literature contains many highly successful practical schemes based on deep learningtechniques [146, 184]. However, the challenge remains to understand when and why
deep learning is effective theoretically. This chapter describes the first theoreticallyjustified scheme for discovering scalar-valued elliptic partial differential equations(PDEs) in three variables from input-output data and provides a rigorous learning
rate. While our novelties are mainly theoretical, we hope to motivate future practicalchoices in PDE learning.
Let D ⊂ R3 be a bounded domain with Lipschitz smooth boundary, L2 (D) be
the space of square-integrable functions defined on D, Hk (D) be the space of k times

weakly differentiable functions in the L2 -sense, and H01 (D) be the closure of Cc∞ (D)in H1 (D). Here, Cc∞ (D) is the space of infinitely differentiable compactly supported

functions on D. Roughly speaking, H01 (D) are the functions in H1 (D) that are zeroon the boundary of D. We suppose that there is an unknown second-order uniformly
elliptic linear PDE operator L : H2 (D) ∩ H01 (D) → L2 (D) [64], which takes the form
Lu(x) = −∇ · (A(x)∇u) + c(x) · ∇u + d(x)u,∗x ∈ D,u|∂D = 0.

(2.1)

This chapter is based on a paper with Alex Townsend [32], published in Foundations of Computational Mathematics. Townsend had an advisory role; I proved the theoretical results and was
the lead author in writing the paper.

15

Here, for every x ∈ D, we have that A(x) ∈ R3×3 is a symmetric positive definite

matrix with bounded coefficient functions so that Aij ∈ L∞ (D), c ∈ Lr (D) withr ≥ 3, d ∈ Ls (D) for s ≥ 3/2, and d(x) ≥ 0 [106]. We emphasize that the regularity
requirements on the variable coefficients are quite weak.
The goal of PDE learning is to discover the operator L from N ≥ 1 input-output

pairs, i.e., {(fj , uj )}N
j=1 , where Luj = fj and uj |∂D = 0 for 1 ≤ j ≤ N . There are
two main types of PDE learning tasks: (1) Experimentally-determined input-output
pairs, where one must do the best one can with the predetermined information and(2) Algorithmically-determined input-output pairs, where the data-driven learningalgorithm can select f1 , . . . , fN for itself. In this chapter, we focus on the PDE
learning task where we have algorithmically-determined input-output pairs and aim
to provide an upper bound on the sample complexity of the Green’s function G
associated with L, i.e. characterize the number of pairs needed to learn G withina prescribed accuracy. In particular, we suppose that the functions f1 , . . . , fN are

generated at random and are drawn from a Gaussian process (GP) (see Section 1.8).
Note that alternative strategies analogue to a power scheme in randomized numericallinear algebra [74, 86, 191, 192] to generate forcing terms iteratively might lead tobetter approximation errors. To keep our theoretical statements manageable, we
restrict our attention to PDEs of the form:
Lu = −∇ · (A(x)∇u) ,x ∈ D,u|∂D = 0.

(2.2)

Lower-order terms in Equation (2.1) should cause few theoretical problems [16],though our algorithm and our bounds get far more complicated.
The approach that dominates the PDE learning literature is to directly learn L byeither (1) learning parameters in the PDE [24, 247], (2) using neural networks (NNs)to approximate the action of the PDE on functions [180, 182, 183, 184, 185], or (3)deriving a model from a library of operators via sparsity considerations [36, 141, 195,
197, 231, 234]. Instead of trying to learn the unbounded, closed operator L directly,

we follow [27, 65, 71] and discover the Green’s function associated with L. That is,
we attempt to learn the function G : D × D → R+ ∪ {∞} such that [64]

uj (x) =G(x, y)fj (y) dy,Dx ∈ D,

1 ≤ j ≤ N.

(2.3)

Seeking G, as opposed to L, has several theoretical benefits:
1. The integral operator in Equation (2.3) is compact [60], while L is only closed [59].
This allows G to be rigorously learned by input-output pairs {(fj , uj )}N
j=1 , as its

range can be approximated by finite-dimensional spaces (see Theorem 2.3).
16

2. It is known that G has a hierarchical low-rank structure [16, Thm. 2.8]: for 0 <P
ϵ < 1, there exists a function Gk (x, y) = kj=1 gj (x)hj (y) with k = O(log4 (1/ϵ))such that [16, Thm. 2.8]

∥G − Gk ∥L2 (X×Y ) ≤ ϵ ∥G∥L2 (X×Ŷ ) ,where X, Y ⊆ D are sufficiently separated domains, and Y ⊆ Ŷ ⊆ D denotesa larger domain than Y (see Theorem 2.4 for the definition). The further apart
X and Y , the faster the singular values of G decay. Moreover, G also has an
off-diagonal decay property [80, 100]:G(x, y) ≤c∥G∥L2 (D×D) ,∥x − y∥2x ̸= y, x ∈ D, y ∈ D,

where c is a constant independent of x and y. Exploiting these structures of G
leads to a rigorous algorithm for constructing a global approximant to G (see Section 2.2).
3. The function G is smooth away from its diagonal, allowing one to efficiently approximate it [80].
Once a global approximation G̃ has been constructed for G using input-output pairs,
given a new righthand side f one can directly compute the integral in Equation (2.3)
to obtain the corresponding solution u to Equation (2.1). Usually, numerically computing the integral in Equation (2.3) must be done with sufficient care as G possesses
a singularity when x = y. However, our global approximation G̃ has a hierarchicalstructure and is constructed as 0 near the diagonal. Therefore, for each fixed x ∈ D,

we simply recommend that D G̃(x, y)fj (y) dy is partitioned into the panels that corresponds to the hierarchical decomposition, and then discretized each panel with aquadrature rule.

There are two main contributions in this chapter: (1) the generalization of therandomized singular value decomposition (SVD) algorithm for learning matrices frommatrix-vector products to Hilbert–Schmidt (HS) operators and (2) a theoretical learning rate for discovering Green’s functions associated with PDEs of the form Equation (2.2). These contributions are summarized in Theorems 2.1 and 2.3.
Theorem 2.1 says that, with high probability, one can recover a near-best rank k
HS operator using k+p operator-function products, for a small integer p. In the boundof the theorem, a quantity, denoted by 0 < γk ≤ 1, measures the quality of the input-

output training pairs (see Sections 2.1.1 and 2.1.4). We then combine Theorem 2.1

17

with the theory of Green’s functions for elliptic PDEs to derive a theoretical learning
rate for PDEs.
In Theorem 2.3, we show that Green’s functions associated with uniformly elliptic
PDEs in three dimensions can be recovered using N = O(ϵ−6 log4 (1/ϵ)) input-output−1/2

pairs (fj , uj )N
j=1 to within an accuracy of O(Γϵlog3 (1/ϵ)ϵ) with high probability, for

0 < ϵ < 1. Our learning rate associated with uniformly elliptic PDEs in three variables
is therefore O(ϵ−6 log4 (1/ϵ)). The quantity 0 < Γϵ ≤ 1 (defined in Equation (2.23))
measures the quality of the GP used to generate the random functions {fj }N
j=1 forlearning G. We emphasize that the number of training pairs is small only if the GP’s

quality is high. The probability bound in Theorem 2.3 implies that the constructed
approximation is close to G with high probability and converges almost surely to theGreen’s function as ϵ → 0.

2.1

Low-rank approximation of Hilbert–Schmidt
operators

In a landmark paper, Halko, Martinsson, and Tropp proved that one could learn
the column space of a finite matrix—to high accuracy and with a high probabilityof success—by using matrix-vector products with standard Gaussian random vectors [86]. We now set out to generalize this from matrices to HS operators. Alternative randomized low-rank approximation techniques such as the generalized Nyströmmethod [159] might also be generalized in a similar manner. Since the proof is relatively long, we state our final generalization now.
Theorem 2.1. Let D1 , D2 ⊆ Rd be domains with d ≥ 1 and F : L2 (D1 ) → L2 (D2 )be an HS operator. Select a target rank k ≥ 1, an oversampling parameter p ≥ 2,

and a D1 × (k + p) quasimatrix Ω such that each column is i.i.d. and drawn fromGP(0, K), where K : D1 × D1 → R is a continuous symmetric positive definite kernel
with eigenvalues λ1 ≥ λ2 ≥ · · · > 0. If Y = F Ω, then
s
!
!1/2
∞
X
1 k(k + p)
E[∥F − PY F ∥HS ] ≤ 1 +
σj2
,
(2.4)
γk p − 1
j=k+1
where γk = k/(λ1 Tr(C−1 )) with Cij =

D1 ×D1vi (x)K(x, y)vj (y) dx dy for 1 ≤ i, j ≤k. Here, PY is the orthogonal projection onto the vector space spanned by the columnsof Y, σj is the jth singular value of F , and vj is the jth right singular vector of F .

18

Assume further that p ≥ 4, then for any s, t ≥ 1, we have
v
!1/2
u
∞
∞
X
X
u
3
k(k
+
p)
λ
j
∥F − PY F ∥HS ≤ t1 + t2 s2
σ2,
γk p + 1 j=1 λ1 j=k+1 j

(2.5)

2

with probability ≥ 1 − t−p − [se−(s −1)/2 ]k+p .
2

We remark that the term [se−(s −1)/2 ]k+p in the statement of Theorem 2.1 is
2bounded by e−s for s ≥ 2 and k + p ≥ 5. The term 0 ≤ γk ≤ 1 is discussed in

Section 2.1.4 and is bounded by the inverse of the harmonic mean of k eigenvalues of
the covariance kernel under some conditions on the kernel eigenvectors and the rightsingular vectors of the Hilbert–Schmidt operator F (see Lemma 2.2). In the rest of
the section, we prove this theorem.

2.1.1

Three caveats that make the generalization non-trivial

One might imagine that the generalization of the randomized SVD algorithm frommatrices to HS operators is trivial, but this is not the case due to three caveats.
First, the randomized SVD on finite matrices always uses matrix-vector productswith standard Gaussian random vectors [86]. However, for GPs, one must always havea continuous kernel K in GP(0, K), which discretizes to a non-standard multivariate
Gaussian distribution. Therefore, we must extend [86, Thm. 10.5] to allow for nonstandard multivariate Gaussian distributions. The discrete version of our extension
is the following:
Corollary 2.1. Let A be a real n2 × n1 matrix with singular values σ1 ≥ · · · ≥σmin{n1 ,n2 } . Choose a target rank k ≥ 1 and an oversampling parameter p ≥ 2. Draw

an n1 × (k + p) Gaussian matrix, Ω, with independent columns where each column
is i.i.d. from a multivariate Gaussian distribution with mean (0, . . . , 0)⊤ and positive
definite covariance matrix K. If Y = AΩ, then the expected approximation error isbounded by


v
u
n1
X
uk + pλ1 E [∥A − PY A∥F ] ≤ 1 + t
p − 1 j=n −k+1 λj


1

∞
X

j=k+1

σj2

!1/2

,

(2.6)

where λ1 ≥ · · · ≥ λn1 > 0 are the eigenvalues of K and PY is the orthogonal projection
onto the vector space spanned by the columns of Y. Assume further that p ≥ 4, thenfor any s, t ≥ 1, we havevuu 3(k + p)
∥A − PY A∥F ≤ 1 + ts · t
p+1

n1
X
j=1

19

λj

!

n1
X



1 
λj
j=n −k+1
1

∞
X

j=k+1

σj2

!1/2

,

2

with probability ≥ 1 − t−p − [se−(s −1)/2 ]k+p .

P 1
Choosing a covariance matrix K with eigenvalue decay so that limn1 →∞ nj=1
λj <2
∞ allows E[∥Ω∥F ] to remain bounded as n1 → ∞. This is of interest when applying
the randomized SVD algorithm to extremely large matrices and is critical for HS

operators. A stronger statement of this result (see Theorem 3.1) shows that priorinformation on A can be incorporated into the covariance matrix to achieve lower
approximation error than the randomized SVD with standard Gaussian vectors.
Secondly, we need an additional essential assumption. The kernel in GP(0, K) is“reasonable” for learning F , where reasonableness is measured by the quantity γk in

Theorem 2.1. If the first k right singular functions of the HS operator v1 , . . . , vk arespanned by the first k + m eigenfunctions of K ψ1 , . . . , ψk+m , for some m ∈ N, then(see Equation (2.9) and Lemma 2.2)kk+m
1
1 X λ1
1 X λ1≤≤.
k j=1 λjγkk j=m+1 λj

In the matrix setting, this assumption always holds with m = n1 −k (see Corollary 2.1)
and one can have γk = 1 when λ1 = · · · = λn1 [86, Thm. 10.5].

Finally, probabilistic error bounds for the randomized SVD in [86] are derived

using tail bounds for functions of standard Gaussian matrices [121, Sec. 5.1]. Unfortunately, we are not aware of tail bounds for non-standard Gaussian quasimatrices.
√
This results in a weaker bounds by a factor of k + p in Corollary 2.1 comparedto [86, Thm. 10.7].

2.1.2

Deterministic error bound

Apart from the three caveats, the proof of Theorem 2.1 follows the outline of theargument in [86, Thm. 10.5]. We define two quasimatrices U and V containing
the left and right singular functions of F so that the jth column of V is vj . We
also denote by Σ the infinite diagonal matrix with the singular values of F , i.e.,
σ1 ≥ σ2 ≥ · · · ≥ 0, on the diagonal. Finally, for a fixed k ≥ 1, we define the D1 × k

quasimatrix as the truncation of V after the first k columns and V2 as the remainder.
Similarly, we split Σ into two parts:
 k ∞ 
Σ1 0
k .
Σ=
0 Σ2
∞

We are ready to prove an infinite dimensional analogue of [86, Thm. 9.1] for HS
operators.
20

Theorem 2.2 (Deterministic error bound). Let F : L2 (D1 ) → L2 (D2 ) be an HS

operator with SVD given in Equation (1.7). Let Ω be a D1 × k quasimatrix andY = F Ω. If Ω1 = V1∗ Ω and Ω2 = V2∗ Ω, then assuming Ω1 has full rank, we have∥F − PY F ∥2HS ≤ ∥Σ2 ∥2HS + ∥Σ2 Ω2 Ω†1 ∥2HS ,where PY = Y(Y∗ Y)† Y∗ is the orthogonal projection onto the space spanned by thecolumns of Y and Ω†1 = (Ω∗1 Ω1 )−1 Ω∗1 .
Proof. First, note that because UU∗ is the orthonormal projection onto the range ofF and U is a basis for the range, we have∥F − PY F ∥HS = ∥UU∗ F − PY UU∗ F ∥HS .
By Parseval’s theorem [194, Thm. 4.18], we have
∥UU∗ F − PY UU∗ F ∥HS = ∥U∗ UU∗ F − U∗ PY UU∗ F V∥HS .
Moreover, we have the equality ∥F − PY F ∥HS = ∥(I − PU∗ Y )U∗ F V∥HS because theP∞P∞P
inner product ⟨ ∞j=1 αj βj = 0. We now take
j=1 βuj ⟩ = 0 if and only if
j=1 αj uj ,A = U∗ F V, which is a bounded infinite matrix such that ∥A∥F = ∥F ∥HS < ∞. The
statement of the theorem immediately follows from the proof of [86, Thm. 9.1].

This theorem shows that the bound on the approximation error ∥F − PY F ∥HS

depends on the singular values of the HS operator and the test matrix Ω.

2.1.3

Probability distribution of Ω1

If the columns of Ω are independent and identically distributed as GP(0, K), thenthe matrix Ω1 in Theorem 2.2 is of size k × ℓ with entries that follow a Gaussian
distribution. To see this, note that
⟨v1 , ω1 ⟩ · · · ⟨v1 , ωℓ ⟩


..
..
..
Ω1 = V1∗ Ω = 
 , ωj ∼ GP(0, K).
.
.
.
⟨vk , ω1 ⟩ · · · ⟨vk , ωℓ ⟩If ω ∼ GP(0, K) with K given in Equation (1.8), then we find that!
∞X⟨v, ω⟩ ∼ N 0,
λj ⟨v, ψj ⟩2
j=1

so we conclude that Ω1 has Gaussian entries with zero mean. Finding the covariances
between the entries is more involved.
21

Lemma 2.1. With the same setup as Theorem 2.2, suppose that the columns of Ω
are independent and identically distributed as GP(0, K). Then, the matrix Ω1 = V1∗ Ω

in Theorem 2.2 has independent columns and each column is identically distributed
as a multivariate Gaussian with positive definite covariance matrix C given by


Cij =

vi (x)K(x, y)vj (y) dx dy,
D1 ×D11 ≤ i, j ≤ k,

(2.7)

where vi is the ith column of V1 .
Proof. We already know that the entries are Gaussian with mean 0. Moreover, thecolumns are independent because ω1 , . . . , ωℓ are independent. Therefore, we focus on
the covariance matrix. Let 1 ≤ i, i′ ≤ k, 1 ≤ j, j ′ ≤ ℓ, then since E[⟨vi , ωj ⟩] = 0 wehavecov(⟨vi , ωj ⟩, ⟨vi′ , ωj ′ ⟩) = E [⟨vi , ωj ⟩ ⟨vi′ , ωj ′ ⟩] = E [Xij Xi′ j ′ ] ,
√ (j)
P
(j)
where Xij = ⟨vi , ωj ⟩. Since ⟨vi , ωj ⟩ ∼ ∞λn cn ⟨vi , ψn ⟩, where cn ∼ N (0, 1), we
n=1
have

cov(⟨vi , ωj ⟩, ⟨vi′ , ωj ′ ⟩) = Elim

m1 ,m2 →∞Xijm1 Xim′ j2′,Xijm1 :=m1 p
X
λn c(j)n ⟨vi , ψn ⟩.
n=1

We first show that limm1 ,m2 →∞ E Xijm1 Xim′ j2′ − E[Xij Xi′ j ′ ] = 0. For any m1 , m2 ≥ 1,

we have by the triangle inequality,E Xijm1 Xim′ j2′ − E[Xij Xi′ j ′ ] ≤ E Xijm1 Xim′ j2′ − Xij Xi′ j ′≤ E (Xijm1 − Xij )Xim′ j2′ + E Xij (Xim′ j2′ − Xi′ j ′ )
h
i1 h
i1
i1 
h
1
2 2
2 2
m1
m2 2 2
m2 2 2′′≤ E Xij − Xij
E Xi′ j ′+ E X i j − Xi′ j ′E |Xij | ,where the last inequality follows from the Cauchy–Schwarz inequality. We now set
out to show that both terms in the last inequality converge to zero as m1 , m2 → ∞.
P
The terms E[|Xim′ j2′ |2 ] and E[|Xij |2 ] are bounded by ∞n=1 λn < ∞, using the Cauchy–
Schwarz inequality. Moreover, we have


2
∞
∞
h
i
X
X
p
2
m1
(j)


E Xij − Xij=E
λn cn ⟨vi , ψn ⟩≤λn −−−−→ 0,
n=m1 +1

because Xij −Xijm1 ∼ N (0,n=m1 +1

P∞

m1 →∞

2
n=m1 +1 λn ⟨vi , ψn ⟩ ). Then, we find that cov(Xij , Xi′ j ′ ) =limm1 ,m2 →∞ E[Xijm1 Xim′ j2′ ] and we obtain
#
"m m
1 X
2 p
X
′
(j )
cov(Xij , Xi′ j ′ ) = lim Eλn λn′ c(j)n cn′ ⟨vi , ψn ⟩⟨vi′ , ψn′ ⟩
m1 ,m2 →∞=lim

m1 ,m2 →∞n=1 n′ =1
m
m2 p
1
XX
n=1 n′ =1

22(j ′ )

λn λn′ E[c(j)n cn′ ]⟨vi , ψn ⟩⟨vi′ , ψn′ ⟩.

(j)(j ′ )The latter expression is zero if n ̸= n′ or j ̸= j ′ because then cn and cn′

are

(j)
independent random variables with mean 0. Since E[(cn )2 ] = 1, we havecov(Xij , Xi′ j ′ ) =

(P
∞
0,

n=1 λn ⟨vi , ψn ⟩⟨vi′ , ψn ⟩,j = j ′,
otherwise.

The result follows as the infinite sum is equal to the integral in Equation (2.7).
To see that C is positive definite, let a ∈ Rk , then a∗ Ca = E[Za2 ] ≥ 0, whereP
2∗
Za ∼ N (0, ∞n=1 λn ⟨a1 v1 + · · · + ak vk , ψn ⟩ ). Moreover, a Ca = 0 implies that a = 0
because v1 , . . . , vk are orthonormal and {ψn } is an orthonormal basis of L2 (D1 ).

Lemma 2.1 gives the distribution of the matrix Ω1 , which is essential to prove
Theorem 2.1 in Section 2.1.6. In particular, Ω1 has independent columns that are each
distributed as a multivariate Gaussian with covariance matrix given in Equation (2.7).

2.1.4

Quality of the covariance kernel

To investigate the quality of the kernel, we introduce the Wishart distribution, which
is a family of probability distributions over symmetric and nonnegative-definite matrices that often appear in the context of covariance matrices [239]. If Ω1 is ak × ℓ random matrix with independent columns, where each column is a multivariate Gaussian distribution with mean (0, . . . , 0)⊤ and covariance C, then A =Ω1 Ω∗1 has a Wishart distribution [239]. We write A ∼ Wk (ℓ, C). We note that∥Ω†1 ∥2F = Tr[(Ω†1 )∗ Ω†1 ] = Tr(A−1 ), where the second equality holds with probability one because the matrix A = Ω1 Ω∗1 is invertible with probability one (see [156,1
Thm. 3.1.4]). By [156, Thm. 3.2.12] for ℓ − k ≥ 2, we have E[A−1 ] = ℓ−k−1
C−1 ,E[Tr(A−1 )] = Tr(C−1 )/(ℓ − k − 1), and conclude thathiE ∥Ω†1 ∥2F =1
k,γk λ1 ℓ − k − 1

γk :=

k
.
λ1 Tr(C−1 )(2.8)

The quantity γk can be viewed as measuring the quality of the covariance kernel
K for learning the HS operator F (see Theorem 2.1). First, 1 ≤ γk < ∞ as C issymmetric positive definite. Moreover, for 1 ≤ j ≤ k, the jth largest eigenvalue of
C is bounded by the jth largest eigenvalue of K as C is a principal submatrix ofV∗ KV [104, Sec. III.5]. Therefore, the following inequality holds,k

1 X λ11≤< ∞,k j=1 λj
γk
23

(2.9)

and the harmonic mean of the first k scaled eigenvalues of K is a lower bound for 1/γk .
In the ideal situation, the eigenfunctions of K are the right singular functions of F ,Pi.e., ψn = vn , C is a diagonal matrix with entries λ1 , . . . , λk , and γk = k/( kj=1 λ1 /λj )is as small as possible.
We now provide a useful upper bound on γk in a more general setting.
Lemma 2.2. Let V1 be a D1 × k quasimatrix with orthonormal columns and assume
that there exists m ∈ N such that the columns of V1 are spanned by the first k + meigenvectors of the continuous positive definite kernel K : D1 × D1 → R. Thenk+m
1
1 X λ1≤,γkk j=m+1 λjwhere λ1 ≥ λ2 ≥ · · · > 0 are the eigenvalues of K. This bound is tight in the sensethat the inequality can be attained as an equality.

Proof. Let Q = [v1 | · · · | vk | qk+1 | · · · | qk+m ] be a quasimatrix with orthonormal

columns whose columns form an orthonormal basis for Span(ψ1 , . . . , ψk+m ). Then,
Q is an invariant space of K and C is a principal submatrix of Q∗ KQ, which haseigenvalues λ1 ≥ · · · ≥ λk+m . By [104, Thm. 6.46] the k eigenvalues of C, denoted byµ1 , . . . , µk , are greater than the first k +m eigenvalues of K: µj ≥ λm+j for 1 ≤ j ≤ k,and the result follows as the trace of a matrix is the sum of its eigenvalues.

2.1.5

Probabilistic error bounds

As discussed in Section 2.1.1, we need to extend the probability bounds of the randomized SVD to allow for non-standard Gaussian random vectors. The following
lemma is a generalization of [86, Thm. A.7].
Lemma 2.3. Let k, ℓ ≥ 1 such that ℓ − k ≥ 4 and Ω1 be a k × ℓ random matrix

with independent columns such that each column has mean (0, . . . , 0)⊤ and positive
definite covariance C. For all t ≥ 1, we have
3 Tr(C−1 ) 2† 2
P ∥Ω1 ∥F >· t ≤ t−(ℓ−k) .
ℓ−k+1

Proof. Since Ω1 Ω∗1 ∼ Wk (ℓ, C), the reciprocals of its diagonal elements follow a scaled
chi-square distribution [156, Thm. 3.2.12], i.e.,((Ω1 Ω∗1 )−1 )jj∼ Xj−1 ,(C−1 )jj

Xj ∼ χ2ℓ−k+1 ,24

1 ≤ j ≤ k.

Let Z = ∥Ω†1 ∥2F = Tr[(Ω1 Ω∗1 )−1 ] and q = (ℓ − k)/2. Following the proof of [86,Thm. A.7], we have the inequality

3 Tr(C−1 ) 2
P |Z| ≥
·t
ℓ−k+1





3 Tr(C−1 ) 2
≤
·t
ℓ−k+1

−q

E [|Z|q ] ,t ≥ 1.

Moreover, by the Minkowski inequality, we haveq

1/q

(E [|Z |])

=

E

"

k
X

−1

[C

]jj Xj−1

j=1

q #!1/q≤kX1/q 3 Tr(C−1 )[C−1 ]jj E |Xj−1 |q
≤
,
ℓ
−
k
+
1
j=1

where the last inequality is from [86, Lem. A.9]. The result follows from the argumentin the proof of [86, Thm. A.7].
Under the assumption of Lemma 2.2, we find that Lemma 2.3 gives the following
bound:
P





v
u
u
†
∥Ω ∥F > t · t
1

k+m
X




3
λ−1 ≤ t−(ℓ−k) .
ℓ − k + 1 j=m+1 j 

In particular, in the finite dimensional case when λ1 = · · · = λn = 1, we recover the
probabilistic bound found in [86, Thm. A.7].

To obtain the probability statement found in Equation (2.11) we require controlof the tail of the distribution of a Gaussian quasimatrix with non-standard covariancekernel (see Section 2.1.6). In the theory of the randomized SVD, one relies on theconcentration of measure results [86, Prop. 10.3]. However, we need to employ adifferent strategy and instead directly bound the HS norm of Ω2 . One difficulty isthat the norm of this matrix must be controlled for large dimensions n, which leads toa weaker probability bound than [86]. While it is possible to apply Markov’s inequality
to obtain deviation bounds, we highlight that Lemma 2.4 provides a Chernoff-typebound, i.e., exponential decay of the tail distribution of ∥Ω2 ∥HS , which is crucial toapproximate Green’s functions (see Section 2.2.4.3).
Lemma 2.4. With the same notation as in Theorem 2.2, let ℓ ≥ k ≥ 1. For all s ≥ 1

we havehiℓ2
P ∥Ω2 ∥2HS > ℓs2 Tr(K) ≤ se−(s −1)/2 .

Proof. We first remark that∥Ω2 ∥2HS ≤ ∥Ω∥2HS =

ℓ
X

Zj ,

j=1

25

Zj := ∥ωj ∥2L2 (D1 ) ,(2.10)

where the Zj are i.i.d. because ωj ∼ GP(0, K) are i.i.d. For 1 ≤ j ≤ ℓ, we have(c.f. Section 1.8),

∞
X

ωj =

c(j)
m

m=1

p
λm ψm ,(j)where cm ∼ N (0, 1) are i.i.d. for m ≥ 1 and 1 ≤ j ≤ ℓ.

First, since the series inEquation (2.10) converges absolutely, we have
∞
X

N
X
(j) 2
Xm ,
Zj =
(cm ) λm = lim
N →∞
m=1
m=1

2
Xm = (c(j)
m ) λm ,where the Xm are independent random variables and Xm ∼ λm χ2 for 1 ≤ m ≤ N .
Here, χ2 denotes the chi-squared distribution [155, Chapt. 4.3].

Let N ≥ 1 and 0 < θ < 1/(2 Tr(K)), we can bound the moment generatingPfunction of N
m=1 Xm as
h

E eθ

PN

m=1 Xm

i

=

N
Y

m=1

N
 θXm  Y
E e
(1 − 2θλm )−1/2 ≤
=
m=1
−1/2

≤ (1 − 2θ Tr(K))

1 − 2θ

,

N
X

m=1

λm

!−1/2

because Xm /λm are independent random variables that follow a chi-squared distribution. Using the monotone convergence theorem, we haveE eθZj ≤ (1 − 2θ Tr(K))−1/2 .

Let s̃ ≥ 0 and 0 < θ < 1/(2 Tr(K)). By the Chernoff bound [41, Thm. 1], we

obtainℓP ∥Ω2 ∥2HS > ℓ(1 + s̃) Tr(K) ≤ e−(1+s̃) Tr(K)ℓθ E eθZj

= e−(1+s̃) Tr(K)ℓθ (1 − 2θ Tr(K))−ℓ/2 .

We can minimize this upper bound over 0 < θ < 1/(2 Tr(K)) by choosing θ =s̃/(2(1 + s̃) Tr(K)), which gives

Choosing s =√P ∥Ω2 ∥2HS > ℓ(1 + s̃) Tr(K) ≤ (1 + s̃)ℓ/2 e−ℓs̃/2 .

1 + s̃ ≥ 1 concludes the proof.

Lemma 2.4 can be refined further to take into account the interaction between theHilbert–Schmidt operator F and the covariance kernel K (see Lemma 3.1).
26

2.1.6

Randomized SVD algorithm for HS operators

We first prove an intermediary result, which generalizes [86, Prop. 10.1] to HS operators. Note that one may obtain sharper bounds using a suitably chosen covariancekernels that yields a lower approximation error (see Chapter 3).
Lemma 2.5. Let Σ2 , V2 , and Ω be defined as in Theorem 2.2, and T be an ℓ × k
matrix, where ℓ ≥ k ≥ 1. Then,E ∥Σ2 V2∗ ΩT∥2HS ≤ λ1 ∥Σ2 ∥2HS ∥T∥2F ,where λ1 is the first eigenvalue of K.

∗
Proof. Let T = UT DT VT
be the SVD of T. If {vT,i }ki=1 are the columns of VT , then
k

 X


∗
E ∥Σ2 V2∗ ΩT∥2HS =E ∥Σ2 Ω2 UT DT VT
vT,i ∥22 ,i=1

where Ω2 = V2∗ Ω. Therefore, we haveE∥Σ2 Ω2 T∥2HS



k
X


=
((DT )ii )2 E ∥Σ2 Ω2 UT (:, i)∥22 .
i=1

Moreover, using the monotone convergence theorem for non-negative random variables, we have
"∞ ℓ
#
XX


2
E ∥Σ2 Ω2 UT (:, i)∥22 = E
σk+n|Ω2 (n, j)|2 UT (j, i)2
n=1 j=1=∞ Xℓ
X
n=1 j=1


2
σk+n
UT (j, i)2 E |Ω2 (n, j)|2 ,where σk+1 , σk+2 , . . . are the diagonal elements of Σ2 . Then, the quasimatrix Ω2 has
independent columns and, using Lemma 2.1, we haveE |Ω2 (n, j)|2 =

vk+n (x)K(x, y)vk+n (y) dx dy,
D1 ×D1

where vk+n is the nth column of V2 . Then, E [|Ω2 (n, j)|2 ] ≤ λ1 , as E [|Ω2 (n, j)|2 ] iswritten as a Rayleigh quotient. Finally, we have

ℓ
∞
k
X
X
X


∗
2
2
2
2
σk+n
= λ1 ∥T∥2F ∥Σ2 ∥2HS ,E ∥Σ2 V2 ΩT∥HS ≤ λ1((DT )ii )UT (j, i)i=1

j=1

by orthonormality of the columns on UT .
27

n=1

We are now ready to prove Theorem 2.1, which shows that the randomized SVD
can be generalized to HS operators.
Proof of Theorem 2.1. Let Ω1 , Ω2 be the quasimatrices defined in Theorem 2.2. The
k × (k + p) matrix Ω1 has full rank with probability one and by Theorem 2.2, we have
1/2 
† 2
2
E [∥(I − PY )F ∥HS ] ≤ E ∥Σ2 ∥HS + ∥Σ2 Ω2 Ω1 ∥HS≤ ∥Σ2 ∥HS + E∥Σ2 Ω2 Ω†1 ∥HS1/2 h † 2 i1/2E ∥Ω1 ∥F,
≤ ∥Σ2 ∥HS + E ∥Σ2 Ω2 ∥2HS

where the last inequality follows from Cauchy–Schwarz inequality. Then, combining
Lemma 2.5 and Equation (2.8), we haveE ∥Σ2 Ω2 ∥2HS ≤ λ1 (k + p)∥Σ2 ∥2HS
and E ∥Ω1 ∥2F ≤1
k,γk λ1 p − 1
P∞2
where γk is defined in Section 2.1.4. The observation that ∥Σ2 ∥2HS =j=k+1 σjconcludes the proof of Equation (2.4).

For the probabilistic bound in Equation (2.5), we note that by Theorem 2.2 wehave,∥F − PY F ∥2HS ≤ ∥Σ2 ∥2HS + ∥Σ2 Ω2 Ω†1 ∥2HS ≤ (1 + ∥Ω2 ∥2HS ∥Ω†1 ∥2F )∥Σ2 ∥2HS ,where the second inequality uses the submultiplicativity of the HS norm. The boundfollows from bounding ∥Ω†1 ∥2F and ∥Ω2 ∥2HS using Lemmas 2.3 and 2.4, respectively.
Remark 2.1. The expectation bound (2.4) in Theorem 2.1 does not control the squareof the HS norm and therefore cannot be used to obtain an expectation bound for the
randomized scheme for learning Green’s functions described in Section 2.2.
The following proposition provides an expectation bound for the randomized SVDof the HS norm squared.
Proposition 2.1. With the notations of Theorem 2.1, we have
! ∞
√
∞
X
X


3
2
k(k
+
p)
λ
j
E ∥F − PY F ∥2HS ≤ 1 +σ2.
γk
p + 1 j=1 λ1 j=k+1 j
Proof. Let Ω1 , Ω2 be the quasimatrices defined in Theorem 2.2. We combine Theorem 2.2 with the submultiplicativity of the HS norm and Cauchy–Schwarz inequality
to obtain
1

1

E[∥F − PY F ∥2HS ] ≤ ∥Σ2 ∥2HS + E[∥Σ2 Ω2 Ω†1 ∥2HS ] ≤ (1 + E[∥Ω2 ∥4HS ] 2 E[∥Ω†1 ∥4F ] 2 )∥Σ2 ∥2HS .
28

We can then control both terms E[∥Ω2 ∥4HS ]1/2 and E[∥Ω†1 ∥4F ]1/2 independently.

First, since Ω1 Ω∗1 ∼ Wk (k + p, C), where C is defined in Lemma 2.1, we have(cf. the proof of Lemma 2.3)((Ω1 Ω∗1 )−1 )jj∼ Xj−1 ,(C−1 )jj

Xj ∼ χ2p+1 ,1 ≤ j ≤ k.

Therefore,kX† 4 1/2∗ −1 2 1/2
E[∥Ω1 ∥ ] = E[Tr((Ω1 Ω1 ) ) ] = E(C−1 )jj Xj−1
j=1

≤ Tr(C−1 )E2 [X1−1 ],!2 1/2
by the triangle inequality for the norm defined as E2 (Z) := E[|Z|2 ]1/2 (see [86,Sec. A.3.1]). Finally, using [86, Lem. A.9], we have E2 [X1−1 ] = 3/(p + 1), whichgivesE[∥Ω†1 ∥4 ]1/2 ≤3
Tr(C−1 ).
p+1

The second term can be bounded as∥Ω2 ∥2HS ≤ ∥Ω∥2HS =

k+p
X

Zj ,

j=1

Zj = ∥ωj ∥2L2 (D1 ) ,where the Zj are i.i.d.. Therefore, by the triangle inequality for the E2 -norm applied
to the random variable ∥Ω∥2HS , we haveE(∥Ω2 ∥4HS )1/2 ≤ (k + p)E[Z12 ]1/2 .
We then characterize the distribution of Z1 following the proof of Lemma 2.4 as
Z1 =

∞
X

λm Y m ,m=1

Ym ∼ χ2 .

Applying triangle inequality to E2 (Z1 ) yields
E[Z12 ]1/2 ≤∞X

λm E[Ym2 ]1/2 = Tr(K)E[Y12 ]1/2 =m=1which concludes the proof.

29√
2 Tr(K),

2.2

Recovering the Green’s function from inputoutput pairs

It is known that the Green’s function associated with Equation (2.2) always exists,
is unique, and is a nonnegative function G : D × D → R+ ∪ {∞} such that
G(x, y)f (y) dy,u(x) =Df ∈ Cc∞ (D).

For each y ∈ Ω and any r > 0, we have G(·, y) ∈ H1 (D \ Br (y)) ∩ W01,1 (D) [80].
Here, Br (y) = {z ∈ R3 : ∥z − y∥2 < r}, W 1,1 (D) is the space of weakly differentiable

functions in the L1 -sense, and W01,1 (D) is the closure of Cc∞ (D) in W 1,1 (D). Since thePDE in Equation (2.2) is self-adjoint, we also know that for almost every x, y ∈ D,
we have G(x, y) = G(y, x) [80].
We now state Theorem 2.3, which shows that if N = O(ϵ−6 log4 (1/ϵ)) and one has

N input-output pairs {(fj , uj )}N
j=1 with algorithmically-selected fj , then the Green’s

function associated with L in Equation (2.2) can be recovered to within an accuracy−1/2of O(Γϵ
log3 (1/ϵ)ϵ) with high probability. Here, the quantity 0 < Γϵ ≤ 1 measures
the quality of the random input functions {fj }N
j=1 (see Section 2.2.4.2).

Theorem 2.3. Let 0 < ϵ < 1, D ⊂ R3 be a bounded Lipschitz domain, and L given
in Equation (2.2). If G is the Green’s function associated with L, then there is arandomized algorithm that constructs an approximation G̃ of G using O(ϵ−6 log4 (1/ϵ))
input-output pairs such that, as ϵ → 0, we have∥G − G̃∥L2 (D×D) = O Γϵ−1/2 log3 (1/ϵ)ϵ ∥G∥L2 (D×D) ,(2.11)

with probability ≥ 1 − O(ϵlog(1/ϵ)−6 ). The term Γϵ is defined by Equation (2.23).
For simplicity, we have not reported the dependence of the bound in Equation (2.11) with respect to the spectral condition number, κC = λmax /λmin 1 , of thecoefficient matrix A(x) in Equation (2.2).
Our algorithm that leads to the proof of Theorem 2.3 relies on the extension of
the randomized SVD to HS operators (see Section 2.1) and a hierarchical partition of
the domain of G into “well-separated” domains. The scheme described in this section
is summarized by Algorithm 1.
1Here, λmax is defined as supx∈D λmax (A(x)) and λmin = inf x∈D λmin (A(x)) > 0.

30

Algorithm 1 Approximation of the Green’s function
Input: Action of the integral operator with kernel G
Output: Approximation G̃ of G
1: Construct a hierarchical partition of the domain D × D
2: Approximate the Green’s function on the admissible domains with the randomized
SVD
3: Neglect G on the non-admissible domains using a decay bound for the Green’s
function near the diagonal

2.2.1

Recovering the Green’s function on admissible domains

Roughly speaking, as ∥x − y∥2 increases G becomes smoother about (x, y), which canbe made precise using so-called admissible domains [13, 15, 84]. For X, Y ⊂ R3 , letdiam X := supx,y∈X ∥x − y∥2 be the diameter of X, and dist(X, Y ) := inf x∈X,y∈Y ∥x −y∥2 be the shortest distance between X and Y . Admissible domains are defined asfollows.

Definition 2.1. For a fixed parameter ρ > 0, we say that two bounded and non-empty
domains X, Y ⊂ R3 are admissible ifdist(X, Y ) ≥ ρ max{diam X, diam Y }.
Otherwise, we say that X × Y is non-admissible.
There exists a weaker definition of admissible domains, which only requires that
dist(X, Y ) ≥ ρ min{diam X, diam Y } [84, p. 59], but we do not consider it.
2.2.1.1

Approximation theory on admissible domains

It turns out that the Green’s function associated with Equation (2.2) has exponentially decaying singular values when restricted to admissible domains. Roughly
speaking, if X, Y ⊂ D are such that X × Y is an admissible domain, then G iswell-approximated by a function of the form [16]Gk (x, y) =kX

gj (x)hj (y),j=1(x, y) ∈ X × Y,

(2.12)

for some functions g1 , . . . , gk ∈ L2 (X) and h1 , . . . , hk ∈ L2 (Y ). This is summarizedin Theorem 2.4, which is a corollary of [16, Thm. 2.8].
Theorem 2.4. Let G be the Green’s function associated with Equation (2.2) andρ > 0. Let X, Y ⊂ D such that dist(X, Y ) ≥ ρ max{diam X, diam Y }. Then, for any
31

0 < ϵ < 1, there exists k ≤ kϵ := ⌈c(ρ, diam D, κC )⌉⌈log(1/ϵ)⌉4 + ⌈log(1/ϵ)⌉ and an
approximant, Gk , of G in the form given in Equation (2.12) such that∥G − Gk ∥L2 (X×Y ) ≤ ϵ∥G∥L2 (X×Ŷ ) ,Ŷ := {y ∈ D, dist(y, Y ) ≤ρdiam Y },2

where κC = λmax /λmin is the spectral condition number of the coefficient matrix A(x)in Equation (2.2) and c is a constant that only depends on ρ, diam D, κC .
Proof. In [16, Thm. 2.8], it is shown that if Y = Ỹ ∩ D and Ỹ is convex, then there
exists k ≤ c3ρ/2 ⌈log(1/ϵ)⌉4 + ⌈log(1/ϵ)⌉ and an approximant, Gk , of G such that
∥G(x, ·) − Gk (x, ·)∥L2 (Y ) ≤ ϵ∥G(x, ·)∥L2 (Ŷ ) ,x ∈ X,(2.13)

where Ŷ := {y ∈ D, dist(y, Y ) ≤ ρ2 diam Y } and cρ/2 is a constant that only depends
on ρ, diam Y , and κC . As remarked by [16], Ỹ can be included in a convex of diameterdiam D that includes D to obtain the constant c(ρ, diam D, κC ). The statement
follows by integrating the error bound in Equation (2.13) over X.
Since the truncated SVD of G on X × Y gives the best rank kϵ ≥ k approximation

to G, Theorem 2.4 also gives bounds on singular values:
X∞

2
σj,X×Y1/2≤ ∥G − Gk ∥L2 (X×Y ) ≤ ϵ∥G∥L2 (X×Ŷ ) ,(2.14)

where σj,X×Y is the jth singular value of G restricted to X × Y .

Since kϵ =j=kϵ +1

4

O(log (1/ϵ)), we conclude that the singular values of G restricted to admissible domains X × Y rapidly decay to zero.
2.2.1.2

Randomized SVD for admissible domains

Since G has exponentially decaying singular values on admissible domains X × Y , weuse the randomized SVD for HS operators to learn G on X × Y with high probability(see Section 2.1).
We start by defining a GP on the domain Y . Let RY ×Y K be the restriction2of the covariance kernel K to the domain Y × Y , which is a continuous symmet-

ric positive definite kernel so that GP(0, RY ×Y K) defines a GP on Y . We choose

a target rank k ≥ 1, an oversampling parameter p ≥ 2, and form a quasimatrixΩ = f1 | · · · | fk+p such that fj ∈ L2 (Y ) and fj ∼ GP(0, RY ×Y K) are identically

distributed and independent. We then extend by zero each column of Ω from L2 (Y ) to2

We denote the restriction operator by RY ×Y : L2 (D × D) → L2 (Y × Y ).

32



L2 (D) by R∗Y Ω = R∗Y f1 | · · · | R∗Y fk+p , where R∗Y fj ∼ GP(0, R∗Y ×Y RY ×Y K). The
zero extension operator R∗Y : L2 (Y ) → L2 (D) is the adjoint of RY : L2 (D) → L2 (Y ).
Given the training data, Y = u1 | · · · | uk+p such that Luj = R∗Y fj and uj |∂D =0, we now construct an approximation to G on X × Y using the randomized SVD(see Section 2.1). Following Theorem 2.1, we have the following approximation error
for t ≥ 1 and s ≥ 2:∞k(k + p) X λj
2 2
1+t s
γk,X×Y p + 1 j=1 λ1

3∥G − G̃X×Y ∥2L2 (X×Y ) ≤−p−s2 (k+p)

with probability greater than 1−t −e

!

X∞

j=k+1

2
σj,X×Y1/2,(2.15). Here, λ1 ≥ λ2 ≥ · · · > 0 are the eigen-values of K, G̃X×Y = PRX Y RX F R∗Y and PRX Y = RX Y((RX Y)∗ RX Y)† (RX Y)∗ isthe orthogonal projection onto the space spanned by the columns of RX Y. Moreover,γk,X×Y is a measure of the quality of the covariance kernel of GP(0, R∗Y ×Y RY ×Y K)(see Section 2.1.4) and, for 1 ≤ i, j ≤ k, defined as γk,X×Y = k/(λ1 Tr(C−1
X×Y )), where[CX×Y ]ij =D×DR∗Y vi,X×Y (x)K(x, y)R∗Y vj,X×Y (y) dx dy,

and v1,X×Y , . . . , vk,X×Y ∈ L2 (Y ) are the first k right singular functions of G restricted
to X × Y .
Unfortunately, there is a big problem with the formula G̃X×Y = PRX Y RX F R∗Y .

It cannot be formed because we only have access to input-output data, so we have no

mechanism for composing PRX Y on the left of RX F R∗Y . Instead, we note that since

the partial differential operator in Equation (2.2) is self-adjoint, F is self-adjoint,
and G is itself symmetric. That means we can use this to write down a formula forG̃Y ×X instead. That is,G̃Y ×X = G̃∗X×Y = RY F R∗X PRX Y ,where we used the fact that PRX Y is also self-adjoint. This means we can construct

G̃Y ×X by asking for more input-output data to assess the quasimatrix F (R∗X RX Y).
Of course, to compute G̃X×Y , we can swap the roles of X and Y in the above argument.
With a target rank of k = kϵ = ⌈c(ρ, diam D, κC )⌉⌈log(1/ϵ)⌉4 + ⌈log(1/ϵ)⌉ and an

oversampling parameter of p = kϵ , we can combine Theorem 2.4 and Equations (2.14)
and (2.15) to obtain the bound∥G − G̃X×Y ∥2L2 (X×Y ) ≤1 + t 2 s2

6kϵ

∞
X
λj

γkϵ ,X×Y j=1 λ133!

ϵ2 ∥G∥2L2 (X×Ŷ ) ,2with probability greater than 1−t−kϵ −e−2s kϵ . A similar approximation error holds forG̃Y ×X without additional evaluations of F . We conclude that our algorithm requires
Nϵ,X×Y = 2(kϵ + p) = O log4 (1/ϵ) input-output pairs to learn an approximant to G

on X × Y and Y × X.

2.2.2

Ignoring the Green’s function on non-admissible domains

When the Green’s function is restricted to non-admissible domains, its singular values
may not decay. Instead, to learn G we take advantage of the off-diagonal decay
property of G. It is known that for almost every x ̸= y ∈ D thenG(x, y) ≤cκC∥G∥L2 (D×D) ,∥x − y∥2(2.16)

where cκC is an implicit constant that only depends on κC (see [80, Thm. 1.1]). Notethat we have normalized [80, Eq. 1.8] to highlight the dependence on ∥G∥L2 (D×D) .

If X × Y is a non-admissible domain, then for any (x, y) ∈ X × Y , we find that∥x − y∥2 ≤ dist(X, Y ) + diam(X) + diam(Y ) < (2 + ρ) max{diam X, diam Y },because dist(X, Y ) < ρ max{diam X, diam Y }. This means that x ∈ Br (y)∩D, wherer = (2 + ρ) max{diam X, diam Y }. Using Equation (2.16), we have




2

X

G(x, y) dx ≤

2

G(x, y)dx ≤ c2κC ∥G∥2L2 (D×D)

Br (y)∩D≤ 4πc2κC r∥G∥2L2 (D×D) .

Br (y)∥x − y∥−22 dx

Noting that diam(Y ) ≤ r/(2 + ρ) and Y 1 dy ≤ 4π(diam(Y )/2)3 /3, we have thefollowing inequality for non-admissible domains X × Y :∥G∥2L2 (X×Y ) ≤2π 2
c2 r4 ∥G∥2L2 (D×D) ,3(2 + ρ)3 κC(2.17)

where r = (2 + ρ) max{diam X, diam Y }. We conclude that the Green’s function

restricted to a non-admissible domain has a relatively small norm when the domain
itself is small. Therefore, in our approximant G̃ for G, we ignore G on non-admissible
domains by setting G̃ to be zero.

34

2.2.3

Hierarchical admissible partition of domain

We now describe a hierarchical partitioning of D × D so that many subdomains
are admissible domains, and the non-admissible domains are all small. For ease ofnotation, we may assume—without loss of generality—that diam D = 1 and D ⊂[0, 1]3 ; otherwise, one should shift and scale D. Moreover, partitioning [0, 1]3 andrestricting the partition to D is easier than partitioning D directly. For the definition√of admissible domains, we find it convenient to select ρ = 1/ 3.

Figure 2.1: Two levels of hierarchical partitioning of [0, 1]3 . The blue and green
domains are admissible, while the blue and red domains are non-admissible.
Let I = [0, 1]3 . The hierarchical partitioning for n levels is defined recursively as:• I1×1×1 := I1 × I1 × I1 = [0, 1]3 is the root for level L = 0.
• At a given level 0 ≤ L ≤ n − 1, if Ij1 ×j2 ×j3 := Ij1 × Ij2 × Ij3 is a node of the tree,
then it has 8 children defined as{I2j1 +nj (1) × I2j2 +nj (2) × I2j3 +nj (3) | nj ∈ {0, 1}3 }.

Here, if Ij = [a, b], 0 ≤ a < b ≤ 1, then I2j = a, a+b
and I2j+1 = a+b
,b .
2
2

The set of non-admissible domains can be given by an unwieldy expression
Pnon-adm =V3[

i=1 |ji −j̃i |≤12n ≤j1 ,j2 ,j3 ≤2n+1 −1
2n ≤j̃1 ,j̃2 ,j̃3 ≤2n+1 −1

Ij1 ×j2 ×j3 × Ij̃1 ×j̃2 ×j̃3 ,(2.18)

where ∧ is the logical “and” operator. The set of admissible domains is given by
Padm =n[

L=1

Λ(Pnon-adm (L − 1))\Pnon-adm (L)),

35(2.19)

where Pnon-adm (L) is the set of non-admissible domain for a hierarchical level of L andΛ(Pnon-adm (L − 1)) =[[Ij1 ×j2 ×j3 ×Ij̃ ×j̃ ×j̃ nj ,nj̃ ∈{0,1}3
1
2
3∈Pnon-adm (L−1)

I×3i=1 2ji +nj (i) × I×3i=1 2j̃i +nj̃ (i) .

Using Equation (2.18)-Equation (2.19), the number of admissible and non-admissibleP
domains are precisely |Pnon-adm | = (3 × 2n − 2)3 and |Padm | = nℓ=1 26 (3 × 2L−1 − 2)3 −(3×2L −2)3 . In particular, the size of the partition at the hierarchical level 0 ≤ L ≤ n
is equal to 8L and the tree has a total of (8n+1 − 1)/7 nodes (see Figure 2.2).
1D

3D

Figure 2.2: For illustration purposes, we include the hierarchical structure of theGreen’s functions in 1D after 4 levels (left) and in 3D after 2 levels (right). The
hierarchical structure in 3D is complicated as this is physically a 6-dimensional tensor
that has been rearranged so it can be visualized.
Finally, the hierarchical partition of D × D can be defined via the partition P =Padm ∪ Pnon-adm of [0, 1]3 by doing the following:
D×D =

[

τ ×σ∈P(τ ∩ D) × (σ ∩ D).

The sets of admissible and non-admissible domains of D × D are denoted by Padm
and Pnon-adm in the next sections.

2.2.4

Recovering the Green’s function on the entire domain

We now show that we can recover G on the entire domain D × D.

36

2.2.4.1

Global approximation on the non-admissible set

Let nϵ be the number of levels in the hierarchical partition D × D (see Section 2.2.3).

We want to make sure that the norm of the Green’s function on all non-admissible
domains is small so that we can safely ignore that part of G (see Section 2.2.2). As
one increases the hierarchical partitioning levels, the volume of the non-admissible
domains get smaller (see Figure 2.3).
Level 2

Level 3

Level 4

Figure 2.3: For illustration purposes, we include the hierarchical structure of theGreen function in 1D. The green blocks are admissible domains at that level, thegray blocks are admissible at a higher level, and the red blocks are the non-admissible
domains at that level. The area of the non-admissible domains decreases at deeper
levels.
Let X × Y ∈ Pnon-adm be a non-admissible domain, the two domains X and Y√
have diameter bounded by 3/2nϵ because they are included in cubes of side length
1/2nϵ (see Section 2.2.3). Combining this with Equation (2.17) yields∥G∥2L2 (X×Y ) ≤ 2π 2 (6 +√3)c2κC 2−4nϵ ∥G∥2L2 (D×D) .

Therefore, the L2 -norm of G on the non-admissible domain Pnon-adm satisfies∥G∥2L2 (Pnon-adm ) =X

X×Y ∈Pnon-adm∥G∥2L2 (X×Y ) ≤ 54π 2 (6 +√3)c2κC 2−nϵ ∥G∥2L2 (D×D) ,where we used |Pnon-adm | = (3 × 2nϵ − 2)3 ≤ 27(23nϵ ). This means that if we select nϵ

to be

l
m
√
nϵ = log2 (54π 2 (6 + 3)c2κC ) + 2 log2 (1/ϵ) ∼ 2 log2 (1/ϵ),(2.20)

then we guarantee that ∥G∥L2 (Pnon-adm ) ≤ ϵ∥G∥L2 (D×D) . We can safely ignore G on

non-admissible domains—by taking the zero approximant—while approximating G
to within ϵ.

37

2.2.4.2

Learning rate of the Green’s function

Following Section 2.2.1.2, we can construct an approximant G̃X×Y to the Green’s
function on an admissible domain X × Y of the hierarchical partitioning using the
HS randomized SVD algorithm, which requires Nϵ,X×Y = O(log4 (1/ϵ)) input-output

training pairs (see Section 2.2.1.2). Therefore, the number of training input-output
pairs needed to construct an approximant to G on all admissible domains is given byX

Nϵ =X×Y ∈Padm

Nϵ,X×Y = O |Padm | log4 (1/ϵ) ,where |Padm | denotes the total number of admissible domains at the hierarchical levelnϵ , which is given by Equation (2.20). Then, we have (see Section 2.2.3):|Padm | =

nϵ
X
ℓ=1

26 (3 × 2ℓ−1 − 2)3 − (3 × 2ℓ − 2)3 ≤ 63 23nϵ ,(2.21)

and, using Equation (2.20), we obtain |Padm | = O(1/ϵ6 ). This means that the total
number of required input-output training pairs to learn G with high probability isbounded by Nϵ = O ϵ−6 log4 (1/ϵ) .
2.2.4.3

Global approximation error

We know that with Nϵ = O(ϵ−6 log4 (1/ϵ)) input-output training pairs, we can construct an accurate approximant to G on each admissible and non-admissible domain.
Since the number of admissible and non-admissible domains depends on ϵ, we now
check that this implies a globally accurate approximant that we denote by G̃.
Since G̃ is zero on non-admissible domains and Padm ∩ Pnon-adm has measure zero,
we have∥G − G̃∥2L2 (D×D) ≤ ϵ2 ∥G∥2L2 (D×D) +X

X×Y ∈Padm∥G − G̃∥2L2 (X×Y ) .

(2.22)

Following Section 2.2.4.2, if X ×Y is admissible then the approximation error satisfies∥G − G̃X×Y ∥2L2 (X×Y ) ≤ 12t2 s2

kϵ

∞
X
λj

γkϵ ,X×Y j=1 λ1ϵ2 ∥G∥2L2 (X×Ŷ ) ,2with probability greater than 1 − t−kϵ − e−2s kϵ . Here, Ŷ = {y ∈ D, dist(y, Y ) ≤√√diam Y /2 3} (see Theorem 2.4 with ρ = 1/ 3). To measure the worst γkϵ ,X×Y , wedefineΓϵ = min{γkϵ ,X×Y : X × Y ∈ Padm }.
38(2.23)

From Equation (2.9), we know that 0 < Γϵ ≤ 1 and that 1/Γϵ is greater than theharmonic mean of the first kϵ scaled eigenvalues of the covariance kernel K, i.e.,
kϵ
1
1 X
λ1
≥
,
Γϵ
kϵ j=1 λj(2.24)

Now, one can see that X × Ŷ is included in at most 53 = 125 neighbours including

itself. Assuming that all the probability bounds hold on the admissible domains, this
implies thatX

X×Y ∈Padm∥G − G̃∥2L2 (X×Y ) ≤X

X×Y ∈Padm≤ 12t2 s2Xkϵ
Tr(K)ϵ2λ1 ΓϵX×Y ∈P≤ 1500t2 s2
1/4

We then choose t = e and s = kϵ∥G − G̃∥2L2 (X×Y )adm∥G∥2L2 (X×Ŷ )kϵ
Tr(K)ϵ2 ∥G∥2L2 (D×D) .
λ1 Γϵso that the approximation bound on each ad√

missible domain holds with probability of failure less than 2e− kϵ . Finally, using
Equation (2.22) we conclude that as ϵ → 0, the approximation error on D × D
satisfies∥G − G̃∥L2 (D×D) = O Γϵ−1/2 log3 (1/ϵ)ϵ ∥G∥L2 (D×D) ,
√

3 3nϵ

with probability ≥ (1 − 2e− kϵ )6 2

= 1 − O(ϵlog(1/ϵ)−6 ), where nϵ is given by Equa-tion (2.20). We conclude that the approximant G̃ is a good approximation to G withvery high probability.

2.3

Discussion

There are several possible extensions of the results presented in this chapter related to
the recovery of hierarchical matrices, the study of other partial differential operators,
and practical deep learning applications, which we discuss further in this section.

2.3.1

Fast and stable reconstruction of hierarchical matrices

We described an algorithm for reconstructing Green’s function on admissible domainsof a hierarchical partition of D × D that requires performing the HS randomized

SVD O(ϵ−6 ) times. We want to reduce it to a factor that is O(polylog(1/ϵ)). A
polylogarithmic function in x is any polynomial in log(x) and is denoted by polylog(x).

39For n×n hierarchical matrices, there are several existing algorithms for recovering
the matrix based on matrix-vector products [25, 130, 143, 144]. There are two main
approaches: (1) the “bottom-up” approach: one begins at the lowest level of the hierarchy and moves up and (2) the “top-down” approach: one updates the approximantby peeling off the off-diagonal blocks and going down the hierarchy. The bottom-up
approach requires O(n) applications of the randomized SVD algorithm [143]. There

are lower complexity alternatives that only require O(log(n)) matrix-vector productswith random vectors [130]. However, the algorithm in [130] is not yet proven to

be theoretically stable as errors from low-rank approximations potentially accumulate exponentially, though this is not observed in practice. For symmetric positive
semi-definite matrices, it may be possible to employ a sparse Cholesky factorization [198, 199]. This leads us to formulate the following challenge:
Algorithmic challenge: Design a provably stable algorithm that canrecover an n × n hierarchical matrix using O(log(n)) matrix-vector
products with high probability.
If one can design such an algorithm and it can be extended to HS operators, thenthe O(ϵ−6 log4 (1/ϵ)) term in Theorem 2.3 may improve to O(polylog(1/ϵ)). Thismeans that the learning rate of partial differential operators of the form of Equation (2.2) will be a polynomial in log(1/ϵ) and grow sublinearly with respect to 1/ϵ.

2.3.2

Extension to other partial differential operators

Our learning rate for elliptic partial differential operators (PDOs) in three variables(see Section 2.2) depends on the decay of the singular values of the Green’s function
on admissible domains [16]. We expect that one can also find the learning rate forother PDOs.
It is known that the Green’s functions associated to elliptic PDOs in two dimensions exist and satisfy the following pointwise estimate [53]:



1
R
1
|G(x, y)| ≤ C+ log, ∥x − y∥2 ≤ R := max(dx , dy ), (2.25)
2γR∥x − y∥22where dx = dist(x, ∂D), γ is a constant depending on the size of the domain D, andC is an implicit constant. One can conclude that G(x, ·) is locally integrable for allx ∈ D with ∥G(x, ·)∥Lp (Br (x)∩D) < ∞ for r > 0 and 1 ≤ p < ∞. We believe that the
pointwise estimate in Equation (2.25) implies the off-diagonal low-rank structure of
G here, as suggested in [16]. Therefore, we expect that the results in this chapter can

40

be extended to elliptic PDOs in two variables. It should also be possible to characterize the learning rate for elliptic PDOs with lower order terms (under reasonableconditions) [54, 94, 106] as the associated Green’s functions have similar regularityand pointwise estimates. The main task is to extend [16, Thm. 2.8] to constructseparable approximations of the Green’s functions on admissible domains.
PDOs in four or more variables are far more challenging since we rely on thefollowing bound on the Green’s function on non-admissible domains [80]:G(x, y) ≤c(d, κC )∥x − y∥22−d ,λminx ̸= y ∈ D,

where D ⊂ Rd , d ≥ 3 is the dimension, and c is a constant depending only on dand κC . This inequality implies that the Lp -norm of G on non-admissible domains is
finite when 0 ≤ p < d/(d − 2). However, for a dimension d ≥ 4, we have p < 2 andone cannot ensure that the L2 norm of G is finite. Therefore, the Green’s function
may not be compatible with the HS randomized SVD.
The low-rank theory of Bebendorf and Hackbush has been recently extended
from elliptic to parabolic operators [16] and combined with pointwise estimates forGreen’s functions [107] to obtain a learning rate parabolic PDEs, expressed in the
L1 -norm [28]. In contrast, we believe that deriving a theoretical learning rate for
hyperbolic PDOs remains a significant research challenge for many reasons. The firstroadblock is that the Green’s function associated with hyperbolic PDOs do not necessarily lie in L2 (D × D). For example, the Green’s function associated with the wave

equation in three variables, i.e., L = ∂t2 − ∇2 , is not square-integrable as
G(x, t, y, s) =δ(t − s − ∥x − y∥2 ),4π∥x − y∥2(x, t), (y, s) ∈ R3 × [0, ∞),where δ(·) is the Dirac delta function.
Finally, while the extension to nonlinear dynamical systems seem out of reach of
the technique presented in this chapter, characterizing the sample complexity of such
systems and learn finite-dimensional approximations of the dynamics using Koopman
operator theory [6, 34, 110] would be an interesting future research direction.

2.3.3

Connection with neural networks

As a concluding remark, we emphasize that the algorithm described in this chapter
to learn Green’s functions is not meant to be applied in practice. The proof of Theorem 2.3 relies on the construction of a hierarchical partition of the domain D × D andthe HS randomized SVD algorithm applied on each admissible domain. While this41

gives an algorithm for approximating Green’s functions with high probability, it would
be prohibitively computationally expensive to employ the hierarchical scheme and thegeneralization of the randomized SVD to large-scale three-dimensional problems.
However, there are more practical approaches based on deep learning that currently do not yet have theoretical guarantees [65, 71]. As we will see in Chapter 5,
deep learning techniques may be more competitive due to their ability to learn non
self-adjoint problems and the fast optimization algorithms, based on stochastic gradient descent, for training neural networks. There are many possible connections
between the work presented in this chapter and neural networks from practical andtheoretical viewpoints.
A promising opportunity that we will explore in Chapter 5 is to design a NN
that can learn and approximate Green’s functions using input-output training pairs{(fj , uj )}N
j=1 . Once a neural network N has been trained such that ∥N − G∥L2 ≤ϵ∥G∥L2 , the solution to Lu = f can be obtained by computing the integral u(x) =N (x, y)f (y) dy. Therefore, this may give an efficient computational approach fordiscovering operators since a NN is only trained once. Incorporating a priori knowlDedge of the Green’s function into the network architecture design could be particularly
beneficial. As an example, one might exploit the low-rank structure by performing
dimensionality reduction with an autoencoder [71, 75]. One could also wrap the selection of the kernel in the GP for generating random functions and training data intoa Bayesian framework. We expect that the theory developed in this chapter could
guide deep learning experiments regarding the choice of training data and the typeneural network architectures used to take advantage of the hierarchical structure ofGreen’s functions and their singularity along the diagonal.
Finally, we wonder how many parameters in a NN are needed to approximate
a Green’s function associated with elliptic PDOs within a tolerance of 0 < ϵ < 1.
Can one exploit the off-diagonal low-rank structure of Green’s functions to reduce
the number of parameters? We expect the recent work on the characterization of
ReLU NNs’ approximation power is useful [82, 175, 242]. The use of NNs with highapproximation power such as rational NNs might also be of interest to approximate
the singularities of the Green’s function near the diagonal, as we shall see in Chapters 4
and 5.

42

Chapter 3
A generalization of the randomized
singular value decomposition∗
The theory behind the randomized SVD has been extended in Chapter 2 to nonstandard covariance matrices and HS operators. However, the probability bounds,
generalizing [86, Thm. 10.7], are not sharp enough to emphasize the improved performance of covariance matrices with prior information over the standard randomized
SVD. In this chapter, we improve the bounds obtained in Chapter 2 when the matrixvector products are with multivariate Gaussian random vectors. Our theory allows for
multivariate Gaussian random input vectors that have a general symmetric positive
semi-definite covariance matrix. A key novelty of this work is that prior knowledgeof the matrix A can be exploited to design covariance matrices that achieve lower
approximation errors than the randomized SVD with standard Gaussian vectors. We
then design a practical algorithm for learning Hilbert–Schmidt (HS) operators using
random input functions, sampled from a Gaussian process (GP). Examples of applications include learning integral kernels such as Green’s functions associated withlinear partial differential equations, as discussed in the previous chapter.
The choice of the covariance kernel in the GP is crucial and impacts both thetheoretical bounds and numerical results of the randomized SVD. This leads us to
introduce a new covariance kernel based on weighted Jacobi polynomials for learning
HS operators. One of the main advantages of this kernel is that it is directly expressed
as a Karhunen–Loève expansion [101, 133] so that it is faster to sample functions fromthe associated GP than using a standard squared-exponential kernel. In addition, we∗This chapter is based on a paper with Alex Townsend [31], published in ICLR 2022. Townsend
had an advisory role; I proved the theoretical results, performed the numerical experiments, andwas the lead author in writing the paper.

43

show that the smoothness of the functions sampled from a GP with the Jacobi kernel
can be controlled as it is related to the decay rate of the kernel’s eigenvalues.

3.1

Theoretical bounds for non-standard covariance matrices

In this section we provide new probability bounds for GPs with nonstandard covariance matrices. Let m ≥ n ≥ 1 and A be an m × n real matrix with singular value
decomposition A = UΣV∗ , where U and V are orthonormal matrices, and Σ be an
m × n diagonal matrix with entries σ1 (A) ≥ · · · ≥ σn (A) ≥ 0. For a fixed target rankk ≥ 1, we define Σ1 and Σ2 to be the k × k and (n − k) × (n − k) diagonal matrices,which respectively contain the first k singular values of A: σ1 (A) ≥ · · · ≥ σk (A), andthe remaining singular values. Let V1 be the n × k matrix obtained by truncating
V1 after k columns and V2 the remainder. In this section, K denotes a symmetricpositive semi-definite n × n matrix with kth largest eigenvalue λk > 0 and Ω ∈ Rn×ℓa Gaussian random matrix with ℓ ≥ k independent columns sampled from a multivariate normal distribution with covariance matrix K. Finally, we define Ω1 := V1∗ Ωand Ω2 := V2∗ Ω. The following theorem is a refinement of Theorem 2.1. While it isformulated with matrices, the same result holds for HS operators in infinite dimensions.
Theorem 3.1. Let A be an m × n matrix, k ≥ 1 an integer, and choose an oversam-

pling parameter p ≥ 4. If Ω ∈ Rn×(k+p) is a Gaussian random matrix, where eachcolumn is i.i.d. from a multivariate Gaussian distribution with covariance matrix
K ∈ Rn×n , and QR = AΩ is the economized QR decomposition of AΩ, then for allu, t ≥ 1,∥A − QQ∗ A∥F ≤!v
uXn
3k βk ut
1 + ut (k + p)
σj2 (A),p + 1 γk
j=k+1
s

(3.1)

2

with failure probability at most t−p +[ue−(u −1)/2 ]k+p . Here, the covariance quality factors are denoted by γk = k/(λ1 Tr((V1∗ KV1 )−1 ))) and βk = Tr(Σ22 V2∗ KV2 )/(λ1 ∥Σ2 ∥2F ),where λ1 is the largest eigenvalue of K.

This result differs from Theorem 2.1 and [86, Thm. 10.5] due to the additional
factors γk and βk , which measure the quality of the covariance matrix to learn A
in Theorem 3.1. They can be respectively bounded (Lemmas 2.2 and 3.2) using the44

eigenvalues λ1 ≥ · · · ≥ λn of the covariance matrix K and the singular values of A

as:

n
1
1 X λ1≤,γkk j=n−k+1 λj

βk ≤

n
X
λj−k

λ1

j=k+1

σj2 (A) X
nσj2 (A).

(3.2)

j=k+1

This shows that the performance of the generalized randomized SVD depends on the
decay rate of the sequence {λj }. The quantities γk and βk depend on how much priorinformation of the k + 1, . . . , n right singular vectors of A is encoded in K. In theideal situation where these singular vectors are known, then one can define K suchthat βk = 0 for λk+1 = · · · = λn = 0. Unlike the weaker but more explicit boundproven in Section 2.1, this highlights that a suitably chosen covariance matrix canoutperform the randomized SVD with standard Gaussian vectors (see Section 3.4.1

for a numerical example).
The proof of Theorem 3.1 will require bounding ∥Ω†1 ∥2F , which we achieve using

Lemma 2.3, as well as the term ∥Σ2 Ω2 ∥2F , which is done in the following lemma.

Lemma 3.1. With the notations introduced at the beginning of the section, for alls ≥ 0, we haveP ∥Σ2 Ω2 ∥2F > ℓ(1 + s) Tr(Σ22 V2∗ KV2 ) ≤ (1 + s)ℓ/2 e−sℓ/2 .

Proof. Let ωj be the jth column of Ω for 1 ≤ j ≤ ℓ and v1 , . . . , vn be the n columnsof the orthonormal matrix V. We first remark that∥Ω2 ∥2F =

ℓ
X

Zj ,

Zj :=

n−k
X

2
∗
σk+n
(A)(vk+nω )2 ,
1
1 j

n1 =1

j=1

where the Zj are i.i.d. because ωj ∼ N (0, K) are i.i.d. Let λ1 ≥ · · · ≥ λn ≥ 0 be theeigenvalues of K with eigenvectors ψ1 , . . . , ψn ∈ Rn . For 1 ≤ j ≤ ℓ, we have,
n
X
p
(j)
ωj =
(ci )2 λi ψi ,
i=1

(j)

where ci ∼ N (0, 1) are i.i.d. for 1 ≤ i ≤ n and 1 ≤ j ≤ ℓ. Then,
Zj =

n
X
i=1

(j)
(ci )2 λi

n−k
X

2
∗
σk+n
(A)(vk+nψ )2 =
1
1 i

n1 =1

Xi

i=1

where the Xi are independent. Let γi = λifor 1 ≤ i ≤ n.

n
X

Pn−k

45

2
∗
2
2
n1 =1 σk+n1 (A)(vk+n1 ψi ) , then Xi ∼ γi χLet 0 < θ < 1/(2Pn

i=1 Xi as

h

E eθ

Pn

i=1 XiiPn

i=1 γi ).

We can bound the moment generating function ofnnY θXi  Y=
E e=
(1 − 2θγi )−1/2 ≤
i=1

i=1

1 − 2θ

n
X

γi

i=1

!−1/2

because the Xi /γi are independent and follow a chi-squared distribution. The rightinequality is obtained by showing by recurrence that, if a1 , . . . , an ≥ 0 are such
Pn
Qn
Pn
that
i=1 ai ≤ 1, theni=1 (1 − ai ) ≥ 1 −i=1 ai . For convenience, we definePn
C1 := i=1 γn , we have shown that
E eθZj ≤ (1 − 2θC1 )−1/2 .
Moreover, we find that
C1 =

n−k
X

2
σk+n
v∗
1 k+n1

n1 =1

n
X
i=1

= Tr(Σ22 V2∗ KV2 ).

ψi∗ λi ψi!

vk+n1 =

n−k
X

2
∗
σk+n
(A)vk+n
Kvk+n111

n1 =1Let s ≥ 0 and 0 < θ < 1/(2C1 ). By the Chernoff bound [41, Thm. 1], we obtainℓP ∥Σ2 Ω2 ∥2F > ℓ(1 + s)C1 ≤ e−(1+s)C1 ℓθ E eθZj
= e−(1+s)C1 ℓθ (1 − 2θC1 )−ℓ/2 .

We minimize the bound over 0 < θ < 1/(2 Tr(K)) by choosing θ = s/(2(1 + s)C1 ),which givesP ∥Σ2 Ω2 ∥2F > ℓ(1 + s)C1 ≤ (1 + s)ℓ/2 e−ℓs/2 .

We now prove Theorem 3.1, which provides a refined probability bound for the
performance of the generalized randomized SVD on matrices.
Proof of Theorem 3.1. Using Theorem 2.2 and the submultiplicativity of the Frobenius norm, we have∥A − QQ∗ A∥2F ≤ ∥Σ2 ∥2F + ∥Σ2 Ω2 ∥2F ∥Ω†1 ∥2F .

(3.3)

Let ℓ = k + p with p ≥ 4. Combining Lemmas 2.3 and 3.1 to bound the terms∥Σ2 Ω2 ∥2F and ∥Ω†1 ∥2F in Equation (3.3) yields the following probability estimate:k+p∥A − QQ∗ A∥2F ≤ ∥Σ2 ∥2F + 3t2 (1 + s)
Tr((V1∗ KV1 )−1 ) Tr(Σ22 V2∗ KV2 )
p+1

 n
(k + p)k βk X 22≤ 1 + 3t (1 + s)
σ (A),
p + 1 γk j=k+1 j
46

with failure probability at most t−p + (1 + s)(k+p)/2 e−s(k+p)/2 . Note that we introduced
γk := k/(λ1 Tr((V1∗ KV1 )−1 ))) and βk := Tr(Σ22 V2∗ KV2 )/(λ1 ∥Σ2 ∥2F ). We conclude the√proof by defining u = 1 + s ≥ 1.
The following Lemma provides an estimate of the quantity βk introduced in thestatement of Theorem 3.1.
Lemma 3.2. Let βk = Tr(Σ22 V2∗ KV2 )/(λ1 ∥Σ2 ∥2F ), then the following inequality holds
βk ≤

n
X
λj−k

j=k+1

λ1

 X
n

σj2 (A)σj2 (A).

j=k+1

Proof. Let µ1 ≥ · · · ≥ µn−k be the eigenvalues of the matrix V2∗ KV2 . Using von
Neumann’s trace inequality [152, 230], we haveTr(Σ22 V2∗ KV2 ) ≤nXµj−k σj2 (A).

j=k+1

Then, the matrix V2∗ KV2 is a principal submatrix of V∗ KV, which has the same
eigenvalues of K. Therefore, by [104, Thm. 6.46], the eigenvalues of V2∗ KV2 areindividually bounded by the eigenvalues of K, i.e., µj ≤ λj for 1 ≤ j ≤ n − k, whichconcludes the proof.

Finally, we highlight that the statement of Theorem 3.1 can be simplified bychoosing p = 5, t = 4, and u = 3 to highlight the difference with the standard bounds
for the randomized SVD.
Corollary 3.1 (Generalized randomized SVD). Let A be an m × n matrix and k ≥ 1
an integer. If Ω ∈ Rn×(k+5) is a Gaussian random matrix, where each column is

i.i.d. from a multivariate Gaussian distribution with symmetric positive semi-definite

covariance matrix K ∈ Rn×n , and QR = AΩ is the economized QR decomposition ofAΩ, thenP ∥A − QQ∗ A∥F ≤

1+9

s


!v
uX
n
βk u
t
k(k + 5)σj2 (A)  ≥ 0.999.
γk
j=k+1

In contrast, a simplification of the theorem for the randomized SVD [86, Thm. 10.7]
by choosing t = 6 and u = 4 gives the following result.

47

Corollary 3.2 (Randomized SVD). Let A be an m×n matrix and k ≥ 1 an integer. IfΩ ∈ Rn×(k+5) is a standard Gaussian random matrix and QR = AΩ is the economized

QR decomposition of AΩ, then



v
uX
n

√
u
P ∥A − QQ∗ A∥F ≤ 1 + 16 k + 5 t
σj2 (A)  ≥ 0.999.


j=k+1

The following proposition bounds the expected approximation error of the randomized SVD with multivariate Gaussian inputs.
Proposition 3.1. Let A be an m × n matrix, k ≥ 1 an integer, and choose an

oversampling parameter p ≥ 2. If Ω ∈ Rn×(k+p) is a Gaussian random matrix, where

each column is sampled from a multivariate Gaussian distribution with covariancematrix K ∈ Rn×n , and QR = AΩ is the economized QR decomposition of AΩ, then,
s
!v
uX
n
βk k(k + p) u∗t
E [∥A − QQ A∥F ] ≤ 1 +
σj2 (A),γk p − 1
j=k+1

where γk = k/(λ1 Tr((V1∗ KV1 )−1 ))) and βk = Tr(Σ22 V2∗ KV2 )/(λ1 ∥Σ2 ∥2F ).

We remark that for standard Gaussian inputs, we have γk = βk = 1 in Proposition 3.1, and we recover the average Frobenius error of the randomized SVD [86,Thm. 10.5] up to a factor of (k + p) due to the non-independence of Ω1 and Ω2 ingeneral. The proof of Proposition 3.1 consists of combining the proof of Theorem 2.1
with the following lemma, which is a refinement of Lemma 2.5.
Lemma 3.3. Let ℓ ≥ 1, Ω ∈ Rn×ℓ be a Gaussian random matrix, where each column
is sampled from a multivariate Gaussian distribution with covariance matrix K, andT be an ℓ × k matrix. Then,E[∥Σ2 V2∗ ΩT∥2F ] = Tr(Σ22 V2∗ KV2 )∥T∥2F .

(3.4)

Proof. Let K = QK ΛQ∗K be the eigenvalue decomposition of K, where QK is orthonormal and Λ is a diagonal matrix containing the eigenvalues of K in decreasingorder: λ1 ≥ · · · ≥ λn ≥ 0. We note that Ω can be expressed as Ω = QK Λ1/2 G, where
G is a standard Gaussian matrix. Let S = Σ2 V2∗ QK Λ1/2 , the proof follows from [86,Prop. A.1], which shows that E∥SGT∥2F = ∥S∥2F ∥T∥2F .

Note that one can bound the term Tr(Σ22 V2∗ KV2 ) by λ1 ∥Σ2 ∥2F , where λ1 is thelargest eigenvalue of K (cf. Lemma 2.5). While this provides a simple upper bound, it

does not demonstrate that the use of a covariance matrix containing prior informationon the singular vectors of A can outperform the randomized SVD with standard
Gaussian inputs.
48

3.2

Randomized SVD for Hilbert–Schmidt operators

We now describe the randomized SVD for learning HS operators (see Algorithm 2).
The algorithm is implemented in the Chebfun software system [56], which is a MATLAB package for computing with functions. The Chebfun implementation of therandomized SVD for HS operators uses Chebfun’s capabilities, which offer continuous analogues of several matrix operations like the QR decomposition and numericalintegration. Indeed, the continuous analogue of a matrix-vector multiplication AΩ
for an HS integral operator F (see Section 1.6 for definitions and properties of HS
operators), with kernel G : D × D → R, is(F f )(x) =G(x, y)f (y) dy,Dx ∈ D, f ∈ L2 (D),where D ⊂ Rd with d ≥ 1.
Algorithm 2 Randomized SVD for HS operators
Input: HS integral operator F with kernel G(x, y), number of samples k > 0
Output: Approximation Gk of G
1: Define a GP covariance kernel K
2: Sample the GP k times to generate a quasimatrix of random functions Ω =[f1 . . . fk ]3: Evaluate the integral operator at Ω, Y = [F (f1 ) . . . F (fk )]4: Orthonormalize the columns of Y , Q = orth(Y ) = [q1 . . . qk ]5: Compute an approximation to G by evaluating the adjoint of F
6: Initialize Gk (x, y) to 0
7: for i = 1 : k do8:Gk (x, y) ← Gk (x, y) + qi (x) D G(z, y)qi (z) dz
The algorithm takes as input an integral operator that we aim to approximate.
Note that we focus here on learning an integral operator, but other HS operators
would work similarly. The first step of the randomized SVD for HS operators consistsof generating a D × k quasimatrix Ω by sampling a GP k times, where k is thetarget rank (see Section 3.3). Therefore, each column of Ω is an object, consisting ofa polynomial approximation of a smooth random function sampled from the GP in
the Chebyshev basis. After evaluating the HS operator at Ω to obtain a quasimatrix
Y , we use the QR algorithm [218] to obtain an orthonormal basis Q for the rangeof the columns of Y . Then, the randomized SVD for HS operators requires the49left-evaluation of the operator F or, equivalently, the evaluation of its adjoint Ftsatisfying:(Ft f )(x) =G(y, x)f (y) dy,Dx ∈ D.

We evaluate the adjoint of F at each column vector of Q to construct an approximation Gk of G. Finally, the approximation error between the operator kernel G
and the learned kernel Gk can be computed in the L2 -norm, corresponding to the HS
norm of the integral operator.

3.3

Covariance kernels

To generate the random input functions f1 , . . . , fk for the randomized SVD for HS
operators, we draw them from a GP, denoted by GP(0, K), for a certain covariancekernel K. A widely employed covariance kernel is the squared-exponential function
KSE [187] given by
KSE (x, y) = exp −|x − y|2 /(2ℓ2 ) ,x, y ∈ D,(3.5)

where ℓ > 0 is a parameter controlling the length-scale of the GP. This kernel isisotropic as it only depends on |x − y|, is infinitely differentiable, and its eigenvalues

decay supergeometrically to 0. Since the bound in Theorem 3.1 degrades as theratio λ1 /λj increases for j ≥ k + 1 (cf. Equation (3.2)), the randomized SVD forlearning HS operators prefers covariance kernels with slowly decaying eigenvalues.
Our randomized SVD cannot hope to learn HS operators where the range of theoperator has a rank greater than k̃, where k̃ is such that the k̃th eigenvalue of KSE
reaches machine precision. In Figure 3.1, we display the squared-exponential kernel
with length-scale parameters ℓ = 1, 0.1, 0.01 together with sampled functions fromGP(0, KSE ). We observe that the functions become more oscillatory as the lengthscale parameter ℓ decreases and hence the numerical rank of the kernel increases or,equivalently, the associated eigenvalues {λj } decay more slowly to zero.

Other popular kernels for GPs include the Matérn kernel [131, 187] and Brownian
bridge [161]. Prior information on the HS operator can also be enforced through the
choice of the covariance kernel. For instance, one can impose the periodicity of thesamples by using the following squared-exponential periodic kernel:



2
x−y
2
KPer (x, y) = exp − 2 sin, x, y ∈ D,ℓ2
where ℓ > 0 is the length-scale parameter.
50

ℓ=1

ℓ = 0.1

1.0

1.00

0.5

ℓ = 0.01

1.0

1.00

1.0

1.00

0.75

0.5

0.75

0.5

0.75

y 0.0

0.50

y 0.0

0.50

y 0.0

0.50

−0.5

0.25

−0.5

0.25

−0.5

0.25

0.00

−1.0
−1.0

0.00

−1.0
−1.0

−1.0
−1.0

−0.5

0.0

0.5

1.0

x

−0.5

0.0

0.5

1.0

x

2

2

4

1

1

2

0

0

0

−1

−1

−2

−2
−1.0

−0.5

0.0

0.5

−2
−1.0

1.0

x

−0.5

0.0

0.5

−4
−1.0

1.0

x

0.00
−0.5

0.0

−0.5

0.0

0.5

1.0

x

0.5

1.0

x

Figure 3.1: Squared-exponential covariance kernel KSE with parameter ℓ = 1, 0.1, 0.01
(top row) and five functions sampled from GP(0, KSE ) (bottom row).

3.3.1

Sample random functions from a Gaussian process

In finite dimensions, a random vector u ∼ N (0, K), where K ∈ Rn×n is a covariancematrix with Cholesky factorization K = LL∗ , can be generated from the matrixvector product u = Lc. Here, c ∈ Rn is a vector whose entries follow the standard
Gaussian distribution. We now detail how this process extends to infinite dimensions

with a continuous covariance kernel. Let K be a continuous symmetric positivedefinite covariance function defined on the domain [a, b] × [a, b] ⊂ R2 with −∞ < a <b < ∞. We consider the continuous analogue of the Cholesky factorization to write
K as [218]K(x, y) =∞X

rj (x)rj (y) = Lc (x)L∗c (y),j=1x, y ∈ [a, b],

where rj is the jth row of Lc , which —in Chebfun’s terminology— is a lower-triangularquasimatrix. In practice, we truncate the series after n terms, either arbitrarily or
when the nth largest kernel eigenvalue, λn , falls below machine precision. Then, if
c ∈ Rn follows the standard Gaussian distribution, a function u can be sampled fromGP(0, K) as u = Lc c. That is,u(x) =nX

cj rj (x),j=1

x ∈ [a, b].

The continuous Cholesky factorization is implemented in Chebfun2 [217], which is theextension of Chebfun for computing with two-dimensional functions. As an example,51

the polynomial approximation, which is accurate up to essentially machine precision,of the squared-exponential covariance kernel KSE with parameter ℓ = 0.01 on [−1, 1]2
yields a numerical rank of n = 503. The functions sampled from GP(0, KSE ) become
more oscillatory as the length-scale parameter ℓ decreases and hence the numericalrank of the kernel increases or, equivalently, the associated eigenvalues sequence {λj }decays more slowly to zero.

3.3.2

Influence of the kernel’s eigenvalues and Mercer’s representation

The covariance kernel can also be defined from its Mercer’s representation asK(x, y) =∞X

λj ψj (x)ψj (y),j=1x, y ∈ D,(3.6)

where {ψj } is an orthonormal basis of L2 (D) and λ1 ≥ λ2 ≥ · · · > 0 [93, Thm. 4.6.5].

We prefer to construct K directly from Mercer’s representations for several reasons.

First, one can impose prior knowledge of the kernel of the HS operator on the eigenfunctions of K (such as periodicity or smoothness). Then, one can often generatesamples from GP(0, K) efficiently using Equation (3.6). Finally, one can control the
decay rate of the eigenvalues of K.

Hence, the quantity γk in the probability bound of Theorem 3.1 measures thequality of the covariance kernel K in GP(0, K) to generate random functions that canlearn the HS operator F. To minimize 1/γk we would like to select the eigenvalues

λ1 ≥ λ2 ≥ · · · > 0 of K so that they have the slowest possible decay rate whileP∞
1maintaining
j=1 λj < ∞. One needs {λj } ∈ ℓ to guarantee that ω ∼ GP(0, K)P
has finite expected squared L2 -norm, i.e., E[∥ω∥2L2 (D) ] = ∞j=1 λj < ∞. The best
sequence of eigenvalues we know that satisfies this property is called the Rissanen
sequence [189] and is given by λj = Rj := 2−L(j) , where
L(j) = log2 (c0 ) + log∗2 (j),log∗2 (j) =

∞
X
i=2

(i)

max(log2 (j), 0),

c0 =

∞
X

∗

2− log2 (i) ,

i=2

(i)

and log2 (j) = log2 ◦ · · · ◦ log2 (j) is the composition of log2 (·) i times. Other options
for the choice of eigenvalues include any sequence of the form λj = j −ν for ν > 1.

3.3.3

Jacobi covariance kernel

If D = [−1, 1], then a natural choice of orthonormal basis of L2 (D) to define the
Mercer’s representation of the kernel are weighted Jacobi polynomials [50, 166]. That52

is, for a weight function wα,β (x) = (1 − x)α (1 + x)β with α, β > −1, and any positive

eigenvalue sequence {λj }, we consider the Jacobi kernel(α,β)
KJac (x, y) =

∞
X

1/2

(α,β)

λj+1 wα,β (x)P̃j

1/2

(α,β)

(x)wα,β (y)P̃j

(y),

j=0

(α,β)

where P̃jx, y ∈ [−1, 1],(3.7)

is the scaled Jacobi polynomial of degree j and parameters (α, β) where

(α,β)
Pj
is defined by Rodrigues’ formula [211, Eq. 4.3.1] as(α,β)

wα,β (x)Pj(α,β)

The polynomials P̃j(x) =(−1)j dj 
wα,β (x)(1 − x2 )j .
2j n! dxj
1/2

(α,β)

are normalized such that ∥wα,β P̃j(α,β)

is chosen such that KJac
sampled asu(x) =∥L2 ([−1,1]) = 1 and {λj }∈ L2 ([−1, 1]2 ). In this case, a random function can be∞Xp
1/2 (α,β)λj+1 cj wα,β P̃j
(x),
j=0

x ∈ [−1, 1],

where cj ∼ N (0, 1) for 0 ≤ j ≤ ∞.

A desirable property of a covariance kernel is to be unbiased towards one spatial

direction, i.e., K(x, y) = K(−y, −x) for x, y ∈ [−1, 1], which motivates us to always(α,β)

select α = β. Moreover, it is desirable to have the eigenfunctions of KJac to bepolynomial so that one can generate samples from GP(0, K) efficiently. This leads us

to choose α and β to be even integers. The choice of α = β = 0 gives the Legendre
kernel [68, 83]. In the rest of this chapter, we will use Equation (3.7) with α = β = 2 toensure that functions sampled from the associated GP satisfy homogeneous Dirichletboundary conditions (see Figure 3.3). We emphasize that covariance kernels on higher
dimensional domains of the form D = [−1, 1]d , for d ≥ 2, can be defined using tensor

products of weighted Jacobi polynomials.

3.3.4

Smoothness of functions sampled from a GP with Jacobi kernel(2,2)

We now connect the decay rate of the eigenvalues of the Jacobi covariance kernel KJac(2,2)

to the smoothness of the samples from GP(0, KJac ). Hence, the Jacobi covariancefunction allows the control of the decay rate of the eigenvalues {λj } as well as thesmoothness of the resulting randomly generated functions. First, Lemma 3.4 assertsthat if the coefficients of an infinite polynomial series have sufficient decay, then the
resulting series is smooth with regularity depending on the decay rate. This result
can be seen as a converse to [220, Thm. 7.1].
53

Lemma 3.4. Let {pj } be a family of polynomials such that maxx∈[−1,1] |pj (x)| = 1 andPdeg(pj ) ≤ j. If fn (x) = nj=0 aj pj (x) with |aj | ≤ j −ν for ν > 1, then fn converges
P
uniformly to f (x) = ∞j=0 aj pj (x) and f is µ times continuously differentiable for any
integer µ such that µ < (ν − 1)/2.
Proof. By Markov brothers’ inequality [142], for all j ≥ 0 and 0 ≤ µ ≤ j, we have
P
P
(µ)
(µ)
(µ)
maxx∈[−1,1] |pj (x)| ≤ j 2µ . Therefore, |fn (x)| ≤ nj=0 |aj |∥pj ∥∞ ≤ nj=0 j 2µ−ν so(µ)|fn (x)| < ∞ if µ < (ν − 1)/2. The result follows from a standard result on uniform
convergence and differentiation [193, Thm. 7.17].

Note that the main application of this lemma occurs when deg(pj ) = j for all j ≥ 0.

We then prove a bound on ultraspherical polynomials in order to apply Lemma 3.4(2,2)

to functions sampled from the GP with the Jacobi covariance kernel KJac . First,
(5/2)
(2,2)
with parameter 5/2 andis a scaled ultraspherical polynomial C̃jnote that P̃jdegree j ≥ 0 so it can be bounded by the following proposition.
(5/2)

Proposition 3.2. Let C̃jbe the ultraspherical polynomial of degree j with param1(5/2)
eter 5/2, normalized such that −1 (1 − x2 )2 C̃j (x)2 dx = 1. Then,(5/2)

max |(1 − x2 )C̃jx∈[−1,1]p(x)| ≤ 2 j + 5/12,

j ≥ 0.

(3.8)

Proof. Let j ≥ 0 and x ∈ [−1, 1], according to [166, Table 18.3.1],sj + 5/2
(5/2)
(5/2)
C
(x),
C̃j (x) = 3(j + 1)(j + 2)(j + 3)(j + 4) j
(5/2)

where Cj

(3.9)(x) is the standard ultraspherical polynomial. Using [166, (18.9.8)], we

have
(3/2)
(3/2)
(j + 3)(j + 4)Cj (x) − (j + 1)(j + 2)Cj+2 (x)(5/2)(1 − x )Cj (x) =.
2

6(j + 5/2)

(3/2)

(3/2)

By using [166, (18.9.7)], we have (Cj+2 (x) − Cj(1/2)(x))/2 = (j + 5/2)Cj+2 (x) andhence,

2 (3/2)(j + 1)(j + 2) (1/2)
Cj+2 (x).
(x) = Cj (x) −3
3
We bound the two terms with [166, (18.14.4)] to obtain the following inequalities:(5/2)(1 − x2 )Cj

(3/2)

|Cj

(x)| ≤(j + 1)(j + 2)
,
2

54

(1/2)

|Cj+2 (x)| ≤ 1.

(a) 2(b) 102

1

0

10

p j+
2

1

x x∈

−1

ma

−2
−1

−0.5

0
x

0.5

100 0
10

1

[−

]
1,1

101

|(1

5/

−

12
)|

) x
/2 (
(5
2 )C̃ j

x

102

103

Degree j(5/2)

Figure 3.2: (a) Normalized ultraspherical polynomials C̃j
up to degree 10. (b)Theoretical bound (blue line) for the maximum of weighted ultraspherical polynomialson [−1, 1], obtained in Proposition 3.2, against the one observed numerically (redline).
(5/2)

Hence, |(1 − x2 )Cj(x)| ≤ 2(j + 1)(j + 2)/3 and following Equation (3.9) we obtain(5/2)|(1 − x2 )C̃js(x)| ≤ 2p(j + 1)(j + 2)(j + 5/2)
≤ 2 j + 5/12,(j + 3)(j + 4)

which concludes the proof.
The bound given in Proposition 3.2 differs initially by a factor of 4/3 from thepnumerically observed upper bound (1.5 j + 5/12 ) as shown by Figure 3.2. We
now state the following theorem about the regularity of functions sampled from
(2,2)
GP(0, KJac ), which guarantees that if the eigenvalues are chosen such that λj =(2,2)

O(1/j ν ) with ν > 3, then f ∼ GP(0, KJac ) is almost surely continuous. Moreover,(2,2)

a faster decay of the eigenvalues of KJac implies higher regularity of the sampled
functions, in an almost sure sense.
Theorem 3.2. Let {λj } ∈ ℓ1 (R+ ) be a positive sequence such that λj = O(j −ν ) for(2,2)

ν > 3. If f is sampled from GP(0, KJac ), then f ∈ C µ ([−1, 1]) almost surely for any

integer µ < (ν − 3)/2.

(2,2)

Proof. Since f ∼ GP(0, KJac ), f ∼P∞

j=0 cjp(2,2)
λj+1 (1 − x2 )P̃j (x), where cj ∼N (0, 1) for j ≥ 0. Let fn denote the truncation of f after n terms. By letting M > 0
be the constant such that λj+1 ≤ M (j + 1)−ν , we find that∥f − fn ∥∞ ≤ Sn ,∞X√
|cj−1 |j (1−ν)/2 ,Sn := 2 M
j=n+2

55

(2,2)

where we used maxx∈[−1,1] |(1 − x2 )P̃j

we have√(x)| ≤ 2 j + 1 (cf. Proposition 3.2). Thus,lim ∥f − fn ∥∞ = 0 ≥ P lim Sn = 0 .

P
n→∞
n→∞
P∞
Here, Sn ∼ Xn = j=n+2 Yj j (1−ν)/2 , where Yj follows a half-normal distribution [123]a.s.

with parameter σ = 1 and the (Yj )j are independent. We want to show that Xn −−→ 0.

For ϵ > 0, using Chebyshev’s inequality, we have:

 ∞


∞ 
∞
1 X
2 X 1
2
1 X 1
1
P(|Xn | ≥ ϵ) ≤ 21−≤ 2 1−,ϵ n=0
π j=n+2 j ν−1ϵπ ν − 2 n=1 nν−2
n=0

∞
X

which is finite if ν > 3. Therefore, using the Borel–Cantelli Lemma [57, Chapt. 2.3],
Xn converges to 0 almost surely and P(limn→∞ Xn = 0) = 1. Finally,
P lim ∥f − fn ∥∞ = 0 ≥ P lim Xn = 0 = 1,
n→∞

n→∞

which proves that {fn } converges uniformly and hence f is continuous with probability
one. The statement for higher order derivatives follows the proof of Lemma 3.4.

This theorem can be seen as a particular case of Driscoll’s zero-one law [55], whichcharacterizes the regularity of functions samples from GPs (see also [99]). Hence, onePmust have ∞j=1 jλj < ∞ so that the series of functions in Equation (3.7) converges(2,2)
uniformly and KJac is a continuous kernel. Under this additional constraint, thebest choice of eigenvalues is given by a scaled Rissanen sequence: λj = Rj /j, forj ≥ 1 (cf. Section 3.3.2). In Figure 3.3, we display the Jacobi kernel of type (2, 2)
with functions sampled from the corresponding GP. We selected eigenvalue sequencesof different decay rates: from the faster 1/j 4 to the slower Rissanen sequence Rj /j(Section 3.3.2). For λj = 1/j 3 and λj = Rj /j, we observe a large variation of therandomly generated functions near x = ±1, indicating a potential discontinuity ofthe samples at these two points as n → ∞. This is in agreement with Theorem 3.2,
which only guarantees continuity (with probability one) of the randomly generatedfunctions if λj ∼ 1/j ν with ν > 3.

3.4

Numerical experiments

We now perform several numerical experiments with the randomized SVD to learn
matrices using random vectors sampled for a multivariate Gaussian distribution and
HS operators.

56λj = 1/j 4

λj = 1/j 3

1.0

1.00

0.5

λj = Rj /j

1.0

1.00

1.0

1.00

0.75

0.5

0.75

0.5

0.75

y 0.0

0.50

y 0.0

0.50

y 0.0

0.50

−0.5

0.25

−0.5

0.25

−0.5

0.25

0.00

−1.0
−1.0

0.00

−1.0
−1.0

−1.0
−1.0

−0.5

0.0

0.5

1.0

x

−0.5

0.0

0.5

1.0

x

2

2

2

1

1

1

0

0

0

−1

−1

−1

−2
−1.0

−0.5

0.0

x

0.5

1.0

−2
−1.0

−0.5

0.0

0.5

1.0

x

−2
−1.0

0.00
−0.5

0.0

−0.5

0.0

0.5

1.0

x

0.5

1.0

x

(2,2)

Figure 3.3: Covariance kernel KJac constructed using Jacobi polynomials of type(2, 2) with λj = 1/j 4 , 1/j 3 , and Rj /j, where Rj is the Rissanen sequence (top).
(2,2)
The bottom panels illustrate functions sampled from GP(0, KJac ) with the different
eigenvalue sequences. The series for generating the random functions are truncatedto n = 500.

3.4.1

Covariance matrix with prior knowledge

The approximation error bound in Theorem 3.1 depends on the eigenvalues of thecovariance matrix, which dictates the distribution of the column vectors of the input
matrix Ω. Roughly speaking, the more prior knowledge of the matrix A that canbe incorporated into the covariance matrix, the better. In this numerical example,
we investigate whether the standard randomized SVD, which uses the identity as
its covariance matrix, can be improved by using a different covariance matrix. We
then attempt to learn the discretized 2000 × 2000 matrix, i.e., the discrete Green’s
function, of the inverse of the following differential operator:
Lu = d2 u/dx2 − 100 sin(5πx)u,x ∈ [0, 1].

We vary the number of columns (i.e. samples from the GP) in the input matrix Ωfrom 1 to 2000.
In Figure 3.4(a), we compare the ratios between the relative error in the Frobenius norm given by the randomized SVD and the best approximation error, obtainedby truncating the SVD of A. The prior covariance matrix K consists of the discretized 2000 × 2000 matrix of the Green’s function of the negative Laplace operator
Lu = −d2 u/dx2 on [0, 1] to incorporate knowledge of the diffusion term in the matrix

A. We see that a nonstandard covariance matrix leads to a higher approximation
57

2.5

2

1.5

1
0

500

1,000

1,500

Number of samples

2,000(b) 0.5(c) 10

0

0.4

10−3

0.3

10−6

Error

Standard covariance
Prior covarianceBest approximation

Computational time (s)Error / Best error

(a) 3

0.2

10−12

0.1
0

10−9

0

500

1,000

1,500

Number of samples

2,000

10−15

0

0.1

0.2

0.3

0.4

Computational time (s)0.5

Figure 3.4: (a) Ratio between the average randomized SVD approximation error(over 10 runs) of the 2000 × 2000 matrix of the inverse of the differential operator
Lu = d2 u/dx2 − 100 sin(5πx)u on [0, 1], and the best approximation error. The error
bars in light colour (blue and red) illustrate one standard deviation. (b) Averagecomputational time of the algorithm (over 10 runs). The eigenvalue decompositionof the covariance matrix has been precomputed offline. (c) Randomized SVD approximation error with standard and prior covariance matrices with respect to thecomputational time.
accuracy, with a reduction of the error by a factor of 1.3-1.6 compared to the standard
randomized SVD.
At the same time, the procedure is only 20% slower1 on average (Figure 3.4(b)) asone can precompute the eigenvalue decomposition of the covariance matrix. Hence,
sampling a random vector from a multivariate normal distribution with an arbitrary
covariance matrix K can be computationally expensive when the dimension, n, of thematrix is large as it requires the computation of a Cholesky factorization, which canbe done in O(n3 ) operations. We highlight that this step can be precomputed once,such that the overhead of the generalized SVD can be essentially expressed as thecost of an extra matrix-vector multiplication. Then, the difference in timings between

standard and prior covariance matrices is marginal as shown by Figure 3.4(b).
We observe in Figure 3.4(c) that using a standard covariance matrix offers a better
trade-off between error and computational time. However, choosing a prior covariancematrix is of interest in applications where the sampling time is much higher than the
numerical linear algebra costs to maximize the accuracy of the approximation matrix
from a limited number of samples.
Additionally, we would like to highlight that prior covariance matrices can be
designed and derived using physical knowledge of the problem, such as its diffusive
1
Timings were performed on an Intel Xeon CPU E5-2667 v2 @ 3.30GHz using MATLAB R2020b
without explicit parallelization.

58

nature, which can also significantly decrease the precomputation cost. In this example, we employ the discretized Green’s function of the negative Laplacian operator
with homogeneous Dirichlet boundary conditions, given by Lu = −d2 u/dx2 on [0, 1],
for which we know the eigenvalue decomposition. Hence, the eigenvalues and normalized eigenfunctions are respectively given by
λn =

1
,
π 2 n2ψn (x) =√2 sin(nπx),x ∈ [0, 1], n ≥ 1.

Therefore, one can employ Mercer’s representation (see Equation (3.6)) to sample therandom vectors and precompute the covariance matrix in O(n2 ) operations. For aproblem of size n = 2000, it takes 0.16s to precompute the matrix.

3.4.2

Randomized SVD for Hilbert–Schmidt operators

We now apply the randomized SVD for HS operators to learn kernels of integral
operators. In this first example, the kernel is defined as [215]G(x, y) = cos(10(x2 + y)) sin(10(x + y 2 )),x, y ∈ [−1, 1],

and is displayed in Figure 3.5(a). We employ the squared-exponential covariancekernel KSE with parameter ℓ = 0.01 and k = 100 samples (see Equation (3.5)) tosample random functions from the associated GP. The learned kernel Gk is represented
on the bottom panel of Figure 3.5(a) and has an approximation error around machine
precision.
As a second application of the randomized SVD for HS operators, we learn the kernel G(x, y) = Ai(−13(x2 y + y 2 )) for x, y ∈ [−1, 1], where Ai is the Airy function [166,
Chapt. 9] defined by

1
Ai(x) =
π

 ∞
0

 3

t
cos
+ xt dt,3x ∈ R.

We plot the kernel and its low-rank approximant given by the randomized SVD for
HS operators in Figure 3.5(b) and obtain an approximation error (measured in theL2 -norm) of 5.04 × 10−14 . The two kernels have a numerical rank equal to 42.
The last example consists of learning the HS operator associated with the kernel
G(x, y) = J0 (100(xy + y 2 )) for x, y ∈ [−1, 1], where J0 is the Bessel function of thefirst kind [166, Chapt. 10] defined as
1
J0 (x) =π πcos(x sin t) dt,
059

x ∈ R,

Kernel

(a) 1

(b)

1

0.50

0.5

y

0

0.0

−1

0

1

0

0.00

−1.0

−1

1.0

−1

0

1

0

0.0

1

−1

0

x

1

−1.0

0

0.0
−0.5

−0.50

−1

0.50

−1

0

1

0

0.00

1

1.0
0.5

y

0

0.0

−0.25
−1

−1

0

x

1

−0.50

−1.0

x

0.25

y

−0.5
−1

1.0
0.5

y

x

0.5

y

1

−0.25

x

1

(c)

0.25

y

−0.5
−1

Learned Kernel

1.0

−0.5
−1

−1

0

1

−1.0

x

Figure 3.5: Kernels of three HS operators (top) together with the kernels learned
by the randomized SVD for HS operators (bottom), using the squared-exponential
covariance kernel KSE with parameter ℓ = 0.01 and one hundred functions sampledfrom GP(0, KSE ).
and plotted in Figure 3.5(c). The rank of this kernel is equal to 91 while its approximation is of rank 89 and the approximation error is equal to 4.88 × 10−13 . We observethat in the three numerical examples displayed in Figure 3.5, the differences between
the learned and the original kernels are not visually perceptible.
Finally, we evaluate the influence of the choice of covariance kernel and number ofsamples in Figure 3.6. Here, we vary the number of samples from k = 1 to k = 100 and
use the randomized SVD for HS operators with four different covariance kernels: the(2,2)

squared-exponential KSE with parameters ℓ = 0.01, 0.1, 1, and the Jacobi kernel KJac

with eigenvalues λj = 1/j 3 , for j ≥ 1. In the left panel of Figure 3.6, we represent the

eigenvalue ratio λj /λ1 of the four kernels and observe that this quantity falls below
machine precision for the squared-exponential kernel with ℓ = 1 and ℓ = 0.1 at j = 13

and j = 59, respectively. In Figure 3.6(right), we observe that these two kernels fail toapproximate kernels of high numerical rank. The other two kernels have a much slower
decay of eigenvalues and can capture (or learn) more complicated kernels. We then see
in the right panel of Figure 3.6 that the relative approximation errors obtained using(2,2)

KJac and KSE are close to the best approximation error given by the squared tail ofP
the singular values of the integral kernel G(x, y), i.e., ( j≥k+1 σj2 )1/2 . The overshootin the error at k = 100 compared to the machine precision is due to the decay ofthe eigenvalues of the covariance kernels. Hence, spatial directions associated with60

small eigenvalues are harder to learn accurately. This issue does not arise in finite
dimensions with the standard randomized SVD because the covariance kernel used
there is isotropic, i.e., all its eigenvalues are equal to one. However, this choice is nolonger possible for learning HS integral operators as the covariance kernel K must
be squared-integrable. The relative approximation errors at k = 100 (averaged over
(2,2)

(2,2)

10 runs) using KJac and KSE with ℓ = 0.01 are Error(KJac ) ≈ 2.6 × 10−11 , andError(KSE ) ≈ 5.7 × 10−13 , which gives a ratio of(2,2)

Error(KJac )/Error(KSE ) ≈ 45.6.

(3.10)

However, the square-root of the ratio of the quality of the two kernels for k = 91 isequal toq(2,2)

γ91 (KSE )/γ91 (KJac ) ≈ 117.8,(3.11)

which is of the same order of magnitude of Equation (3.10) as predicted by Theo(2,2)
rem 3.1. In Equation (3.11), γ91 (KSE ) ≈ 5.88 × 10−2 and γ91 (KJac ) ≈ 4.24 × 10−6
are both computed using Chebfun.
101

101
(2,2)

KJac , λj = 1/j 3
KSE , ` = 0.01
KSE , ` = 0.1
KSE , ` = 1

10−2
Relative error

λj /λ1

10−2
10−5
10−8
10−11
10−14

10−5
10−8
10−11

0

100

200

300

400

10−14

500

j

0

20

40
60
Number of samples

80

100

(2,2)

Figure 3.6: Left: Scaled eigenvalues of the Jacobi covariance kernel KJac withsequence λj = 1/j 3 and squared-exponential kernels KSE with parameters ℓ =0.01, 0.1, 1, respectively. Right: Average (over 10 runs) relative approximation error in the L2 -norm between the Bessel kernel G(x, y) = J0 (100(xy + y 2 )) and its
low-rank approximation Gk (x, y), obtained from the randomized SVD by sampling
the GPs k times. The error bars in light colour (blue and red) illustrate one standard
deviation and the black line indicates the best approximation error given by the tailof the singular values of G.
In conclusion, this section provides numerical insights to motivate the choice ofthe covariance kernel to learn HS operators. Following Figure 3.6, a kernel with slowly
decaying eigenvalues is preferable and yields better approximation errors or higher
61learning rate with respect to the number of samples, especially when learning a kernel
with a large numerical rank. The optimal choice from a theoretical viewpoint is toselect a covariance kernel whose eigenvalues have a decay rate similar to the Rissanen
sequence [189], but other choices may be preferable in practice to ensure smoothnessof the sample functions (cf. Section 3.3.4).

62

Chapter 4
Rational neural networks∗
A key question in designing deep learning architectures is the choice of the activation
function to reduce the number of trainable parameters of the network while keeping the same approximation power [75]. While smooth activation functions suchas sigmoid, logistic, or hyperbolic tangent are widely used, they suffer from the“vanishing gradient problem” [19] because their derivatives are zero for large inputs. Neural networks (NNs) based on polynomial activation functions are an alternative [40, 47, 77, 81, 139, 228], but can be numerically unstable due to large gradientsfor large inputs [19]. Moreover, polynomials do not approximate non-smooth functions efficiently [220], which can lead to optimization issues in classification problems.
A popular choice of activation function is the Rectified Linear Unit (ReLU) defined
as ReLU(x) = max(x, 0) [97, 158]. It has numerous advantages, such as being fastto evaluate and zero for many inputs [73]. Many theoretical studies characterize andunderstand the expressiveness of shallow and deep ReLU neural networks from theperspective of approximation theory [52, 129, 150, 213, 242].
ReLU networks also suffer from drawbacks, which are most evident during training. The main disadvantage is that the gradient of ReLU is zero for negative real
numbers. Therefore, its derivative is zero if the activation function is saturated [140].
Several adaptations to ReLU have been proposed over the past few years, such as
Leaky ReLU [140], Exponential Linear Unit (ELU) [45], Parametric Linear Unit
(PReLU) [87], and Scaled Exponential Linear Unit (SELU) [109], to improve theinitialization and optimization of neural networks and avoid the use of batch normalization layers [95]. These modifications outperform ReLU in image classification∗

This chapter is based on a paper with Yuji Nakatsukasa and Alex Townsend [30], published in
NeurIPS 2020. Nakatsukasa and Townsend had advisory roles and proved Lemma 4.1. I proved theother theoretical results, performed the numerical experiments, and was the lead author in writing
the paper.

63

applications, and some of these activation functions have trainable parameters, which
are learned by gradient descent at the same time as the hyperparameters of the network. To obtain significant benefits for image classification and partial differential
equation (PDE) solvers, one can also perform an exhaustive search over trainable
activation functions constructed from standard units [96, 186].
In this chapter, we are motivated by designing activation functions with greaterapproximation power than ReLU. We study rational neural networks, which are neuralnetworks with activation functions that are trainable rational functions. These will
have improved theoretical guarantees on expressivity compared to ReLU, as we shallsee in Section 4.2.

4.1

Definitions

We consider neural networks whose activation functions consist of rational functions
with trainable coefficients ai and bj , i.e., functions of the form:
PrP
ai x iP (x)F (x) =
, aP ̸= 0, bQ ̸= 0,= Pri=0QjQ(x)j=0 bj x(4.1)

where rP and rQ are the polynomial degrees of the numerator and denominator,
respectively. We say that F (x) is of type (rP , rQ ) and degree max(rP , rQ ).
The use of rational functions in deep learning is motivated by the theoretical workof Telgarsky, who proved error bounds on the approximation of ReLU neural networks
by high-degree rational functions and vice versa [214]. On the practical side, neuralnetworks based on rational activation functions are considered by Molina et al. [153],
who defined a safe Padé Activation Unit (PAU) as
PrPii=0 ai xP
F (x) =.
rQ
1 + | j=1
bj x j |The denominator is selected so that F (x) does not have poles located on the real axis.
PAU networks can learn new activation functions and are competitive with state-ofthe-art neural networks for image classification. However, this choice results in a
non-smooth activation function and makes the gradient expensive to evaluate during
training. In a closely related work, Chen et al. [39] propose high-degree rational activation functions in a neural network, which have benefits in terms of approximation
power. However, this choice can significantly increase the number of parameters in
the network, causing the training stage to be computationally expensive.

64

In this chapter, we use low-degree rational functions as activation functions, which
are then composed together by the neural network to build high-degree rational functions. In this way, we can leverage the approximation power of high-degree rational
functions without making training expensive. We highlight the approximation powerof rational networks and provide optimal error bounds to demonstrate that rational
neural networks theoretically outperform ReLU networks. Motivated by our theoretical results, we consider rational activation functions of type (3, 2), i.e., rP = 3 andrQ = 2. This type appears naturally in the theoretical analysis due to the composition property of Zolotarev sign functions (see Section 4.2.1): the degree of the overall
rational function represented by the rational neural network is an enormous 3#layers ,while the number of trainable parameters only grows linearly with respect to thedepth of the network. A low-degree activation function keeps the number of trainable
parameters small, while the implicit composition in a neural network gives us theapproximation power of high-degree rationals. This choice is also motivated empirically, and we do not claim that the type (3, 2) is the best choice for all situations as
the configurations may depend on the application as shown later by Figure 4.5. Our
experiments1 on the approximation of smooth functions and generative adversarial
networks (GANs) suggest that rational neural networks are an attractive alternative
to ReLU networks (see Section 4.3).

4.2

Theoretical results on rational neural networks

Here, we demonstrate the theoretical benefit of using neural networks based on rational activation functions due to their superiority over ReLU in approximating functions. We derive optimal bounds in terms of the total number of trainable parameters(also called size) needed by rational networks to approximate ReLU networks as well
as functions in the Sobolev space W n,∞ ([0, 1]d ), where n, d ≥ 1 are integers. Throughout this chapter, we take ϵ to be a small parameter with 0 < ϵ < 1. We first show thatan ϵ-approximation on the domain [−1, 1]d of a ReLU network (NReLU ) by a rational
neural network (NRational ) must have the following size (indicated in brackets):
NRational [Ω(log(log(1/ϵ)))] ≤ NReLU ≤ NRational [O(log(log(1/ϵ)))],(4.2)

where the constants only depend on the size and depth of the ReLU network. Here,the upper bound means that all ReLU networks can be approximated to within ϵ bya rational network of size O(log(log(1/ϵ))). The lower bound means that there is a1

All code and hyperparameters are publicly available at [29].

65ReLU network that cannot be ϵ-approximated by a rational network of size less thanC log(log(1/ϵ)), for some constant C > 0. In comparison, the size needed by a ReLU
network to approximate a rational neural network within the tolerance of ϵ is givenby the following inequalities:
NReLU [Ω(log(1/ϵ))] ≤ NRational ≤ NReLU [O(log(1/ϵ))3 ],(4.3)

where the constants only depend on the size and depth of the rational neural network. This means that all rational networks can be approximated to within ϵ by a
ReLU network of size O(log(1/ϵ))3 , while there is a rational network that cannot be

ϵ-approximated by a ReLU network of size less than Ω(log(1/ϵ)). A comparison between (4.2) and (4.3) suggests that rational networks could be more expressive than
ReLU.

4.2.1

Approximation of ReLU networks by rational neuralnetworks

Telgarsky showed that neural networks and rational functions can approximate eachother in the sense that there exists a rational function of degree O(polylog(1/ϵ)) thatis ϵ-close to a ReLU network [214, Thm. 1.1], where ϵ > 0 is a small number.
Theorem 4.1 (Telgarsky). Let 0 < ϵ < 1 and let ∥ · ∥1 denote the vector 1-norm.
The following two statements hold:1. Let k be a nonnegative integer and p : [0, 1]d → [−1, 1], q : [0, 1]d → [2−k , 1] bepolynomials of degree ≤ r, each with ≤ s monomials. Then, there exists a ReLU
network NReLU : [0, 1]d → R of size

such that
O k 7 log(1/ϵ)3 + min srk log(sr/ϵ), sdk 2 log(dsr/ϵ)2 ,supx∈[0,1]dNReLU (x) −

p(x)
≤ ϵ.
q(x)

2. Let NReLU : [−1, 1]d → R be a ReLU network with M layers and at most k nodes
per layer, where each node computes x 7→ ReLU(a⊤ x + b) and the pair (a, b) (possibly

distinct across nodes) satisfies ∥a∥1 + |b| ≤ 1. Then, there exists a rational function
R : [−1, 1]d → R with degree (maximum of numerator and denominator)
O k M log(M/ϵ)M ,such thatsup |NReLU (x) − R(x)| ≤ ϵ.

x∈[−1,1]d66

To prove this statement, Telgarsky used a rational function constructed with Newman polynomials [163] to obtain a rational approximation to the ReLU function thatconverges with square-root exponential accuracy. That is, Telgarsky needed a rational function of degree Ω(log(1/ϵ)2 ) to achieve a tolerance of ϵ. A degree r rational
function can be represented with 2(r + 1) coefficients, i.e., a0 , . . . , ar and b0 , . . . , brin Equation (4.1). Therefore, the rational approximation to a ReLU network constructed by Telgarsky requires at least Ω(polylog(1/ϵ)) parameters. In contrast, for
any rational function, Telgarsky showed that there exists a ReLU network of sizeO(polylog(1/ϵ)) that is an ϵ-approximation on [0, 1]d .

Our key observation is that by composing low-degree rational functions together,

we can approximate a ReLU network much more efficiently in terms of the size (rather
than the degree) of the rational network. Our theoretical work is based on a family ofrationals called Zolotarev sign functions, which are the best rational approximationin the infinity norm on [−1, −ℓ] ∪ [ℓ, 1], with 0 < ℓ < 1, to the sign function [3, 176],
defined as−1, x < 0,sign(x) = 0,
x = 0,

1,
x > 0.

We first show that a rational function can approximate the absolute value function |x|on [−1, 1] with square-root exponential convergence using a composition of Zolotarev
functions.
Lemma 4.1. For any integer k ≥ 0, we havemin max ||x| − xr(x)| ≤ 4e−π√k/2r∈Rk,k x∈[−1,1],

where Rk,k is the space of rational functions of type at most (k, k). Thus, xr(x) is aQrational approximant to |x| of type at most (k + 1, k). Moreover, if k = pi=1 ki forsome p ≥ 1 and integers k1 , . . . , kp ≥ 2, then r can be written as r = Rp ◦ · · · ◦ R1 ,where Ri ∈ Rki ,ki .

Proof. Let 0 < ℓ < 1 be a real number and consider the sign function on the domain[−1, −ℓ] ∪ [ℓ, 1], i.e.,(−1, x ∈ [−1, −ℓ],sign(x) =+1, x ∈ [ℓ, 1].

By [17, Equation (33)], we find that for any k ≥ 0,


min
max
|sign(x) − r(x)| ≤ 4 expr∈Rk,k x∈[−1,−ℓ]∪[ℓ,1]67π22 log(4/ℓ)

−k

.

Let r(x) be the rational function of type (k, k) that attains the minimum [17, Equation (12)]. We refer to such r(x) as the Zolotarev sign function. It is given byQ⌊(k−1)/2⌋ 2
x + c2jsn2 (jK(κ)/k; κ)
j=1
r(x) = M x Q⌊k/2⌋, cj = ℓ2.
2 (jK(κ)/k; κ)
2+c
1
−
sn
x
2j−1
j=1

Here, M is a real constant selected so that sign(x) − r(x) equioscillates on [−1, −ℓ] ∪√[ℓ, 1], κ = 1 − ℓ2 , sn(·) is the first Jacobian elliptic function, and K is the complete
elliptic integral of the first kind. Since |x| = x·sign(x) we have the following inequality,max

x∈[−1,−ℓ]∪[ℓ,1]||x| − xr(x)| =x∈[−1,−ℓ]∪[ℓ,1]max|x · sign(x) − xr(x)|≤x∈[−1,−ℓ]∪[ℓ,1]max|sign(x) − r(x)| .

The last inequality follows because |x| ≤ 1 on [−1, −ℓ] ∪ [ℓ, 1]. Moreover, since

xr(x) ≥ 0 for x ∈ [−1, 1] (see [17, Equation (12)]) we havemax ||x| − xr(x)| ≤ max |x| ≤ ℓ.

x∈[−ℓ,ℓ]x∈[−ℓ,ℓ]Therefore,(max ||x| − xr(x)| ≤ max ℓ, 4 exp

x∈[−1,1]π22 log(4/ℓ)

−k ).

Now, we select 0 < ℓ < 1 to minimize this upper bound. One finds that ℓ =p
4 exp(−π k/2) and the result follows immediately.

For the final claim, let r be the Zolotarev sign function Zk (· ; ℓ) of type (k, k)
Qon [−1, −ℓ] ∪ [ℓ, 1], with k = pi=1 ki . By definition, Zk (·; ℓ) is the best rational
approximation of degree k to the sign function on [−1, −ℓ]∪[ℓ, 1]. We know from [117,160] that there exist p Zolotarev sign functions R1 , . . . , Rp , where each Ri is of type(ki , ki ), such thatr(x) := Zk (x; ℓ) = Rp (· · · (R2 (R1 (x))) · · · ).

(4.4)

A composition of k ≥ 1 Zolotarev sign functions of type (3, 2) has type (3k , 3k − 1)but can be represented with 7k parameters instead of 2 × 3k + 1. This property
enables the construction of a rational approximation to ReLU using compositions of

low-degree Zolotarev sign functions with O(log(log(1/ϵ))) parameters in Lemma 4.2.
The proof of Lemma 4.2 is a direct consequence of the previous lemma and theproperties of Zolotarev sign functions.
68Lemma 4.2. Let 0 < ϵ < 1. There exists a rational network NRational : [−1, 1] →[−1, 1] of size O(log(log(1/ϵ))) such that

∥NRational − ReLU∥∞ := max |NRational (x) − ReLU(x)| ≤ ϵ.
x∈[−1,1]Moreover, no rational network of size smaller than Ω(log(log(1/ϵ))) can achieve this.
Proof. Let 0 < ϵ < 1, 0 < ℓ < 1, k ≥ 1, and r be the Zolotarev sign function Z3k (· ; ℓ)of type (3k , 3k − 1). Again from [117, 160], we see that there exist k Zolotarev signfunctions R1 , . . . , Rk of type (3, 2) such that their composition equals Z3k (x; ℓ), i.e.,r(x) := Z3k (x; ℓ) = Rk (· · · (R2 (R1 (x)) · · · ).
Following the proof of Lemma 4.1, we have the inequality√kmax ||x| − xr(x)| ≤ 4e−π 3 /2 ,x∈[−1,1]p
3k /2). Now, we takeln(2/π 2 ) + 2 ln(ln(4/ϵ))
k=
,
ln(3)

(4.5)

(4.6)

where we chose ℓ = 4 exp(−π(4.7)

so that the right-hand side of Equation (4.6) is bounded by ϵ. Finally, we use theidentity|x| + x, x ∈ R,2
to define a rational approximation to the ReLU function on the interval [−1, 1] as
1 xr(x)r̃(x) =+x .
2 1+ϵ
ReLU(x) =Therefore, we have the following inequalities for x ∈ [−1, 1],
1
xr(x)
1
|x| −
≤
(||x| − xr(x)| + ϵ|x|)2
1+ϵ2(1 + ϵ)
ϵ
≤
≤ ϵ.
1+ϵ

|ReLU(x) − r̃(x)| =Then, r is a composition of k rational functions of type (3, 2) and can be represented
using at most 7k coefficients (see Equation (4.4)). Moreover, using Equation (4.7),
we see that k = O(log(log(1/ϵ))), which means that r̃ is representable by a rational
network of size O(log(log(1/ϵ))). Finally, |r̃(x)| ≤ 1 for x ∈ [−1, 1].

The lower bound on the rational networks size will be proved separately later in

Proposition 4.1.
69

The upper bound on the complexity of the neural network obtained in Lemma 4.2
is optimal, as proved by Vyacheslavov [232].
Theorem 4.2 (Vyacheslavov). The following inequalities hold:√√C1 e−π k ≤ max ||x| − rk (x)| ≤ C2 e−π k ,x∈[−1,1]k ≥ 0,(4.8)

where rk is the best rational approximation to |x| in [−1, 1] from Rk,k . Here, C1 , C2 >0 are constants that are independent of k.

We first deduce the following corollary, giving lower and upper bounds on the
optimal rational approximation to the ReLU function.
Corollary 4.1. The following inequalities hold:C1 −π√k
C2 −π√ke≤ ∥ReLU − rk ∥∞ ≤e,22

k ≥ 0,(4.9)

where rk is the best rational approximation to ReLU on [−1, 1] in Rk,k and C1 , C2 > 0
are constants given by Theorem 4.2.
Proof. Let k be an integer and let rk ∈ Rk,k be any rational function of degree ≤ k.
Now, define rabs (x) = 2rk (x) − x. Since ReLU(x) = (|x| + x)/2, we have1
1
1
(rabs (x) + x) − (|x| + x) = max |rabs (x) − |x||x∈[−1,1] 2
x∈[−1,1] 2
2√
1≥ C1 e−π k ,2∥ReLU − rk ∥∞ = max

where the inequality is from Theorem 4.2. Now, let rk ∈ Rk,k be the best rational
approximation to |x| on [−1, 1]. Now, define rReLU (x) = (rk (x) + x)/2. We find that11
1(|x| + x) − (rk (x) + x) = max ||x| − rk (x)|x∈[−1,1] 2
x∈[−1,1] 2
2√
1≤ C2 e−π k ,2∥ReLU − rReLU ∥∞ = max

which proves that the best approximation to ReLU satisfies the upper bound.
We now show that a rational neural network must be at least Ω(log(log(1/ϵ))) insize (total number of nodes) to approximate the ReLU function to within ϵ.
Proposition 4.1. Let 0 < ϵ < 1. A rational neural network that approximates the
ReLU function on [−1, 1] to within ϵ has size of at least Ω(log(log(1/ϵ))).

70

Proof. Let NRational : [−1, 1] → R be a rational neural network with k1 , . . . , kM ≥ 1

nodes at each of its M layers, and assume that its activation functions are rational

functions of type at most (rP , rQ ). Let dr = max(rP , rQ ) be the maximum of thePMdegrees of the activation functions of NRational . Such a network has size
i=1 ki .

Note that NRational itself is a rational function of degree d, where from additions
QM
and compositions of rational functions we have d ≤ dM
r
i=1 ki . If NRational is anϵ-approximation to the ReLU function on [−1, 1], we know by Corollary 4.1 that
C1 −π√d
e
≥ ϵ,
2

d≥



1
ln
π



C1
2ϵ

2

.

The statement follows by minimizing the size of NRational , i.e.,
dM
r

M
Y
i=1

That is,

ki ≥



1
ln
π



C1
2ϵ

2

(4.10)
PM

i=1 ki subject to.

  
C1
ln(ki ) + M ln(dr ) ≥ 2 ln ln− 2 ln(π).
2ϵ
i=1

M
X

(4.11)

We introduce a Lagrange multiplier λ ∈ R and define the Lagrangian of this opti-

mization problem as#  
MX
C1− 2 ln(π) −ln(ki ) − M ln(dr ) .
L(k1 , . . . , kM , λ) =ki + λ 2 ln ln
2ϵ
i=1
i=1
"

M
X

One finds using the Karush–Kuhn–Tucker conditions [115] that k1 = · · · = kM = λ.
Then, using Equation (4.11), we find that λ satisfies   2
C1
ln(λ) ≥ln ln
− ln(π) − ln(dr ) =: ln(λ∗ ).
(4.12)M
2ϵTherefore, the rational network NRational with M layers that approximates the ReLU
function to within ϵ on [−1, 1] has a size of at least s(M ) := M λ∗ , where λ∗ is given
by Equation (4.12) and depends on M . We now minimize s(M ) with respect to thenumber of layers M ≥ 1. We remark that minimizing s is equivalent of minimizingln(s), where   

C1
2
ln ln
− ln(π) − ln(dr ).
ln(s(M )) = ln(M ) + ln(λ ) = ln(M ) +M
2ϵ∗

One finds that one should take k1 = · · · = kM = λ∗ = O(1) and M = Ω(log(log(1/ϵ))).
The result follows.

71

The proof of Proposition 4.1 shows that the bound obtained in Lemma 4.2 isoptimal in the sense that a rational network requires at least Ω(log(log(1/ϵ))) parameters to approximate the ReLU function on [−1, 1] to within the tolerance ϵ > 0.
The convergence of the Zolotarev sign functions to the ReLU function is much faster,with respect to the number of parameters, than the rational constructed with Newman polynomials (see Figure 4.1(left)). We also include in this panel the algebraic
convergence of O(1/ϵ) obtained by polynomials [220] as a comparison.
100∥ReLU − rN ∥∞

1

10−3
10−6

O(1/)

0.8

O(log(1 2
/) )

0.6
0.4

10−9

0.2

10−12
10−15

O(log(log(1/)))
0

50

100

150

Polynomial
NewmanZolotarev

200

0
−0.2
−1

250

Number of parameters

ReLU
Rational

−0.5

0

0.5

1

x

Figure 4.1: Left: Approximation error ∥ReLU − rN ∥∞ of the Newman (blue),
Zolotarev sign functions (red), and best polynomial approximation [168] of degree
N − 1 (green) rN to ReLU with respect to the number of parameters required torepresent rN . Right: Best rational function of type (3, 2) (red) that approximates
the ReLU function (blue). We use this to initialize the rational activation functions
when training a rational neural network.
The converse of Lemma 4.2, which is a consequence of a theorem proved by Telgarsky [214, Theorem 1.1], shows that any rational function can be approximated bya ReLU network of size at most O(log(1/ϵ)3 ).
Lemma 4.3. Let 0 < ϵ < 1. If R : [−1, 1] → [−1, 1] is a rational function, thenthere exists a ReLU network NReLU : [−1, 1] → [−1, 1] of size O(log(1/ϵ)3 ) such that∥R − NReLU ∥∞ ≤ ϵ.

Proof. Let 0 < ϵ < 1 and R : [−1, 1] → [−1, 1] be a rational function. Take R̃(x) =R(2x − 1), which is still a rational function. Without loss of generality, we canassume that R̃ is an irreducible rational function (otherwise cancel factors till it isirreducible). Since R̃ is a rational, it can be written as R̃ = p/q with maxx∈[0,1] |q(x)| =1. Moreover, we know that R̃(x) ∈ [−1, 1] for x ∈ [0, 1] so we can assume thatq(x) ≥ 0 for x ∈ [0, 1] (it is either positive or negative by continuity). Since R is72

continuous on [−1, 1], there is an integer n ≥ 1 such that q(x) ∈ [2−n , 1] for x ∈ [0, 1].
Furthermore, we find that |p(x)| ≤ 1 for x ∈ [0, 1] because |R(x)| ≤ 1 and |q(x)| ≤ 1
for x ∈ [0, 1]. By [214, Theorem 1.1], there exists a ReLU network NReLU : [0, 1] → R
of size O(n7 log(1/ϵ)3 ) such thatmax NReLU (x) −x∈[0,1]

p(x)
ϵ
≤ .
q(x)
2

We now define a scaled ReLU network ÑReLU (x) = NReLU (x)/(1 + ϵ/2) such that|ÑReLU (x)| ≤ 1 for x ∈ [0, 1]. Therefore, for all x ∈ [0, 1],
NReLU (x) p(x)1p(x)ϵ p(x)ÑReLU (x) − R̃(x) =−≤ ϵ.
≤
NReLU (x) −+1 + ϵ/2q(x)1 + ϵ/2q(x)2 q(x)

Therefore, x 7→ ÑReLU ((x + 1)/2) is a ReLU neural network of size O(log(1/ϵ)3 ) thatis an ϵ-approximation to R on [−1, 1].

To demonstrate the improved approximation power of rational neural networks
over ReLU networks (O(log(log(1/ϵ))) versus O(log(1/ϵ)3 )), it is known that a ReLU
network that approximates x2 , which is rational, to within ϵ on [−1, 1] must be ofsize at least Ω(log(1/ϵ)) [129, Theorem 11].
We can now state our main theorem based on Lemmas 4.2 and 4.3. Theorem 4.3
provides bounds on the approximation power of ReLU networks by rational neuralnetworks and vice versa. We regard Theorem 4.3 as an analogue of [214, Thm. 1.1] for
our Zolotarev sign functions, where we are counting the number of training parametersinstead of the degree of the rational functions. In particular, our rational networks
have high degrees but can be represented with few parameters due to compositions,making training more computationally efficient. While Telgarsky required a rational
function with O(k M log(M/ϵ)M ) parameters to approximate a ReLU network withfewer than k nodes in each of M layers to within a tolerance of ϵ, we construct a
rational network that only has size O(kM log(log(M/ϵ))).

Theorem 4.3. Let 0 < ϵ < 1 and let ∥ · ∥1 denote the vector 1-norm. The following
two statements hold:1. Let NRational : [−1, 1]d → [−1, 1] be a rational network with M layers and at most
k nodes per layer, where each node computes x 7→ r(a⊤ x + b) and r is a rational

function with Lipschitz constant L (a, b, and r are possibly distinct across nodes).
Suppose further that ∥a∥1 + |b| ≤ 1 and r : [−1, 1] → [−1, 1]. Then, there exists a

ReLU network NReLU : [−1, 1]d → [−1, 1] of size

O kM log(M LM /ϵ)373

such that maxx∈[−1,1]d |NRational (x) − NReLU (x)| ≤ ϵ.

2. Let NReLU : [−1, 1]d → [−1, 1] be a ReLU network with M layers and at most k

nodes per layer, where each node computes x 7→ ReLU(a⊤ x + b) and the pair (a, b)(possibly distinct across nodes) satisfies ∥a∥1 + |b| ≤ 1. Then, there exists a rational
network NRational : [−1, 1]d → [−1, 1] of size

O(kM log(log(M/ϵ)))such that maxx∈[−1,1]d |NReLU (x) − NRational (x)| ≤ ϵ.
Proof. The statement of Theorem 4.3 comes in two parts, and we prove them separately. The structure of the proof closely follows [214, Lemma 1.3].
1. Consider the subnetwork H of the rational network NRational , consisting of

the layers of NRational up to the Jth layer for some 1 ≤ J ≤ M − 1. Let HReLU
denote the ReLU network obtained by replacing each rational function rij in H by
a ReLU network approximation frij at a given tolerance ϵj > 0 for 1 ≤ j ≤ J

and 1 ≤ i ≤ kj , such that |HReLU (x)| ≤ 1 for x ∈ [−1, 1] (see Lemma 4.3). Let

x 7→ ri,J+1 (a⊤i,J+1 H(x) + bi,J+1 ) be the output of the rational network NRational at layer
J + 1 and node i for 1 ≤ i ≤ kJ . Now, approximate node i in the (J + 1)st layer by a

ReLU network fri ,J+1 with tolerance ϵJ+1 > 0 (see Lemma 4.3). The approximation
error Ei,J+1 between the rational and the approximating ReLU network at layer J + 1
and node i satisfies⊤
Ei,J+1 = |fri,J+1 (a⊤i,J+1 HReLU (x) + bi,J+1 ) − ri,J+1 (ai,J+1 H(x) + bi,J+1 )|⊤≤ |fri,J+1 (a⊤i,J+1 HReLU (x) + bi,J+1 ) − ri,J+1 (ai,J+1 HReLU (x) + bi,J+1 )|
|
{z
}
(1)

⊤
+ |ri,J+1 (a⊤i,J+1 HReLU (x) + bi,J+1 ) − ri,J+1 (ai,J+1 H(x) + bi,J+1 )| .

|

{z

}

(2)

The first term is bounded by(1) ≤ max ri,J+1 (x) − fri,J+1 ≤ ϵJ+1 ,x∈[−1,1]

since a⊤i,J+1 HReLU (x) + bi,J+1 ≤ ∥ai,J+1 ∥1 + |bi,J+1 | ≤ 1 by assumption. The second

term is bounded as the Lipschitz constant of ri,J+1 is at most L. That is,(2) ≤ L∥ai,J+1 ∥1 max ∥HReLU (x) − H(x)∥∞ ≤ L max ∥HReLU (x) − H(x)∥∞ ,x∈[−1,1]dx∈[−1,1]d74

where we used the fact that ∥ai,J+1 ∥1 ≤ 1 and ∥HReLU (x)∥∞ ≤ 1 for x ∈ [−1, 1]d . We
find that we have the following set of inequalities:
max Ei,j+1 ≤ L max Ei,j + ϵj+1 ,1≤i≤kj+11≤i≤kj1 ≤ i ≤ kj ,1 ≤ j ≤ J + 1,

with Ei,0 = 0. If we select ϵj = ϵLj−J−1 /(J +1), then we find that max1≤i≤kJ+1 Ei,J+1 ≤ϵ. When J = M − 1, the ReLU network approximates the original rational network,
NRational , and the ReLU network has size
O k

M
X
j=1

log



M
Lj−M ϵ3 !

.

where we used the fact that kj ≤ k for 1 ≤ j ≤ M . This can be simplified a little
since

M
X
j=1

log



M
Lj−M ϵ

3

=

M
X
j=1

3

log(M LM /ϵ) + j log(1/L) = O M log(M LM /ϵ)3 .

2. Telgarsky proved in [214, Lemma 1.3] that if HR is a neural network obtainedby replacing all the ReLU activation functions in NReLU by rational functions R

for 1 ≤ j ≤ M , which satisfies R(x) ∈ [−1, 1] and |R(x) − ReLU(x)| ≤ ϵ/M forx ∈ [−1, 1], thenmax |NReLU (x) − HR (x)| ≤ ϵ.

x∈[−1,1]d

Let R̃ be a rational neural network approximating ReLU with a tolerance of ϵ/M ,constructed by Lemma 4.2. Then, R̃ is rational network of size O(log(log(M/ϵ))) andthus, HR̃ is a rational neural network of size O(M k log(log(M/ϵ))).

Theorem 4.3 highlights the improved approximation power of rational neural networks over ReLU networks. ReLU networks of size O(polylog(1/ϵ)) are required toapproximate rational networks while rational networks of size only O(log(log(1/ϵ)))are sufficient to approximate ReLU networks.

4.2.2

Approximation of functions by rational networks

A important question is the required size and depth of deep neural networks to approximate smooth functions [129, 154, 242]. In this section, we consider the approximation theory of rational networks. In particular, we consider the approximation75

of functions in the Sobolev space W n,∞ ([0, 1]d ), where n ≥ 1 is the regularity of thefunctions and d ≥ 1. The norm of a function f ∈ W n,∞ ([0, 1]d ) is defined as∥f ∥W n,∞ ([0,1]d ) = max ess sup |Dn f (x)|,|n|≤n x∈[0,1]dwhere n is the multi-index n = (n1 , . . . , nd ) ∈ {0, . . . , n}d , and Dn f is the correspond-

ing weak derivative of f . In this section, we consider the approximation of functions

from
Fd,n := {f ∈ W n,∞ ([0, 1]d ),∥f ∥W n,∞ ([0,1]d ) ≤ 1}.

By the Sobolev embedding theorem [33], Fd,n contains the functions in C n−1 ([0, 1]d ),which is the class of functions whose first n − 1 derivatives are Lipschitz continuous.

Yarotsky derived upper bounds on the size of neural networks with piecewise linear
activation functions needed to approximate functions in Fd,n [242, Thm. 1]. In par-ticular, Yarotsky constructed an ϵ-approximation to functions in Fd,n with a ReLU
network of size at most O(ϵ−d/n log(1/ϵ)) and depth smaller than O(log(1/ϵ)).
Theorem 4.4 (Yarotsky). Let d ≥ 1, n ≥ 1, 0 < ϵ < 1, and f ∈ Fd,n . There exists
a ReLU neural network NReLU of sizeO(ϵ−d/n log(1/ϵ))
and maximum depth O(log(1/ϵ)) such that ∥f − NReLU ∥∞ ≤ ϵ.
The term ϵ−d/n in Theorem 4.4 is introduced by a local Taylor approximation,
while the log(1/ϵ) term is the size of the ReLU network needed to approximate monomials, i.e., xj for j ≥ 0, in the Taylor series expansion. We now present an analogueof Theorem 4.4 for a rational neural network.
Theorem 4.5. Let d ≥ 1, n ≥ 1, 0 < ϵ < 1, and f ∈ Fd,n . There exists a rational
neural network NRational of sizeO(ϵ−d/n log(log(1/ϵ)))and maximum depth O(log(log(1/ϵ))) such that ∥f − NRational ∥∞ ≤ ϵ.
The proof of Theorem 4.5 consists of approximating f by a local Taylor expansion.
One needs to approximate the piecewise linear functions and monomials arising in
the Taylor expansion by rational networks. The main distinction between Yarotsky’s
argument and the proof of Theorem 4.5 is that monomials can be represented byrational neural networks with a size that does not depend on the accuracy of ϵ. In
76

contrast, ReLU networks require O(log(1/ϵ)) parameters. Meanwhile, while ReLU

neural networks can exactly approximate piecewise linear functions with a constantnumber of parameters, rational networks can approximate them with a size of a mostO(log(log(1/ϵ))) (see Lemma 4.2). That is, rational neural networks approximate
piecewise linear functions much faster than ReLU networks approximate polynomials.

This allows the existence of a rational network approximation to f with exponentiallysmaller depth (O(log(log(1/ϵ)))) than the ReLU networks constructed by Yarotsky.
We first show that the construction in Lemma 4.2 can approximate any piecewise
linear function on [−1, 1].
Proposition 4.2. Let 0 < ϵ < 1 and let g : [0, 1] → R be any continuous piecewise
linear function with m ≥ 1 breakpoints and Lipschitz constant L > 0. Then, there
exists a rational neural network NRational : [0, 1] → R of size at mostO(m log(log(L/ϵ)))such that maxx∈[0,1] |g(x) − NRational (x)| ≤ ϵ.
Proof. Let 0 ≤ b1 < · · · < bM ≤ 1 be the breakpoints of g. In a similar way to theproof of [242, Proposition 1], we first express ρ as the following sum:g(x) = c0 ReLU(b1 − x) +

m
X
j=1

cj ReLU(x − bj ) + cm+1 ,(4.13)

for some constants c0 , . . . , cm+1 ∈ R. Therefore, g can be exactly represented using a
ReLU network with m + 1 nodes and one layer, i.e.,
ReLU(−x + b1 )
 ReLU(x − b1 ) g(x) = c0 c1 · · · cm 
 + cm+1 .
..

.
ReLU(x − bm )Since g has a Lipschitz constant of L, we find that |c0 | ≤ L andPm

j=1 |cj |≤ L.

Using Lemma 4.2 we can approximate a ReLU function on [−1, 1] with toleranceϵ/(2L) by a rational network RReLU of size O(log(log(2L/ϵ))). Now, we construct

NRational : [0, 1] → R as a rational network obtained by replacing the ReLU functions
in g by RReLU . We have the following error estimate:mX
ϵ ϵmax |g(x)−NRational (x)| ≤ |c0 |∥ReLU−RReLU ∥∞ +|cj |∥ReLU−RReLU ∥∞ ≤ + ≤ ϵ.
x∈[0,1]
2 2
j=1

The result follows as NRational is of size O(m log(log(L/ϵ))).
77

We remark that the size of the rational network required to approximate a piecewise linear function depends on ϵ. In contrast, ReLU neural networks can represent
piecewise linear functions exactly. In the next proposition, we show that a rational
neural network can represent xn , for some integer n, exactly.
Proposition 4.3. Let n ≥ 1, rP ≥ 2, and rQ ≥ 0. There exists a rational network NRational , with rational activation functions of type (rP , rQ ), of size at most
5⌊logrP (n)⌋2 + 1 such that NRational (x) = xn for all x ∈ R.
Proof. We start by expressing n in base rP , i.e.,⌊logr (n)⌋

n=

P
X

cℓ rPℓ ,ℓ=0cℓ ∈ {0, 1, . . . , rP − 1}.

This means we can represent xn asnx =⌊logr (n)⌋PYℓxcℓ rP .

(4.14)ℓ=0ℓ

Note that xcℓ rP is just xrP composed ℓ times as well as composed with xcℓ so canbe represented by a rational neural network with ℓ + 1 layers, each with one node.
ℓ

Therefore, all the xcℓ rP terms can be represented in rational networks that in total
have size⌊logr (n)⌋PX
13(ℓ + 1) = (⌊logrP(n)⌋)2 + ⌊logrP(n)⌋ + 1.
2
2
ℓ=0
ℓ

The function xn can be formed by multiplying all the xcℓ rP terms together. Since

xy = (x2 + y 2 − (x − y)2 )/2, there is a rational network with one layer and three nodes
that represents the multiplication operation. Therefore, multiplying all the terms
together requires a rational network of size at most 3⌊logrP(n)⌋ (see Equation (4.14)).
The result follows by noting that x2 /2 + 9x/2 + 1 ≤ 5x2 + 1 for x ≥ 1.
We can now prove Theorem 4.5 using the two previous propositions.
Proof of Theorem 4.5. The proof is based on the proof of [242, Theorem 1] and consists of replacing the piecewise linear functions and monomials arising in the local
Taylor approximation of the function f by rational networks using the previous approximation results.

78

1.2
1
0.8
0.6
0.4
0.2
0

0

0.2

0.4

0.6

0.8

1

x
Figure 4.2: Partition of unity: ψ0 (red), ψ1 (blue), and ψ2 (green), for N = 2.
Let N ≥ 1 be an integer and consider a partition of unity of (N + 1)d functions
ϕm on the domain [0, 1]d , i.e.,Xϕm (x) = 1,

ϕm (x) =dYψmk (xk ),x = (x1 , . . . , xd ),k=1

m∈{0,...,N }dwhere m = (m1 , . . . , md ), and ψmk is given by1,
ψmk (x) = 0,

2 − 3N xk − mNk ,1,
if xk − mNk < 3N
mk
2
if xk − N > 3N,otherwise.

Examples of the functions ψmk are shown in Figure 4.2 when N = 2. We now define
a local Taylor approximation of f by
fN (x) =X

ϕm (x)Pm (x),m∈{0,...,N }dwhere Pm denotes the degree n − 1 Taylor polynomial of f at x = m/N . That is,
Pm (x) =X Dn f ( m ) 
N

|n|<n

where |n| =dPd

k=1 nk , n! =

Qd

n!

x−

m n
,
N

n
k=1 nk !, and (x − m/N ) =Qdx ∈ [0, 1] and note that
mk
1
support(ϕm ) ⊂ x = (x1 , . . . , xd ) : xk −
<
,
N
N

79

(4.15)nk
k=1 (xk − mk /N ) .

m ∈ {0, . . . , N }d .

LetHence, the approximation error between f and its local Taylor approximation satisfies|f (x) − fN (x)| =X

ϕm (f (x) − Pm (x)) ≤m∈{0,...,N }dXm
m: xk − Nk|f (x) − Pm (x)|

|
|



n
n
2d dn 12d dn 1n≤
max ess sup |D f (x)| ≤.
|n|=n x∈[0,1]d
n!
N
n!
N
1
<N

We now select (see [242, Theorem 1] for a similar idea)&−1/n 'n! ϵ,N=
2d dn 2so thatmax |f (x) − fN (x)| ≤ ϵ/2.

(4.16)

x∈[0,1]d

We now approximate the function fn by a rational network using Propositions 4.2
and 4.3. First, we write fN as

X
X
m n
am,n ϕm (x) x −fN (x) =,(4.17)
Ndm∈{0,...,N } |n|<n

where |am,n | ≤ 1 and the monomials are uniformly bounded by 1 (see Equation (4.15)).

Equation (4.17) consists of at most dn (N + 1)d terms of the form ϕm (x)(x − m/N )n .
The monomial part (x − m/N )n in Equation (4.17) is representable by a rational

network of size O(d log(n)2 ) using Proposition 4.3, including the fact that the multiplication is a rational network with one layer and three nodes. Let 0 < δ < 1 bea small number, for each mk ∈ {0, . . . , N } the piecewise linear function ψmk has a
Lipschitz constant of L = 3N . Therefore, it can be approximated with a tolerance
δ by a rational network ψ̃mk of size O(log(log(N/δ))) (see Proposition 4.2). We canassume ∥ψ̃mk ∥∞ = 1 by increasing the size of the network by a constant. This yields
the following approximation error between a term in Equation (4.17) and the rational
network constructed using ψ̃mk :


d
d
d

Y
Y
m n Ym nϕm (x) x −−ψ̃mk (xk ) x −≤ψmk (xk ) −ψ̃mk (xk )
N
N
k=1
k=1
k=1

≤ ψm1 (x1 ) − ψ̃m1 (x1 )

d
Y

k=2

≤ ψm1 (x1 ) − ψ̃m1 (x1 ) +
≤δ+

d
Y

k=2

ψmk (xk ) −ψmk (xk ) + ψ̃m1 (x1 )

d
Y

k=2

d
Y

k=2

d
Y

k=2

ψmk (xk ) −ψ̃mk (xk ) ≤ dδ.
80

d
Y

k=2

ψ̃mk (xk )ψmk (xk ) −

d
Y

k=2

ψ̃mk (xk )Here, the final inequality is derived by repeating the argument of the previous inequalities for x2 , . . . , xd . If we denote by NRational the rational network approximation

to fN constructed above, then, for all x ∈ [0, 1]d , we have|fN (x) − NRational (x)| ≤XXm∈{0,...,N }d |n|<n≤ 2d dn+1 δ.



d

m n
m n Y|am,n | ϕm (x) x −−ψ̃mk (xk ) x −NN
k=1

Therefore, we select δ = ϵ/(2d+1 dn+1 ) so that maxx∈[0,1]d |fN (x)− f˜N (x)| ≤ ϵ/2. Then,by Equation (4.16), we havemax |f (x) − NRational (x)| ≤x∈[0,1]dϵϵ+ ≤ ϵ.
2 2

The statement of the theorem follows as the rational network NRational has size at
mostO(dn (N + 1)d log(log(N/δ))) = O(ϵ−d/n log(log(1/ϵ1+1/n ))) = O(ϵ−d/n log(log(1/ϵ))).

A theorem proved by DeVore et al. [52] gives a lower bound of Ω(ϵ−d/n ) on thenumber of parameters needed by a neural network to express any function in Fd,nwith an error ϵ, under the assumption that the weights are chosen continuously.
Comparing O(ϵ−d/n log(log(1/ϵ))) and O(ϵ−d/n log(1/ϵ)), we find that rational neuralnetworks require exponentially fewer nodes than ReLU networks with respect to the
optimal bound of Ω(ϵ−d/n ) to approximate functions in Fd,n .

4.3

Experiments using rational neural networks

In this section, we consider neural networks with trainable rational activation functions of type (3, 2). We select the type (3, 2) based on empirical performance; roughly,
a low-degree (but higher than 1) rational function is ideal for generating high-degree
rational functions by composition, with a small number of parameters. The rational
activation units can be easily implemented in the open-source TensorFlow library [1]by using the polyval and divide commands for function evaluations. The coefficients of the numerators and denominators of the rational activation functions aretrainable parameters, determined at the same time as the weights and biases of theneural network by backpropagation and a gradient descent optimization algorithm.
81One crucial question is the initialization of the coefficients of the rational activation functions [39, 153]. A badly initialized rational function might contain poleson the real axis, leading to exploding values, or converge to a local minimum inthe optimization process. Our experiments, supported by the empirical results ofMolina et al. [153], show that initializing each rational function with the best rational
approximation to the ReLU function (as described in Lemma 4.2) produces good performance. The underlying idea is to initialize rational networks near a network with
ReLU activation functions, widely used for deep learning. Then, the adaptivity of therational functions allows for further improvements during the training phase. We represent the initial rational function used in our experiments in Figure 4.1(right). The
coefficients of this function are obtained by using the minimax command, available
in the Chebfun software [56, 67] for numerically computing rational approximations,and are given in Table 4.1.
Table 4.1: Initialization coefficients of the rational activation functions.
a0

a1

a2

a3

b0

b1

b2

1.19151.59570.50000.02182.38300.00001.0000In the following experiments, we use a single rational activation function of type(3, 2) at each layer, instead of different functions at each node to reduce the numberof trainable parameters and the computational training expense. This adds 7 degrees
of freedom per layer.

4.3.1

Approximation of functions

Raissi, Perdikaris, and Karniadakis [180, 184] introduce a framework called deep hidden physics models for discovering nonlinear partial differential equations (PDEs)
from observations. This technique requires to solving the following interpolation problem: given the observation data (ui )1≤i≤N at the spatio-temporal points (xi , ti )1≤i≤N ,find a neural network N (called the identification network), that minimizes the loss
functionN
1 X
L=|N (xi , ti ) − ui |2 .
(4.18)
N i=1
This technique has successfully discovered hidden models in fluid mechanics [185],solid mechanics [85], and nonlinear PDEs such as the Korteweg–de Vries (KdV) equation [184]. Raissi et al. use an identification network, consisting of 4 layers and 50
82

nodes per layer, to interpolate samples from a solution to the KdV equation. Moreover, they observe that networks based on smooth activation functions, such as thehyperbolic tangent (tanh(x)) or the sinusoid (sin(x)), outperform ReLU neural networks [180, 184]. However, the performance of these smooth activation functions
highly depends on the application.
Moreover, these functions might not be adapted to approximate non-smooth or
highly oscillatory solutions. Recently, Jagtap, Kawaguchi, and Karniadakis [96] proposed and analyzed different adaptive activation functions to approximate smoothand discontinuous functions with physics-informed neural networks. More specifically, they use an adaptive version of classical activation functions such as sigmoid,
hyperbolic tangent, ReLU, and Leaky ReLU. The choice of these trainable activation
functions introduces another parameter in the design of the neural network architecture, which may not be ideal for use for a black-box data-driven PDE solver.
20
15
10

1.5

5

1.0

x0

0.5

−5

0.0

−10

−0.5

−15
−20

Validation loss

2.0

0

10

20

30

40

−1.0

101
100
10 1
10 2
10 3
10 4
10 5
10 6
10 7
10 8 010t

ReLU
Sinusoid
Rational
Polynomial
101
102
103
Epochs104

Figure 4.3: Solution to the KdV equation used as training data (left) and validation
loss of a ReLU (blue), sinusoid (green), rational (red), and polynomial (purple) neuralnetworks with respect to the number of optimization steps (right).
We illustrate that rational neural networks can address the issues mentioned above
due to their adaptivity and approximation power (see Section 4.2). Similarly to
Raissi [180], we use a solution u to the KdV equation:
ut = −uux − uxxx ,u(x, 0) = − sin(πx/20),

as training data for the identification network (see the left panel of Figure 4.3). We
use the TensorFlow implementation2 of the deep hidden physics model framework tobuild and train the identifier network N that approximates a solution u to the KdV

equation. The true solution is computed on the domain (x, t) ∈ [−20, 20] × [0, 40] by
2

We adapt the code that is publicly available [181].

83

Raissi [180] using the Chebfun package [56] with a spectral Fourier discretization of
512 and a time-step of ∆t = 10−4 . Moreover, the solution is stored after every 2000
time steps, giving a testing data set of approximately 105 spatio-temporal points in[−20, 20] × [0, 40]. We then constituted the training and validation sets (of 104 pointseach) by randomly subsampling the solution at 2 × 104 points in [−20, 20] × [0, 40].

In a similar manner to [180], we use a fully connected identification network toapproximate u with 4 hidden layers with 50 nodes per layer. The network is trainedusing the L-BFGS optimization algorithm with 10,000 iterations. We train and compare four networks with the following activation functions: ReLU, sinusoid, trainablerational functions of type (3, 2), and trainable polynomials of degree 3. Furthermore,
the rational activation functions are initialized to be the best approximation to the
ReLU function, using the initial coefficients reported in Table 4.1.
The mean squared error (MSE) of the neural networks on the validation set
throughout the training phase is reported in the right panel of Figure 4.3. We observethat the rational neural network outperforms the sinusoid network, despite having thesame asymptotic convergence rate. The network with polynomial activation functions(chosen to be of degree 3 in this example) is harder to train than the rational network,
as shown by the non-smooth validation loss (see the right panel of Figure 4.3). We
highlight that rational neural networks are never much bigger in terms of trainable
parameters than ReLU networks since the increase is only linear with respect to thenumber of layers. Here, the ReLU network has 8000 parameters (consisting of weights
and biases), while the rational network has 8000 + 7 × #layers = 8035. The ReLU,

sinusoid, rational, and polynomial networks achieve the following mean square errors
after 104 epochs:
MSE(uReLU ) = 1.9 × 10−4 ,MSE(uSinusoid ) = 3.3 × 10−6 ,MSE(uRational ) = 1.2 × 10−7 ,MSE(uPolynomial ) = 3.6 × 10−5 .

The rational neural network is approximately five times more accurate than thesinusoid network used by Raissi and twenty times more accurate than the ReLU
network. The absolute approximation errors between the different neural networks
and the exact solution to the KdV equation is illustrated in Figure 4.4. We find thatthe approximation errors made by the ReLU network are not uniformly distributedin space and time and located in specific regions, indicating that a network with
non-smooth activation functions is not appropriate to resolve smooth solutions to
PDEs.

84

ReLU

Sinusoid

20

0.05

15

0.010

15
0.04

10
5

0.03

x 0

0.02

−5
−10

0.01

−15
−20

Rational

20

0.008

10
5

0.006

x 0

0.004

−5
−10

0.002

−15
0.00
0

10

20

t

30

40

−20

0.000
0

10

20

30

t

40

20

0.00200

15

0.00175100.0015050.00125x 00.00100−50.00075−100.00050−150.00025−200.00000
0

10

20

t

30

40

Validation loss

Figure 4.4: Approximation errors of the neural networks with ReLU, sinusoid, andrational activation layers. Note the different scales of the errors.

101
100
10 1
10 2
10 3
10 4
10 5
10 6
10 7
10 8 010(2, 2)(3, 2)(4, 3)(5, 4)

101

102

103

104

EpochsFigure 4.5: Validation loss of rational networks of types (2, 2), (3, 2), (4, 3), and (5, 4)
with respect to the number of epochs.

85

Finally, in Figure 4.5, we compare rational neural networks with different degree
activation functions (each initialized to approximate the ReLU function using the
MATLAB code initial rational coeffs.m available at [29]) and find that they all
performed better than ReLU networks. While a type (3, 2) rational offers a goodtrade-off between the number of parameters and quality of approximation according
to the theoretical results presented in Section 4.2, the type of rational function mightwell depend on the application considered.

4.3.2

Generative adversarial networks

Generative adversarial networks are used to generate synthetic examples from anexisting dataset [76]. They consist of two networks: a generator to produce synthetic samples and a discriminator to evaluate the samples of the generator withthe training dataset. Radford et al. [179] describe deep convolutional generative
adversarial networks (DCGANs) to build good image representations using convolutional architectures. They evaluate their model on the MNIST and ImageNet image
datasets [51, 120].
This section highlights the simplicity of using rational activation functions inexisting neural network architectures by training an Auxiliary Classifier GAN (ACGAN) [165] on the MNIST dataset. In particular, the neural network, referred to as
the ReLU network in this section, consists of convolutional generator and discriminator networks with ReLU and Leaky ReLU [140] activation units (respectively) andis used as a reference GAN. We adapt the Keras example in [43] to train an Auxiliary Classifier GAN with rational activation functions on the MNIST. The hyperparameters used for the GAN experiment are given in Table 4.2. Moreover, theGAN is trained on 20 epochs with a batch size of 100 by Adam’s optimization algorithm [108] and the following parameters: α = 0.0002 and β1 = 0.5, as suggested
by [179].
As in the experiment described in Section 4.3.1, we replace the activation unitsof the generative and discriminator networks by a rational function with trainablecoefficients (see Figure 4.1). We initialize the activation functions in the training
phase with the best rational function that approximates the ReLU function on [−1, 1].
We show images of digits from the first five classes generated by a ReLU andrational GANs at different epochs of the training in Figure 4.6 (the samples are
generated randomly and are not manually selected). We observe that a rational
network can generate realistic images with a broader range of features than the ReLU

86

Table 4.2: Hyper-parameters of the GAN experiment, BN denotes the presence of a
Batch normalization layer. The Generator and Discriminator networks are trainedwith ReLU and rational activation functions, initialized with the coefficients reported
in Table 4.1. Transposed convolution layers and rational activation functions arerespectively abbreviated as “Transp. Conv.” and “Rat.”.
Operation

Strides

Features

BN

Dropout

Activation

N/A
5×5
5×5
5×5

N/A
1×1
2×2
2×2

3456
192
96
1

✗
✓
✓
✗

0.0
0.0
0.0
0.0

ReLU / Rat.
ReLU / Rat.
ReLU / Rat.
Tanh

3×3
3×3
3×3
3×3
N/A

2×2
1×1
2×2
1×1
N/A

32
64
128
256
11

✗
✗
✗
✗
✗

0.3
0.3
0.3
0.3
0.0

Leaky ReLU / Rat.
Leaky ReLU / Rat.
Leaky ReLU / Rat.
Leaky ReLU / Rat.
Soft-Sigmoid

ReLU

Generator
LinearTransp. Conv.
Transp. Conv.
Transp. Conv.
Discriminator
Convolution
Convolution
Convolution
Convolution
LinearKernelepoch 10epoch 15

epoch 20

epoch 5epoch 10epoch 15

epoch 20

Rational

epoch 5

MNIST images

Figure 4.6: Digits generated by a ReLU (top) and rational (bottom) auxiliary classifier
generative adversarial network. The right panel contains samples from the first fiveclasses of the MNIST dataset for comparison.

87network, as illustrated by the presence of bold numbers at the epoch 20 in the bottom
panel of Figure 4.6.
We report in Figure 4.7 samples of the 10 classes present in the MNIST dataset(right) and images generated at the 20th epoch by the GAN with ReLU/Leaky ReLU
units (left) and rational activation functions (middle). We observe that the digits one
generated by the rational network are identical, suggesting that the rational GAN
suffers from mode collapse. It should be noted that generative adversarial networks
are notoriously tricky to train [75]. The hyper-parameters of the reference model areintensively tuned for a piecewise linear activation function (as shown by the use of
Leaky ReLU in the discriminator network). Moreover, many stabilization methods
have been proposed to resolve the mode collapse and non-convergence issues in training, such as Wasserstein GAN [10], Unrolled Generative Adversarial Networks [149],
and batch normalization [95]. These techniques could be explored and combined withrational networks to address the mode collapse issue observed in this experiment.

88

ReLU

Rational

MNIST images

Figure 4.7: Forty images generated by a ReLU network and a rational network after
20 epochs, together with real images from the MNIST dataset.

89

Chapter 5
Data-driven discovery of Green’s
functions with deep learning∗
Deep learning (DL) holds promise as a scientific tool for discovering elusive patterns
within the natural and technological world [75, 118]. These patterns hint at undiscovered partial differential equations (PDEs) that describe governing phenomena withinbiology and physics. From sparse and noisy laboratory observations, we aim to learn
mechanistic laws of nature [35, 102]. Recently, scientific computing and machinelearning have successfully converged on PDE discovery [36, 195, 197, 245], PDE learning [65, 71, 126, 135, 180, 185], and symbolic regression [201, 224] as promising means
for applying machine learning to scientific investigations. These methods attempt todiscover the coefficients of a PDE model or learn the operator that maps excitations
to system responses. The recent DL techniques addressing the latter problem arebased on approximating the solution operator associated with a PDE by a neuralnetwork (NN) [65, 71, 126, 135, 180]. While excellent for solving PDEs, we consider
them as “black box” and focus here on a data-driven strategy that improves humanunderstanding of the governing PDE model.
We then offer a radically different, alternative approach that is backed by theory [32] and infuse an interpretation in the model by learning well-understood mathematical objects that imply underlying physical laws. We devise a DL method, employed for learning the Green’s functions [207] associated with unknown governing
linear PDEs, and train the neural networks by collecting physical system responses
from random excitation functions drawn from a Gaussian process (GP). The empirically derived Green’s functions relate the system’s response (or PDE solution) to a∗This chapter is based on a paper with Christopher Earls and Alex Townsend [27], published in
Scientific Reports. Earls and Townsend had an advisory role; I designed the deep learning method,
performed the numerical experiments, and was the lead author in writing the paper.

90

forcing term, and can then be used as a fast reduced-order PDE solver. The existinggraph kernel network [126] and DeepGreen [71] techniques also aim to learn solution
operators of PDEs based on Green’s functions. While they show competitive performance in predicting the solution of the PDE for new forcing functions, the errors
between the exact and learned Green’s functions are relatively large, which makes theextraction of qualitative and quantitative features of the physical system challenging.
Our secondary objective is to study the discovered Green’s functions for clues regarding the physical properties of the observed systems. Our approach relies on therational neural networks introduced in the previous chapter, which have higher approximation power than standard networks and carry human-understandable featuresof the PDE, such as shock and singularity locations, as we shall see later.
In this chapter, we use techniques from deep learning to discover the Green’s
function of linear differential equations Lu = f from input-output pairs (f, u), as

opposed to directly learning L, or model parameters. In this sense, our approach isagnostic to the forward PDE model, but nonetheless offers insights into its physicalproperties. There are several advantages to learning the Green’s function. First, once

the Green’s function is learned by a neural network, it is possible to compute thesolution, u, for a new forcing term, f , by evaluating an integral (see Equation (5.2));
which is more efficient than training a new NN. Second, the Green’s function associated with L contains information about the operator, L, and the type of boundary

constraints that are imposed; which helps uncover mechanistic understanding fromexperimental data. Finally, as discussed in Chapter 2, it is easier to train NNs toapproximate Green’s functions, which are square-integrable functions under sufficientregularity conditions [53, 80, 207], than trying to approximate the action of the linear
differential operator, L, which is not bounded [113]. Also, any prior mathematicaland physical knowledge of the operator, L, can be exploited in the design of the
NN architecture, which could enforce a particular structure such as symmetry of theGreen’s function.

5.1

Learning Green’s functions

We consider linear differential operators, L, defined on a bounded domain Ω ⊂ Rd ,where d ∈ {1, 2, 3} denotes the spatial dimension. The aim of our method is to

discover properties of the operator, L, using N input-output pairs {(fj , uj )}N
j=1 , consisting of forcing functions, fj : Ω → R, and system responses, uj : Ω → R, which are91

solutions to the following equation:
Luj = fj ,D(uj , Ω) = g,

(5.1)

where D is a linear operator acting on the solutions, u, and the domain, Ω; with gbeing the constraint. We assume that the forcing terms have sufficient regularity,and that the operator, D, is a constraint so that Equation (5.1) has a unique so-lution [207]. An example of constraint is the imposition of homogeneous Dirichletboundary conditions on the solutions: D(uj , Ω) := uj |∂Ω = 0. Note that boundary

conditions, integral conditions, jump conditions, or non-standard constraints, are all
possible (see Section 5.4.1).

5.1.1

Definitions

A Green’s function [9, 64, 157, 207] of the operator, L, is defined as the solution to
the following equation:

LG(x, y) = δ(y − x),x, y ∈ Ω,

where L is acting on the function x 7→ G(x, y) for fixed y ∈ Ω, and δ(·) denotes

the Dirac delta function. The Green’s function is well-defined and unique under
mild conditions on L, and suitable solution constraints imposed via an operator, D(see Equation (5.1)) [207]. Moreover, if (f, u) is an input-output pair, satisfying
Equation (5.1) with g = 0, thenu(x) =G(x, y)f (y) dy,Ωx ∈ Ω.

Therefore, the Green’s function associated with L can be thought of as the rightinverse of L.
Let uhom be the homogeneous solution to (5.1), so that
Luhom = 0,D(uhom , Ω) = g.

Using superposition, we can construct solutions, uj , to Equation (5.1) as uj = ũj +uhom , where ũj satisfies
Lũj = fj ,D(ũj , Ω) = 0.

Then, the relation between the system’s response, uj , and the forcing term, fj , canbe expressed via the Green’s function as

uj (x) =G(x, y)fj (y) dy + uhom (x),Ω92x ∈ Ω.

In this chapter, we focus on learning Green’s functions and homogeneous solutions
from a fixed boundary constraint g but one could also approximate a second Green’s
function associated with uhom from multiple boundary constraints. Therefore, we
train two NNs: NG : Ω × Ω → R ∪ {±∞} and Nhom : Ω → R, to learn the Green’s
function, and also the homogeneous solution associated with L and the constraint
operator D. Note that this procedure allows us to discover boundary conditions,or constraints, directly from the input-output data without imposing it in the loss
function (which often results in training instabilities [236]).

5.1.2

Theoretical justification

Our approach for learning Green’s functions associated with linear differential operators has a theoretically rigorous underpinning. Indeed, we showed in Chapter 2 thatuniformly elliptic operators in three dimensions have an intrinsic learning rate, whichcharacterizes the number of training pairs needed to construct an ϵ-approximationin the L2 -norm of the Green’s function, G, with high probability, for 0 < ϵ < 1.
The number of training pairs depends on the quality of the covariance kernel used
to generate the random forcing terms, {fj }N
j=1 . Our choice of covariance kernel (Section 5.2.1) is motivated by the GP quality measure (cf. Section 2.1.4), to ensure that

our set of training forcing terms is sufficiently diverse to capture the action of thesolution operator, f 7→ u(x) = Ω G(x, y)f (y) dy, on a diverse set of functions.
Similarly, the choice of rational NNs to approximate the Green’s function, andthe homogeneous solution, is justified by the higher approximation power of these
networks over ReLU as observed in Chapter 4. Other adaptive activation functions
have been proposed for learning or solving PDEs with NNs [96], but they are only
motivated by empirical observations. Both theory and experiments support rational
NNs for regression problems. The number of trainable parameters, consisting of
weight matrices, bias vectors, and rational coefficients, needed by a rational NN toapproximate smooth functions within 0 < ϵ < 1, can be completely characterized [30].
This motivates our choice of NN architecture for learning Green’s functions.

5.2

Deep learning method

Our DL approach (see Figure 5.1) begins with excitations (or forcing terms), {fj }N
j=1 ,sampled from a Gaussian process having a carefully designed covariance kernel, andcorresponding system responses, {uj }N
j=1 (see Chapter 3). It is postulated that there
is an unknown linearized governing PDE so that Luj = fj . The selection of random93

forcing terms is theoretically justified by Chapter 2 and enables us to learn the dominant eigenmodes of the solution operator, using only a small number, N , of training
pairs. The Green’s function, G, and homogeneous solution, uhom , which encodes theboundary conditions associated with the PDE, satisfyuj (x) =x ∈ Ω,G(x, y)fj (y) dy + uhom (x),
Ω

(5.2)

and are approximated by two rational neural networks: NG and Nhom .
(a) GP covariance kernel
y

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2
0.0
0.00

(d)

0.2
0.50

0.75

1.00

(e)

x

(b)

Algorithmically informed
samples from theory

Excite

Train

Train

0.0
0.25

y

x

x

DL Hom.
solution

DL Green’s
functionuG

Source/forcing
Unknown system

Observe

Boundary
conditions on(f)(c)
Response

Human readable featureson

Extract &visualize

Extract &visualize

Homogeneous solution

Green’s function

1.0

1.0

0.25

0.8

0.20

0.6

0.15

0.4

0.10

−0.5

0.2

0.05

−1.0
0.00

0.0
0.00

0.5

y

0.0

0.25

0.50

x

0.75

1.00

0.00
0.25

0.50

0.75

1.00

x

Figure 5.1: Schematic of our DL method for learning Green’s functions from inputoutput pairs. (a) The covariance kernel of the Gaussian process, which is used togenerate excitations. (b) The system’s response to each excitation is computed andrecorded (c). (d) A loss function is minimized to train rational NNs (e). (f) The
learned Green’s function and homogeneous solution are visualized by sampling theNNs.
The parameters of the NNs representing the Green’s function and homogeneous solution are simultaneously learned through minimization of the loss function displayedin Figure 5.1(d). We discretize the integrals in the loss function at the specified meausurement locations {xi }N
i=1 , within the domain, Ω, and forcing term sample points,Nf{yi }i=1, respectively, using a quadrature rule.

In this section, we detail the deep learning method used to learn Green’s functions.

Our DL technique is data-driven and requires minimal by-hand parameter tuning. In

94

fact, all the numerical examples described in this chapter are performed using a single
rational NN architecture, initialization procedure, and optimization algorithm1 .

5.2.1

Generating the training data

We create a training dataset, consisting of input-output functions, {(fj uj )} for 1 ≤j ≤ N , in three steps: (1) Generating the forcing terms by sampling random functions
from a Gaussian process, (2) Solving Equation (5.1) for the generated forcing terms,and (3) Sampling the forcing terms, fj , at the points {y1 , . . . , yNf } ⊂ Ω and thesystem’s responses, uj , at {x1 , . . . , xNu } ⊂ Ω. Here, Nf and Nu are the forcing

and solution discretization sizes, respectively. We recommend that all the forcing
terms are sampled on the same grid and similarly for the system’s responses. This
minimizes the number of evaluations of NG during the training phase and reduces
the computational and memory costs of training.

The spatial locations of points {yi } and the forcing discretization size, Nf , arechosen arbitrarily to train the NNs as the forcing terms are assumed to be known
over Ω. In practice, the number, Nu , and location of the measurement points, {xi },are imposed by the nature of the experiment, or simulation, performed to measure
the system’s response. When Ω is an interval, we always select Nf = 200, Nu = 100,and equally-spaced sampled points for the forcing and response functions.
Unless otherwise stated, the training data comprises N = 100 forcing and solution pairs, where the forcing terms are drawn at random from a Gaussian process,GP(0, KSE ), where KSE is the squared-exponential covariance kernel [187] defined as|x − y|2
KSE (x, y) = exp −, x, y ∈ Ω.
(5.3)
2ℓ2
As discussed in Section 3.3, the parameter ℓ > 0 in Equation (5.3) is called thelength-scale parameter, and characterizes the correlation between the values of f ∼GP(0, KSE ) at x and y for x, y ∈ Ω. A small parameter, ℓ, yields highly oscillatory
random functions, f , and determines the ability of the GP to generate a diverse set oftraining functions. This last property is crucial for capturing different modes within
the operator, L, and for learning the associated Green’s function accurately [32].
Other possible choices of covariance kernels include the periodic kernel [187]:
2 sin2 (π|x − y|)
KPer (x, y) = exp −, x, y ∈ Ω,ℓ2
1

All data and codes used in this chapter are publicly available on the GitHub and Zenodo repositories at https://github.com/NBoulle/greenlearning/ [26] to reproduce the numerical experiments and figures. A software package, including additional examples and documentation, is alsoavailable at https://greenlearning.readthedocs.io/.

95

which is used to sample periodic random functions for problems with periodic boundary conditions (Figure 5.8(b)). Another possibility is a kernel from the Matérn family [187] or the Jacobi kernel introduced in Section 3.3.3.
When Ω is an interval [a, b], we introduce a normalized length-scale parameter
λ = ℓ/(b − a), so that the method described does not depend on the length of
the interval. In addition, we choose λ = 0.03, so that the length-scale, ℓ, is larger

than the forcing spatial discretization size, which allows us to adequately resolve thefunctions sampled from the GP with the discretization. More precisely, we makesure that ℓ ≥ (b − a)/Nf so that 1/Nf ≤ λ. In Figure 5.2, we display the squared-

exponential covariance kernel on the domain Ω = [−1, 1], along with ten randomfunctions sampled from GP(0, KSE ).
(a)

1.0

1.0
0.8

0.5

y

(b)

4
2

0.6
0.0

0
0.4

−0.5
−1.0
−1.0

0.2
0.0
−0.5

0.0

0.5

1.0

x

−2
−4
−1.0

−0.5

0.0

0.5

1.0

x

Figure 5.2: Random forcing terms. (a) Squared exponential covariance kernel KSE
on [−1, 1]2 with normalized length-scale λ = 0.03 (b) 10 functions sampled from theGaussian process GP(0, KSE ).
When (5.1) is a boundary-value problem, we generate training pairs by solving the
PDE with a spectral method [219] using the Chebfun software system [56], written
in MATLAB, and using a tolerance of 5 × 10−13 . We also solve the homogeneous

problem with zero-forcing, to compare the learned and exact homogeneous solutions.
The exact homogeneous solution is not included in the training dataset. When theNhomogeneous solution is zero, the solutions, {uj }N
j=1 , and forcing terms, {fj }j=1 , arerescaled, so that max1≤j≤N ∥uj ∥L∞ (Ω) = 1. By doing this, we facilitate the training of

the NNs by avoiding disproportionately small-scale or large-scale data. In the presenceof real data, with no known homogeneous solution, one could instead normalize theoutput of the NNs, NG and Nhom , to facilitate the training procedure.

5.2.2

Rational neural networks

As introduced in Chapter 4, rational NNs consist of NNs with adaptive rational
activation functions x 7→ σ(x) = p(x)/q(x), where p and q are two polynomials, whose96coefficients are trained at the same time as the other parameters of the networks, suchas the weights and biases. These coefficients are shared between all the neurons in agiven layer but generally differ between the network’s layers. This type of network
was proven to have better approximation power than standard Rectified Linear Unit
(ReLU) networks [73, 242], which means that they can approximate smooth functions
more accurately with fewer layers and network parameters (see Section 4.2). It is also
observed in Section 4.3 that rational NNs require fewer optimization steps in practiceand therefore can be more efficient to train than other activation functions.
The NNs, NG and Nhom , which approximate the Green’s function and homoge-

neous solution associated with Equation (5.1), respectively, are chosen to be rational
NNs with 4 hidden layers and 50 neurons in each layer. We choose the polynomials,
p and q, within the activation functions to be of degree 3 and 2, respectively, and initialize the coefficients of all the rational activation functions so that they are the best
(3, 2) rational approximant to a ReLU (see Section 4.3 for details). The motivationis that the flexibility of the rational functions brings extra benefit in the training andaccuracy over the ReLU activation function. We highlight that the increase in thenumber of trainable parameters, due to the adaptive rational activation functions, isonly linear with respect to the number of layers and negligible compared to the total
number of parameters in the network as:
number of rational coefficients = 7 × number of hidden layers = 28.
The weight matrices of the NNs are initialized using Glorot normal initializer [72],while the biases are initialized to zero.
Another advantage of rational NNs is the potential presence of poles, i.e., zeros of
the polynomial q. While the initialization of the activation functions avoids training
issues due to potential spurious poles, the poles can be exploited to learn physicalfeatures of the differential operator (see Section 5.4.5). Therefore, the architectureof the NNs also supports the aim of a human-understandable approach for learning
PDEs. In higher dimensions, such as d = 2 or d = 3, the Green’s function is notnecessarily bounded along the diagonal, i.e., {(x, x), x ∈ Ω}; thus making the polesof the rational NNs crucial.

Finally, we emphasize that the enhanced approximation properties of rational
NNs make them ideal for learning Green’s functions and, more generally, approximating functions within regression problems. These networks may also be of benefit97

to other approaches for solving and learning PDEs with DL techniques, such as DeepGreen [71], Neural operator [126], Fourier neural operator [127], DeepONet [135], andPINNs [184].

5.2.3

Loss function

The NNs, NG and Nhom , are trained by minimizing a mean square relative error (inthe L2 -norm) regression loss, defined as:
2
 

N
1 X
1
Loss =
uj (x) − Nhom (x) − NG (x, y)fj (y) dydx.
N j=1 ∥uj ∥2L2 (Ω) Ω
Ω

(5.4)

Unless otherwise stated, the integrals in Equation (5.4) are discretized by a trapezoidal
rule [210] using training data values that coincide with the forcing discretization grid,Nfu, and measurement points, {xi }N{yi }i=1
i=1 . As an example, for 1 ≤ j ≤ N , the squared
L2 -norm of uj , on a one-dimensional domain Ω = [a, b] ⊂ R, is approximated as b∥uj ∥2L2 (Ω) =auj (x)2 dx ≈NuX
uj (xi−1 )2 + uj (xi )2
i=2

2

∆xi ,

where ∆xi = xi − xi−1 is the length of the ith subinterval [xi−1 , xi ].

Later in Section 5.3.4, we compare the results obtained by using trapezoidal inte-

gration, described above, and a Monte-Carlo integration [21]:
Nu
b−aX
2
∥uj ∥L2 (Ω) ≈uj (xi )2 ,Nu i=1

which has a lower convergence rate to the integral with respect to the number ofpoints, Nu . This integration technique is, however, particularly suited for approximating integrals in high dimensions, or with complex geometries [21]. One could also
use a mesh of the domain and compute the integrals with a quadrature rule on eachcell.
It is also possible to incorporate some prior knowledge about the Green’s functionin the loss function, by adding a penalty term. If the differential operator is selfadjoint, then depending on the constraint operator D, the associated Green’s function
is symmetric, i.e., G(x, y) = G(y, x) for all x, y ∈ Ω. In this case, one can train asymmetric NN NG defined as

NG (x, y) = N (x, y) + N (y, x),x, y ∈ Ω.

However, our numerical experiments reveal that the NNs can learn both boundary
conditions and symmetry properties directly, from the training data, without additional constraints on the loss function or network architectures.
98

5.2.4

Optimization algorithm

The NNs are implemented with single-precision floating-point format within the TensorFlow DL library [1], and are trained2 using a two-step optimization procedure
to minimize the loss function. First, we use Adam’s algorithm [108] for the first
1000 optimization steps (or epochs), with default learning rate 0.001 and parameters
β1 = 0.9, β2 = 0.999. Then, we employ the limited memory BFGS, with boundconstraints (L-BFGS-B) optimization algorithm [37, 132], implemented in the SciPylibrary [229], with a maximum of 5 × 104 iterations. This training procedure is usedby Lu et al. to train physics-informed NNs (PINNs) and mitigate the risk of the

optimizer getting stuck at poor local minima [136]. The L-BFGS-B algorithm is also
successful for PDE learning [180] and PDE solvers using DL techniques [136, 184].
Moreover, this optimization algorithm takes advantage of the smoothness of the loss
function by using quasi-Newton approximations to second-order derivatives and often converges in fewer iterations than Adam’s algorithm and other methods basedon stochastic gradient descent [136]. Within this setting, rational NNs are beneficial
because the activation functions are smooth while maintaining an initialization close
to ReLU.
102

Loss function

L-BFGS-B

Adam

101
100
10−1
10−2
10−3
10

−4

10−5 010

Rational
ReLU
Tanh
101

102
103
Epochs104

Figure 5.3: Loss function during training. Loss function magnitudes for the ReLU,
tanh, and rational NNs with respect to the number of epochs. The networks aretrained to learn the Green’s function of the Helmholtz operator with homogeneous
Dirichlet boundary conditions and frequency K = 15. Adam’s optimizer is used until
1000 epochs (before the dashed line) and L-BFGS-B is employed thereafter.
In Figure 5.3, we display the value of the loss function during the training of
the NNs with different activation functions: rational, ReLU, and hyperbolic tangent
The numerical experiments are performed on a desktop computer with a Intel® Xeon® CPU
E5-2667 v2 @ 3.30GHz and a NVIDIA® Tesla® K40m GPU.
2

99

(tanh). In this example, we aim to learn the Green’s function of a high-frequency
Helmholtz operator with homogeneous Dirichlet boundary conditions on Ω = [0, 1]:d2 u
Lu = 2 + K 2 u,dxu(0) = u(1) = 0,

(5.5)

where K = 15 denotes the Helmholtz frequency. Note that the operator defined by
Equation (5.5) is indefinite but invertible. We first remark in Figure 5.3 that therational NN is easier to train than the other NNs, as it minimizes the loss function to10−5 with ≈ 15000 epochs, while a ReLU NN requires three times as many epochs toreach 10−4 . We also see that the loss function for the ReLU and rational NN becomes
more oscillatory [18] and harder to minimize before epoch 1000, while it converges
much faster after switching to L-BFGS-B. In theory, one could introduce a variable
learning rate that improves the behavior of Adam’s optimizer [69, 204]. However,that introduces an additional parameter, which is not desirable in the context of
PDE learning. We aim to design an adaptive and easy-to-use method that does not
require extensive hyperparameter tuning. We also observe that the tanh NN has asimilar convergence rate to the rational NN due to the smoothness of the activation
function, but this network exhibits instability during training, as indicated by thehigh value of the loss function when the optimization terminates. Rational NNs do
not suffer from this issue, thanks to the initialization close to a ReLU NN, as can beobserved in Figure 5.3, when focusing on the value of the loss function corresponding
to the early optimization steps.

5.2.5

Measuring the results

Once the NNs have been trained, we visualize the Green’s functions by sampling the
networks on a fine 1000 × 1000 grid of Ω × Ω. In the case where the exact Green’s

function Gexact is known, we measure the accuracy of the trained NN, NG , using a
relative error in the L2 -norm:
Relative Error = 100 × ∥Gexact − NG ∥L2 (Ω) /∥Gexact ∥L2 (Ω) .

(5.6)

Here, we multiplied by 100 to obtain the relative error as a percentage (%). Thisillustrates an additional advantage of using a Green’s function formulation: we cancreate test case problems with known Green’s functions and evaluate the method
using relative error and offer performance guarantees on benchmark problems. The
standard approaches in the literature often use best-case and worst-case examples
as testing procedures and therefore do not guarantee that the solution operator is100accurately learned. The “worst-case” examples can be misleading if they consist offunctions with similar behavior to the forcing terms already included in the training
dataset. Furthermore, since the space of possible forcing terms is of infinite dimension,
it is not possible to evaluate the trained NNs on all these functions to obtain a true
worst-case example.

5.3

Robustness of the method

We test the robustness of our DL method for learning Green’s functions and homogeneous solutions of differential equations, with respect to the number of training pairs,
the discretization of the solutions and forcing terms, and the noise perturbation of the
training solutions, {uj }N
j=1 . For consistency, we perform numerical experiments where
we learn the Green’s function of the Helmholtz operator with parameter K = 15 andhomogeneous Dirichlet boundary conditions (see Equation (5.5)). The performanceis measured using the relative error in the L2 -norm defined in Equation (5.6) between
the trained network, NG , and the exact Green’s function, Gexact , whose analytic expression is given by

Gexact (x, y) =( sin(15x) sin(15(y−1)), if x ≤ y,

15 sin(15)sin(15y) sin(15(x−1)),
15 sin(15)if x > y,where x, y ∈ [0, 1].

5.3.1

Influence of the activation function on the accuracy

We first compare the performances of different activation functions for learning theGreen’s functions of the Helmholtz operator by training the NNs, NG and Nhom ,with rational, ReLU, and tanh activation functions. The numerical experiments arerepeated ten times to study the statistical effect of the random initialization of the
network weights and the stochastic nature of Adam’s optimizer. The rational NN
achieves a mean relative error of 1.2% (with a standard deviation of 0.2%), while
the ReLU NN reaches an average error of 3.3% (with a standard deviation of 0.2%),
which is about three times larger. Note that the ten times difference in the loss
function between ReLU and Rational NNs, displayed in Figure 5.3, is consistent withthe factor of three in the relative error since the loss is a mean squared error and√
10 ≈ 3. This indicates that the rational neural networks are not overfitting thetraining dataset. One of the numerical experiments with a tanh NN terminated early
due to the training instabilities mentioned in Section 5.2.3, achieving a relative error101

of 99%. We excluded this problematic run when comparing the ReLU and rational
NN’s accuracy, limiting ourselves only to cases where the training was successful. The
ReLU and rational NNs did not suffer from such issues and were always successful.
The averaged relative error of the tanh NN, over the nine remaining experiments, isequal to 3.9% (with a standard deviation of 1.4%), which is slightly worse than the
ReLU NN, with higher volatility of the results.
Exact Green’s function
1.0

Rational NN (1.1% error)
1.0

0.10

0.8

0.05

y 0.6

0.10

0.8

0.05

y 0.6

0.00

0.4

0.00

0.4

0.2
0.0
0.00

0.25

0.50

0.75

1.00

−0.05

−0.05

0.2

−0.10

0.0
0.00

0.25

x
ReLU NN (4.2% error)

y

0.00

0.4

0.75

1.00

0.05

0.6
0.00
0.4

−0.05

0.2

0.10

0.8

0.05

0.50

−0.10

Tanh NN (2.7% error)

0.6

0.25

1.00

1.0

0.10

0.8

0.0
0.00

0.75

x

1.0

y

0.50

−0.10

x

−0.05

0.2
0.0
0.00

0.25

0.50

0.75

1.00

−0.10

x

Figure 5.4: Comparison of activation functions. Exact and learned Green’s functionsof the Helmholtz operator by a rational, ReLU, and tanh NN. The relative error in
the L2 norm is reported in the titles of the panels.
The exact and learned Green’s functions with rational, ReLU, and tanh NNs
are displayed in Figure 5.4. We see that the rational and tanh NNs are smoothapproximations of the exact Green’s function, while visual artifacts are present for
the ReLU NN as it is piecewise linear, despite its good accuracy.

5.3.2

Number of training pairs and spatial measurements

This section describes our method’s accuracy as we change the number of training
pairs and the size of the spatial discretization. First, we fix the number of spatial measurements to be Nu = 100, and then vary the number of input-output pairs,{(fj , uj )}N
j=1 , of the training dataset for the Helmholtz operator with Dirichlet bound-ary conditions (see Equation (5.5)). As we increase N from 1 to 100, we report the102relative error of the Green’s function learned by a rational NN in Figure 5.5(a). Next,
in Figure 5.5(b), we display the relative error on the learned Green’s function as we
increase Nu from 3 to 100, with N = 100 input-output pairs. Note that we only perform the numerical experiments once since we obtained a low variation of the relative
errors in Section 5.3.1 when the networks, NG and Nhom , have rational activation
functions. We observe similar behavior in Figure 5.5(a) and (b), where the relative

error first rapidly (exponentially) decreases as we increase the number of functions inour dataset or spatial measurements of the solutions to the Helmholtz equations withrandom forcing terms. One important thing to notice is our method’s ability to learn
the Green’s function of a high-frequency Helmholtz operator, with only 1% relative
error, using very few training pairs. The learning rate of our deep learning technique
for this operator appears to be poly-logarithmic, i.e., the number of input-output
pairs required to learn the Green’s function within accuracy 0 < ϵ < 1 behaves likeO(polylog(1/ϵ)), as predicted by the remark in Section 2.3.1.

Relative error (%)(a) 103(b) 103(c)103

102

102

102

101

101

101

100

100

100

10−1 020406080Input-output pairs (N )100

10−1 020406080Measurements (Nu )100

10−1 0102030Noise (%)

40

50

Figure 5.5: Robustness of the method. Relative error of the learned Green’s functionof the Helmholtz operator with respect to the number of input-output pairs (a),spatial measurements (b), and level of Gaussian noise perturbation (c).
The performance reaches a plateau at N ≈ 20 and Nu ≈ 20, respectively, andceases to improve. However, the stagnation of the relative error for more numerous

training data and spatial measurements is expected and can be explained by our choiceof covariance kernel length-scale, which restricts the GP’s ability to generate a wide
variety of forcing terms. Following Section 5.2.1, we chose a normalized length-scaleparameter λ = 0.03, which yields approximately 20 eigenvalues greater than 10−2 .
This issue can be resolved by decreasing the length-scale parameter and concomitantly
increasing the forcing discretization size or choosing another covariance kernel with aless pronounced eigenvalue decay rate (see Section 3.3). In summary, the number of103spatial measurements should be larger than 1/λ to resolve the forcing terms and thenumber of input-output pairs should correspond to the number of covariance kernel
eigenvalues greater than 10−2 .

5.3.3

Noise perturbation

The impact of noise in the training dataset on the accuracy of the learned Green’s
function is gauged experimentally by perturbing the system’s response measurementswith Gaussian noise as
unoise(xi ) = uj (xi )(1 + δci,j ),
j

(5.7)

where the coefficients ci,j are i.i.d. Gaussian random variables for 1 ≤ i ≤ Nu and1 ≤ j ≤ N , and δ denotes the noise level (in percent). We then vary the levelof Gaussian noise perturbation from 0% to 50%, train the NNs, NG and Nhom , foreach choice of the noise level, and report the relative error in Figure 5.5(c). We
first observe a low impact of the noise level on the accuracy of the learned Green’s
function, as a perturbation of the system’s responses measurements with 20% noise
only increases the relative error by a factor of 2 from 1.5% (no noise) to 2.7%. Whenthe level of noise exceeds 25%, we notice large variations of the relative errors andassociated higher volatility in results, characterized by a large standard deviation inerror associated with repeated numerical experiments. We consider our DL approach
relatively robust to noise in the training dataset.

5.3.4

Location of the measurements

As described in Section 5.2.1, by default, we use a uniform grid for spatial measurements of the training dataset, and thus we discretize the integrals in the loss function(cf. Equation (5.4)) using a trapezoidal rule. We conducted additional numericalexperiments on the Helmholtz example to study the influence of the measurements’location and quadrature rule on the relative error of the learned Green’s function.
We report the relative errors between the learned and exact Green’s functions inTable 5.1, using a Monte-Carlo or a trapezoidal rule to approximate the integrals
and uniform or random spatial measurements. In the latter case, the measurementupoints {xi }N
i=1 are independently and identically sampled from a uniform distribution,

U(0, 1), where Ω = [0, 1] is the domain. We find that the respective relative errors
vary between 0.96% and 1.3%. Therefore, we do not observe statistically significant
differences in the relative error computed by rational NNs. These results support the104

claim that our method is relatively robust to the type of spatial measurements in the
training dataset.
Table 5.1: Choice of quadrature rules. Relative error of the Green’s function of theHelmholtz operator with frequency K = 15 learned by a rational NN with respect tothe type of spatial measurements and quadrature rule (Monte-Carlo or trapezoidalrule) used.
Spatial measurements Monte-Carlo Trapezoidal rule
Random
1.1%
1.3%
1.3%
0.96%
Uniform

5.3.5

Missing measurements data

Since experimental data may be partially corrupted or unavailable at some spatial
locations, we assess our method’s accuracy with respect to missing measurement
data in the training dataset. We consider the high-frequency Helmholtz operator,
defined on the domain Ω = [0, 1] by Equation (5.5), with homogeneous Dirichletboundary conditions. We introduce a gap in the spatial measurements located atx ∈ [0.5, 0.7] by sampling the system’s responses, {uj }N
j=1 , uniformly on the domain[0, 0.5] ∪ [0.7, 1]. Note that the forcing terms, {fj }N
j=1 , are still sampled uniformlyon the whole domain since they are assumed to be known. The Green’s function
and homogeneous solution learned by the rational NNs are displayed in Figure 5.6(a)and (b), respectively. Surprisingly, we find that the NN, NG , can capture the high-frequency pattern of the Green’s function and achieves a relative error of 8.2%, despite
the large gap within the measurement data for x ∈ [0.5, 0.7]. Another interestingoutcome of this numerical experiment is that the lack of spatial measurements in a
specific interval does not influence the accuracy of our method outside this location,i.e., for x ∈ [0, 0.5]∪[0.7, 1] and y ∈ [0, 1] in this example. This phenomenon might be

explained by the existence of non-local effects when expressing the solution operator
associated with the PDE as an integral operator.

5.4

Human-understandable features

The trained NNs contain both the desired Green’s function and homogeneous solution, which we evaluate and visualize to glean novel insights concerning the underlying
governing PDE (Figure 5.7). In this way, we achieve one part of our human interpretation goal: finding a link between the properties of the Green’s function and that of
the underlying differential operator and solution constraints.
105

(a) 1.0
0.8

y 0.6
0.4

0.10

(b) 0.0010

0.05

0.00050.00

0.0000Exact
Learned−0.05 −0.0005

0.2
0.0
0.00

0.25

0.50

0.75

1.00

−0.10 −0.0010
0.00

0.25

0.50

x

0.75

1.00

x

Figure 5.6: Gap in measurements. (a) Green’s function of the Helmholtz operator
and its homogeneous solution (b) learned by a rational NN with no measurement
points for x ∈ [0.5, 0.7]. The space between the vertical black lines indicates where
there is a lack of spatial measurements.

(a)
y

(c)

Green’sfunction
LearnedGreen’sfunction

1.0

22

0.5

11

0.0

0

−0.5

−11

−1.0
−1.0

(b)

−0.5

0.0

0.5

1.0

x

Homogeneous
solution
Homogeneous solution

3.0

−22

22

0.5

11

0.0

0

−0.5

−11

y

Extract features

−1.0
−1.0

(d)

−0.5

0.0

0.5

1.0

x

−22

Dominant
eigenvalues
Dominant eigenvalues
102
100

10−2
10−4

(f)

Singularity
location
Singularity location
50

1

0

0

−1.5

−50

−1

−0.5

0.0

x

0.5

1.0

−100
−1.0

−0.5

0.0

x

0.5

1.0

101

102

Dominant
eigenmodes
Dominant eigenmodes
2

0.0

−3.0
−1.0

100

n

100
Exact
Learned1.5(e)Symmetries
Symmetries
1.0

−2
−1.0

−0.5

0.0

0.5

1.0

x

Figure 5.7: Feature extraction from learned Green’s functions. The NNs for thelearned Green’s function (a) and homogeneous solution (b) enable the extraction ofqualitative and quantitative features associated with the differential operator. For
example, the symmetries in the Green’s function reveal PDE invariances (c), poles ofrational NNs identify singularity type and location (d), the dominant eigenvalues (e)and eigenmodes (f) of the learned Green’s function are related to the eigenvalues andeigenmodes of the differential operator.

106

As an example, if the Green’s function is symmetric, i.e., G(x, y) = G(y, x) for all
x, y ∈ Ω, then the operator L is self-adjoint. Another aspect of human interpretability

is that the poles of the trained rational NN tend to cluster in a way that reveal thelocation and type of singularities in the homogeneous solution, discussed further in
Section 5.4.5 below. Finally, there is a direct correspondence between the dominanteigenmodes and eigenvalues (as well as the singular vectors and singular values) of
the learned Green’s function and those of the differential operator. The correspondence gives insight into the important eigenmodes that govern the system’s behavior(see Sections 5.4.2 and 5.4.3 below). This section highlights that several features ofthe differential operators can be extracted from the learned Green’s function, which
supports our aim of uncovering mechanistic understanding from input-output pairsof forcing terms and solutions.

5.4.1

Linear constraints and symmetries

We first remark that boundary constraints, such as the constraint operator, D, of

Equation (5.1), can be recovered from the Green’s function, G, of the differential
operator, L. Let f ∈ Cc∞ (Ω) be any infinitely differentiable function with compact

support on Ω, and u be the solution to Equation (5.1) with forcing term, f , such that
G(x, y)f (y) dy + uhom (x),u(x) =Ωx ∈ Ω.

Under sufficient regularity conditions, the linearity of the operator, D, implies that

D(G(·, y), Ω) = 0 for all y ∈ Ω. For instance, if D is the Dirichlet operator: D(u, Ω) =u|∂Ω , then the Green’s function satisfies G(x, y) = 0 for all x ∈ ∂Ω.
(a)(b) 1.0

1.0
0.4

0.8

0.5

y

0.2

y 0.6

0.0

0.4

0.0
−0.5
−1.0
−1.0

−0.2
−0.5

0.0

0.5

1.0

x

0.02
0.00
−0.02

0.2
0.0
0.00

0.25

0.50

0.75

1.00

x

Figure 5.8: Extraction of linear constraints. (a) Learned Green’s functions ofa second-order differential operator with an integral constraint defined in Equation (5.8). (b) Green’s function of the Helmholtz operator with periodic boundary
conditions learned by a rational NN.

107

As an example, we display in Figure 5.8(a) the learned Green’s function of thefollowing second-order differential operator on Ω = [−1, 1] with an integral constraint
on the solution:

 1

du2
Lu = 2 + x2 u,
dxu(x) dx = 2.

u(−1) = 1,

(5.8)

−1

We observe that G(−1, y) = 0 for all y ∈ [−1, 1] and one can verify that the relation
1−1

G(x, y) dx = 0 holds for any y ∈ [−1, 1]. In a second example, we learn the Green’s

function of the Helmholtz operator on Ω = [0, 1] with frequency K = 15 and periodic
boundary conditions: u(0) = u(1). One can see in Figure 5.8(b) that the Green’s
function itself is periodic and that G(0, y) = G(1, y) for all y ∈ [0, 1], as expected.

The periodicity of the Green’s function in the y-direction: G(x, 0) = G(x, 1) forx ∈ [0, 1], is due to the fact that the Helmholtz operator is self-adjoint, which implies

symmetry in the associated Green’s function. Furthermore, any linear constraint
C(u) = 0 such as linear conservation laws or symmetries [167], satisfied by all thesolutions to Equation (5.1), under forcing f ∈ Cc∞ (Ω), is also satisfied by the Green’s
function, G, such that C(G(·, y)) = 0 for all y ∈ Ω, and is therefore witnessed by theGreen’s function.

5.4.2

Eigenvalue decomposition

Let L be a self-adjoint operator and consider the following eigenvalue problem:
Lv = λv,

D(v, Ω) = 0,

(5.9)

where v is an eigenfunction of the differential operator, L, satisfying the homogeneous

constraints with associated eigenvalue, λ > 0. The eigenfunction, v, can be expressedusing the Green’s function, G, of L as
v(x) = λG(x, y)v(y) dy,Ωx ∈ Ω,

which implies that v is also an eigenfunction of the integral operator with kernel G,
but with eigenvalue 1/λ. Consider now the eigenvalue problem associated with theGreen’s function, itself:G(x, y)w(y) dy = µw(x),Ωx ∈ Ω,

where µ > 0. Then, we find that (w, 1/µ) are solutions to the eigenvalue problem (5.9). Consequently, the differential operator, L, and integral operator with kernel, G, share the same eigenfunctions, but possess reciprocal eigenvalues [207]. Thus,
108

(a) 1.0

Exact Green’s function

Learned Green’s function
0.25

1.0

0.25

0.8

0.20

0.8

0.20

y 0.6

0.15

y 0.6

0.15

0.4

0.10

0.4

0.10

0.2

0.05

0.2

0.05

0.00

0.0
0.00

0.0
0.00

0.25

0.50

0.75

1.00

0.00
0.25

x

(b)

y

0.50

0.75

1.00

x

Exact Green’s function

Learned Green’s function

1.0

0.0

1.0

0.0

0.8

−0.2

0.8

−0.2

0.6

−0.4

0.6

−0.4

0.4

−0.6

0.4

−0.6

0.2

−0.8

0.2

−0.8

−1.0

0.0
0.00

0.0
0.00

0.25

0.50

0.75

1.00

y

x

0.25

0.50

0.75

1.00

−1.0

x

Figure 5.9: Laplace and advection-diffusion operators. Exact and learned Green’s
functions of the Laplace (a) and advection-diffusion (b) operators.
we can effectively compute the lowest eigenmodes of L from the learned Green’s

function.

We now evaluate our method’s ability to accurately recover the eigenfunctions of
the Green’s function that are associated with the largest eigenvalues, in magnitude,from input-output pairs. We train a NN to learn the Green’s function of the Laplace
operator Lu = −d2 u/dx2 on [0, 1], with homogeneous Dirichlet boundary conditions,and numerically compute its eigenvalue decomposition. In Figure 5.9(a), we display
the learned and exact Green’s function, whose expression is given for x, y ∈ [0, 1] by(x(1 − y), if x ≤ y,
Gexact (x, y) =y(1 − x), if y < x.
The one hundred largest eigenvalues in magnitude, along with the corresponding
first five eigenfunctions, are visualized for the exact and learned Green’s functions inFigure 5.10. Note that the eigenvectors of the learned Green’s functions are normalized and flipped to match the ones of the exact Green’s function because eigenfunctions are unique up to a scalar multiple when the eigenvalues are all distinct. We findthat we can recover the largest eigenvalues and eigenfunctions of the learned Green’s
function and that the first 20 largest eigenvalues remain accurate. Therefore, the approximation error between the learned and exact Green’s functions mainly affects the109

(a) 100

D

V
2

10−1

2

0.2

0.4

1

2

0.2

0.4

3

4

5

1

10−2
−3

10

0

10−4
10−5

−1

10−6
10−7 0
10

1

101
n

−2

102

(b) 100

0

2

−1

x

0.6

3

0.8

4

1

5

10

1

10−2
−3

10

0

10−4
10−5

−1

10−6
10−7 0
10

101
n

−2

102

0

x

0.6

0.8

1

Figure 5.10: Eigenvalue decomposition. The first 100 largest eigenvalues and first
five eigenfunctions of the exact (a) and learned (b) Green’s functions of the Laplace
operator. The eigenvalues are represented in the left panels, while the right panels
illustrate the first five eigenfunctions of the Green’s function.

110

smallest eigenvalues. This is an essential feature of our method since the dominanteigenmodes of the differential operator L are associated with the largest eigenvalues of

the Green’s functions, which can be learned accurately. The exponential decay of thesmallest eigenvalues of the learned Green’s function in the left panel of Figure 5.10(b)is because the rational NN is a smooth approximation to the exact Green’s function.

5.4.3

Singular value decomposition

When the Green’s function of the differentiation operator, L, is square-integrable, its

associated Hilbert–Schmidt integral operator admits a singular value decomposition(SVD) (see Section 1.6). Then, there exist a positive sequence σ1 ≥ σ2 ≥ · · · > 0,
and two orthonormal bases, {ϕj } and {ψj }, of L2 (Ω) such that
G(x, y)f (y) dy + uhom (x) =u(x) =
Ω

∞
X

j=1
σj >0

σj ⟨ϕj , f ⟩ψj (x) + uhom (x),x ∈ Ω, (5.10)

where u is the solution to Equation (5.1) with forcing term f , and ⟨·, ·⟩ denotes theinner product in L2 (Ω). Therefore, the action of the solution operator f 7→ u can beapproximated using the SVD of the integral operator. Similarly to Section 5.4.2 withthe eigenvalue decomposition, the dominant terms in the expansion of Equation (5.10)
are associated with the largest singular values of the integral operator.
We now show that one can accurately recover the first singular values and singular
vectors from the Green’s function learned by a rational NN. We train a rational NN
to learn the Green’s function of an advection-diffusion operator L on Ω = [0, 1] with
Dirichlet boundary conditions, defined as
Lu =1 d2 u du++ u,
4 dx2 dxu(0) = 1, u(1) = −2.

(5.11)The learned Green’s function is illustrated in Figure 5.9(a), next to the exact Green’s
function given by:(
4x(y − 1) exp(−2(x − y)), if x ≤ y,
Gexact (x, y) =(x − 1)y,
if y < x,
for x, y ∈ [0, 1]. In Figure 5.11, we display the first five left and right singular vectors

and the singular values of the exact and learned Green’s functions. We observethat the first fifteen singular values of the learned Green’s functions are accurate.
This leads us to conclude that our method enables the construction of a low-rank
representation of the solution operator associated with the differential operator, L,

and allows us to compute and analyze its dominant modes.
111

(a)

U
2

1

2

Σ

3

4

5

2

10−1

1

10−3
10−5

0.2

0.4

1

2

x

0.6

0.8

1

10−7 0
10

0.2

0.4

1

2

0.2

0.4

3

4

5

−1

10−6
0

2

0

10−4

−1

1

1

10−2

0

−2

V

100

101
n

102

−2

0

y

0.6

0.8

1

(b)
2

3

4

100

5

2

−1

3

4

5

10
1

10−3

0

0

10−4
10−5

−1
−2

1

10−2

−1

10−6
0

0.2

0.4

x

0.6

0.8

1

10−7 0
10

101
n

102

−2

0

y

0.6

0.8

1

Figure 5.11: Singular value decomposition. Singular value decomposition of the exact(a) and learned (b) Green’s functions of the advection-diffusion operator defined by
Equation (5.11). The left and right panels, respectively, show the first five left andright singular vectors, {ϕ}5n=1 and {ψ}5n=1 , of the exact and learned Green’s functions.
The singular values of the Green’s functions are plotted in the middle panel.

1125.4.4

Schrödinger equation with double-well potential

We highlight the ability of our DL method to learn physical features of an underlying
system by considering the steady-state one-dimensional Schrödinger operator on Ω =[−3, 3]:d2 u
L(u) = −h2 2 + V (x)u, u(−3) = u(3) = 0,dx
with double-well potential V (x) = x2 +1.5 exp(−(4x)4 ) and h = 0.1 [222]. The potential V (x) is illustrated in Figure 5.12, along with the Green’s function learned by therational NN from pairs of forcing terms and the system’s responses. First, the shapeof the well potential can be visualized along the diagonal of the Green’s functionin Figure 5.12(b). Next, in Figure 5.12, we compute the first ten eigenstates of the
Schrödinger operator in Chebfun [56] and plot them using a similar representationas [222, Figure 6.9]. Similarly to Section 5.4.2, we compute the eigenvalue decomposition of the Green’s function learned by a rational NN and plot the eigenstates(shifted by the corresponding eigenvalues) in Figure 5.12. Note that the eigenvalues of the operator and the Green’s functions are reversed. We observe a perfectagreement between the first ten exact and learned eigenstates. These energy levels
capture information about the states of atomic particles modeled by the Schrödinger
equation.

5.4.5

Singularity location and typeThe input-output function of a rational NN is a high-degree rational function, whichmeans that it has poles (isolated points for which it is infinite). In rational function
approximation theory, it is known that the poles of a near-optimal rational approximant tend to cluster near a function’s singularities [223]. The clustering of the poles
near the singularity is needed for the rational approximant to have excellent global
approximation [205, 206]. Moreover, the type of clustering (algebraic, exponential,
beveled exponential) can reveal the type of singularity (square-root, blow-up, nondifferentiable) at that location. This feature of rational approximants is used in othersettings [20].
We show that the rational NNs also cluster poles in a way that identifies its
location and type. In Figure 5.13(c), we display the complex argument of the trained
rational NN for the Green’s function of a second-order differential operator with ajump condition, defined on Ω = [0, 1] as
Lu = 0.2

d2 u du+ ,dx2 dxu(0) = u(1) = 0, u(0.7− ) = 2, u(0.7+ ) = 1.
113

(a) 10

(b)

8

3.0

10
8

1.5

6

y

6
0.0

4

4
−1.5

2
0
−3.0

(c) 2.5

−1.5

0.0

1.5

−3.0
−3.0

3.0

x

Exact eigenstates

(d)

1.5

1.5

1.0

1.0

0.5

0.5
0.0
−1.5

1.5

x

0.0

1.5

3.0

x

Learned eigenstates
2.0

0.0

0
−1.5

2.5

2.0

0.0
−1.5

2

0.0

1.5

x

Figure 5.12: Schrödinger equation. (a) Double well potential V (x) = x2 +1.5 exp(−(4x)4 ). (n) Learned Green’s function of the Schrödinger equation withpotential V (x). (c) First ten exact eigenstates computed numerically from the
Schrödinger operator and (d) eigenstates computed from the learned Green’s function
displayed in (b). The eigenfunctions are shifted by an amount corresponding to theeigenvalue. The double-well potential is shown as a black curve.

114

These diagrams are known as phase portraits and are useful for illustrating complex
analysis [235]. A pole of the rational function can be identified as a point in thecomplex plane for which the full colormap goes around that point in a clockwise
fashion. In particular, in Figure 5.13(c), we see that the poles of the rational function
cluster quite closely to the real-line (where Im(z) = 0) at x = 0.7. If the clustering is
examined more closely, it may be possible to reveal that the singularity in the Green’s
function at x = 0.7 is due to a jump condition.
0.0

0.75

−0.2

y 0.50

−0.4

0.25
0.00
0.00

−0.6
0.25

0.50

0.75

1.00

(b) 2.0(c) 0.50

1.5

0.25

3π/2

0.00

π

−0.25

π/2

Im(z)

(a) 1.00

1.0
Exact
Learned

0.5
0.0
0.00

0.25

0.50

x

0.75

1.00

−0.50
0.00

2π

0
0.25

x

0.50

0.75

1.00

Re(z)

Figure 5.13: Singularity location. (a) Learned Green’s function of a second-orderdifferential operator with a jump condition at x = 0.7. Homogeneous solution of theoperator with jump condition (b) and argument of the rational NN representing thehomogeneous solution in the complex plane (c).
Rational NNs are also important for resolving Green’s function with boundary
layers as the NN can resolve the boundary layer by clustering its poles in the complex
plane. In Figure 5.14, we see a learned Green’s function of a differential equation
with a boundary layer at x = 0 with ν = 10−2 :Lu = −ν

d2 u du− ,dx2 dxu(0) = u(1) = 0,

Ω = [0, 1].

The analytical expression for the Green’s function is given by the following equation:
(

1
−xν
1/ν
y/ν
(1
−
e
)
e
−
e
, if x ≤ y,1/νGexact (x, y) = e 1 −1(1−x)/νy/ν
1−e
1−e, if y < x,e1/ν −1

While the Green’s function is not smooth, our rational NN still resolves it with relatively good accuracy, as shown by the sharp interface along the diagonal.

5.5

Viscous shock and multiphysics examples

In this section, we focus on two physical models and analyse the Green’s functions
discovered by our deep learning approach.
115

(a)

(b)

1.0
0.8

y 0.6

1.00

0.000100.75

0.00005

0.50

0.00000

Exact
Learned

0.4
0.25
0.2
0.0
0.00

0.00
0.25

0.50

0.75

1.00

−0.00005−0.00010
0.00

x

0.25

0.50

0.75

1.00

x

Figure 5.14: Boundary layer. Learned Green’s function (a) and homogeneous solution(b) to a differential equation with a boundary layer around x = 0.

5.5.1

Viscous shock

As a first example, we consider a second-order differential operator having suitablevariable coefficients to model a viscous shock at x = 0 [122]:
Lu = 10−3

dud2 u+ 2x ,
2dxdx

u(−1) = −1, u(1) = 1.

The system’s responses are obtained by solving the PDE, with Dirichlet boundary
conditions, using a spectral numerical solver for each of the N = 100 random forcing terms, sampled from a GP having a squared-exponential covariance kernel [32].
The learned Green’s function is displayed in Figure 5.15(a) and satisfies the following
symmetry relation: G(x, y) = G(−x, −y), indicating the presence of a reflective symmetry group within the underlying PDE. Indeed, if u is a solution to Lu = f withhomogeneous boundary conditions, then u(−x) is a solution to Lv = f (−x). We
also observe in Figure 5.15(b) and (c) that the homogeneous solution is accurately
captured and that the poles of the homogeneous rational NN cluster near the real axis
around x = 0: the location of the singularity induced by the shock (cf. Section 5.4.5).
Next, we reproduce the same viscous shock numerical experiment, except that thistime we remove measurements of the system’s response from the training dataset in
the interval [−0.2, 0.2]: adjacent to the shock front. By comparing Figure 5.15(d)-(f)and Figure 5.15(a)-(c), we find that the Green’s function and homogeneous solution,
learned by the rational NNs, may not be affected in the region outside of the interval with missing data. In some cases, the NNs can still accurately capture themain features of the Green’s function and homogeneous solution in the region lacking
measurements. The robustness of our method to noise perturbation and corrupted
or missing data is of significant interest and promising for real applications withexperimental data.
116

−2

0.5

0.0

−4

0.0

−0.5

−6

−0.5

−1.0

−0.5

0.0

0.5

1.0

0

−1.0

1.0

−2

0.5

0.0

−4

0.0

−0.5

−6

−0.5

−1.0

−0.5

0.0

0.5

1.0

−8

0.5

−2.0

−0.5

−2.5
−0.5

0.0

x

0.5

1.0

−3.0

3π/2

0.0

π

−0.5

π/2

−1.0

Exact
Learned

−0.5

0.0

0.5

0
−0.5

0.0

0.5

1.0

Re(z)

2

Exact
Learned

1

0

−1
−0.5

0.0

x

0.5

1.0

3π/2

0.0

π

−0.5

π/2

−1.0

x

2π

0.5

−1.0

1.0

(i)

−1.0

2π

(f) 1.0−1.5

0.0

Phase portrait

0.5

−1.0

1.0

x

−0.5
−1.0

−1.0

−1.0

(h)
0.0

0.5

−1.0

0.0

−1.0

x

(g) 1.0

−0.5

(e)

0.5

−1.0

Exact
Learned−1.0x(d) 1.0y−8(c) 1.0

Homogeneous solution

1.0

0.5

−1.0

y

(b)

Im(z)

0

Im(z)

y

Green’s function

Im(z)(a) 1.0

0
−0.5

0.0

0.5

1.0

Re(z)

1.0

2π

0.5

3π/2

0.0

π

−0.5

π/2

−1.0

−1.0

0
−0.5

0.0

0.5

1.0

Re(z)

Figure 5.15: Green’s functions learned by rational neural networks. (a) Green’s
function of a differential operator with a viscous shock at x = 0, learned by a rational
NN. (b) Learned and exact (computed by a classical spectral method) homogeneous
solution to the differential equation with zero forcing term. (c) Phase portrait of thehomogeneous rational NN evaluated on the complex plane. (d)-(f) Similar to (a)-(c),but without any system’s response measurements in x ∈ [−0.2, 0.2] (see vertical blacklines) near the shock. (g) Learned Green’s function and homogeneous solution (h) ofan advection-diffusion operator with advection occurring for x ≥ 0. (i) Phase portraitof the homogeneous NN on the complex plane.

1175.5.2

Advection-diffusion operator

We next apply our DL method to discover the Green’s function and homogeneous
solution of an advection-diffusion operator, where the advection is dominant only
within the right half of the domain:
Lu = 0.1d2 u
du+ I(x≥0) ,2dxdx

u(−1) = 2, u(1) = −1,

on Ω = [−1, 1]. Here, I(x≥0) denotes the characteristic function on x ≥ 0. The result-ing equation is diffusive on the left half of the domain, while the advection is turned
on for x ≥ 0. The output of the Green’s function NN is plotted in Figure 5.15(g),where we observe the disparate spatial behaviors of the dominant physical mechanisms. This can be recognized when observing the restriction of the Green’s function
to the subdomain [−1, 0] × [−1, 0], where the observed solution is reminiscent of theGreen’s function for the Laplacian; thus indicating that the PDE is diffusive on theleft half of the domain. Similarly, the restriction of the learned Green’s function to[0, 1] × [0, 1] is characteristic of advection.

In Figure 5.15(h) and (i), we display the homogeneous solution NN, along with thephase of the rational NN, evaluated on the complex plane. The agreement between
the exact and learned homogeneous solution illustrates the ability of the DL method
to accurately capture the behavior of a system within “multiphysics” contexts. The
choice of rational NNs is crucial here: to deepen our understanding of the system,as the poles of the homogeneous rational NN characterize the location and typeof singularities in the homogeneous solution. Here the change in behavior of thedifferential operator from diffusion to advection is delineated by the location of thepoles of the rational NN.

5.6

Two-dimensional operators and systems

Our deep learning technique for learning Green’s functions generalizes well in two
dimensions and for systems of linear partial differential equations as we will see inthis section.

5.6.1

Differential operators in two dimensions

We demonstrate the ability of our method to learn Green’s functions associated with
two-dimensional operators by repeating the numerical experiment of [126], which118consists of learning the Green’s function of the Poisson operator on the unit disk
Ω = D(0, 1), with homogeneous Dirichlet boundary conditions:
Lu = ∇2 u,u|∂D(0,1) = 0.

This experiment is a good benchmark for PDE learning techniques as the analyticalexpression of the Green’s function in Cartesian coordinates can be expressed as [157]:
(x − x̃)2 + (y − ỹ)2
1
Gexact (x, y, x̃, ỹ) =ln,
4π(xỹ − x̃y)2 + (xx̃ + y ỹ − 1)2
where (x, y), (x̃, ỹ) ∈ D(0, 1).
The training dataset for this numerical example is created as follows. First, we
generate N = 100 random forcing terms using the command randnfundisk of the
Chebfun software [56, 66, 237] with a frequency parameter of λ = 0.2, and then solve
the Poisson equation, with corresponding right-hand sides, using a spectral method.
Then, the forcing terms and system responses (i.e. solutions) are sampled at the Nu =Nf = 673 nodes of a disk mesh, generated using the Gmsh software [70]. Moreover,the mesh structure ensures that the repartition of the sample points is approximately
uniform in the disk (Figure 5.16(c)) and that the boundary is accurately captured.
The Green’s function and homogeneous rational NNs have four hidden layers andwidth of 50 neurons, with 4 and 2 input nodes, respectively, as the Green’s function
is defined on Ω × Ω. The two-dimensional integrals of the loss function (5.4) are
discretized using uniform quadrature weights: wi = π/Nf for 1 ≤ i ≤ Nf . In Figure 5.16(d)-(g), we visualize four two-dimensional slices of the learned Green’s func-tion together with two slices of the exact Green’s function in panels (a) and (b). Because of the symmetry in the Green’s function, due to the self-adjointness of L and theboundary constraints, the exact Green’s function satisfies G(x, y, 0, 0) = G(0, 0, x, y)for (x, y) ∈ D(0, 1). Therefore, we compare Figure 5.16(a) to Figure 5.16(d)-(e),and similarly for Figure 5.16(b) and Figure 5.16(f)-(g). We observe that the Green’s
function is accurately learned by the rational NN, which preserves low approximation
errors near the singularity at (x, y) = (x̃, ỹ), contrary to the neural operator technique [126]. The visual artifacts present in Figure 5.16(e)-(g) are likely due to the low

spatial discretization of the training data. One could increase the number of spatial
measurements or use a high-order quadrature rule.

119

(a)

Exact G(x, y, 0, 0)
0.0

1

(d)

0.0

−0.2
y

−0.4

0

(b)

−1

0
x

1

y

0.0

−0.4

0

−1

(f)

−1

0
x

1

(c)

−1

0
x

1

−0.8

Exact hom. solution
0.010

1

ỹ

−1

−0.8

(g)

0.0

0

0
x

1

1

−0.2
ỹ

−0.4

0

−0.4

0

−0.6
−1

−1

0
x

1

(h)

−0.8

0.0

−0.2
x̃

−0.6

−0.8

−1

−1

0
y

1

−0.8

Learned hom. solution
0.010

1

0.005
y

0.000

−1

0
x̃

Learned G(0, y, 0, ỹ)

0

0.000

−0.005
−1

−1

1

0.005
y

−0.4

0

−0.6

Learned G(x, 0, x̃, 0)
1

−0.6
−1

−0.2

−0.4

0

−0.2
x̃

0.0

−0.6

−0.8

Exact G(x, 0, x̃, 0)
1

Learned G(0, 0, x̃, ỹ)
1

−0.2

−0.6
−1

(e)

Learned G(x, y, 0, 0)
1−0.005−1−0.010−10x

1−0.010Figure 5.16: Poisson equation on the disk. Exact (a)-(b) and learned (d)-(f) Green’s
function of the Poisson operator on the unit disk, evaluated at two-dimensional slices.
The colorbar is scaled to remove the singularity of the Green’s function at (x, y) =(x̃, ỹ). (c) Exact homogeneous solution with sample points for the training functionsand (h) homogeneous solution learned by the rational NN.

120

5.6.2

System of differential equations

The method extends also naturally to systems of differential equations. Let f =

⊤
 1
⊤
f · · · f nf : Ω → Rnf be a vector of nf forcing terms and u = u1 · · · unu :Ω → Rnu be a vector of nu system responses such that       
u1
f1
u1
g1
 ..   ..  ..    .. L  .  =  .  , D  .  , Ω =  .  .
unu
f nf
unu
g nu

(5.12)The solution to Equation (5.12) with f = 0 is called the homogeneous solution and de⊤u
. Similarly to the scalar case, we can express thenoted by uhom = u1hom · · · unhom

relation between the system’s response and the forcing term using Green’s functions
and an integral formulation asiu (x) =

nf 
X
j=1

Ω

Gi,j (x, y)f j (y) dy + uihom (x),x ∈ Ω,(5.13)

for 1 ≤ i ≤ nu . Here, Gi,j : Ω × Ω → R ∪ {±∞} is a component of the Green’s
matrix for 1 ≤ i ≤ nu and 1 ≤ j ≤ nf , which consists of a nu × nf matrix of Green’s

functions:
· · · G1,nf (x, y)

..
..
,
.
.
Gnu ,1 (x, y) · · · Gnu ,nf (x, y)G1,1 (x, y)
..
G(x, y) = .

x, y ∈ Ω.

Following Equation (5.13), we remark that the differential equations decouple, andtherefore we can learn each row of the Green’s function matrix independently. Thatis, for each row 1 ≤ i ≤ nu , we train nf NNs to approximate the components

Gi,1 , . . . , Gi,nf , and one NN to approximate the ith component of the homogeneous
solution, uihom .
As an example, we consider the following system of ordinary differential equations(ODEs) on Ω = [−1, 1]:d2 u− v = f 1,
dx2
−d2 v
+ xu = f 2 ,2
dx(5.14a)(5.14b)with boundary conditions: u(−1) = 1, u(1) = −1, v(−1) = v(1) = −2. In Fig-ure 5.17, we display the different components of the Green’s matrix and the exact
solution (computed by a spectral method), along with the learned homogeneous solutions. We find that the Green’s function matrix provides insight on the coupling
121

G1,1

(a) 1.0

G1,2
1.0

0.0

0.00

uhom

(b)
1

0.5

y

0.5
−0.2 y

0.0
−0.5
−1.0
−1.0

0.0

0.5

0.0

−1.0
−1.0

1.0

x

−0.15
−0.5

G2,1

0.5

−1

−1.0

1.0

x

0.00

0.0
−0.5

y

0.0

0.4

−1.90

0.2

−1.95

−0.5
−0.02
0.0

x

0.5

1.0

0.0

0.5

1.0

0.5

1.0

x
vhom

0.5

−0.5

−0.5

1.0

0.02

0.5

−1.0
−1.0

0.0

Exact
Learned

G2,2

1.0

y

0

−0.10

−0.5

−0.4
−0.5

−0.05

−1.0
−1.0

0.0
−0.5

0.0

0.5

−2.00

1.0

x

−1.0

Exact
Learned
−0.5

0.0

x

Figure 5.17: Green’s matrix of system of ODEs. (a) Matrix of Green’s function
learned from the system of ordinary differential equations (5.14). (b) Homogeneous
solutions associated with the system of ODEs.
between the two system variables, u and v, as shown by the diagonal components
G1,2 and G2,1 of the Green’s matrix in Figure 5.17(a). Similarly, the components G1,1and G2,2 are characteristic of diffusion operators, which appear in Equation (5.14).
In this case, the Green’s matrix can be understood as a 2 × 2 block inverse [137] of
the linear operator, L.

5.7

Nonlinear and vector-valued equations

We can also discover Green’s functions from forcing terms and concomitant solutions to nonlinear differential equations possessing semi-dominant linearity as well asGreen’s functions associated with vector-valued equations.

5.7.1

Linearized models of nonlinear operators

We demonstrate that our DL method can be used to linearize and extract Green’s
functions from nonlinear boundary value problems of the form
Lu + ϵN (u) = f,

D(u, Ω) = g,where L denotes a linear operator, N is a nonlinear operator, and ϵ < 1 is a small
parameter controlling the nonlinearity. We demonstrate this ability on the three122

nonlinear boundary value problems, dominated by the linearity, used in [71]. In
Figure 5.18(a)-(c), we visualize the Green’s function NNs of three operators withcubic nonlinearity considered in [71].
(a) 2π
y

0.00

−0.25

π

0
0

π

2π

(b) 2π
y

−0.50

0.00

−0.12

π

0
0

x

2π

y

−0.24

0.14

0.07

π

0

0.00
0

π

x

G1,1

(d)1.0π(c) 2π

x

G1,2
25

2π

1.0

6

Exact velocity(e)1.0

1.0
0.8

y

0.5

10

y

0.5

0

y

0.6
0.5
0.4
0.2

0.0
0.0

0.5

1.0

−5

0.0
0.0

x

0.5

1.0

−6

0.0

0.0
0.0

0.5

x

G2,1

x

(f)

G2,2

1.0

6

1.0

1.0

25

Learned velocity
1.0

1.0
0.8

y

0.5

0

y

0.5

10

y

0.6
0.5
0.4
0.2

0.0
0.0

0.5

1.0

−6

x

0.0
0.0

0.5

1.0

−5

0.0

0.0
0.0

x

0.5

1.0

x

Figure 5.18: Linearized models and Stokes flow. (a)-(c) Green’s functions of three differential operators: Helmholtz, Sturm–Liouville, and biharmonic, with cubic nonlinearity. (d) Matrix of Green’s functions of a two-dimensional Stokes flow in a lid-driven
cavity, evaluated at the two-dimensional slice (x, y, 0.5, 0.5). Velocity magnitude andstreamlines of the exact (e) and learned (f) homogeneous solution to the Stokes equations with zero applied body force.
First, Figure 5.18(a) illustrates the learned Green’s function of a cubic Helmholtz
system on Ω = [0, 2π] with homogeneous Dirichlet boundary conditions:d2 u+ αu + ϵu3 = f (x),dx2where α = −1 and ϵ = 0.4. Next, in Figure 5.18(b), we consider a nonlinear Sturm–
Liouville operator of the form:[−p(x)u′ ]′ + q(x)(u + ϵu3 ) = f (x),123u(0) = u(2π) = 0,

with p(x) = 0.4 sin(x) − 3, q(x) = 0.6 sin(x) − 2, and ϵ = 0.4. The notation u′

denotes the derivative with respect to x, du/dx. Finally, the example represented in
Figure 5.18(c) is the learned Green’s function of a nonlinear biharmonic operator:[−p(x)u′′ ]′′ + q(u + ϵu3 ) = f (x),u(0) = u(2π) = 0,

where p = −4, q = 2, and ϵ = 0.4.
The nonlinearity does not prevent our method from discovering a Green’s functionof an approximate linear model, from which one can understand features such as symmetry and boundary conditions. This property is crucial for tackling time-dependentproblems, where the present technique may be extended and applied to uncover linear
propagators.

5.7.2

Lid-driven cavity problem

Finally, we consider a classical benchmark in fluid dynamics consisting of Stokes flow
in a two-dimensional lid-driven cavity problem [62]. We aim to discover the matrix ofGreen’s functions of the Stokes flow [22], which is modelled by the following systemof equations on the domain Ω = [0, 1]2 ,µ∇2 u − ∇p = f ,∇ · u = 0.
Here, u = (ux , uy ) is the fluid velocity, p is the pressure, f = (fx , fy ) is an applied
body force (i.e. a forcing term), and µ = 1/100 is the dynamic viscosity. The fluid
velocity satisfies no-slip boundary conditions on the walls, except on the top wall
where u = (1, 0). We first generate one hundred forcing terms, f , with two smooth
random components, in the Chebfun software [56, 66] using the randnfun2 commandwith wavelength parameter λ = 0.1. The Stokes equations are then discretized with
Taylor–Hood finite elements [23, 212] for the velocity and pressure on a mesh with 96×
96 square cells and subsequently solved using the Firedrake finite element library [188].
We illustrate in Figure 5.19 an example of applied body force and velocity solution
obtained by solving the system of PDEs. We then create the training dataset for theNNs by sampling the applied body forces and corresponding velocity solutions, u, ona regular 25 × 25 grid.
In this context, the relation between the system’s responses and the forcing terms
can be expressed using a Green’s matrix, which consists of a two-by-two matrix ofGreen’s functions and whose components reveal features of the underlying system124

(a) 1.0

4(b) 1.0

1.00

3

y 0.5

2

0.75

y 0.5

0.50

1
0.0

0.25

0
0.0

0.5

0.0

1.0

0.00
0.0

x

0.5

1.0

x

Figure 5.19: Training functions for Stokes flow. (a) Magnitude of a random applied
body force used as a forcing term in the Stokes equations. (b) Velocity magnitude
and streamlines of the system’s response.
such as symmetry and coupling (Figure 5.18(d) and Section 5.6.2). The four Green’s
functions and two homogeneous NNs have the same architecture as the one describedin Section 5.2.2, except that they have respectively four and two input nodes (insteadof two and one) due to the current spatial dimension. Figure 5.18(e) and (f) illustrate that the homogeneous solution to the Stokes equation is accurately capturedby the homogeneous rational NN, despite the corner singularities and coarse measurement grid. The four components of the Green’s matrix for the Stokes flow areevaluated on the two-dimensional slice (x, y, 0.5, 0.5), for x, y ∈ [0, 1], and displayedin Figure 5.18(d). This figure allows us to visualize the system’s response to a point
force, f = (fx , fy ), located at (0.5, 0.5), with the system’s response being denoted as
u = (ux , uy ), whereux (x, y) = G1,1 (x, y, 0.5, 0.5)fx + G1,2 (x, y, 0.5, 0.5)fy ,uy (x, y) = G2,1 (x, y, 0.5, 0.5)fx + G2,2 (x, y, 0.5, 0.5)fy ,for x, y ∈ [0, 1]. The visualization of the G2,2 component in Figure 5.18(d), cor-

responding to the system’s response to a unitary vertical point force f = (0, 1) isreminiscent of [61, Figure 1].
Finally, we evaluate the components of the Green’s matrix at three other twodimensional slices: (x, 0.5, x̃, 0.5), (0.5, y, 0.5, ỹ), (0.5, 0.5, x̃, ỹ) and display them respectively in Figures 5.20 to 5.22. These figures illustrate the different symmetries of the Green’s matrix, which are captured by the rational NNs. As an example, we see in Figures 5.20 and 5.21 that G1,1 (x, 0.5, x̃, 0.5) = G2,2 (0.5, x, 0.5, x̃)and G2,2 (x, 0.5, x̃, 0.5) = G1,1 (0.5, x, 0.5, x̃), for x, x̃ ∈ [0, 1]. Similarly, we findin Figure 5.22 that G1,1 (0.5, 0.5, x̃, ỹ) = G1,1 (0.5, 0.5, ỹ, x̃) and G1,2 (0.5, 0.5, x̃, ỹ) =G2,1 (0.5, 0.5, x̃, ỹ), for x̃, ỹ ∈ [0, 1]. The G1,2 and G2,1 components of the Green’s
125G1,1 (x, 0.5, x̃, 0.5)

G1,2 (x, 0.5, x̃, 0.5)

1.0

25

1.0

1

x̃ 0.510x̃ 0.5

0

−5

0.0

0.0
0.0

0.5

1.0

0.0

0.5

x

x

G2,1 (x, 0.5, x̃, 0.5)

G2,2 (x, 0.5, x̃, 0.5)

1.0

−1

1.0

1

1.0

25

x̃ 0.50x̃ 0.5

10

0.0
0.0

0.5

1.0

−1

0.0
0.0

x

0.5

1.0

−5

x

Figure 5.20: 2nd Green’s matrix slice of Stokes flow. The four components of theGreen’s matrix learned by a rational neural network evaluated at the two-dimensionalslice (x, 0.5, x̃, 0.5).

G1,1 (0.5, y, 0.5, ỹ)

G1,2 (0.5, y, 0.5, ỹ)

1.0

25

1.0

1

ỹ 0.510ỹ 0.5

0

−5

0.0

0.0
0.0

0.5

1.0

0.0

y

0.5

1.0

−1

y

G2,1 (0.5, y, 0.5, ỹ)G2,2 (0.5, y, 0.5, ỹ)

1.0

1

1.0

25

ỹ 0.50ỹ 0.5

10

0.0
0.0

0.5

1.0

−1

0.0
0.0

y

0.5

1.0

−5

y

Figure 5.21: 3rd Green’s matrix slice of Stokes flow. The four components of theGreen’s matrix learned by a rational neural network evaluated at the two-dimensionalslice (0.5, y, 0.5, ỹ).

126

G1,1 (0.5, 0.5, x̃, ỹ)

G1,2 (0.5, 0.5, x̃, ỹ)

1.0

25

1.0

6

ỹ 0.510ỹ 0.5

0

−5

0.0

0.0
0.0

0.5

1.0

0.0

0.5

x̃

1.0

−6

x̃

G2,1 (0.5, 0.5, x̃, ỹ)G2,2 (0.5, 0.5, x̃, ỹ)

1.0

6

1.0

25

ỹ 0.50ỹ 0.5

10

0.0
0.0

0.5

1.0

−6

0.0
0.0

x̃

0.5

1.0

−5

x̃

Figure 5.22: 4th Green’s matrix slice of Stokes flow. The four components of theGreen’s matrix learned by a rational neural network evaluated at the two-dimensionalslice (0.5, 0.5, x̃, ỹ).
matrix in Figure 5.20 highlight a singularity along the diagonal (x, 0.5, x, 0.5) forx ∈ [0, 1]. However, this singularity does not prevent the rational NNs from accurately

learning the different components of the Green’s matrix displayed in Figure 5.18(d)and Figures 5.20 to 5.22.

5.8

Time-dependent equations

In this section, we show that one can use a time-stepping scheme to discretize a timedependent PDE and learn the Green’s function associated with the time-propagator
operator τ : un → un+1 , where un is the solution of the PDE at time t = n∆t fora fixed time step ∆t. As an example, we consider the time-dependent Schrödinger
equation with a harmonic trap potential V (x) = x2 given byi∂ψ(x, t)1 ∂ 2 ψ(x, t)=−+ x2 ψ(x, t),
∂t
2 ∂x2

x ∈ [−3, 3],(5.15)

with homogeneous Dirichlet boundary conditions. We use a Crank–Nicolson timestepping scheme with time step ∆t = 2 × 10−2 to discretize Equation (5.15) in timeand obtainψn+1 − ψn1
1 d2 ψn+1
1 d2 ψn22i=−+ x ψn+1 −+ x ψn .
∆t
2
2 dx2
2 dx2
127

Our training dataset consists of one hundred random initial forcing functions ψn attime t and associated response ψn+1 at time t + ∆t. The functions ψn have real andimaginary parts sampled from a Gaussian process with periodic kernel and lengthscale parameter λ = 0.5 (see Section 5.2.1), and multiplied by the Gaussian damping
6function g(x) = e−x /20 to ensure that the functions decay to zero before reaching
the domain boundaries. We then train a rational neural network to learn the Green’s
function G associated with the time-propagator operator such that
 3

G(x, y)ψn (y) dy = ψn+1 (x),τ (ψn )(x) =−3x ∈ [−3, 3].

Note that since ψ takes complex values, we in fact split Equation (5.15) into a systemof equations for the real and imaginary parts of ψ, and learn the Green’s matrix
associated with the system (see Section 5.6.2).
G1,1(a) 3.0

G1,2
3

1.5

y

2

0.0

0.0

1.5

0.0

0

−3.0
−3.0

3.0

x

−1.5

0.0

1.5

3.0

0.0

0

−1.5

x

−2
−3.0

−1.5

y

1.5

3.0

−4

3
2

0.0

−3.0
−3.0

0.0

1.5

3.0

1.5

3.0

x
v

3.0

2
1

Exact
Predicted

0
1

−1.5
0.0

−4

x

1.5

−1.5

0

G2,2
4

1.5

−3.0
−3.0

Exact
Predicted

−1

G2,1
3.0

y

u

(b) 2
1

−1.5

0
−1.5

4

1.5

y

1

−1.5
−3.0
−3.0

3.0

0
−1.5

0.0

x

1.5

3.0

−1
−2
−3.0

−1.5

0.0

x

Figure 5.23: Green’s matrix of the time-dependent Schrödinger equation. (a) The
four components of the Green’s matrix for the time propagator operator of the timedependent Schrödinger equation discretized using a time-stepping scheme. (b) Realand imaginary components of the worst case prediction of the solution at the nexttime step.
We report the Green’s matrix of the time-propagator operator for the Schrödinger
equation in Figure 5.23(a) and observe that the four components are dominated by
the diagonal, which is expected for a small time-step. Additionally, we evaluate theaccuracy of the learned Green’s functions by generating a testing dataset with one
hundred initial functions ψn , sampled from the same distribution, and associatedsolution ψn+1 at time t + ∆t. We then compute the average (over the one hundred128

test cases) relative error in the L2 norm between the exact solution ψn+1 and the one
pred
predicted using the learned Green’s functions, ψn+1
, as
pred
relative error = ∥ψn+1 − ψn+1
∥L2 ([−3,3]) /∥ψn+1 ∥L2 ([−3,3]) ,pred
where ψn+1
is defined as
pred
ψn+1
(x) =

 3

G(x, y)ψn (y) dy,−3x ∈ [−3, 3].

Finally, we obtain an average relative error of 1.3% with standard deviation 0.2%
across the 100 test cases, confirming the good accuracy of our method. We display
the worst-case prediction of the solution ψn+1 in Figure 5.23(b).

129Conclusions
This thesis derived theoretical results and a practical deep learning algorithm forapproximating Green’s functions associated with linear partial differential equations(PDEs) from pairs of forcing terms and solutions to a PDE.
By generalizing the randomized singular value decomposition (SVD) to Hilbert–
Schmidt (HS) operators in Chapter 2, we showed that one can rigorously learn theGreen’s function associated with an elliptic PDE in three dimensions. We derived alearning rate associated with elliptic partial differential operators in three dimensions
and bounded the number of input-output training pairs required to recover a Green’s
function approximately with high probability. The random forcing functions are
sampled from a Gaussian process (GP) with mean zero and are characterized by the
associated covariance kernel. One practical outcome of this work is a measure for thequality of covariance kernels, which may be used to design efficient GP kernels for
PDE learning tasks.
We then explored the practical extensions of the randomized SVD to Gaussian
random vectors with correlated entries (i.e., nonstandard covariance matrices) andHS operators in Chapter 3. This chapter motivates new computational and algorithmic approaches for constructing the covariance kernel based on prior information tocompute a low-rank approximation of matrices and impose properties on the learned
matrix and random functions from the GP. We performed numerical experiments
to demonstrate that covariance matrices with prior knowledge can outperform thestandard identity matrix used in the literature and lead to near-optimal approximation errors. In addition, we proposed a covariance kernel based on weighted Jacobi
polynomials, which allows the control of the smoothness of the random functions
generated and may find practical applications in PDE learning [27, 30] as it imposes
prior knowledge of Dirichlet boundary conditions. The algorithm presented in thischapter is limited to matrices and HS operators and does not extend to unbounded
operators such as differential operators. Additionally, the theoretical bounds only offer probabilistic guarantees for Gaussian inputs, while sub-Gaussian distributions [98]130

of the inputs would be closer to realistic application settings.
Motivated by the theoretical results obtained in Chapters 2 and 3, we wanted
to design an efficient deep learning architecture for learning Green’s functions. In
Chapter 4, we investigated rational neural networks, which are neural networks withsmooth trainable activation functions based on rational functions. We proved theoretical statements quantifying the advantages of rational neural networks over ReLU
networks. In particular, we remarked that a composition of low-degree rational functions has a good approximation power but a relatively small number of trainable
parameters. Therefore, we showed that rational neural networks require fewer nodes
and exponentially smaller depth than ReLU networks to approximate smooth functions to within a certain accuracy. This improved approximation power has practical consequences for large neural networks, given that a deep neural network is
computationally expensive to train due to expensive gradient evaluations and slower
convergence. The experiments conducted in the chapter demonstrate the potential
applications of these rational networks for solving PDEs and generative adversarial networks. The practical implementation of rational networks is straightforwardin the TensorFlow framework and consists of replacing the activation functions bytrainable rational functions. The main benefits of rational NNs are their fast approximation power, the trainability of the activation parameters, and the smoothness of
the activation function outside poles.
Our primary objective in Chapter 5 was to uncover mechanistic understanding
from input-output data using a human-understandable representation of an underlying hidden differential operator. This representation took the form of a rational NN
for the Green’s function. We extensively described all the physical features of theoperator that can be extracted and discovered from the learned Green’s function andhomogeneous solutions, such as linear conservation laws, symmetries, shock front andsingularity locations, boundary conditions, and dominant modes. Our deep learning
method for learning Green’s functions and extracting human-understandable properties of partial differential equations benefits from the adaptivity of rational neuralnetworks and its support for qualitative feature detection and interpretation. We successfully tested our approach with noisy and sparse measurements as training data in
one and two dimensions. The design of our network architecture, and the covariancekernel used to generate the system forcings, are guided by rigorous theoretical statements, obtained in Chapters 2 to 4, that offer performance guarantees. This showsthat our proposed deep learning method may be used to discover new mechanistic
understanding with machine learning.
131

The deep learning method naturally extends to the case of three spatial dimensions
but these systems are more challenging due to the GPU memory demands required
to represent the six-dimensional inputs used to train the neural network representing
the Green’s function. However, alternative optimization algorithms than the one weused, such as mini-batch optimization [108, 125], may be employed to alleviate thecomputational expense of the training procedure. While our method is demonstrated
on linear differential operators, it can be extended to nonlinear, time-dependent problems that can be linearized using an implicit-explicit time-stepping scheme [12, 171]or an iterative method [105]. This process should allow us to learn Green’s functionsof linear time propagators and understand physical behavior in time-dependent problems from input-output data such as the time-dependent Schrödinger equation. The
numerical experiments conducted in Chapter 5 highlight that our approach can generalized to discover Green’s functions of some linearization of a nonlinear differential
operator.
There are many future research directions exploring the potential applications ofrational networks beyond PDE learning, in fields such as image classification, timeseries forecasting, and generative adversarial networks. These applications already
employ nonstandard activation functions to overcome various drawbacks of ReLU.
Another exciting and promising field is the numerical solution and data-driven discovery of partial differential equations with deep learning. We believe that popular
techniques such as physics-informed neural networks [184] could benefit from rational NNs to improve the robustness and performance of PDE solvers, both from atheoretical and practical viewpoint.
Finally, while the ideas present in Chapter 2 have been recently applied to derivea learning rate for Green’s functions associated with parabolic PDEs [28], obtaining
theoretical results for more general classes of PDEs, such as hyperbolic, fractional, or
stochastic PDEs, remain highly challenging. Such studies are essential to understandwhich mathematical models can be learned from data, obtain performance guarantees,
and, more generally, deepen our knowledge of PDE learning techniques.

132

Bibliography[1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,
S. Ghemawat, G. Irving, M. Isard, et al., TensorFlow: A System forLarge-Scale Machine Learning, in 12th USENIX Conference on Operating Systems Design and Implementation, 2016, pp. 265–283.
[2] H. Abdi and L. J. Williams, Principal component analysis, Wiley Interdiscip. Rev. Comput. Stat., 2 (2010), pp. 433–459.
[3] N. I. Achieser, Theory of Approximation, Courier Corporation, 2013.
[4] N. Ailon and B. Chazelle, Approximate nearest neighbors and the fast
Johnson–Lindenstrauss transform, in Proceedings of the Thirty-Eighth Annual
ACM Symposium on Theory of Computing, 2006, pp. 557–563.
[5], The fast Johnson–Lindenstrauss transform and approximate nearest
neighbors, SIAM J. Comput., 39 (2009), pp. 302–322.

[6] R. Alexander and D. Giannakis, Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques, Physica D, 409(2020), p. 132520.
[7] O. Alter, P. O. Brown, and D. Botstein, Singular value decomposition
for genome-wide expression data processing and modeling, Proc. Natl. Acad.
Sci. USA, 97 (2000), pp. 10101–10106.
[8] M. Anthony and P. Bartlett, Neural network learning: Theoretical foundations, Cambridge University Press, 1999.
[9] G. Arfken, H. Weber, and F. E. Harris, Mathematical Methods for
Physicists, Academic Press, 7th ed., 2012.

133

[10] M. Arjovsky, S. Chintala, and L. Bottou, Wasserstein Generative Adversarial Networks, in Proc. 34th International Conference on Machine Learning(ICML), 2017, pp. 214–223.
[11] S. Arridge, P. Maass, O. Öktem, and C.-B. Schönlieb, Solving inverse
problems using data-driven models, Acta Numer., 28 (2019), pp. 1–174.
[12] U. M. Ascher, S. J. Ruuth, and R. J. Spiteri, Implicit-explicit RungeKutta methods for time-dependent partial differential equations, Appl. Numer.
Math., 25 (1997), pp. 151–167.
[13] J. Ballani and D. Kressner, Matrices with hierarchical low-rank structures,in Exploiting Hidden Structure in Matrix Computations: Algorithms and Applications, Springer, 2016, pp. 161–209.
[14] A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind,
Automatic differentiation in machine learning: a survey, J. Mach. Learn. Res.,18 (2018), pp. 1–43.
[15] M. Bebendorf, Hierarchical Matrices: A Means to Efficiently Solve Elliptic
Boundary Value Problems, Springer-Verlag, 2008.
[16] M. Bebendorf and W. Hackbusch, Existence of H -matrix approximants
to the inverse FE-matrix of elliptic operators with L∞ -coefficients, Numer.
Math., 95 (2003), pp. 1–28.
[17] B. Beckermann and A. Townsend, On the Singular Values of Matrices
with Displacement Structure, SIAM J. Matrix Anal. A., 38 (2017), pp. 1227–
1248.
[18] Y. Bengio, Practical recommendations for gradient-based training of deep architectures, in Neural networks: Tricks of the trade, Springer, 2012, pp. 437–478.
[19] Y. Bengio, P. Simard, and P. Frasconi, Learning Long-Term Dependencies with Gradient Descent is Difficult, IEEE T. Neural Netw., 5 (1994),
pp. 157–166.
[20] W. T. Beyene, Pole-clustering and rational-interpolation techniques for simplifying distributed systems, IEEE T. Circuits-I, 46 (1999), pp. 1468–1472.

134

[21] K. Binder, D. M. Ceperley, J.-P. Hansen, M. Kalos, D. Landau,
D. Levesque, H. Mueller-Krumbhaar, D. Stauffer, and J.-J. Weis,
Monte Carlo Methods in Statistical Physics, Springer Science & Business Media,
2012.
[22] J. R. Blake, A note on the image system for a Stokeslet in a no-slip boundary,Math. Proc. Camb. Philos. Soc., 70 (1971), pp. 303–310.
[23] D. Boffi, F. Brezzi, and M. Fortin, Mixed Finite Element Methods andApplications, Springer, 2013.
[24] A. Bonito, A. Cohen, R. DeVore, G. Petrova, and G. Welper,
Diffusion coefficients estimation for elliptic partial differential equations, SIAM
J. Math. Anal., 49 (2017), pp. 1570–1592.
[25] W. Boukaram, G. Turkiyyah, and D. Keyes, Randomized GPU algorithms for the construction of hierarchical matrices from matrix-vector operations, SIAM J. Sci. Comput., 41 (2019), pp. C339–C366.
[26] N. Boullé, NBoulle/GreenLearning - Software and datasets (version v1.0).
Zenodo. https://doi.org/10.5281/zenodo.4656020, 2021.
[27] N. Boullé, C. J. Earls, and A. Townsend, Data-driven discovery
of Green’s functions with human-understandable deep learning, Sci. Rep., 12(2022).
[28] N. Boullé, S. Kim, T. Shi, and A. Townsend, Learning Green’s functions
associated with time-dependent partial differential equations, J. Mach. Learn.
Res., 23 (2022), pp. 1–34.
[29] N. Boullé, Y. Nakatsukasa, and A. Townsend, GitHub repository.
https://github.com/NBoulle/RationalNets/, 2020.
[30] N. Boullé, Y. Nakatsukasa, and A. Townsend, Rational neural networks, in Advances in Neural Information Processing Systems (NeurIPS),
vol. 33, 2020, pp. 14243–14253.
[31] N. Boullé and A. Townsend, A generalization of the randomized singularvalue decomposition, in International Conference on Learning Representations(ICLR), 2022.

135

[32] N. Boullé and A. Townsend, Learning elliptic partial differential equations
with randomized linear algebra, Found. Comput. Math., (2022).
[33] H. Brezis, Functional Analysis, Sobolev Spaces and Partial Differential Equations, Springer Science & Business Media, 2010.
[34] S. L. Brunton, M. Budišić, E. Kaiser, and J. N. Kutz, Modern Koopman theory for dynamical systems, arXiv preprint arXiv:2102.12086, (2021).
[35] S. L. Brunton, B. R. Noack, and P. Koumoutsakos, Machine Learning
for Fluid Mechanics, Annu. Rev. Fluid Mech., 52 (2020), pp. 477–508.
[36] S. L. Brunton, J. L. Proctor, and J. N. Kutz, Discovering governing
equations from data by sparse identification of nonlinear dynamical systems,
Proc. Natl. Acad. Sci. USA, 113 (2016).
[37] R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu, A limited memory algorithm
for bound constrained optimization, SIAM J. Sci. Comput., 16 (1995), pp. 1190–
1208.
[38] T. Chen and H. Chen, Universal approximation to nonlinear operators byneural networks with arbitrary activation functions and its application to dynamical systems, IEEE Trans. Neur. Netw., 6 (1995), pp. 911–917.
[39] Z. Chen, F. Chen, R. Lai, X. Zhang, and C.-T. Lu, Rational Neural Networks for Approximating Graph Convolution Operator on Jump Discontinuities,
in IEEE International Conference on Data Mining (ICDM), 2018, pp. 59–68.
[40] X. Cheng, B. Khomtchouk, N. Matloff, and P. Mohanty, Polynomial
Regression As an Alternative to Neural Nets, arXiv preprint arXiv:1806.06850,(2018).
[41] H. Chernoff, A measure of asymptotic efficiency for tests of a hypothesis
based on the sum of observations, Ann. Math. Stat., (1952), pp. 493–507.
[42] S. Cho, H. Dong, and S. Kim, Global estimates for Green’s matrix of second
order parabolic systems with application to elliptic systems in two dimensional
domains, Potential Anal., 36 (2012), pp. 339–372.
[43] F. Chollet et al., Keras. https://keras.io, 2015.

136

[44] K. L. Clarkson and D. P. Woodruff, Low-rank approximation and regression in input sparsity time, J. ACM, 63 (2017), pp. 1–45.
[45] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, Fast and Accurate
Deep Network Learning by Exponential Linear Units (ELUs), arXiv preprint
arXiv:1511.07289, (2015).
[46] G. Cybenko, Approximation by superpositions of a sigmoidal function, Math.
Control Signals Syst., 2 (1989), pp. 303–314.
[47] J. Daws Jr. and C. G. Webster, A Polynomial-Based Approach for Architectural Design and Learning with Deep Neural Networks, arXiv preprint
arXiv:1905.10457, (2019).
[48] C. de Boor, An alternative approach to (the teaching of ) rank, basis, anddimension, Lin. Alg. Appl., 146 (1991), pp. 221–229.
[49] M. V. de Hoop, N. B. Kovachki, N. H. Nelsen, and A. M. Stuart,
Convergence rates for learning linear operators from noisy data, arXiv preprint
arXiv:2108.12515, (2021).
[50] P. Deheuvels and G. V. Martynov, A Karhunen–Loeve decompositionof a Gaussian process generated by independent pairs of exponential randomvariables, J. Funct. Anal., 255 (2008), pp. 2363–2394.
[51] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 248–255.
[52] R. A. DeVore, R. Howard, and C. Micchelli, Optimal nonlinear approximation, Manuscripta Math., 63 (1989), pp. 469–478.
[53] H. Dong and S. Kim, Green’s matrices of second order elliptic systems withmeasurable coefficients in two dimensional domains, Trans. Am. Math. Soc.,
361 (2009), pp. 3303–3323.
[54], Green’s function for nondivergence elliptic operators in two dimensions,
SIAM J. Math. Anal., 53 (2021), pp. 4637–4656.

[55] M. F. Driscoll, The reproducing kernel Hilbert space structure of the sample
paths of a Gaussian process, Zeit. Wahrscheinlichkeitstheorie Verwandte Geb.,
26 (1973), pp. 309–316.
137

[56] T. A. Driscoll, N. Hale, and L. N. Trefethen, Chebfun Guide, Pafnuty
Publications, 2014.
[57] R. Durrett, Probability: Theory and Examples, Cambridge University Press,
5th ed., 2019.
[58] C. Eckart and G. Young, The approximation of one matrix by another oflower rank, Psychometrika, 1 (1936), pp. 211–218.
[59] D. E. Edmunds and W. D. Evans, Spectral theory and differential operators,
Oxford University Press, 2018.
[60] D. E. Edmunds, V. M. Kokilashvili, and A. Meskhi, Bounded and compact integral operators, Springer Science & Business Media, 2013.
[61] M. Ekiel-Jeżewska, R. Boniecki, M. Bukowicki, and M. Gruca,
Stokes velocity generated by a point force in various geometries, Eur. Phys.
J. E, 41 (2018), pp. 1–7.
[62] H. C. Elman, D. J. Silvester, and A. J. Wathen, Finite Elements andFast Iterative Solvers: With Applications in Incompressible Fluid Dynamics,
Oxford University Press, 2nd ed., 2014.
[63] B. Engquist and H. Zhao, Approximate separability of the Green’s functionof the Helmholtz equation in the high frequency limit, Comm. Pure Appl. Math.,
71 (2018), pp. 2220–2274.
[64] L. C. Evans, Partial Differential Equations, AMS, Providence, RI, 2nd ed.,2010.
[65] J. Feliu-Faba, Y. Fan, and L. Ying, Meta-learning pseudo-differential operators with deep neural networks, J. Comput. Phys., 408 (2020).
[66] S. Filip, A. Javeed, and L. N. Trefethen, Smooth random functions,random ODEs, and Gaussian processes, SIAM Rev., 61 (2019), pp. 185–205.
[67] S.-I. Filip, Y. Nakatsukasa, L. N. Trefethen, and B. Beckermann,
Rational Minimax Approximation via Adaptive Barycentric Representations,
SIAM J. Sci. Comput., 40 (2018), pp. A2427–A2455.
[68] J. Foster, T. Lyons, and H. Oberhauser, An optimal polynomial approximation of Brownian motion, SIAM J. Numer. Anal., 58 (2020), pp. 1393–1421.
138[69] A. P. George and W. B. Powell, Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming, Mach. Learn., 65(2006), pp. 167–198.
[70] C. Geuzaine and J.-F. Remacle, Gmsh: A 3-D finite element mesh generator with built-in pre-and post-processing facilities, Int. J. Numer. MethodsEng., 79 (2009), pp. 1309–1331.
[71] C. R. Gin, D. E. Shea, S. L. Brunton, and J. N. Kutz, DeepGreen:
deep learning of Green’s functions for nonlinear boundary value problems, Sci.
Rep., 11 (2021), pp. 1–14.
[72] X. Glorot and Y. Bengio, Understanding the difficulty of training deep
feedforward neural networks, in Proc. 13th International Conference on Artificial
Intelligence and Statistics (AISTATS), 2010, pp. 249–256.
[73] X. Glorot, A. Bordes, and Y. Bengio, Deep Sparse Rectifier Neural
Networks, in Proc. 14th International Conference on Artificial Intelligence andStatistics (AISTATS), 2011, pp. 315–323.
[74] G. H. Golub and C. F. Van Loan, Matrix Computations, JHU Press,
4th ed., 2013.
[75] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT
Press, 2016.
[76] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, Generative Adversarial
Nets, in Advances in Neural Information Processing Systems (NeurIPS), 2014,
pp. 2672–2680.
[77] M. Goyal, R. Goyal, and B. Lall, Improved polynomial neural networks
with normalised activations, in International Joint Conference on Neural Networks (IJCNN), 2020, pp. 1–8.
[78] A. Graves and N. Jaitly, Towards end-to-end speech recognition with recurrent neural networks, in International Conference on Machine Learning (ICML),
PMLR, 2014, pp. 1764–1772.

139

[79] G. Green, An essay on the application of mathematical analysis to the theories
of electricity and magnetism, J. für die Reine und Angew. Math., 47 (1854),
pp. 161–221.
[80] M. Grüter and K.-O. Widman, The Green function for uniformly elliptic
equations, Manuscripta Math., 37 (1982), pp. 303–342.
[81] S. Guarnieri, F. Piazza, and A. Uncini, Multilayer feedforward networks
with adaptive spline activation function, IEEE Trans. Neural Networ., 10 (1999),
pp. 672–683.
[82] I. Gühring, G. Kutyniok, and P. Petersen, Error bounds for approximations with deep ReLU neural networks in W s,p norms, Anal. Appl., 18 (2020),
pp. 803–859.
[83] K. Habermann, A semicircle law and decorrelation phenomena for iterated
Kolmogorov loops, J. London Math. Soc., (2019).
[84] W. Hackbusch, Hierarchical Matrices: Algorithms and Analysis, Springer,
2015.
[85] E. Haghighat, M. Raissi, A. Moure, H. Gomez, and R. Juanes, A
physics-informed deep learning framework for inversion and surrogate modelingin solid mechanics, Comput. Methods Appl. Mech. Eng., 379 (2021).
[86] N. Halko, P.-G. Martinsson, and J. A. Tropp, Finding structure withrandomness: Probabilistic algorithms for constructing approximate matrix decompositions, SIAM Rev., 53 (2011), pp. 217–288.
[87] K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in Proc. IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1026–1034.
[88] C. F. Higham and D. J. Higham, Deep learning: An introduction for appliedmathematicians, SIAM Rev., 61 (2019), pp. 860–891.
[89] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly,
A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al., Deepneural networks for acoustic modeling in speech recognition: The shared viewsof four research groups, IEEE Signal Process. Mag., 29 (2012), pp. 82–97.

140

[90] S. Hochreiter and J. Schmidhuber, Long short-term memory, NeuralComput., 9 (1997), pp. 1735–1780.
[91] S. Hofmann and S. Kim, The Green function estimates for strongly elliptic
systems of second order, Manuscripta Math., 124 (2007), pp. 139–172.
[92] H. Hotelling, Analysis of a complex of statistical variables into principal
components., J. Educ. Psychol., 24 (1933), p. 417.
[93] T. Hsing and R. Eubank, Theoretical foundations of functional data analysis, with an introduction to linear operators, John Wiley & Sons, 2015.
[94] S. Hwang and S. Kim, Green’s function for second order elliptic equations in
non-divergence form, Potential Anal., 52 (2020), pp. 27–39.
[95] S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network
Training by Reducing Internal Covariate Shift, in Proc. 32nd International Conference on Machine Learning (ICML), 2015, pp. 448–456.
[96] A. D. Jagtap, K. Kawaguchi, and G. E. Karniadakis, Adaptive activation functions accelerate convergence in deep and physics-informed neuralnetworks, J. Comput. Phys., 404 (2020).
[97] K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, What is thebest multi-stage architecture for object recognition?, in Proc. IEEE International
Conference on Computer Vision (ICCV), 2009, pp. 2146–2153.
[98] J.-P. Kahane, Propriétés locales des fonctions à séries de Fourier aléatoires,Stud. Math., 19 (1960), pp. 1–25.
[99] M. Kanagawa, P. Hennig, D. Sejdinovic, and B. K. Sriperumbudur,
Gaussian processes and kernel methods: A review on connections and equivalences, arXiv preprint arXiv:1807.02582, (2018).
[100] K. Kang and S. Kim, Global pointwise estimates for Green’s matrix of second
order elliptic systems, J. Differ. Equ., 249 (2010), pp. 2643–2662.
[101] K. Karhunen, Über lineare methoden in der wahrscheinlichkeitsrechnung,Ann. Acad. Science Fenn., Ser. A. I., 37 (1946), pp. 3–79.

141

[102] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang,
and L. Yang, Physics-informed machine learning, Nat. Rev. Phys., 3 (2021),
pp. 422–440.
[103] T. Karras, S. Laine, and T. Aila, A style-based generator architecture forgenerative adversarial networks, in Conference on Computer Vision and Pattern
Recognition (CVPR), 2019, pp. 4401–4410.
[104] T. Kato, Perturbation Theory for Linear Operators, Springer Science & Business Media, 2013.
[105] C. T. Kelley, Iterative Methods for Linear and Nonlinear Equations, SIAM,
1995.
[106] S. Kim and G. Sakellaris, Green’s function for second order elliptic equations with singular lower order coefficients, Commun. Partial. Differ. Equ., 44(2019), pp. 228–270.
[107] S. Kim and L. Xu, Green’s function for second order parabolic equations withsingular lower order coefficients, Commun. Pure Appl. Anal., 21 (2022), pp. 1–
21.
[108] D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, inInternational Conference on Learning Representations (ICLR), 2015.
[109] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, SelfNormalizing Neural Networks, in Advances in Neural Information Processing
Systems (NeurIPS), 2017, pp. 971–980.
[110] B. O. Koopman, Hamiltonian systems and transformation in Hilbert space,
Proc. Natl. Acad. Sci.s, 17 (1931), pp. 315–318.
[111] N. Kovachki, S. Lanthaler, and S. Mishra, On universal approximationand error bounds for Fourier Neural Operators, J. Mach. Learn. Res., 22 (2021),
pp. 1–76.
[112] N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya,
A. Stuart, and A. Anandkumar, Neural operator: Learning maps between
function spaces, arXiv preprint arXiv:2108.08481, (2021).
[113] E. Kreyszig, Introductory Functional Analysis with Applications, Wiley, 1978.
142

[114] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ImageNet Classification
with Deep Convolutional Neural Networks, in Advances in Neural Information
Processing Systems (NeurIPS), 2012, pp. 1097–1105.
[115] H. W. Kuhn and A. W. Tucker, Nonlinear Programming, in Proc. Second Berkeley Symp. on Math. Statist. and Prob., Univ. of Calif. Press, 1951,
pp. 481–492.
[116] S. Lanthaler, S. Mishra, and G. E. Karniadakis, Error estimates for
DeepONets: A deep learning framework in infinite dimensions, Trans. Math.
Appl., 6 (2022).
[117] V. I. Lebedev, On a Zolotarev problem in the method of alternating directions,
USSR Comp. Math. Math+, 17 (1977), pp. 58–76.
[118] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature, 521 (2015),
pp. 436–444.
[119] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel, Handwritten digit recognition with a back-propagation
network, Advances in Neural Information Processing Systems (NeurIPS), 2(1989).
[120] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-basedlearning applied to document recognition, Proc. IEEE, 86 (1998), pp. 2278–2324.
[121] M. Ledoux, The concentration of measure phenomenon, Math. Surveys.
Monog. 89, AMS, Providence, RI, 2001.
[122] J.-Y. Lee and L. Greengard, A fast adaptive numerical method for stiff
two-point boundary value problems, SIAM J. Sci. Comput., 18 (1997), pp. 403–
429.
[123] F. C. Leone, L. S. Nelson, and R. B. Nottingham, The folded normal
distribution, Technometrics, 3 (1961), pp. 543–550.
[124] D. Li, K. Xu, J. M. Harris, and E. Darve, Coupled time-lapse fullwaveform inversion for subsurface flow problems using intrusive automatic differentiation, Water Resour. Res., 56 (2020).

143

[125] M. Li, T. Zhang, Y. Chen, and A. J. Smola, Efficient mini-batch training
for stochastic optimization, in Proc. 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014, pp. 661–670.
[126] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya,
A. Stuart, and A. Anandkumar, Neural operator: Graph kernel network
for partial differential equations, arXiv preprint arXiv:2003.03485, (2020).
[127], Fourier neural operator for parametric partial differential equations, inInternational Conference on Learning Representations (ICLR), 2021.

[128] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, A. Stuart, K. Bhattacharya, and A. Anandkumar, Multipole graph neural operator for parametric partial differential equations, in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 6755–6766.
[129] S. Liang and R. Srikant, Why Deep Neural Networks for Function Approximation?, in International Conference on Learning Representations (ICLR),
2017.
[130] L. Lin, J. Lu, and L. Ying, Fast construction of hierarchical matrix representation from matrix–vector multiplication, J. Comput. Phys., 230 (2011),
pp. 4071–4087.
[131] F. Lindgren, H. Rue, and J. Lindström, An explicit link between Gaussian
fields and Gaussian Markov random fields: the stochastic partial differential
equation approach, J. R. Stat. Soc. B, 73 (2011), pp. 423–498.
[132] D. C. Liu and J. Nocedal, On the limited memory BFGS method for large
scale optimization, Math. Program., 45 (1989), pp. 503–528.
[133] M. Loève, Fonctions aleatoire de second ordre, Rev. Sci., 84 (1946), pp. 195–
206.
[134] Z. Long, Y. Lu, X. Ma, and B. Dong, PDE-NET: Learning PDEs from
data, in International Conference on Machine Learning (ICML), 2018, pp. 3208–
3216.
[135] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis, Learning
nonlinear operators via DeepONet based on the universal approximation theoremof operators, Nat. Mach. Intell., 3 (2021), pp. 218–229.
144

[136] L. Lu, X. Meng, Z. Mao, and G. E. Karniadakis, DeepXDE: A
deep learning library for solving differential equations, SIAM Rev., 63 (2021),
pp. 208–228.
[137] T.-T. Lu and S.-H. Shiou, Inverses of 2× 2 block matrices, Comput. Math.
Appl., 43 (2002), pp. 119–129.
[138] J. Ma, R. P. Sheridan, A. Liaw, G. E. Dahl, and V. Svetnik, Deep neural nets as a method for quantitative structure–activity relationships, J. Chem.
Inf. Model., 55 (2015), pp. 263–274.
[139] L. Ma and K. Khorasani, Constructive feedforward neural networks using Hermite polynomial activation functions, IEEE Trans. Neural Networ., 16(2005), pp. 821–833.
[140] A. L. Maas, A. Y. Hannun, and A. Y. Ng, Rectifier Nonlinearities Improve Neural Network Acoustic Models, in International Conference on Machine
Learning (ICML), 2013.
[141] S. Maddu, B. L. Cheeseman, I. F. Sbalzarini, and C. L. Müller,
Stability selection enables robust learning of differential equations from limited
noisy data, Proc. R. Soc. A, 478 (2022).
[142] A. A. Markov, On a question by D. I. Mendeleev, Zapiski Imp. Akad. Nauk,
62 (1889), pp. 1–24.
[143] P.-G. Martinsson, A fast randomized algorithm for computing a hierarchically semiseparable representation of a matrix, SIAM J. Matrix Anal. Appl., 32(2011), pp. 1251–1274.
[144], Compressing rank-structured matrices via randomized sampling, SIAM J.
Sci. Comput., 38 (2016), pp. A1959–A1986.

[145] P.-G. Martinsson and J. A. Tropp, Randomized numerical linear algebra:
Foundations and algorithms, Acta Numer., 29 (2020), p. 403–572.
[146] X. Meng, Z. Li, D. Zhang, and G. E. Karniadakis, PPINN: Parareal
physics-informed neural network for time-dependent PDEs, Comput. Methods
Appl. Mech. Eng., 370 (2020).

145

[147] X. Meng and M. W. Mahoney, Low-distortion subspace embeddings in
input-sparsity time and applications to robust linear regression, in Proceedingsof the Forty-Fifth Annual ACM Symposium on Theory of Computing, 2013,
pp. 91–100.
[148] J. Mercer, Functions of positive and negative type, and their connection thetheory of integral equations, Philos. T. R. Soc. A, 209 (1909), pp. 415–446.
[149] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, Unrolled Generative Adversarial Networks, in International Conference on Learning Representations (ICLR), 2017.
[150] H. N. Mhaskar, Neural Networks for Optimal Approximation of Smooth andAnalytic Functions , Neural Comput., 8 (1996), pp. 164–177.
[151] L. Mirsky, Symmetric gauge functions and unitarily invariant norms, Q. J.
Math., 11 (1960), pp. 50–59.
[152], A trace inequality of John von Neumann, Monatsh. Math., 79 (1975),
pp. 303–306.

[153] A. Molina, P. Schramowski, and K. Kersting, Padé Activation Units:
End-to-end Learning of Flexible Activation Functions in Deep Networks, in International Conference on Learning Representations (ICLR), 2019.
[154] H. Montanelli, H. Yang, and Q. Du, Deep ReLU networks overcome
the curse of dimensionality for generalized bandlimited functions, J. Comput.
Math., 39 (2021), pp. 801–815.
[155] A. M. Mood, F. A. Graybill, and D. C. Boes, Introduction to the Theoryof Statistics, McGraw-Hill, 3rd ed., 1974.
[156] R. J. Muirhead, Aspects of multivariate statistical theory, John Wiley & Sons,
2009.
[157] T. Myint-U and L. Debnath, Linear Partial Differential Equations for Scientists and Engineers, Birkhäuser Basel, 2007.
[158] V. Nair and G. E. Hinton, Rectified Linear Units Improve Restricted Boltzmann Machines, in Proc. 27th International Conference on Machine Learning(ICML), 2010, pp. 807–814.
146

[159] Y. Nakatsukasa, Fast and stable randomized low-rank matrix approximation,
arXiv preprint arXiv:2009.11392, (2020).
[160] Y. Nakatsukasa and R. W. Freund, Computing fundamental matrix decompositions accurately via the matrix sign function in two iterations: The
power of Zolotarev’s functions, SIAM Rev., 58 (2016), pp. 461–493.
[161] N. H. Nelsen and A. M. Stuart, The random feature model for input-output
maps between Banach spaces, SIAM J. Sci. Comput., 43 (2021), pp. A3212–
A3243.
[162] J. Nelson and H. L. Nguyên, OSNAP: Faster Numerical Linear Algebra
Algorithms via Sparser Subspace Embeddings, in IEEE 54th Annual Symposium
on Foundations of Computer Science, 2013, pp. 117–126.
[163] D. J. Newman, Rational approximation to |x|, Mich. Math. J., 11 (1964),
pp. 11–14.

[164] E. J. Nyström, Über die praktische Auflösung von integralgleichungen mit
Anwendungen auf randwertaufgaben, Acta Math., 54 (1930), pp. 185–204.
[165] A. Odena, C. Olah, and J. Shlens, Conditional image synthesis with auxiliary classifier GANs, in International Conference on Machine Learning (ICML),
vol. 70, 2017, pp. 2642–2651.
[166] F. W. J. Olver, D. W. Lozier, R. F. Boisvert, and C. W. Clark,
NIST Handbook of Mathematical Functions , Cambridge University Press, 2010.
[167] P. J. Olver, Applications of Lie groups to differential equations, SpringerVerlag, 2nd ed., 1993.
[168] R. Pachón and L. N. Trefethen, Barycentric-Remez algorithms for best
polynomial approximation in the chebfun system, BIT, 49 (2009), pp. 721–741.
[169] G. Pang, L. Lu, and G. E. Karniadakis, fPINNs: Fractional physicsinformed neural networks, SIAM J. Sci. Comput., 41 (2019), pp. A2603–A2626.
[170] G. Pang, L. Yang, and G. E. Karniadakis, Neural-net-induced Gaussian
process regression for function approximation and PDE solution, J. Comput.
Phys., 384 (2019), pp. 270–288.

147

[171] L. Pareschi and G. Russo, Implicit–explicit runge–kutta schemes and applications to hyperbolic systems with relaxation, J. Sci. Comput., 25 (2005),
pp. 129–155.
[172] D. S. Parker, Random Butterfly Transformations with Applications in Computational Linear Algebra, Tech. Rep. CSD-950023, UCLA, 1995.
[173] A. Paterek, Improving regularized singular value decomposition for collaborative filtering, in Proc. KDD cup and workshop, 2007, pp. 5–8.
[174] K. Pearson, On lines and planes of closest fit to systems of points in space,
Lond .Edinb. Phil. Mag., 2 (1901), pp. 559–572.
[175] P. Petersen and F. Voigtlaender, Optimal approximation of piecewisesmooth functions using deep ReLU neural networks, Neural Netw., 108 (2018),
pp. 296–330.
[176] P. P. Petrushev and V. A. Popov, Rational Approximation of Real Functions, Cambridge University Press, 2011.
[177] E. Qian, I.-G. Farcas, and K. Willcox, Reduced operator inference
for nonlinear partial differential equations, SIAM J. Sci. Comput., 44 (2022),
pp. A1934–A1959.
[178] E. Qian, B. Kramer, B. Peherstorfer, and K. Willcox, Lift & learn:
Physics-informed machine learning for large-scale nonlinear dynamical systems,
Phys. D, 406 (2020).
[179] A. Radford, L. Metz, and S. Chintala, Unsupervised Representation
Learning with Deep Convolutional Generative Adversarial Networks, in International Conference on Learning Representations (ICLR), 2016.
[180] M. Raissi, Deep hidden physics models: Deep learning of nonlinear partial
differential equations, J. Mach. Learn. Res., 19 (2018), pp. 932–955.
[181], GitHub repository.

https://github.com/maziarraissi/DeepHPMs/,

2020.
[182] M. Raissi and G. E. Karniadakis, Hidden physics models: Machine learning of nonlinear partial differential equations, J. Comput. Phys., 357 (2018),
pp. 125–141.
148

[183] M. Raissi, P. Perdikaris, and G. E. Karniadakis, Multistep neural networks for data-driven discovery of nonlinear dynamical systems, arXiv preprint
arXiv:1801.01236, (2018).
[184] M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physics-informed
neural networks: A deep learning framework for solving forward and inverse
problems involving nonlinear partial differential equations, J. Comput. Phys.,
378 (2019), pp. 686–707.
[185] M. Raissi, A. Yazdani, and G. E. Karniadakis, Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations, Science, 367(2020), pp. 1026–1030.
[186] P. Ramachandran, B. Zoph, and Q. V. Le, Searching for Activation
Functions, arXiv preprint arXiv:1710.05941, (2017).
[187] C. E. Rasmussen and C. Williams, Gaussian processes for machine learning, MIT Press, 2006.
[188] F. Rathgeber, D. A. Ham, L. Mitchell, M. Lange, F. Luporini, A. T.
McRae, G.-T. Bercea, G. R. Markall, and P. H. Kelly, Firedrake:
automating the finite element method by composing abstractions, ACM Trans.
Math. Softw., 43 (2016), pp. 1–27.
[189] J. Rissanen, A universal prior for integers and estimation by minimum description length, Ann. Stat., 11 (1983), pp. 416–431.
[190] G. F. Roach, Green’s Functions, Cambridge University Press, 2nd ed., 1982.
[191] V. Rokhlin, A. Szlam, and M. Tygert, A randomized algorithm for principal component analysis, SIAM J. Matrix Anal. Appl., 31 (2010), pp. 1100–
1124.
[192] S. Roweis, EM algorithms for PCA and SPCA, Advances in Neural Information Processing Systems (NeurIPS), 10 (1997), pp. 626–632.
[193] W. Rudin, Principles of mathematical analysis, International series in pureand applied mathematics, McGraw-Hill, 3rd ed., 1976.
[194], Real and complex analysis, McGraw-Hill, 3rd ed., 1986.

149

[195] S. H. Rudy, S. L. Brunton, J. L. Proctor, and J. N. Kutz, Data-driven
discovery of partial differential equations, Sci. Adv., 3 (2017).
[196] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning representations by back-propagating errors, Nature, 323 (1986), pp. 533–536.
[197] H. Schaeffer, Learning partial differential equations via data discovery andsparse optimization, Proc. Math. Phys. Eng. Sci., 473 (2017).
[198] F. Schäfer and H. Owhadi, Sparse recovery of elliptic solvers from matrixvector products, arXiv preprint arXiv:2110.05351, (2021).
[199] F. Schäfer, T. J. Sullivan, and H. Owhadi, Compression, inversion,and approximate PCA of dense kernel matrices at near-linear computational
complexity, Multiscale Model. Sim., 19 (2021), pp. 688–730.
[200] E. Schmidt, Zur Theorie der linearen und nicht linearen Integralgleichungen
Zweite Abhandlung, Math. Ann, 64 (1907), pp. 161–174.
[201] M. Schmidt and H. Lipson, Distilling free-form natural laws from experimental data, Science, 324 (2009), pp. 81–85.
[202] K. Shukla, P. C. Di Leoni, J. Blackshire, D. Sparkman, and G. E.
Karniadakis, Physics-informed neural network for ultrasound nondestructive
quantification of surface breaking cracks, J. Nondestruct. Eval., 39 (2020), pp. 1–
20.
[203] J. Sirignano and K. Spiliopoulos, DGM: A deep learning algorithm forsolving partial differential equations, J. Comput. Phys., 375 (2018), pp. 1339–
1364.
[204] L. N. Smith, Cyclical learning rates for training neural networks, in IEEE
Winter Conference on Applications of Computer Vision, 2017, pp. 464–472.
[205] H. Stahl, Best uniform rational approximation of |x| on [−1, 1], Mat. Sb., 183(1992), pp. 85–118.
[206], Best uniform rational approximation of xα on [0, 1], Bull. Amer. Math.
Soc., 28 (1993), pp. 116–122.

[207] I. Stakgold and M. J. Holst, Green’s Functions and Boundary Value Problems, John Wiley & Sons, 2011.
150

[208] G. W. Stewart, Matrix Algorithms: Volume 1: Basic Decompositions, SIAM,
1998.
[209] A. M. Stuart, Inverse problems: a Bayesian perspective, Acta Numer., 19(2010), pp. 451–559.
[210] E. Süli and D. F. Mayers, An Introduction to Numerical Analysis, Cambridge University Press, 2003.
[211] G. Szego, Orthogonal polynomials, AMS, Providence, RI, 4th ed., 1939.
[212] C. Taylor and P. Hood, A numerical solution of the Navier-Stokes equations
using the finite element technique, Comput. Fluids, 1 (1973), pp. 73–100.
[213] M. Telgarsky, Benefits of depth in neural networks, in Conference on Learning Theory (COLT), 2016, pp. 1517–1539.
[214], Neural networks and rational functions, in International Conference on
Machine Learning (ICML), vol. 70, 2017, pp. 3387–3393.

[215] A. Townsend, Pretty functions approximated by Chebfun2. https://www.
chebfun.org/examples/approx2/PrettyFunctions.html, 2013.
[216], Computing with functions in two dimensions, PhD thesis, University of
Oxford, 2014.

[217] A. Townsend and L. N. Trefethen, An extension of Chebfun to two dimensions, SIAM J. Sci. Comput., 35 (2013), pp. C495–C518.
[218], Continuous analogues of matrix factorizations, P. Roy. Soc. A, 471 (2015).

[219] L. N. Trefethen, Spectral Methods in MATLAB, SIAM, 2000.
[220], Approximation Theory and Approximation Practice, Extended Edition,
SIAM, 2019.

[221] L. N. Trefethen and D. Bau III, Numerical linear algebra, SIAM, 1997.
[222] L. N. Trefethen, A. Birkisson, and T. A. Driscoll, Exploring ODEs,
SIAM, 2017.

151

[223] L. N. Trefethen, Y. Nakatsukasa, and J. Weideman, Exponential nodeclustering at singularities for rational approximation, quadrature, and PDEs,
Numer. Math., 147 (2021), pp. 227–254.
[224] M.-L. Udrescu and M. Tegmark, AI Feynman: A physics-inspired method
for symbolic regression, Sci. Adv., 6 (2020).
[225] S.-M. Udrescu, A. Tan, J. Feng, O. Neto, T. Wu, and M. Tegmark,AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity, in Advances in Neural Information Processing Systems (NeurIPS), vol. 33,
2020, pp. 4860–4871.
[226] Y. Urano, A fast randomized algorithm for linear least-squares regression via
sparse transforms, Master’s thesis, New York University, 2013.
[227] A.-J. Van Der Veen, E. F. Deprettere, and A. L. Swindlehurst,
Subspace-based signal analysis using singular value decomposition, Proc. IEEE,
81 (1993), pp. 1277–1308.
[228] L. Vecci, F. Piazza, and A. Uncini, Learning and approximation capabilities of adaptive spline activation function neural networks, Neural Netw., 11(1998), pp. 259–270.
[229] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy,
D. Cournapeau,

E. Burovski,

P. Peterson,

W. Weckesser,

J. Bright, et al., SciPy 1.0: fundamental algorithms for scientific computing
in Python, Nat. Methods, 17 (2020), pp. 261–272.
[230] J. von Neumann, Some matrix-inequalities and metrization of matrix-space,
Tomsk Univ. Rev., 1 (1937), pp. 286–300.
[231] H. U. Voss, J. Timmer, and J. Kurths, Nonlinear dynamical system identification from uncertain and indirect measurements, Int. J. Bifurc. Chaos Appl.
Sci. Eng., 14 (2004), pp. 1905–1933.
[232] N. S. Vyacheslavov, On the uniform approximation of |x| by rational functions., Sov. Math. Dokl., 16 (1975), pp. 100–104.
[233] S. Wang, H. Wang, and P. Perdikaris, Learning the solution operator ofparametric partial differential equations with physics-informed DeepONets, Sci.
Adv., 7 (2021).
152

[234] Z. Wang, X. Huan, and K. Garikipati, Variational system identificationof the partial differential equations governing the physics of pattern-formation:
inference under varying fidelity and noise, Comput. Methods Appl. Mech. Eng.,
356 (2019), pp. 44–74.
[235] E. Wegert, Visual Complex Functions: An Introduction with Phase Portraits,
Springer Science & Business Media, 2012.
[236] C. L. Wight and J. Zhao, Solving Allen-Cahn and Cahn-Hilliard Equations Using the Adaptive Physics Informed Neural Networks, Commun. Comput. Phys., 29 (2021), pp. 930–954.
[237] H. Wilber, A. Townsend, and G. B. Wright, Computing with functionsin spherical and polar geometries II. The disk, SIAM J. Sci. Comput., 39 (2017),
pp. C238–C262.
[238] C. Williams and M. Seeger, Using the Nyström method to speed up kernel
machines, in Advances in Neural Information Processing Systems (NeurIPS),
vol. 14, 2001, pp. 682–688.
[239] J. Wishart, The generalised product moment distribution in samples from anormal multivariate population, Biometrika, (1928), pp. 32–52.
[240] S. Wold, K. Esbensen, and P. Geladi, Principal component analysis,
Chemometr. Intell. Lab., 2 (1987), pp. 37–52.
[241] F. Woolfe, E. Liberty, V. Rokhlin, and M. Tygert, A fast randomized
algorithm for the approximation of matrices, Appl. Comput. Harmon. Anal., 25(2008), pp. 335–366.
[242] D. Yarotsky, Error bounds for approximations with deep ReLU networks,
Neural Netw., 94 (2017), pp. 103–114.
[243] A. Yazdani, L. Lu, M. Raissi, and G. E. Karniadakis, Systems biology
informed deep learning for inferring parameters and hidden dynamics, PLoS
Comput. Biol., 16 (2020).
[244] D. Zhang, L. Guo, and G. E. Karniadakis, Learning in modal space: Solving time-dependent stochastic PDEs using physics-informed neural networks,
SIAM J. Sci. Comput., 42 (2020), pp. A639–A665.

153

[245] J. Zhang and W. Ma, Data-driven discovery of governing equations for fluid
dynamics based on molecular simulation, J. Fluid Mech., 892 (2020).
[246] S. Zhang and G. Lin, Robust data-driven discovery of governing physicallaws with error bars, Proc. R. Soc. A, 474 (2018).
[247] H. Zhao, B. D. Storey, R. D. Braatz, and M. Z. Bazant, Learning
the physics of pattern formation from images, Phys. Rev. Lett., 124 (2020).

154