ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning
Authors: Nilo Schwencke
Cyril Furtlehner
Venue: ICLR 2025 Poster (2025)
Keywords: PINNs
SciML
PDEs
Natural Gradient
Neural Tangent Kernel
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a new natural gradient that scales as $\min(P^2S, S^2P)$, $P$ the number of parameters and $S$ the batch size, focusing on its application to PINNs traning.

Abstract: In the recent years, Physics Informed Neural Networks (PINNs) have received strong interest as a method to solve PDE driven systems, in particular for data assimilation purpose. This method is still in its infancy, with many shortcomings and failures that remain not properly understood.
In this paper we propose a natural gradient approach to PINNs which contributes to speed-up and improve the accuracy of the training.
Based on an  in depth analysis of the differential geometric structures of the problem, we come up with two distinct contributions:
(i) a new natural gradient algorithm that scales as $\min(P^2S, S^2P)$, where $P$ is the number of parameters, and $S$ the batch size;
(ii) a mathematically principled reformulation of the PINNs problem that allows the extension of natural gradient to it, with proved connections to Green's function theory.

Published as a conference paper at ICLR 2025
ANAGRAM: A NATURAL GRADIENT RELATIVE TO ADAPTED MODEL FOR EFFICIENT PINNS LEARNING
Nilo Schwencke
Tau team, INRIA-Saclay A&O team, LISN, Université Paris-Saclay 91190, Gif-sur-Yveter France
nilo. www.eeecccceeeerttomaalil..
Cyril Furtlehner
Tau team, INRIA-Saclat
A&O team, LISN, Université Paris-Saclay 91190, fnf-sur-Yvette, France cyril. wut..feeeneeernnriaaa fr
ABSTRACT
the recent years, Physics Informed Neural Networks (PINNs) have received strong interest method to solve PDE driven systems, in particular for data assimilation purpose. This method is still in its infancy, with many shortcomings and failures that remain not properly understood. In this paper we propose a natural gradient approach to PINNs which contributes to speed-up and improve the accuracy of the training. Based structures of the problem, we come up with two distinct contributions: (i) a new natural gradient algorithm that scales as min S2S, S²P). where / is the number of parameters, and S the batch size; (ii) a mathematically principled reformulation of the PINNs problem that allows the extension of natural gradient to with proved connections to Green's 's function theory.
in depth analysis of the differential geometric
INTRODUCTION
Following the spectacular success of neural networks for over a decade (LeCun et al. 2015), intensive work has been carried out to apply these methods to numerical analysis (Cuomo et al., 2022).
In particular, following the pioneering work of Dissanayake and Thinn (1994) and Lagaris
et al. (1998), Raissi et al. (2019)) have introduced Physics Informed Neural Networks (PINNs), a method designed to approximate solutions of partial differential equations (PDEs), using deep neural networks. Theoretically based the universal approximation theorem of neural networks (Leshno et al., 1993), and put into practice by automatic differentiation (Baydin et al.) (018) for the computation of differential operators, this method has enjoyed number of successes in fields as diverse as fluid mechanics (Raissi et al., 2019b C,,Sun et al.] 2020;; Raissi et al.) [2020] Jin et al.
2021: de Wolff al. (20TT), bio-engineering (Sahli Costabal et al.) (2000] Kissas et al.) 2020) or free boundary problems (Wang and Perdikaris 2021). Nevertheless, many limitations have been pointed out, notably the inability of these methods in their current formulation to obtain high-precision approximations when no additional data is provided (Krishnapriyan et al.) 2021 Wang al. 2021
Karnakov et al. 2022 Zeng et al. (022)). Recent work by Müller and Zeinhofer (2023), however, has substantially altered this state of affairs, proposing algorithm similar to natural gradient methods in case of linear operator (cf. Section that achieves accuracies several orders of magnitude above previous methods.
Contributions: Müller and Zeinhofer (2024) argue for the need to take function-space geometry into account in order to further understand and improve scientific machine- -eaaiinelearning methods. With this paper, intend to support and extend their approach by making several contributions:

---

Published as a conference paper at ICLR 2025
(i) We highlight principled mathematical framework that restates natural gradient in equivalent, yet simpler way, leading us to propose ANaGRAM, a purprsse natural gradient algorithm of reduced complexity Ommin(P²S, #porameters and
compared to O where
#batch samples.
(ii We reinterpret the PINNs framework from functional analysis perspective in order to extend ANaGRAM to the PINN's context in straightforward manner.
(ii) We establish a direct correspondence between ANaGRAM for PINNs and the Green's function of the operator the tangent space.
The rest of this article is organized as follows:
in Section after introducing neural networks and parametric models in Section 2.1 from functional analysis perspective, review two concepts crucial to our work: PINNs framework in Section 2.2 and natural gradient in Section 2.3 In Section 3 introduce the notions of empirical tangent space and expression for the corresponding notion of empirical natural gradient leading to ANaGRAM, described in algorithm [ reinterpreting PINNs as regression problem from the right functional perspective in Section yielding ANaGRAM algorithm 2 for PINNs, we state in Section 4.2 that natural gradient matches the Green's function of the operator on the tangent space and analyse the consequence of this on the interpretation of PINNs training process under ANaGRAM. Finally, in Section [ we show empirical evidences of the performance ANaGRAM on a selected benchmark of PDES.
In Section after
POSITION OF THE PROBLEM
2.1
NEURAL NETWORKS AND PARAMETRIC MODEL
Our starting point is the following functional definition of parametric models, of which neural networks are non-linear special case:
Definition 1 (Parametric model). Given a domain P of R", K (R, and a Hilbert space H consisting of functions 52 Km parametric model is differentiable functional:
RRP - H 0
(1)
(ar E f u(x;
To prevent confusion, we will write u instead of u( 0). for all Ω
Since a parametric model is differentiable by definition, we can define its differential:
Definition 2 (Differential of parametric model). Let IRP H be parametric model and e R².. Then the differential of the parametric model in the parameter 0 is:
RP
H
duje
- >
hp
au p20
To simplify notations, we will write for all
& p & P and for all f RP, up instead du "0,-
Given a parametric model u, we can define the following two objects of interest:
The image set of this the set of functions reached by u, i.e.
M = Im u := {u O RR* c H.
(3)
Although not strictly rigorous M is often considered in deep-learning as a differential
submanifold of H. so we will keep this analogy in mind for pedagogical purposes.
The tangent space at this is the image set of the differential at 0, i.e. the linear subspace of H. consisting of the functions reached by duig. i.e.
TeM = Im dujø = Span (2p40 1 H. Once again, this definition is made with reference to differential geometry.
(4)
We give several examples of Parametric models in Section
We now introduce PINNs.
In particular, because u may not be injective.
2

---

Published as a conference paper at ICLR 2025
2.2 PHYSICS INFORMED NEURAL NETWORKS (PINNS)
As in Definition 1 let us consider domain Ω of R" endowed with a probability measure E (R. Cj. M its boundary endowed with probability measure o, and H. Hilbert space composed of functions 92 - km Then let us consider two functional (inter alia differential) operators:
D { H - 1.2 14
(,2 R) -B]]
B
D
that we will assume to be differentiable alia PDE):
We can then consider the functional equation (inter
f R, B(u) = (7n R,o) on 2
in
The PINNs framework, as introduced by Raissi et al. (2019a) consists then in approximating a solution to the PDE by making the ansatz ujo. with ui neural network, sampling points (x.) Isses, in 52 according to μ. (a Jsses in 2s according to a and then to optimize the loss:
1 Sn
1 2SB
((0)
p (Duuolaa)--((et -
Σ (B(uol(z")-ghee
by classical gradient descent techniques, used in the context of deep learning, such Adam (Kingma lnd Ba (014)) or L-BFGS (Liu and Nocedal 1989). One of the cornrrtonns of Raissi et al. (201 9a) also use automatic differentiation (Baydin al. 2018 compute the operators D and B, thus obtaining quasi-exact calculations, whereas most classic techniques require either approximating operators with Finite Differences, or carrying out the calculations manually with Finite Elements.
Although appealing due to its simplicity and relative ease of implementation, this approach suffers from several well-documented empirical pathologies (Krishnapriyan al. [021] Wang al. I022] Grossmann et al. 2024). which can be understood as an ill conditioned problem (De Ryck et al.
2024 Liu et al.] 2024) and for which several ad hoc procedures has been proposed (Karnakov et al. 2022 Zeng et al. 2022; McClenny and Braga-Neto (022). Following Müller and Zeinhofer (2024).
we argue this work that the key point rather to theoretically understand the geometry of the problem and adapt PINNs training accordingly.
2.3 NATURAL GRADIENT
Natural gradient has been introduced, in the context of Information Geometry by Amari and Doug las 1998). Given a loss: f 0 - IR+, the gradient descent:
oc+1 O. 17 VL,
(8)
replaced by the update:
0++1 < O, - Fd ve,
with Fe, being the Gram-Matrix associated to a Fisher-Rao information metric (Amari 2016 or equivalently, the Hessian of some Kullback-Leiller divergence (Kullback and Leibler 1951) and the Moore-Penrooe Penrose pseudo-inverse. This notion has been later further extended to the more abstract setting of Riemannian metrics in the context of nettol-s by ( In this case, given a Riemannian- (pseudo) metric Ge. the gradient-descent update is replaced by:
0.+1 < 0, GnG Ve,
(10)
where Go.p.g == Go. (0, d,. is the Gram matrix of partial derivatives relative Go.- Despite its mathemattcally principled advantage, natural gradient suffers from its computational cost, which makes it prohibitive, if not untractable for real world applications. Indeed:
Computation of the Gram matrix Ge, is at least quadratic in the number of parameters.
. Inversion of Go, is cubic in the number of parameters.
It can be shown that, if D) and B are defined and differentiable on C - [K")) then such H always exists; of chapter 12 of Berezansky al. (996).
3

---

Published as a conference paper at ICLR 2025
Different approaches have been proposed to circumvent this limitations. The most prominent one is K-FAC introduced by Heskes (2000) and further extended by Martens and Grosse (2015); Grosse and Martens (016). which approximates the Gram matrix by block-diagonal matrices. This approximation can be understood as making the ansatz that the partial derivatives of weights belonging to different layers
orthogonal. A refinement of this method has been proposed by George et al. (2018), in which the eigen-structure of the block- -oick-aaagoaal matrices are carefully taken into account in order to provide a better approximation of the diagonal rescaling induced by the inversion of the Gram matrix. This method has been extented to PINNs by Dangel et al. (2025). In completely different vein, Ollivier (2017) has proposed a statistical approach that has been proved to converge to the natural gradient update in the learning rate limit.
To conclude this section, let us give more geometric interpretation of natural gradient. To this end, let us consider the standard quadratic regression problem:
(()) = 2S Σ (mp(a;) fx=)))²²
(11)
with dio a parametric model, for instance a neural-network, (1:)) sampled from some probability measure on some domain Ω of RN, In the limit S - (population limit), this loss can be reinterpreted as the evaluation at ui the functional loss:
C E (22 14)
2 1u リー fliaaa..
(12)
Taking the Fréchet derivative, one gets: for all v. 1 (52,
dClo(h) = (v- f ())
(13)
Lee the functional gradient of L is VL :- :- As noted for instance in Verbockhaven er al. (2024). Natural gradient has then to be interpreted from the functional point view as the projection of V Luuo onto the tangent space T&M with respect the L'( μ1) metric. However, this functional update must be converted into a parameter space update. Since the parameter space RP somehow identified with T via the differential application du o. it would be sufficient to take the inverse of this application to obtain the parametric update. In general d is not invertible but at least it admits pseudo-inverse dul g. Moreover, since T Im duj by definition, duje defined on all ToM Thus, we have that the natural gradient the population limits corresponds the update:
r.+1 0. - du'p
(Itia. (VL(mee))
(14)
Note that the use of the pseudo-inverse implies that the update in the parameter space happens in the subspace (Ker ) c R..
3 EMPIRICAL NATURAL GRADIENT AND ANAGRAM
In practice, one cannot reach the population limit and thus Equation is only an asymptotic update. Nevertheless,
derive a more accurate update, when (I:){=1 that is usually called a batch. Following Jacot et al. 20118) we know that standard quadratic gradient descent update with respect batch in the vanishing learning rate limit 7j 0, rewrites in the functional space as:
rely only finite set of points
dujo (x) dt
Σ (1, (u,) v.). NTKo(z,)) = Σ (Q)))(1)) (o)))(
(15)
Furthermore, Rudner et al. 2019) y Bai et al. (022) show that under natural gradient descent, the Neural Tangent Kernel (NTK) should be replaced Equation (1) by the Natural NTK (NNTK):
N () = Σ (O)u)(z)) Gbpa (2)4)())) Gep. == (oou1o ),40)n (16)
EEPgER
4

---

Published as a conference paper at ICLR 2025
As consequence, may that the update under natural gradient descent with respect to a batch (r:) happens a subspace of the tangent space, namely the empirical Tangent Space:
KNNTK M Span(NNTKo(s
(sie)) = ToM.
(17)
Subsequently, Equation can then adapted to define the empirical Natural Gradient update:
0++1 - 0: - dd -0-7du
[IĀNNTK M (V (VLiup))
(18)
0.(0))
Note that this update can be understood from the functional perspective the standard Nyström method (Sun et al. 2015), bridging the gap between our work and the many methods developed K o kernel cannot be computed explicitly our case, since requires priori inverting the Gram matrix, which adds further challenge. With this in mind, present a first result, encapsulated in the following theorem, which is one our main contributions:
in this field. Nevertheless, the NNTK'
Theorem (ANaGRAM). Let us define for all 1 i for all p
@os.p ( dulo
Viyuee Viiu (2i) vil( - f(x;).
Then:
II
M V (.(r)))
(o) Emmetric)
V + E
(19)
where Eg and Eg are correction terms specified Equations (57 and (58 in Section C.3 respec- tively accounting for the metric's 's impact on empirical tangent space defintion, and the substraction of the evaluation of the orthogonal par of the functional gradient.
proof of this theorem, well more comprehensive introduction to empirical natural gradient, encompassing a détour through RKHS theory, can be found in Section C
Remark 1. In some important cases the correction terms Fg E-metric and Fg vanishes. This happens for instance for Eg when solving D D[u]
with D linear and MLP (see Section B.2 We refer to
Proposition
2 and Remark at the end of Section C.3 Expetric cancels out in the following case:
Proposition There exist points (ix) such that NNTT M TgM. Then notably Emgetric (2.).)
As first approximation, can neglect those two terms, yielding the following vanilla algorithm: Algorithm 1: vanilla ANaGRAM
Input: u RRP (1, // neural network architecture
OD // initialization of the neural network
(E, // target function of the quadratic regression (x // batch in ?
e 0>0 // cutoff level to compute the pseudo inverse
repeat
S0. (ōpu1 (Ii) 1518 // Computed via futorniffeeentiiee
Cn. So, Do. SVD (
So, (Do.p if Do.p > c celse 0) /SpEr
VC - (u (I;) -f(xx)) lsiss
de. - Vo, Ag 0%. VL
arg min Σ (f(x.) u1o upo--ndo (zi))* // Using e.g. line search
nER+
1ggg
0 0, Ni do,
until stop criterion met
Note that algorithm equivalent Gauss-Newton algorithm (Cai et al. 2019) applied to the empirical loss in Equation (11) also considered recently in Jnini et al. (2024) with different setting.
orthogonal to the whole tangent space To.M.
5

---

Published as a conference paper at ICLR 2025
Nevertheless, work aims at general approach, giving rise different algorithms depending on the approximations of Egeric and Eg. One the pleasant byproducts the ANaGRAM framework is also that it leads to straightforward criterion to choose points in the batch, namely:
(xt = arg min l /N (VL) V (ar))es
(20)
which amenable to various approximations, subject to further investigations. Taking the best advantage of this criterion should eventually allow us to use natural gradient mini-batch setting while staying close to the convergence rate of the full batch natural gradient as characterized in Xu We will now show how ANaGRAM can be applied to the PINNs framework.
all 2024).
ANAGRAM FOR PINNS
Generalizing ANaGRAM PINNs only requires to change the problem perspective.
4.1 PINNS AS A LEAST-SQUARES REGRESSION PROBLEM
The only difference between the losses of Equation and Equation (11 is the use of the differential operator D and the boundary operator B in Equation (7). More precisely, PINNS and standard quadratic regression problems essentially similar, except that in the of PINNS compound model (D, B) u instead of u directly, where, using the definitions of Equation (
the
RP H
1.3 (2,, 05) 1.3(2 R. (2 IR, o) (D( B[uo])
(D, B) ou:
(21)
0
210
The derivation of vanilla ANaGRAM in PINNs
context then
straightforward:
Algorithm 2: vanilla ANaGRAM for PINNs
Input: u: - H // neural network architecture
0p // initialization of the neural network
D:H- - 1. R, // di differential operator
B:H 11: - R,o) // boundary operator
feL R, // source term gEL - o) // boundary value (x.)) // batch in Ω (x,)) ES // batch in M
e> // cutoff level to compute the pseudo inverse
repeat
2o.
(0, ((.3)) p-1 // vía autodiff
Ve. So. 0b. -VD())))
So.
Bo.r if Bo. e clsse 0) Igrgp
VC (Diujo,1(s0) [(x;) f(x?)) IgiESp
(B(yoo (I. g(x:3)
do. Vo. Ag. 0% VL
Σ (f(xP) Diuo, ndo, -nd, {(o)))
Ti arg min
25D
nER+
15i88p
Σ
B Bluje.-mue,lle -no, 1(0 11 Using e.g. line search
25
1g
0 0, Th. do.
until stop criterion met

---

Published as a conference paper at ICLR 2025
Adaptation of the definitions in Sections 2.3 a to the of PINNS are detailed in Sections C.4 and C.5 respectively. We now present the link between PINN's natural gradient and Green's 's function.
4.2 PINNS NATURAL GRADIENT IS A GREEN'S FUNCTION
Knowing the Green's 's function of linear operator is one of the most optimal ways of solving the associated PDE, since it then suffices to estimate integral to approximate a solution (Duffy 2015). However, this requires prior knowledge of the Green's 's function, which is not always possible. Here, show that using the natural gradient for PINNs implicitly uses the operator's 's Green's 's function. In Section D we briefly recall the main definitions required to state and prove the following theorem:
Theorem 2. Let D H - (22 - p.) be a linear differential operator and u : RRP - H a parametric model. Then for all 0 e RP, the generalized Green's function of D on ToM Im dujo is given by: for all x. y E
[To((E
Σ LEPREP
(x))(() (lu
(22)
with: for all SPQ.PP
Gpg := (0p 0g D(u (3(0
(23)
In particular, the natural gradient of PINNA can be rewritten:
[
0++1 - 0, 0.- d x x
91o, (1, Loo (v))(dd))
(24)
A few comments should be made about Equation understood as the least-square's solution of D[u] However, it does not hold a priori that:
24
First, if 1, then the natural gradient can be order 1, i.e. the affine space U10, + To, M.
D[uio + Me. correctly approximates / 1.3 p)).
u.e. + To.M correctly approximates the image space M 1uI 0 E RP).
Multiplying by a learning rate y & then essential. In this way, natural gradient can be understood in the affine space ujo, + M, and thus getting closer to the solution, while expecting that the change induced by this update will improve the approximation space u1 Te... M¹1 On the other hand, when we approach the end of the optimization, i.e. when the space D[ [[ + M) approximates / "well enough", while dujo. approximates "well enough" MM then it is in our best interest to solve the equation completely, i.e. take learning rates 1) close 1. This is why the use of line search in ANaGRAM (cf. line 6 in Algorithm 2 is essential. We should then conclude that the quality of the solution found by the parametric model u depends only on:
as moving in the direction of the solution of Dlu]
How well r
[D[uj]] 0 RP) can approximate the source f L.2( + R.
The curvature of T. More precisely, its non-linear structure prevents convergence to D[ug D[we] such that f D [[mn]] is non-negligible, while being orthogonal to the tangent space
M].
If we assume now that D is also nonlinear, then all the above analysis also holds for the linear operators dD) me, the difference being that the operator changes at each step. This means that the case of non-linear operators, we have deal with both the non-linearity of D and u, but that does not change the overall dynamic.
Finally, assuming that both D and u are linear (this is for instance the case when we assume to be linear combination of basis functions, like in Finite Elements, or Fourier Series). Then "learning" We with natural gradient (and learning rate 1) corresponds to solve the equation in the least-squares sense with generalized Green's 's function.
'cf. last line of Table 17 in Section C.4
"To our best knowledge, rigorous proof of this phenomenon has not yet been provided. We can therefore only rely on empirical evidence, which is illustrated in Section
7

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
We test ANaGRAM on four problems: 2D Laplace equation ; 1+1 D heat equation 5D Laplace equation and I+1 D Allen-Cahn equation. The first three problems comes from Müller and Zeinhoter (2023). while the last one is proposed in Lu et al. (202I).
For training, we multilayer perceptrons with varying layer sizes and tanh activations, along with fixed batches of points: a batch of size Sp to discretize & and a batch of size S3 to discretize A. The layer size specifications, cutoff factor e, values of So and Sp, and discretization procedures specified separately for each problem. Currently, the cutoff factor is chosen manually and warrants further investigation.
For these various problems, we display as a function of gradient descent steps, the medians over 10 different initializations, of L error EL and test loss Eisst with shaded area indicating the range between the first and third quartiles. E, is defined as: given test points (zx) E
1 E [1;();) - u* (x:)]² where u is a known solution to the PDE and S is taken 10 times V 5,2
bigger than the batch size Sp, while Fress is the empirical PINNS loss of Equation 7 computed with distinct set of points, of size times bigger than the ! batch size Sp. We compare ANaGRAM
to Energy Natural Gradient descent (E-NGD) (Müller and Zeinhofer, 2023). vanilla gradient descent (GD) with line-search, Adam (Kingma and Ba, (2014) with exponentially decaying learning rate after 10 steps as in Müller and Zeinhofer (2023) as well as L-BFGS (Liu and Nocedal, 1989). The corresponding CPU times are also provided in tables for reference. The code is made avaible at wthhtps::// iigiithuu t.ub. com/ II/ Mecoee/MNaGaaR and further implementation and computation details provided in Section A.1
2D Laplace equation : We consider the two dimensional Laplace equation and its solution:
Au
2nx² sin(( sin(T.)2
in Ω
[0, 1]2
2* (x1, ( sin(
(25)
a
109
10a
10-1
10
10-1
L error
Test loss
10-1
10-5
10-7
10
10-9-
10-11
10-11.
10
10"
10
10*
10
10a
0
Gradient descent step
Gradient descent step
ANaGRAM
Adam
E-NGD
GO
L-BFGS
Figure 1: Median absolute 2 errors and Test losses for the 2D Laplace equation.
We choose Sp 900 qqui-distantly spaced points in
CPU time (s)
Per step
Full
the interior of Ω and Sg = 120 equally spaced points on the boundary 2 (30 on each side). ANaGRAM, E-
ANaGRAM
7.16e-02
1.25e+02
NGD and L--BGGS are applied for 2000 iterations each,
Adam
1.23e-02
2.44++02
while GD and Adam are trained for x 10 iterations.
E-NGD
1.94e-01
1.88e+02
The network consists of single hidden layer with width of 32, resulting a total P 129 parameters. The cutoff factor is set to 1 10-6
GD
2.07e-02
4.13e+02
L-BFGS
1.95e-01
1.95e+02
1+1 Heat equation
We consider the (1 + 1) dimensional Heat equation and its solution:
Ō₂u 0 u 0
in Ω
[0. 1]2
n²t
u 0
2 2penter = [0, 1] x (0, 1); u²(1,x) exp
sin (wx). (26)
4
u(0,x) x) = sin(FT)
on 2D = -0) x [0,
8

---

Published as a conference paper at ICLR 2025
10
10-1
10
10
12 eerrrr
Test loss
10
10
10-1
10
10
10
10
10
10a
10
10
10
10a
Gradient descent step
Gradient descent step
ANaGRAM
Actam
F-NAD
Gn
I-RFGS
Figure 2: Median absolute 1,2 errors and Test losses for the Heat equation.
We choose Sn 900 equi-distantly spaced points in the interior of Q and Sg = 90 equally spaced points
CPU time (s)
Per step
Full
on the boundary 2n (30 on 2g2 and 30 on each side
ANaGRAM
1.29e-01
3.78e+02
of 28 ponner ANaGRAM, E-NGD and L-BFGS are
Adam
2.12e-02
4. 45e5++02
applied for 2000 iterations each, while GD and Adam
E-NGD
1.78e-01
4.04e+02
are trained for 20 x 10 iterations. The network consists of a single hidden layer with a width of 64, resulting in total of P 257 parameters. The cutoff factor is set
GD
3.87e-02
7.68e+02
L-BFGS
1.30e-01
3.91e+02
to c = x 10
5D Laplace equation : We consider the five dimensional Laplace equation and its solution:
Au n² - sin()TZ))
Ω = [0, 10,
u (x)
(T(T)))
(27)
2 sin (TT(k
on A
Rm
10a
10a
109
10a
12 error
10
Test loss
10:1
10
10-3
10
10 5
10
10
10
10
10a
10a
10
10a
10a
20P
10*
Gradient descent step
Gradient descent step
ANsGRAM
Adam
E.NGD
GD
LBFGS
Figure Median absolute 2 errors and Test losses for the 5D Laplace equation.
We choose Sp 4000 uniformly drawn points in the interior of Ω and SB 500 uniformly drawn points the boundary A ANaGRAM, E-NGD and L-BFGS applied for 1000 iterations each, while GD and Adam are trained for 20 x 10 iterations. The network consists of single hidden layer with width of 64, resulting in a total of P = 449 parameters. The cutoff 5.10-7 7 x Agmax. where A@max is the maximal eigenvalue of Ge (cf. line of algorithm
CPU time (s)
Per step
Full
ANaGRAM
7.18e-01
4. 4.88e+02
Adam
6.65e-02
1 1.29e+03
E-NGD
6.52c+00
4.96c+03
GD
2.69e-01
5,38e+03
L-BFGS
2.96e-01
2.96e+02
factor is set
1+1 D Allen-Cahn equation
We consider the (1 1) dimensional Allen-Cahn equation:
Ō₂u 10-3 Oezu 5(u
u²) = in Ω = [0. 1] x [-1,1]
-1 u(0, .) = x² cos(TT))
on M.hnder [0, 1] (-1,1)
(28)
on 20 = (0) [-1,1]

---

Published as a conference paper at ICLR 2025
10a
109
error
10
Test loss
10
1
10a*
10:3
10a
10a
10
10*
10
10
10a
10*
Gradient descent step
Gradient descent step
ANaGRAM
Actmm
F-NAD
Gn
I-RFGS
Figure 4: Median absolute T.2 errors and Test losses for the Allen-Cahn equation.
We choose Sp = 900 equi-distantly spaced points in the interior of 5i and S 90 equally spaced points on the boundary an (30 on 2g0 and 30 on each side of (. ANaGRAM and L-BFGS are applied for 4000 iterations each, E-NGD for 1000 iterations, while classical gradient descent (GD) and Adam are trained for 50 x 10 iterations. The network consists of three total
CPU time (s)
Per step
Full
ANaGRAM
6.01e-01
2.16e+03
Adam
2.82e-02
1.18e+03
E-NGD
1.30e+00
6.52e+03
GD
8.59e-02
4.28e+03
hidden layers with a width of 20, resulting of P 921 parameters. The cutoff factor is set to e = 5.10-7 x Aemax. where Agmax the maximal eigenvalue of de (cf. line of algorithm 2)
L-BFGS
4.07c-01
1.60c+03
We demonstrated that our approach can achieve comparable accuracy to Müller and Zeinhofer (2023) on linear problems, consistent with the equivalence established in Section E while maintaining per-step computational cost at most reasonably higher than that of Adam. Excluding Adam and GD, which consistently get stuck at high error levels. the bottom line is that ANaGRAM consistently outperforms both E-NGD and L.-BGGS -ofeon by significant margin-on at least one even both criteria: precision and computation time. The cases where the computation times E-NGD and ANaGRAM are similar occur when small-sized architectures are sufficient for the problem.
Results summary
6 CONCLUSION AND PERSPECTIVES
We introduce empirical Natural Gradient, new kind of natural gradient that scales linearly with respect to the number of parameters and extend to PINNs framework through a mathematically principled reformulation. We show that this update implicitly corresponds to the of the Green's function of the operator. We give empirical evidences that this optimization in its simplest form (vanilla ANaGRAM) already achieves highly accurate solutions, comparable Müller and Zeinhofer (2023) for linear PDEs at a fraction the computational cost, and with significant improvements for non-linear equations, for which equivalence of the two algorithms does not hold anymore.
Still, the present formulation of the algorithm has two limitations: one concerns the chosing procedure of the batch points, which is far limited simple heuristies; the second is the hyperparameter tunning, more specifically the cutoff factor, which is so far chosen by hand, while it may probably be automatically chosen based on the spectrum of δe.
Important perspectives include exploring approximations schemes for terms E. (e.g. using Nyström's 's methods, of. Sun et al. 22015) and Eg (e.g. using Cohen and Migliorati] (017 introduced in Theorem [ the design of optimal collocation points procedure, coupled with SVD cut-off factor adaptation strategy for ANaGRAM, as well as incorporation of common optimization techniques, such momentum. From thcoretical point of view, it seems particularly important to to include data assimilation in this theoretical setting, and understand its regularizing effect, while establishing connections to classical solvers such FEMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The work has been supported by the french ANR grant Scalp (ANN-24-CE23-1320)
We wish to thank Matthieu Dolbeault for insightful discussions sampling methods for - approximations, which eventually led to the concept of empirical tangent space.
REFERENCES
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, May 2015. ISSN 1476-4687. 103./000ttttte144400 14539
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific Machine Learning Through Itysies-IIffrred Neural Networks: Where we are and What's Next. Journal of Scientific Computing, 92(3):88, July 2022. ISSN 1573-7691. doi 110110077/1001150022-113333
M. W. M. G. Dissanayake and N. Phan-Thien. Neural-network-basedd approximations for solving partial differential equations. Communications in Numerical Methods in Engineering, 10(3): 195-201, March 1994. ISSN 1069-8299, 1099-0887. 11t_10.1002/mmm1144011111
Isaac E. Lagaris, Aristidis Likas, and Dimitrios I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987-1000, 1998.
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, February 2019a. ISSN 00219991.
1oi 201.:/............... 10.045
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993.
Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: A survey. Journal of Marchine Learning Research, 18:1-43, 2018
Maziar Raissi, Zhicheng Wang, Michael S. Triantafyllou, and George Em Karniadakis. Deep learning of vortex-induced vibrations. Journal of Fluid Mechanics, 861: 19-137, February 2019b. ISSN 0022- 1122-120: 120, 1469-7645. doi 100.1017//m..218..7777
Maziar Raissi, Hessam Babaee, and Peyman Givi. Deep learning of turbulent scalar mixing. Physical Review Fluids, 4( 4122):12401,, December 2019c. doi Fow//PyyseeeFuuus..1122222
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang. Surrogate modeling for fluid flows based on poystic-c deep learning without simulation data. Computer Methods in Applied Mechan- ics and Engineering, 361:112732, April 2020. ISSN 0045-7825. doi 1.0.10116/..mmm. cmw. 112221
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367 February 2020. doi 126....ee...ceeeee... aaw474
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations. Journal of Com- putational Physics, 426: 109951, February 2021. ISSN 0021-9991. 2000 109951
Taco de Wolff, Hugo Carrillo, Luis Martí, and Nayat Sanchez-Pi. Assessing physics informed neural networks in ocean modelling and climate change applications. In Al: Modeling Oceans and Climate Change Workshop at ICLR 2021, 2021.
Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E. Hurtado, and Ellen Kuhl. Physics- Informed Neural Networks for Cardiac Activation Mapping. Frontiers in Physics, 8, February 2020. ISSN 2296-424X. doi T0010...88///yyy22000000
11

---

Published as a conference paper at ICLR 2025
Georgios Kissas, Yibo Yang, Eileen Hwuang, Walter R. Witschey, John A. Detre, and Paris Perdikaris. Machine learning in cardiovascular flows modeling: Predicting arterial blood pres- sure from non-invasivv 4D flow MRI data using physics-ing innsice-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 358: 11223,, January 2020. ISSN 0045-7825. 1o. amt.:l...ma..22.112266.
Sifan Wang and Paris Perdikaris. Deep learning of free boundary and Stefan problems. Journal of Computational Physics, 428: 109914, 2021.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. In Advances in Neural Information Processing Systems, volume 34, pages
Curran Associates, Inc., 2021.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-iformmd neural networks. SIAM Journal on Scientific Computing. 43(5):A3055 A3081, 2021.
Petr Karnakov, Sergey Litvinov, and Petros Koumoutsakos. Solving inverse problems in physies by optimizing discrete loss is much more faster and accurate without neural networks, 2022.
Qi Zeng, Spencer H. Bryngelson, and Florian Tobias Schaefer. Competitive Physics Informed Networks. ICLR 2022 Workshop on Gamification and Multiagent Solutions, April 2022.
Johannes Müller and Marius Zeinhofer. Achieving high accuracy with PINNs via energy natural gradient descent. In International Conference on Machine Learning, pages 25471-25495 PMLR, 2023.
Johannes Müller and Marius Zeinhofer. Position: Optimization in SciML Should Employ the Function Space Geometry. In Forty-First International Conference on Machine Learning, February 2024.
Yurij M. Berezansky, Zinovij G. Sheftel, and Georgij F. Us. Functional Analysis. Vol. II, volume 86 of Operator Theory Advances and Applications. Birkhäuser, 1996.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint orXiv:1412.6996 2014.
Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale op- timization. Mathematical Programming, 45(1):503-528, August 1989. ISSN 1436-4646. doi 1001/000000//1115800
Tamara G Grossmann, Urszula Julia Komorowska, Jonas Latz, and Carola-Bibiane Schönlieb. Can physics-informed neural networks beat the finite element method? IMA Journal of Applied Mathematics, 89(1):143-174, January 2024. ISSN 0272-4960. 10. (0000000003/mmmma//xxeeeel00
Tim De Ryck, Florent Bonnet, Siddhartha Mishra, and Emmanuel de Bézenac. An operator precondi- tioning perspective training in thysics-informed machine learning, May 2024.
Songming Liu, Chang Su, Jiachen Yao, Zhongkai Hao, Hang Su, Youjia Wu, and Jun Zhu. Precondi- Neural Networks, February 2024.
tioning for
Levi McClenny and Ulisses Braga-Neto. Self Self-Adaptive Physics-Informan Neural Networks using Soft Attention Mechanism, April 2022.
Shun-Ichi Amari and Scott C. Douglas. Why natural gradient? In Proceedings of the 1998 IEEE International Conference on Acousties, Speech and Signal Processing, ICASSP'95 '98 (Cat. No. 98CH36181), volume 2, pages 1213-1216. IEEE, 1998.
Shun-ichi Amari. Information Geometry and Its Applications, volume 194, Springer, 2016.
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1))79-86, 1951. ISSN 0003-4851.
Yann Ollivier. Riemannian metrics for neural networks I: Feedforward networks. Information and Inference: Journal of the IMA, 4(2): 100)108-153 153, 2015.
12

---

Published as a conference paper at ICLR 2025
Tom Heskes. On "Natural" Learning and Pruning in Multilayered Perceptrons. Neural Computation, 12(4):881-9000 901, April 2000. ISSN 0899-7667. 4ooi1000112000997660000000000000
James Martens and Roger Grosse. Optimizing Neural Networks with Kronecker-factored Approxi- mate Curvature. In Proceedings of the 32nd International Conference on Machine Learning, pages 2408-2417. PMLR, June 2015.
Roger Grosse and James Martens. Kronecker-factored approximate Fisher matrix for convolution layers. In Proceedings of The 33rd International Conference on Machine Learning, pages 372-582. PMLR, June 2016.
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast Approximate Natural Gradient Descent Kronecker Factored Eigenbasis. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Felix Dangel, Johannes Müller, and Marius Zeinhofer. Kronecker-Pactored Approximate Curvature for
Neural Networks. In Advances in Neural Information Processing Systems, volume 37, pages 34582-34636, January 2025.
Yann Ollivier. True Asymptotic Natural Gradient Optimization, December 2017.
Manon Verbockhaven, Sylvain Chevallier, and Guillaume Charpiat. Growing tiny networks: Spotting expressivity bottlenecks and fixing them optimally. arXiv preprint arXiv:2405. 19816, 2024.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Tim GJ Rudner, Florian Wenzel, Yee Whye Teh, and Yarin Gal. The natural neural tangent kernel: Neural network training dynamics under natural gradient descent. In 4th Workshop on Bayesian Deep Learming (NeurIPS 2019), 2019.
Qinxun Bai, Steven Rosenberg, and Wei Xu. A Geometric Understanding of Natural Gradient, February 2022.
Shiliang Sun, Jing Zhao, and Jiang Zhu. A review of Nyström methods for large-scale machine learning. Information Fusion, 26:36-48, 2015.
Tianle Cai, Ruiqi Gao, Jikai Hou, Siyu Chen, Dong Wang, Di He, Zhihua Zhang, and Liwei Wang. Gram-Gauss-Nerr Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems, September 2019.
Anas Jnini, Flavio Vella, and Marius Zeinhofer. Gauss-Newton Natural Gradient Descent for Physics- Informed Computational Fluid Dynamics, February 2024.
Xianliang Xu, Ting Du, Wang Kong, Ye Li, and Zhongyi Huang. Convergence Analysis of Natural Gradient Descent for bver-paramettrizzd Physics-Infommn Neural Networks, August 2024.
Dean G. Duffy. Green's 's Functions with Applications. Chapman and Hall/CRC, 2015.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Karniadakis. DeepXDE: deep learning library for solving differential equations. SIAM Review, (3(1):208-222 January 2021. ISSN 0036-1445, 1005-7200. doi 0...1377/9MM1244000
Albert Cohen and Giovanni Migliorati. Optimal weighted least-squares methods. The SMAI Journal of computational mathematics, 3:181-203, 2017. ISSN 2426-8399. doi 10.5802/smai-jcmmm22
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pages 249-256 249-256,700 2010.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang JAX: Composable transformations of Python+NumPy programs, 2018.
13

---

Published as a conference paper at ICLR 2025
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Coumapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Fric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, R. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors, SciPy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261-220 272, 2020.
DeepMind, Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fan- tacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King. Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Miloš Stanojević, Wojciech Stokowiee, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020.
Frank Rosenblatt, The perceptron: A probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
Bernhard Schölkopf, Alexander J. Smola, and Francis Bach. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT press, 2002.
Vern I. Paulsen and Mrinal Raghupathi. An Introduction the Theory of Reproducing Kernel Hilbert Spaces, volume 152. Cambridge university press, 2016.
Ben Adcock and Daan Huybrechs. Frames and Numerical Approximation. SIAM Review, 61(3): 443-473, January 2019. ISSN 0036- 1445, 1095-7200. do:10.113777MMI146999
Mauricio A. Alvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for Vector-Valued Functions: A Review, April 2012.
Milton Abramowitz and Irene A. Stegun. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, volume 55. US Government printing office, 1968.
14

---

Published as a conference paper at ICLR 2025
A COMPLEMENTARY MATERIAL FOR THE EXPERIMENTS
A.1 DESCRIPTION THE EXPERIMENTAL SETTING
Description of the Method We base our code on Müller and Zeinhofer (2023). For our 4 numerical As in Müäler and Zeinhofer (2023), choose the interval [0, 1] for the line search determining the learning rate, I corresponding to solving the (linearized PDE) with the Green's 's function (cf. Section The neural network weights initialized using the Glorot normal initialization (Glorot and Bengio 2010).
experiments, we apply ANaGRAM gradient steps as defined in the Algorithm 2
As in Müller and Zeinhofer (2023), our implementation relies on JAX (Brad- bury ct al. 2018), where all derivatives are computed using JAX automatic differentiation, and the singular value decomposition computation carried out by the scipy (Virtanen et al. 2020) implemen- tation of JAX. Stochastic gradient descent, Adam, well as L-BFGS relies the implementation of DeepMind et al. 2020). All experiments were run on a Ith Gen Intel@ TM 17-1185G7 @ 3.00GHz Laptop CPU in double precision (float64).
Computation Details
A.2
FIGURES RELATIVE TO COMPUTATION TIME
A.2.1
2D LAPLACE
10a
10
10-4
10
10-3
L2 error
Test T loss
20
10
10-7
10
10
10-11
10
10-13
100
10
10
10e
10
10'
Running time (s)
Running time (s)
ANaGRAM
Adam
E-NGD
L-3FGS
Figure 5: Median absolute 2 errors and Test losses for the 2D Laplace equation across 10 different initializations for the five optimizers, relative to computation time. The shaded area indicates the range between the first and third quartiles.
A.2.2
HEAT
10.1
10
10
10-3
error
Test loss
10-3
20-5
10
10
10-3
10-9
10-0
10c
10a
10a
10a
10f
10a
10a
10a
Running time (s)
Running time (s)
ANaGRAM
Adam
E-NGD
GD
L-BFGS
Figure 6: Median absolute 2 enrors and Test losses for the Heat equation across 10 different initializations for the five optimizers, relative to computation time. The shaded area indicates the range between the first and third quartiles.
15

---

Published as a conference paper at ICLR 2025
A.2.3 5D LAPLACE
10:
10*
109
10-
1 error
10"
Test loss
10
10
10
10
10-5
10
10"²
10
10"
Running time (s)
Running time (s)
ANaGRAM
Adamm
E-NGD
GD
L-3FGS
Figure 7: Median absolute 2 errors and Test losses for the 5 Laplace equation across 10 different initializations for the five optimizers, relative to computation time (except for ENGD for which we only took 3 initializations). The shaded area indicates the range between the first and third quartiles.
A.2.4 ALLEN-CAHN
109
10"2
erroo
10-1
Test loss
10-2
12
10
10
4.
10:3
10a
10a
10*
100
10:
10a
Running time (s)
Running time (s)
ANaGRAM
Adam
E-NGD
GD
L-BFGS
Figure 8: Median absolute T,2 errors and Test losses for the Allen-Cahn equation across 10 different initializations for the five optimizers, relative to computation time (except for ENGD for which only took 3 initializations). The shaded area indicates the range between the first and third quartiles.
A.3 STATISTICAL TABLES OF RESULTS
A.3.1 2D LAPLACE
Table 1: Median, Maximum and Minimum 2 Ler-rro of the optimizers for the 2 Laplace equation.
Median
Minimum
Maximum
ANaGRAM
2.42e-09
1.70e-10
1.19e-08
Adam
2.05e-03
1.67e-03
2.86e-03
E-NGD
1.31e-06
8,43e-07
3.87e-05
GD
4.25c-02
1.01e-02
1.25e-01
L-BFGS
1.15e-02
3.08e-03
1.55e-02
16

---

Published as a conference paper at ICLR 2025
Table 2: Median, Maximum and Minimum of the test loss of the optimizers for the 2 Laplace equation.
Median
Minimum
Maximum
ANaGRAM
3.85e-13
8.49e-15
1.43e-12
Adam
3.51e-04
2.29e-04
4.31e-04
E-NGD
2.91e-09
1.0le-10
3.57e-08
GD
3.42e-02
4.32e-03
1.76e-01
L-BFGS
2.37e-03
8.91e-04
9.09e-03
Table 3: Mean and Standard deviation of 1.2-errors of the optimizers for the 2D Laplace equation.
mean
ANaGRAM
3.49e-09
3.58e-09
Adam
2.19e-03
4.18e-04
E-NGD
5.37e-06
1,18e-05
GD
5.41e-02
1.57e-02
L-BFGS
1.13e-02
2.94e-03
Table 4: Mean and Standard deviation of of the test loss of the optimizers for the 2D Laplace equation.
mean
std
ANaGRAM
4.27c-13
4.66e-13
Adam
3.37e-04
7.66e-05
E-NGD
7.00e-09
1.11e-08
GD
5.39e-02
5.39e-02
L-BFGS
3.04e-03
2.23e-03
A.3.2 HEAT
Table 5: Median, Maximum and Minimum L--errors of the optimizers for the Heat equation.
Median
Minimum
Maximum
ANaGRAM
6.48e-07
3.67e-07
6.15e-06
Adam
1.07e-03
5.96e-04
3.94c-03
E-NGD
2.50e-06
1.02e-06
6.38e-06
GD
2.02c-02
1.04e-02
2.39c-02
L-BFGS
2.97e-03
5.14e-(4
6.73e-03
17

---

Published as a conference paper at ICLR 2025
Table Median, Maximum and Minimum test loss of the optimizers for the Heat equation.
Median
Minimum
Maximum
ANaGRAM
6.82e-11
1.90-111
2.50e-10
Adam
2.37e-04
1.11e-04
1.41e-03
E-NGD
2.18e-09
5.62e-10
1.35e-08
GD
1.0le-22
4.70e-03
1.93e-02
L-BFGS
6.84e-04
5.85e-05
3.34e-03
Table 7: Mean and Standard deviation of L---errrrs the optimizers for the Heat equation.
mean
std
ANaGRAM
1.28e-06
1.75e-06
Adam
1.55e-03
5.19e-04
E-NGD
2.89e-06
1.77e-06
GD
1.92e-02
9.60e-04
L-BFGS
3.09e-03
1.74e-03
Table 8: Mean and Standard deviation of loss of the optimizers for the Heat equation.
mean
std
ANaGRAM
8.56e-11
7.05e-11
Adam
3.63e-04
3.93e-04
E-NGD
3.53e-09
3.83e-09
GD
1.20e-02
1.05e-03
L-BFGS
8.54e-04
9.16e-04
A.3.3 5D LAPLACE
Table 9: Median, Maximum and Minimum Lererrors of the optimizers for the 5 D Laplace equation.
Median
Minimum
Maximum
ANaGRAM
3.76e-05
6.99e-06
8.23e-05
Adam
4.86e-02
3.41e-02
6.08e-02
E-NGD
1.40e-05
1.18e-05
1.64e-05
GD
8.44e-02
1.50e-02
1.28e-01
L-BFGS
9.08e-02
1.55e-02
1.71e-01
Table 10: Median, Maximum and Minimum test loss of the optimizers for the D Laplace equation.
Median
Minimum
Maximum
ANaGRAM
3.68e-08
1.03e-08
2.20e-07
Adam
1.53e-02
1.02e-02
2.54e-02
E-NGD
1.51e-08
1.50e-08
2.51e-08
GD
6,00c-02
6.37e-03
1.11e-01
L-BFGS
7.65e-02
5.34e-03
2.25e-01
18

---

Published as a conference paper at ICLR 2025
Table 11: Mean and Standard deviation of L. -erorss of the optimizers for the 5D Laplace equation.
std
ANaGRAM
4.00e-05
2.93e-05
Adam
4.83e-02
8.06e-03
E-NGD
1.41c-05
2.29c-06
GD
7.64e-02
1.75e-02
L-BFGS
1.00e-01
2.19e-02
Table 12: Mean and Standard deviation of loss of the optimizers for the 5 5D Laplace equation.
mean
std
ANaGRAM
6.37e-08
7.01e-08
Adam
1.63e-02
4.19e-03
E-NGD
1.84e-08
5.55e-09
GD
5.64e-02
3.60e-02
L-BFGS
8.20e-02
6.80e-02
A.3.4 ALLEN-CAHN
Table 13: Median, Maximum and Minimum L -errrrors of the optimizers for the Allen-Cahn equation.
Median
Minimum
Maximum
ANaGRAM
2.51e-03
6.14e-04
2.04e-02
Adam
3.90e-01
3.78e-01
4.80e-01
E-NGD
7.39e-01
7.32e-01
8.10e-01
GD
5.86e-01
5.43e-01
8.37e-01
L-BFGS
5,40e-01
4.33e-01
7,45e-01
Table 14: Median, Maximum and Minimum test loss of the optimizers for the Allen-Cale equation.
Median
Minimum
Maximum
ANaGRAM
6.22e-05
1.45e-05
1.38e-03
Adam
7.10e-04
6.29e-04
1.18e-03
E-NGD
2.74e+00
2.58e+00
3,43e+00
GD
2.41e-03
1.70e-03
1.93e-02
L-BEGS
1.48e-03
9.08e-04
5.09e-03
Table 15: Mean and Standard deviation of L-2-rroor of the optimizers for the Allen-Cahn equation.
mean
std
ANaGRAM
4.32e-03
5,93e-03
Adam
4.02e-01
5.19c-04
E-NGD
7.60e-01
4.24e-02
GD
6.06e-01
5.99e-02
L-BFGS
5.49e-01
6.85e-02
19

---

Published as a conference paper at ICLR 2025
Table 16: Mean and Standard deviation of test loss of the optimizers for the Allen-Cahn equation.
mean
std
ANaGRAM
2.19e-04
4.16c-04
Adam
7.81e-04
1.01e-04
E-NGD
2.92++00
4.51e-01
GD
3,95e-03
5,41e-03
L-BFGS
1.77e-03
1.19e-03
A.4 ADDITIONAL EXPERIMENT: BURGERS EQUATION
We consider the (1 1) dimensional Burger equation:
oru + uuuuuu v2 = 0 in Ω [0, 1] [-1, 1]
Munder [0, (-1,1)
(29)
e00,, z) -sin(TTz)
on 2gg (0) [-1,1]
10a
10-1
10-1
10:2
10-2
Test loss
10-3
10-2
10
Loss
10-4
10-2
10-5
10
10-6
1
10-1
10-1
10
102
10a
102
10a
Gradient descent step
Gradient descent step
1.0e+00
1.0e-01
1.0e-02
".0e-03
Figure 9: Test and train losses for the Burgers equation for various viscosities
20

---

Published as a conference paper at ICLR 2025
150
-205
(a) 1=
(b) 10
100
(c) 10
(d) 10
Figure 10: PINN solution profiles for the Burgers equation for various viscosities v
We considered the burger with 4 viscosities (1, 10-1, 10-2, 20 Since shock develops in finite time as - using a regular grid did not yield sufficiently accurate results. Consequently, we manually adjusted the grid. Specifically, we defined two discretizations De, Dr, corresponding to [o, 1] and [-1, 1], respectively, as follows:
D,: A regular grid discretization with 240 points.
Dz: A non-regular grid discretization constructed as:
A regular-gris seriaar-rridoe of 60 240
41 points on [-1, -0.25),
grid defined as -2- Dws on [00.55, 0), where Diog is a regular grid 60 point [2,32].
240
4
The point 0,
grid defined as 2 Ding on 0.25., where Diog, is regular grid 60 240
point
4
on [2, 32).
regular grid of 60 240 points (0.25, 1].
With these definitions, the following grids were constructed:
Gi = Dr Dz: The grid for the domain = [0, x 1-,
D, {-1, 13: The grid for the boundary M0mmder [0, (-1,1),
G
[0)
Dz: The grid for the boundary 29n (0) [-1.1].
21

---

Published as a conference paper at ICLR 2025
We applied ANaGRAM for 1000 iterations, using neural network with three hidden layers, each containing 32 neurons, resulting in a total
2241 parameters. The cutoff factor is set 5.10- x Am where Am represents the largest eigenvalue of óg (see line of Algorithm 2 In Figure 9
to E
we present the training and losses for the different viscosities. The test loss was calculated using a grid constructed similarly to the training grid but with five times as many points.
For viscosity of y = 10 the test loss does not appear to converge, whereas the train loss does. This behavior can be attributed the grid computation method, which disproportionately emphasizes points near the shock. This explanation is supported by the solution profile shown in Figure 10 where, for V = 10-3, the shock in the learned solution is slightly shifted from x = 0 toward x
B EXAMPLES OF PARAMETRIC MODELS
B.1 PARTIAL FOURIER'S S SERIES
Let us fix a dimension d E N. We then define the N N-partial Fourier's Serie in [0, 1] as:
RI " + (10, 11 C)
SN
(OR)....k)))
x E [0, 1] -
Σ
N
Σ
0kk.....
24ar (Ii krwz)
(30)
h-=-N
We see that for all E N[ and 0 RU--N,I OnSNIO (x 52 e (or()))) As
consequence: for all 0 E RRI-N.N
dSN = SN₂
an thus: for all 0 E R[-NNN1
M = TMM = Span (a E [0, 11 2ix kaz) : ke -elI-Na INN.N
(31)
This precisely means that Sy is a linear parametric model.
B.2 MULTILAYER PERCEPTRON
Historically, Multilayer perceptrons (MLPs) were the first neural network models to be proposed (Rosenblati, 1958). Without going into unnecessarily formal description, we will define MI.Ps of depth 7. E IN function R" RM defined by induction:
Initialization (Input Layer) no = n, and
(0)
== x RD0
Inductive Step (Hidden Layers) for all E
ni N, o() R- R, and (z(),
( a
b()
a
-
ER
componenttwse
Final Step (Output Layer)
(w...o)) (x):= ( (L-1))
b(L)
(32)
E -
ERm
Equipped with this definition, we define a parametric model u associated to [(wo)bo) by considering any differentiable parametrization yp RR. - I1 R1-110 x Ri, and then defining:
u: RP H
(33)
fo(0)
i.e. using to encode the coefficients of the weights W and biases ((0) in the coordinates of vector in RP, Note that if sp is bijective, then P X _t_ + 1)wy.
Remark 2. A parametric model u associated to an MLP, as defined in Equation (3), is not linear, if activations (o(") istes, are not, and L > 2 (yielding the qualifier "deep" in "deep learning"). Nevertheless, if sp is linear, we may note that still linear with respect to the parameters associated to weight (L) and bias (L) In particular, for all 0 R.. ue Im duje.
22

---

Published as a conference paper at ICLR 2025
C COMPREHENSIVE INTRODUCTION TO EMPIRICAL NATURAL GRADIENT AND ANAGRAM FRAMEWORK
In this section, propose more comprehensive introduction to the concepts introduced in Section 3 as well as proofs for Proposition!! !1 Theorem T and Proposition 2 which are stated therein. To this end, need to review the notions of Neural Tangent Kernel (NTK) in Section a.T
and Reproducing Kernel Hilbert Space (RKHS) in Section C.2 before introducing empirical Natural Gradient in which is the key theoretical concept behind ANaGRAM.
Section C.3
C.1 NEURAL TANGENT KERNEL (NTK)
Neural Tangent Kernel (NTK) has been introduced by Jacot et al. (2018) as fundamental tool connecting neural networks to kernel methods, another very popular tool Machine-learning (Schōlkopf et al. 2002). More precisely, it shows that for an empirical quadratic loss:
1 $
(()) == 25
f(x)))
the gradient descent:
0 0 Ve,
can be reinterpreted in the functional space, in the limit 77 + 0, as the the differential equation in 210 : R - L (12 - R), with the initial data u10 = uo E L (S2 R):
dujo.
8
dt (x) = N (x:) v.), (T = Σ (r (e)u
15
By the same observations for Equation [4 we observe that Equation 15) induces the following differential equation 0: R+ RP, with the initial data ((0) Og E IR.
d0 dt
du
Σ NTKa, (ue ys)
2 du (NTKa, (2, (1))
-v:)
(34)
Using Euler's approximation method, this can of course be rewritten as the discrete upate:
0.+1 = 0: - Σ du'o, (N 2.) (nn (z4) 31).
(35)
Note that if we assume duja be inversible, we can implicitly compute the inverse, yielding:
dujo. (NT
Σ do (2x) elp),
(36)
making Equation (35) effectively correspond to the usual gradient descent in Equation ( Rudner
et al. (219)) further extended this framework to the of natural gradient descent in the context of information geometry. They demonstrate that the learning dynamic this scenario governed by a new kernel, named the Natural Neural Tangent Kernel (NNTK), which is defined for all 0 R.
NNTKo(æ,9)
Σ (Z)u)((z) pa (C)))()))
(37)
LEP.EP
where Fg is the Fisher information matrix. the more general context of Riemannian Geometry, Bai al. (202)) show that the NNTK is given by: for all 0 e Rº
NNTKo
Σ (2)))((1)) (0um(on
(38)
IEP-EP
with Go being the Gram matrix relative to Riemannian metric Go as introduced in Section for all 0 e Rº, for all p.q & P
Gop.q
Goldp u e, Ggm)))
(39)
23

---

Published as a conference paper at ICLR 2025
In particular, when Go is given by the metric of an ambient Hilbert space H. this yields: for all @ e RP.
for & p.q & P. for all z, y E Ω
NNTKo(E,y)
E
(2pu)(x)) Gopo (o)) Gøp.g = <o ) H
16
I.p.
For the quadratic problem of Equation [1 natural gradient then yields the functional dynamics:
duja:
(2,
y1)-
(40)
dt
In the following, will further explore the (N)NTK and its connection the natural gradient in the context of Reproducing Kernel Hilbert Space (RKHS) theory.
C.2 A PERSPECTIVE ON REPRODUCING KERNEL HILBERT SPACES (RKHS)
In this subsection we will carefully review the intimate link between neural tangent kernels, projec- tions and reproducing kernels. To begin, let us define what a kernel function is, following Paulsen and Raghupathi 2016 Definition 2.12):
Definition 3. A function k: S x Ω - IK, K E {R, CJ is called kernel function provided that: for all
N, for all (x,) E Ω2,, for all a KN
Σ
a;k(x1 >
(41)
LEiEN
Equipped with Definition 3 we can state the following theorem, which binds different perspectives on RKHS:
Theorem An Hilbert space consisting of functions Ω - K, e (R, CJ, is Reproducing Kernel Hilbert Space if and only if one of the following equivalent conditions are met:
1. There is kernel 1 Q R such that H Span (xxx,)) e Q2 and: for all æ, y E Q
(I<(x,-), k(y, 17H k(I, y).
(42)
particular, we have the reproducing property: for all v H. for all x e
v(x) = (k:(x,,), i)M
(43)
2. For all x e St the evaluation form e₂ : H ~ f() is continuous.
proof of this theorem can be found Paulsen and Raghupathi (0066 Definition 1.1, Proposi- tion 2.13 and Moore's Theorem 2.14). We now draw some easy but essential consequences from Theorem 3
Corollary 1. Any finite dimensional Hilbert space H. is a RKHS
Proof. Since H finite dimensional, all norms equivalent. In partieular ll:lla / H supsen [f(()) is equivalent to Il-l Then by point 2 in Theorem 3 H is a RKHS, since for all æ e Ω
eg continuous for ll
In order to set out important theorem that highlights the link between RKHS and projections, need the following definition:
Definition 4. linear operator A : H - H is an integral operator given that there k Ω x Ω K, K E (R, C). such that: for all f H. for all Ω
A A() (z) (k(x, :), fru-
(44)
We state:
24

---

Published as a conference paper at ICLR 2025
Theorem 4. Ho is RKHS, then the orthogonal projection onto Hy, 1a H. - Hu is integral operator whose kernel is the reproducing kernel l of Hy, i.e. for all f E H. for all E Q
11 (f)(z) - (k(2,-), f)H
(45)
In addition, k is given by: for all Y. E
k(x,1) v) Σ L,())((v))
(46)
iEN
where (Li)ien is any orthonormal basis of Ho-
Proof. The second claim is essentially equivalent to Paulsen and Raghupathi (016 Theorem 2.4).
To prove the first claim, let us notice that the reproducing property: for all Ho. for all $2
v(x) (k(ar,-) :), V)H:
(47)
exactly means that the identity on Ho is an integral operator whose kernel the reproducing kernel of Ho- Since the restriction of Ho to Ho is the identity Ho. it is sufficient conclude the proof to show that: for all f & Ho. for all x E Ω
(k(z, =0x= =0
(48)
But this is an immediate consequence of the second claim.
Remark 3. Theorem encaasulates finite dimensional case, since one may take L; for greater than D E N. yielding dim(Ho) < D, which implies particular that Hy is an RKHS by Corollary Remark 4. The assumption Ho is an RKHS essential, since there is no guaranty that such a space (finite dimension aside) is indeed RKHS. One may think for instance to the case where the (L,) are the Fourier's ' polynomials defined Section B.I In this case the associated kernel is the Dirichlet kernel, which is well known to be non convergent, neither pointwise, nor in L (JO, 2m)).
Theorem prompts the question of how construct such orthogonal basis (I,). Assume that already have basis for Hy. i.e., Ho- = Span(up e N) c H. While Cram-Schmin procedure could be used, there is another approach that, in certain sense, is far more optimal. For the sake of simplicity, us use suppose that Ho finite dimensiona 6 Then:
Lemma 1. Let us be Ho Span(u,
p & P) H and consider the Gram matrix
Gpg == (up, und of (up) and its eigen- decomposition G = UA. Then:
Lp = Σ 4p
(49)
1E9EP
is an orthonormal basis of Ho. In particular, 119 is an integral operator whose kernel is:
M(,
Σ Up(x)G "g
(50)
IEP.UEP
Furthermore Top are the left-singular vector of the so-called synthesis operato
R.
Ha Σ 1SPEP
T: a
(51)
up
Proof. Since Ho is generated by the finite basis (u,) 1sps P, it is RKHS by Corollary I and there exist (for instance by the Gram-Schmidt procedure), an orthonormal basis (V)) of Ho. Then by Theorem 4 the operator II defined by: for all H
I()) = Σ VV
(52)
ISPEP
'The infinite- dimenti-ad case is more technical, have be careful with continuity linear applications. As we are only considering caately-parameterian model, this is beyond the scope of our present work. "Nme and notation taken from Adeock and Huybrechs 2019
25

---

Published as a conference paper at ICLR 2025
is the orthogonal projection onto Ho. But the basis (pp) of Equation 449 is precisely orthonor- mal. Indeed:
(Tp 1ag)
Σ ISkEP
ugUR
pp pt
E U
lEtEP
Σ lEkE
AlUna (ux, un U1 - GUUGSee
=((p AUU'U = 8pg
where 6ng is the Kronecker symbol such that 6pg 1 if and only if q. Now building II upon this basis yields: for all f E H.
(T)) = Σ
T (>,
E E
Σ
UKUR, ( 5)
PEpEP
1sp P leke PIEEE
Σ Σ Uk
Σ LEPEP
4 Upd
(u: f)
E
unChi (ui, f)
E.REPEEEE
LERIEP
Thus, the kernel of the projection lTo onto Ho is precisely:
k(x,V)
Σ
Finally, us write the SVD the synthesis operator: Va E R.
T(a) Σ
up E Ha.
(53)
PEPEP
Then in particular, we have: for &
U/ = T(WAA)),
and: for all 1 PPEP
T(el))).
This implies that: for all & p.
Gp. (up Uq)
((e(o)), T(e(i))
53
(n) e (uk. v) e ( w
-but
This means that (W,) and (A2) are respectively the eigenvectors and eigenvalues of The result follows by unicity of eigen-decomposition and respective identification of (W,) to (Up) and (Ap) to (Ap) in Equation (49
This observation will enable us to establish the main result of this section, linking RKHS theory, NTK and natural gradient, in the following corollary.
Corollary The NNTK Ke defined Equation (66) is the kernel of the projection TIAM H H onto M.
Proof. This is a direct consequence of Lemmal since T&M = Span( : 1<PEP).
In the following, we will derive some consequences from NNTK theory, leading to the concept of the empirical Natural Gradient (eNG).
26

---

Published as a conference paper at ICLR 2025
C.3 EMPIRICAL NATURAL GRADIENT (ENG)
To begin, we need to make key observation:
Equation
[5). namely:
dujo,
(x)
NTKo (s,, (1:) ys)
dt
shows that the empirical dynamics under gradient descent happens in the space:
ANTK M == Span(NTKa(;,.5 (24)) ToM.
(54)
Likewise
dllp:
N
(x)
NNTKo, (1, (xi) - yz),
(40
dt
shows that the empirical dynamies under natural gradient descent happens the space:
fNNTK M == Span(XNTToo(;2)) (Ii) = ToM.
(x(()))
Both spaces (.(x;)
ÍNTK M and NNTT .M are subspaces of the tangent space T = Im du There- (,(4))
fore, it remains true that the empirical functional dynamics occurs within M. However, T M
and f-N NINT M are the smallest subspaces in which the empirical functional dynamics take place,
(w..) respectively for classical and natural gradient descent. We encapsulate it in defintion:
Definition 5 (empirical tangent space). Given a parametric model u RR - H. and a batch of points (1s) the empirical tangent space relative to the points ( the space:
NNNTK M == (Dan(NNTKo(s.a)) m;)) 0,(a))
(1))) < Te
When the context is clear, its name will be abbreviated empirical tangent space and it will be denoted:
TeM == Span (NNTKa(s, : (z)))s))))
(55)
The second key observation is the following since natural gradient descent, in the limit S 50 (population limit), is given by the update (cf. end of Section 2.3):
i - 0, - 12 dule. (I+.,, M (V
one may also define a similar update in the empirical tangent space T. This observation motivates the following definition, already introduced in Equation (18):
Definition 6 (empirical Natural Gradient (cNG)). The empirical Natural Gradient (eNG) the update given by the projection of the functional gradient on the empirical tangent space ToM, i.e.
0 - 0, - dd (II/ V.
(56)
The problem now is to find tractable procedure compute the update of Equation (56). This is the aim Theorem [ stateed in Section that we now recall and prove:
Theorem (ANaGRAM). Let us define for all & i for all & Tor.p == (
VL "oo V
:= V (II) = u - f(x;).
Then:
du-o
TI + V
3 + Emmetric
( + F
(19)
where Emeerrcc and Eg are correction terms specified in Equations 57 and 58 in Section C.3 respec- tively accounting for the metric's 's impact on empirical tangent space defintion, and the substraction of the evaluation of the orthogonal par of the functional gradient.
orthogonaal to the whole tangent space To.M
27

---

Published as a conference paper at ICLR 2025
Theorem specifications
The corrections terms Egeni and Eg given by:
Lngettic
D
Vo (l, (L) ( (1. 8,0
(57)
with:
II = Ep-1 e e(n),
the projection onto the r first coordinates of R..
Ip, the identity of R?
Vi = SV ((0e)
. for all E p.. & P. Goy.a (0pu10 )0210)
Ep = -(NNTKK(()))) vLnu - V((x.) lsigs
(- (Tr VE) (a:)
=
(58)
lEigs
Proof. First of all, following Lemma we see that the projection kernel into TeM is given by: for all z, y E M(x,y)
Σ N,,
(59)
LEjEs
with: for all i,j & 8, for all 0 RP
Goi,- = (NNTKa(E..)) -,, NNNTKK(K,, (
(60)
where last equality comes from the fact that NNTKg is the reproducing kernel of ToM. We can then simplify Equation (56):
dujo, (Io..m VLimje) = dufo. - Viuuo >x)
(61)
Σ dulo. (NNTKo -) Golds (T))))) VLimee
(62)
Σ
dujo. (DrMee Ci (u (os) (N V L >
LEP.EE 181
(63)
Σ P.P.gEP 183Js8
Go.p. u. (Ii) Gals ( (E,)), VL
(64)
Using the empirical jacobian and for all & i S, for all
jacobbin matrix introduced in the statement of Theorem |
namely: for all 0 E R.
@ = 20 (z+).
(65)
Equation 660
rewrites: for all 0 RP
Go = Go
(66)
Introducing also: for all 0 R and for all 1 <iES
VCh, NNNKK((EE)) ) Vmmo)
(67)
Equation 64 then rewrites:
dulo, TT M V Viniee
= 2o. (āo. (bo.C., Gb. 3e.)
) Vcll. 'V
(68)
Using now the SVD of se. also introduced in the statement of Theorem namely: for all 0 E R
So DoAoVo
(69)
28

---

Published as a conference paper at ICLR 2025
may express the pseudo-inverse inverse of de as:
T6 = 0o
(70)
and rewrite Equation (66) as:
Go - U.Do Vo
(71)
Let use denote S the rank of 60. and introduce 1I, e first r coordinates of the canonical basis (elw)) P of Rp. Observe that, particular, A 1T,-
(n) the projection onto the
Then, noting that To is orthogonal, and Åo diagonal:
G. - = (I, (o.. L/0g C
(72)
where
Σg = 11, Vi Voll.
(73)
Inserting Equation (22) and Equation 69 Equation 668) we get:
dulo. (ut VIu )
Go
(74)
=
(75)
-Vo((Ir - II,) II, 7,
(76)
(77)
+ +Vo (Ip 11,) Vo 1I, 2/ VCo
(78)
(Vo + Egg
VCl
(0)
F.g
Emmetric
VC
(79)
Finally, by decomposing V Lupo into its collinear and orthogonal components to ToM, i.e. for all 0 e RP
VC u = IToM (VL(mee + T (VLImi)).
(80)
and using the notation for all 1 siES S
VC up introduced in Theorem statement, we have: for all 0 e R".
Aimpo= Vuy (r,) IIt.mm (VCmm) (xx) 17gM (VD(m)o) (xxs)
(81)
-NNTTKO(E))) V H Trg M
(Viiiuje) (2s)
(82)
-CC. -
(83)
where Equation (22
comes from the fact that NNTKo is the kernel defining 11 and Equa- tion 83 uses the defintion given by Equation (67 and the notation Eg introduced in Theorem
statement, specified in Equation (58) Thus VLg Ainn + Fy. which concludes.
Remark 5. Note that the implicit inversion made in order to obtain Equation (64) is now exact, in the
sense that we do not need to assume that duye, is inversible anymore as in Equation (36), since the eigenvectors and eigenvalues of Go, respectively match singular vectors and singular values of du as stated in Lemma]] I
We call this the exact implicit inversion trick.
For the sake of understanding, let us suppose, that So is of rank P. Then in particular $ > and Equation (72) rewrites:
(āo
(3b
Go
(84)
29

---

Published as a conference paper at ICLR 2025
Since S we also have 08 (8)
Ip and thus, Equation (68)
simplifies to:
du'o. 0. (I VLlmee)
(x))
G Gl VL = 06 V.
(85)
where last equality comes from the fact that 1 V
E Im Go by its own definition and the one of NNKK, This means that under those conditions, the term Eg of Theorem vanishes.
Unexpectedly, the assumption do has rank P can be satisfied for a specific subset of points those guaranteed by Proposition D
restated below, which we will now prove:
Proposition 1. There exist P points (f:) such that NNNTK M TaM. Then notably Eg Emmetric 0.
0,(2.)
Proof. Let d == dim(To.M & P. By definition of NNTKo (cf. Equation (66). we have for all a Ω:
p Σ
NNTKo(-2))
a e e TeM,
(86)
with for all p Op = E G/ p.g
())))()) E R. Therefore Ta. = ToM. We will start
by showing that Span( N WTTKKl Ke(..z)
2 E 92 = ToMM
: x 92) = TeM
is clear from Equation 86). Let us now be a E TaM S
x T 92) Since
u E Span(NNTKK) r) x e Ω we have: for all x Ω
(NNTKe((,)) u) u(x);
where last equality comes from the fact that NNTK the reproducing kernel of TeM (cf. Theo- rem D. Therefore u = 0 and thus:
TeM n Span (NNTKo(.2) : x {0),
Le. ToM = Npan(NNTKo(=:2) xE Ω2 Span(NNTKol: ludes. Now, since To is of finite dimension d & P, so is Span WNNKKooss5 e Q), and since (NNTKot (o(:K=)()) x))mn is generating family, one may extract free subfamily out of it, which will be of cardinal d & P, i.e. there exist d & P points (ft) such that ToM Span (NNTKo(,, (.) 1 & d) ToM and thus:
E Ω), which con-
II MM
VL oT VC.
If d P, the sequence leiedd can be extended with additional P d arbitrary points.
Finally, in some cases, we have TigM (VC and thus VL = i.. VL, = VL.. This is the focus of Proposition ? stated and proved hereafter:
Proposition 2. If u can be factorized as uie = Li [Ciog] with O = -01,0,) RP+Ps CBR+P,C.B.... H.1. L: P F(H H.) linear in 0,, and 0. then Eg = 0.
Proof. From the discussion of Section 2.3 more precisely the identification of the Fréchet derivative
of the functional loss Equation we have that the functional gradient for quadratic regression is:
Cup 01o
(87)
Assuming that f = 0, this reduces to:
VLng = u1
(88)
Now using the assumption and notations of Proposition 2 that: Ōe dio (0,) dLja, (0:) [c, Lio, [Cio,] (1(0,00)) u u
(89)
where the second equality comes from the linearity L with respect to 0, particular this implies that uh E ToM, which conludes.
30

---

Published as a conference paper at ICLR 2025
Remark 6. The question arises to whether Proposition has any concrete application, i.e. whether this situation occurs in real applications. As it happens, the hypothesis Proposition 2 verified in particular when solving the functional equation:
D[u] = 0,
(90)
using MLP parametric model with D lincar. Indeed. refering the definition of an MLP in Section B.2, and specifically to the definition of the last layer in Equation (32). we see that MLP can be decomposed into "1(0, 02) L [Ci,]], with 0 being the parameters encoding the last layer. Now, forming the coumpound model defined in Equation (21) of Section 4.1 with the operator D yields:
Dou= Do Lo, [Co=]
D 10: [Cie.] L [Cios]
(91)
=.T.
and thus the coumpound model is still verifying the assumption of Proposition f being null according to Equation (90), we have that all the hypotheses of the proposition are verihed. In real-life applications, boundary conditions also need be taken into account, mentionned in Equation (21). However, these are a simple L.² regression problem when the boundary conditions are Dirichlet, and therefore do not present the same conditioning difficulties for regression with respect the differential operator. This last fact, combined with Proposition 2 in our view partly explains the strong discrepancy between results for linear and non-linear problems.
In future work, we plan to carry out an in-depth analysis of the estimation the F.g and E terms, and their impact on both the overall theoretical framework and the training dynamics. We also aim to develop a more accurate method for approximating them.
C.4 ADAPTING NATURAL GRADIENT DEFINITIONS TO PINNS
In Section 4.1 we highlighted the fact that PINNs can understood classical quadratic regression problem, simply by substituting the model u with the compound model (D, B) introduced in:
RP - H -
L (Q,00) = L' - R, x L2 - R,
(D, ou:
0 21a
(D(uo), B(up))
We then pointed that this simple remark made it possible extend ANaGRAM, and more generally natural gradient, to the case of PINNs without further difficulty.
To make this adaptation more explicit, compare in Table 17 below the objects introduced in Section define natural gradient and empirical natural gradient in classical quadradic regression context, with their equivalents for the PINNs case.
Natural neural tangent kernel and empirical tangent space pose certain technical difficulties and are therefore dealt with separately in Section C.5 below.
31

---

Published as a conference paper at ICLR 2025
Table 17: Comparison of objects introduced in Section 2 for classical and PINNs regression.
Classical regression
PINNs regression
Image set of the model
M = Im u
: Im ((D, B) u))
{upp 0 R²}
{D[ue)) p[uo]) : R*)
c L (92
LL a (2, 20) = L.2 (9) L (02)
Model
differen- tial
dujp E Lin (R² - L- (2))
d((D, B) o u) E Lin (RP - 1,2 (9)
such that:
such that:
u1a+h u1 + (h)
(D, B) 10 = ((D, B) o u))o
+ d ((D, B) o u (h)
+ =(1l1)22)
=((hll2)
Partial deriva- tives
2y2i dup(e(
3p ((D, B) o u) =d (dD, 2)10 (e(o))
410 #10+))())) e
= (2p O B(uel)
lim
= (d 1u (6)2)0)
dB dB (ō
Tangent space
ToM == Im duja
Tol = Im Imd((D, u)
Span 11)
Span (2 B(yo))
P
Σ
0p0
Σ
hp (c) B(uo)))
p-1
p=1
(hp) RP]
(hp) E [R!]
Functional
loss
L: (92) - IR+ - Il - /1L
12(0,00) - R++
L: E
- 1/2 ||v (I.))li
Functional gradient
V VL juy
VL = [[ B)cu)
0 ʃ
= ((D o u) (f.g)
(Q)
el,((2,20))
Natural gradient
dujo (II (VLo))
a( B) o w))l (1 (VCo)
32

---

Published as a conference paper at ICLR 2025
C.5 NATURAL NEURAL TANGENT KERNEL AND EMPIRICAL TANGENT SPACE OF PINNS
Seeing PINNs as a quadratic regression problem with respect to the coumpound model of Equa- tion (21), established in Section 4.1, we that the "natural" defintion of NNTK that arises from Lemma is: for all 0 E Rº, for all T, y (S2 x 222
NNTKo(
E ((D, o) (z) ((D, B) 1) (v)².
P.P
with:
Go Go: (o,
0g ((D, w))e (2(0000)
(92)
The problem lies in the fact, that in order to define the empirical Tangent Space, we would like to be able separate 22 and 7 contributions. To do this, we have to remark that the coumpound model defined in Equation (21) outputs functions that have a two-dimensional output, i.e. the function is vector-valuated and not scalar-valuated anymore. More precisely,
have that for all f Im((D, B) o u) = - c (2, 22) = (2 - RR) x La2 - RR, there exist Sa E L' - R) and fen E I. ( - R) such that f = (fa, fan) Thus for all x (æn, Ta0) E Ω x A
f(x) = (fa(zn), fan E R².
(93)
Hence, the associated reproducing kernel should be bit revisited. In particular the reproducing property rewrites (Alvarez et al., 2012, Section 3.2): for all f (Ja, ToT = 12, 0Q), for all x Tan) f2 x al and for all c R?.
( (NNTK((.))))) [xx) c= fa(an)a fan(san))2.
(94)
In particular, have: fn(zn)
fan( {f, xNTK)((=.))))))) This means that the contributions coming from S and A are lin- early independent and can therefore be separated. More precisely, this can be done by defining the partial NNTKs:
for all y E Q 02, for all an e Ωt for all zan E aQ:
N (y, = NNTK K joly (og, zan))e
(95)
Σ a,((D,)) o u) oly) 2D ())()))
(96)
IEP
for all E Ω x M, for all ran E 0n, for all vn Ω:
20 NNTK
0 (y. xaa) = NNTTK (rn, =20))
(97)
= Σ
C,((D, w) ( w))((a)))
(98)
IEP.EP
Then in particular: for all (In, Tan) Ω x 02, for all f Tor
fa(øn) (f (NTK(((.a)) (f. NNTK
fan(xan) = (1 O
(J, NNNN ol x))22))
This allows to define empirical tangent space for PINNs in the same way in Equation 117). namely: Given two batches (xf)) E S252 and (I. e Saa we define the associated empirical tangent space:
NNTTK (222)) Span
60 29) (.
(1 lsis Sa, & &Sn (99)
Empirical Natural Gradient and associated ANaGRAM derivation poses then no particular difficulty and are done in similar way to Section C3
To make this more concrete, we plot below some NTK and NNTK for the heat equation and the 2D Laplace equation. What observe is that the NNTK yields much more specialized kernel that
33

---

Published as a conference paper at ICLR 2025
NTK, in the sense that it is much more localized and also perfectly centered the reference points. This localization property is expected to be more pronounced as the complexity the model is increased. The drastic discrepancy observed between NTK and NNTK explain why PINNs fail to train under classical descent while empirical natural gradient solves this issue. The excellent locality of the NNTK kernel leads indeed much better optimization schema because the residues indeed arbitrarily shrunk in small regions around each sample batch point by independent modifications of the function which is obviously impossible with the NTK. When increasing the complexity of the model the spatial range of the NNTK is expected to decrease accordingly and then more points will be needed "percolate" the optimization over the domain. In such case if the batch size increases too much various principled strategies can be considered to control the complexity of the empirical natural gradient, precisely because the NNTK defines natural distance between points which can be leverage to define an approximate block Gram matrix. In our point of view this is one of the great advantages the empirical tangent space over the "parameters" tangent space, because no such good metric is given for free in parameters space.
C.5.1
NTK AND NNTK PLOTS OF LAPLACE 2 D EQUATION
Figure 11: NTK at initialization for Laplace equation in 2D. Reading: the red cross on each subfigure
representing point 2;, the plot represents the function NT NT,e, (;,2:))
34

---

Published as a conference paper at ICLR 2025
400
440
A00
Figure 12: NNTK initialization for Laplace equation in 2D. Reading: the red cross each subfigure representing point z;, the plot represents the function N NNTKe, (1,z;)
Figure 13: NTK at the end of optimization for Laplace equation 2D. Reading: the red cross on
each subfigure representing point 2,, the plot represents the function NTK, K (,,2;)
35

---

Published as a conference paper at ICLR 2025
Figure 14: NNTK the end of optimization for Laplace equation in 2D. Reading: the red cross on each subfigure representing point 2i, the plot represents the function NNT Kgmd (;,
C.5.2 NTK AND NNTK PLOTS OF 1+1 D HEAT EQUATION
Figure 15: NTK at initialization for Heat equation in I+ID. Reading: the red cross on each subfigure
representing point 2;, the plot represents the function NT,,, (,,2:)
36

---

Published as a conference paper at ICLR 2025
HON
Figure 16: NNTK initialization for Heat equation in 1+1 Reading: the red cross each subfigure representing point z;, the plot represents the function N (NTKe,
Figure 17: NTK at the end of optimization for Heat equation in I+ID. Reading: the red cross on each subfigure representing point 2,, the plot represents the function NTK, Kond (1,2;)
37

---

Published as a conference paper at ICLR 2025
Figure 18: NNTK at the end of optimization for Heat equation 1+1 D. Reading: the red cross on each subfigure representing point z;, the plot represents the function NTT Koma (,,04)
D CONNECTION OF NATURAL GRADIENT OF PINNS TO GREEN'S FUNCTION
In this section, we will establish the relationship between Natural Gradient and Green's 's function. To set the stage, let us first introduce some definitions.
D.1 PRELIMINARY DERNITIONS
Definition 7 (Green's 's function). Let D 7 - 1.3(S - R4) be a linear differential operator. A Green's function of D is then any kernel function g g IR such that the operator:
D[H] - H
R:
(NEl- Sa (,,(,,,){((s(((()))))) (){())((ss))
(100)
is a right-inverse to D, Le. such that D R IH. Remark 7. We can rephrase Definition 7 by saying that g is a Green's 's function if: for all æ, s Ω
D[g( s))(2) = (x)s),
where 5g is the Dirac's distribution centered in
Remark 8. Definition 7 implies that, D[u] E D[H] is solved by u(x:) 9a, (s))s)
In order to obtain more meaningful results, will need the following generalizations:
Definition 8 (Solution in the least-squares sense). Let D H L R, (1) be linear differential
operator, Hn H subspace isometrically embedded in H and E I.? R, n)) We call un Ho a solution of the equation D[u] = f in the least-squares sense if ug verifies:
inf [D[ [[ - -1 f1 HeHo
(101)
Remark 9, If f D[Ho], then inf
[D[u] f1 0 and thus un is a classical solution.
weHo
38

---

Published as a conference paper at ICLR 2025
Definition 9 (generalized Green's 's function). Let D, Ho and be as in Definition A generalized Green's function of D on Hy is then any kernel function : 52 - R such that the operator:
LA - R. p)) + H
Rxo
- (rEL- In (((ss(()
verifies the equation:
Do RHo =IT DP
(102)
Remark 10. Due to the identity II pad
In Definition 9 is indeed generalization of
Definition 7
We will now state and prove result required prove Theorem
2
D.2 PROOF OF THEOREM 2
Proposition 3. Let D : L' jà) be linear differential operator, and Ho Span(up)
1 & P) = H subspace isometrically embedded H. Then the generalized Green's 's function of D on Ho is given by: for all x. y Ω
(z, =
Σ P.PEE
up(x) DTul(V))
(103)
with: for all < P.
Gpg == (D(up) D(
(104)
Proof. By definition:
D[H[] [D[u] e Ho}
Thus Lemma Tppliies and yields that: for all x, y
k(x,8))= = Σ D[un](x) G Dlu)(y),
(105)
P.gEN
with: for all P.PPPP.
Giy == (Dlupl, (2
(106)
the kernel of the projection [1H)] 1.2(S - R, - 1.3(2 - R, μ)) onto the RKHS D[Ho]-
This means that: Vf L +)
(f)H)( k(z, v)f(()
Σ (z) P(ug(e) fly) pddy) (107)
Ja LEPHEP
Σ
D(,p D(unI())(()) m(dy)
(108)
1.p.gsP
Σ
up(+)
P. [ DlugI(w) ((w) p(dy) (x),
(109)
ISP.PS
where Equation (109) comes from the linearity of D. Refering Detinition 9 this exactly means that the kernel defined by: for all x, E Ω
(x, == [up(2) ((u,(())
p.q
is the generalized Green's 's function of the operator D on Ho.
We now in position present the proof of:
39

---

Published as a conference paper at ICLR 2025
Theorem 2. Let D H - (2 - p)) be linear differential operator and RR H
parametric model. Then for all E R", the generalized Green' 's function of D on TaM Im d is given by: for all z. y ?
)TM((,yy)
= Σ
20)1((2) Gp D(mpo)())
(22)
ISPASP
with: for all P.QQPP
Gm == (opD(oo) P 1200)))))))))
(23)
In particular, the natural gradient of PINN can be rewritten:
0++1 - 0. - -1 ( [
9To, M V (v)p(dy)
(24)
Proof. This simple application of Proposition 3 to the space Hy Span (ō,21 1 < p P) Im dinp ToMM To conclude with the proof of Equation (24). let us note that Equation T02 can be rewritten as:
Rwo Pio [D
(110)
Specifically, we have:
((Dow) IT D[Ht] dujo Dt I1 IT dule Rxo
(111)
Since, by definition, RNo is the operator associated with the Green's 's function gHo. this directly implies that the natural gradient of PINNs:
01+1 (D) (II+ (VLo.))
can be expressed as:
IIE 12 0++1 0, 1,
gTe, VV
which concludes.
D.3 A PRACTICAL EXAMPLE: DERIVATION OF GENERALIZED GREEN'S ' FUNCTION FOR LAPLACIAN OPERATOR, BASED ON PINN'S NATURAL GRADIENT FORMULATION ON FOURIER'S 'S BASIS
We will illustrate Theorem 2 on a parametric model given by partial Fourier's series (cf. Section B.1 for the Laplace operator 17
{
H2 ( 1|- -
- 10 C)
Σー dwv
For this purpose, let us then fix N E N and consider the associated partial Fourier's Serie SN as defined in Equation 30):
RII-N.11
[2 C)
Sx
(ak)
re 10,
Σ
Σ
ag
Ca 2iml
kg
(It Amars)
-==-N
ka=-N
We will then derive according to Theorem the generalized Green's 's function of on the tangent space of Sy defined in Equation (31 namely:
E [0, 1jd 2 kzz) KE BE ([-N,
M T Span
of last line of Table 17 in Scction C.4
40

---

Published as a conference paper at ICLR 2025
To this end, let define: for all & E N
[0, 1]
C
Ck
keai)
(112)
a
and compute: for all E [[-N,, NJ I for
all
& m
&
(2TT)²
k ck:
Ck
then: for all E NNN, NJ
A 6x (2x)²
Σ
2
ck:
m==
and thus: for all ki, k2 E [
d
Gk = (A Cki
A cka) = (2Tr)
Σ
hi
Ōk
m
m-11
where k2 is the Kronecker symbol such that diz A/2
1 if and only if ki 12. This implies that:
-2
Ghi = (2TT) -4
Σ
klm
(1 0k) S
m-
yielding:
(ToM((,)))
Σ
Cbi (x) ky A (xg(
Σ
Ck, (2))
Σ ki
(1
5k kk
(22m)
Σ k2
exs( (v)
N.N1
1=
mm-1
-1
Σ N+I-NNM]'
eb(r) (x(2)
Σ
exp 24m
-i==
ka 91)
(2T)²
Σ mm=1
42
(2
11=1 k7
Ne[-NNN]I'
Finally, in order add some consistency illustration, let show for d that in the limit N - o0, we indeed find the classical Green's 's function of the Laplacian operator. In this case let us first remark, that have:
exp (2iTk( x v))
cos(2 - v))
9TaM(I,N) v) (2
Σ
K2
2T
Since |cos(2 (os(2xrkix-- 1))l 1, the serie is absolutly convergent and thus in the limit N oc, have:
C2(27(1 - 8)). with C2(2) = Σ cos cos(k2)
(x0 y) = lim graM(E, 2nT2 N
k22
From Abramowitz and Stegun 1968 page 1005), we know that for all E (0, 2m):
2²
n2
22
C2(2)
6 2
Thus, thanks to the parity of C2 we have for all (x y) (0,1):
n 72
x - vl
(2 (a y))²
y/
(x -
1)
(0 u)
2
6
4
12
2
2
which is indeed a Green function for the Laplacian in 1d.
41

---

Published as a conference paper at ICLR 2025
D.4 ILLUSTRATION OF ESTIMATED GREEN'S FUNCTION FOR PINNS
D.4.1 GREEN'S FUNCTION PLOTS OF LAPLACE EQUATION IN D
Figure 19: Green's ' function of the operator on the tangent space at initialization for Laplace equation in 2 D. Reading: the red cross on each subfigure representing point v,, the plot represents function gTo, M (1,2a))
Figure 20: Green's function of the operator on the tangent space at the end of optimization for Laplace equation 2D D. Reading: the red cross on each subfigure representing point T;, the plot represents the function GTo... M
42

---

Published as a conference paper at ICLR 2025
D.4.2 GREEN'S FUNCTION PLOTS OF HEAT EQUATION IN 1+1D
Kow
Figure 21: Green's 's function of the operator on the tangent space at initialization for Heat equation 1+1 D. Reading: the red cross on each subfigure representing a point æ;, the plot represents the function gTo ((,
M
Figure 22: Green's function of the operator on the tangent space at the end of optimization for
Heat equation in 1+1 D. Reading: the red cross on each subfigure representing a point 2;, the plot represents the function gT M.)
43

---

Published as a conference paper at ICLR 2025
E MATHEMATICAL EQUIVALENCE OF ANAGRAM AND ENERGY NATURAL GRADIENT IMPLEMENTATION (MÜLLER AND ZEINHOFER, 2023) FOR LINEAR OPERATORS
Let us begin by briefty recalling the definition of Energy Natural Gradient, as introduced Müller and Zeinhofer 2023). Given Energy (Equation in Müller and Zeinhofer (223
F(u)
Jn2 (D( 2) + (B(u) g)²
(113)
associated operators D: : (2 ( (t). B H ( R. and parametrie model Rº H. we have the following definition (Definition in Müller and Zeinhofer (2023))
Definition (Energy Natural Gradient). Consider the problem mingeR L(0), where
L( := o(u
(114)
Denote the Euclidean gradient by (L))). Then we call
((0)) == (L)(())()),
(115)
the energy natural gradient (E-NG), where GE is Gram matrix defined by: for all 1 & p.q & P
GB(0) pg D2 (uu))(() ug G0 up):
(116)
with D2 being the second derivative of E the Fréchet sense.
As noted in Equation (9) of Müller and Zeinhofer (2023), in the case where D and B are linear, Equation (16)) reduces to:for all p.4
GB P = S. DIeg,
uel v
Blee, ue](s) (s)(s)
(117)
Jan
Note that in this case, by setting and taking μ and a uniform, this corresponds exactly to the Gram matrix:for all & p. q &
Gēpa (op((D) 0. o (,(
(118)
L2(2220))
where the compound model (D, B) u and the space L. (2, a52) are those introduced in Equation 21 Now the key element lies the sentence quoted from page 6, section 4.1 1. of Müller and Zeinhofer (2023) (which is confirmed in practice by the code):
"The integrals (15) are computed using the same collocation points as the definition of the PINN loss function L in (14)."
This means that the same set of collocation points is used to evaluate the integrals defining L(0) in Equation (14) and the integrals defining G)((0) in Equation (177).
Let us fix some collocations points (x: in Q and (=:)) ) in 22. Then Equation ( 114) dis-
cretization exactly corresponds up factor 2. the scalar loss of PINNs defined in Equation
Spp Σ (D[mel(a!)
Sn
(00)
2(xP))² -())))
1
Σ
(B(uo)(a)))
g(x:")²² -(("))
25D
25B
The Euclidean gradient VL(O) is then approximated by the Euclidean gradient vi(() given by:
for all 1 <
Sp
(V((O)), =
Σ
a Dr l (x
(D(u))() -(xx)
Sp
(119)
+
0 B
u (x (B[up](a - -g(x{))
S
Let us define:
44

---

Published as a conference paper at ICLR 2025
for all < p P. for all 1 & i & Sp. and for all 1 j < Sg:
ob p.a == 2 DI u (x?):
@ pj == ()
(120)
for all < SD and for all < SB:
V ; == D(upl(=))) - -(xx))
VLg, = Blupl(x))) -(xx))
(121)
and finally:
6 = 100
2B)
v ==
P A b
A =
Sn
ISD
(122)
Isp/
SB
Thus Equation 119 can be rewritten as:
V σ Lo Lo-
(123)
But since the same set of collocation points is used to evaluate the integrals defining GE(O), we also have that Equation (116) is discretized by:
GE(0) GE(O) ZoA36.
(124)
This implies that Equation 1155 is approximated by:
A L()) = VE = GE V((0) VC.
(125)
which corresponds indeed to the update direction line iin in ANaGRAM algorithm
To conclude, should be noted that when 1) or B are not linear, then the equivalence between Equation (117) and Equation (118) not longer holds, with the result that E-NGD and ANaGRAM are no longer equivalent either.
45


Reviews:
[{"review_id": "9YXWQiVCRm", "reviewer": "Reviewer_psCC", "summary": "The paper \"ANaGRAM: A Natural Gradient Relative to Adapted Model for Efficient PINNs Learning\" introduces an innovative natural gradient optimization algorithm tailored for Physics-Informed Neural Networks (PINNs). Named ANaGRAM, this approach leverages differential geometric perspectives to optimize PINNs, aiming to enhance computational efficiency and accuracy in solving Partial Differential Equations (PDEs) in a machine learning framework.", "questions": "1. Implementing an adaptive selection strategy for the SVD cutoff factor based on the eigenvalue spectrum could make the method easier to use.\n\n2. Developing an adaptive method for selecting collocation points based on the geometry of the domain could improve convergence rates.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper's use of differential geometry to formulate a natural gradient specifically for PINNs counts as a theoretical contribution. By aligning gradient updates with the functional space geometry, ANaGRAM offers a alternative to gradient-based methods, potentially addressing some of the known limitations in PINN optimization.\n\n2 NaGRAM's computational complexity, which scales favorably with parameter and batch sizes, is an improvement over traditional natural gradient methods.\n\n3. Experimental results indicate that ANaGRAM consistently achieves lower error rates compared to baseline methods like E-NGD and standard optimizers (Adam, L-BFGS).", "weaknesses": "1. The current implementation of ANaGRAM requires manual tuning of hyperparameters, such as the cutoff factor for SVD. \n\n2. The choice of collocation points (batch points) is limited to heuristics in this study. Developing an adaptive method for selecting these points could further improve ANaGRAM's performance.\n\n3. The results indicate that the efficacy of ANaGRAM may diminish for nonlinear differential operators, where the method's equivalence with traditional approaches is less certain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1QKK26yRi2", "reviewer": "Reviewer_EAmy", "summary": "The paper proposes a new algorithm -- ANaGRAM-- that is well suited for learning PDEs with boundary conditions. The algorithm makes use of a clever manipulation of natural gradient of Amari & Douglas. The natural gradient, PINN and some elements from differential geometry are introduced. The proposed algorithm can be interpreted as follows: compute the integration of the euclidean gradient using by \"weighting\" with the appropriate Green's function and use pseudo-inverse of the differential as a pre-conditioner (Theorem 1). Experiements show that ANaGRAM learns faster than Adam, GD, E-NGD across well-known low-dimensional PDEs.", "questions": "I find the empirical investigations sufficient for this paper. It might be even better to have a high dimensional example. For example, would ANaGRAM still perform better than E-NGD when tested on 100 D Laplace eqn?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written. The proposed method is elegant and well motivated from the theory perspective. Empirical results are well-done and show good evidence for the proposed ANaGRAM.", "weaknesses": "I quite like the paper. Even though I am not an expert of PINNs at all, I am guessing that this paper may be exciting for that community. I found the math well written, which may be not very common for an ICLR submission. \n\nTypos:\n- Eq 2: $h_i \\to h_p $\n- line 147: \"his\" comp. cost $ \\to $ \"its\" comp. cost\n- line 199: maybe add what $\\Delta_{\\theta_r}$ means here (singular values) \n\nThe performance of ANaGRAM is quite similar to E-NGD for Heat eqn and 5D Laplace eqn. ANaGRAM is faster by a tiny margin. I did not find a discussion of this in the paper. Can the authors comment on this? In particular, what would be the expected advantage of  ANaGRAM over E-NGS in words?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DbAwNAYyPG", "reviewer": "Reviewer_UHpU", "summary": "This paper proposed a natural gradient based optimizer called ANaGRAM for PINNs. The key algorithmic contribution of the paper includes a novel approach to discretize the natural gradient by connecting the natural gradient with a linear regression in L2 space, which is then combined with the fact that PINNs can be regarded as a least square regression to adapt the method for PINNs. A major theoretical contribution of this is showing that the natural gradient for PINNs is actually Green’s function under some conditions. \n\nIn summary, this paper proposed a novel optimization approach which can be used to help improve the convergence of the usually ill-condition problems in PINNs.", "questions": "1. It's interesting that ANaGRAM usually has the largest variance among all methods, is there a reason for that? I was thinking it should have small variance since it's more well conditioned.\n\n2. For some experiments, it seems the difference between different methods is more stark for L2 loss compared to test Loss. Any explanation for this?\n\n3. Could you explain more about the relationship between NTK and equation 19 (line 223)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper has an interesting and supportive theoretical results for their results.", "weaknesses": "1. There are many typos 126 il conditioned -> ill condition 186 hapens -> happens 201: writes -> write 274 quadraic....\n2. The flow of the paper may be better by revisiting some parts. For example, at the end of the section 2, authors illustrate the relationship between the natural gradient and the linear regression problem. However, the section 3.1 has the title indicating the same thing. Maybe the end of section 2 can be moved to section 3.1, otherwise, I don't see the reason why section 3.1 has the current titles.\n3. The experiments are not enough. Usually people also include Burgers, Schrodinge, etc.\n4. The paper's main theoretical result is the equivalence of Green's function and natural gradient. It would be helpful to include some toy examples to justify this finding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wtvv8bzDOx", "reviewer": "Reviewer_iNHt", "summary": "This paper studies the natural gradient descent, which is theoretically and empirically believed to improve the convergence of training PINNs. Theoretical analysis on the natural gradient and training PINNs bridges connections on Green function theory.", "questions": "Please see the weakness part.\n\nOne typo: Lines 126-127: ill conditioned", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "(1) The natural gradient descent approach is theoretically sound, offering a promising solution for improving the convergence of regression problems and PINNs. Notably, while gradient descent may struggle with PINNs involving nonlinear PDEs, natural gradient descent effectively addresses this limitation.\n\n\n(2) The paper's presentation is clear and well-structured.", "weaknesses": "(1) The algorithm in the paper is not novel. The general natural GD method requires O(P^3) computational costs, due to the inversion of the Gram matrix of \\hat{\\phi} (G = \\hat{\\phi} \\hat{\\phi}^{T} is in size of P \\times P). This paper applies SVD directly on \\hat{\\phi}. Moreover, the line 6 of the algorithm is exactly equivalent to the vanilla natural GD algorithm (rewriting the pseudo-inverse with the SVD). Therefore, I do not find the novelty of the algorithm. \n\n\n(2) In Equations (18) and (19), it seems that the paper tries to explain the consistency by approximating the integral by finite sum of some observations. However, the results of Corollary 1 are relatively weak. The existence of certain points, as stated, follows naturally from standard regularity conditions (e.g., smoothness) of functions. The result is therefore not surprising and lacks substantial implications. It is better if the paper can alternatively prove that if we randomly pick P training samples as observations and with high probability, the difference between du and \\hat{\\phi} \\hat{f} is upper bounded, e.g., O(1/P^{\\alpha}), where \\alpha is a constant depending on function’s conditions and dimensionality. Such findings would be more meaningful for machine learning.\n\n\n(3) The ANaGRAM for PINNs closely resembles existing natural GD for training PINNs (https://arxiv.org/abs/2408.00573). For example, in Equation (31) of the paper, the pseudo-inverse of (J J^{T}) is equivalent to the SVD of \\hat{\\phi} in this paper. \n\n\n(4) The natural GD requires the full-batch training, which limits its adaptability to stochastic algorithms. We know that the SGD-like algorithms are widely used in deep learning due to their benefits, such as improving the generalization and bypassing spurious local minima. A more practical approach would be one that can adapt effectively to a stochastic setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XSlKCgpe0j", "reviewer": "Reviewer_ogx7", "summary": "The manuscript proposes a natural gradient method related to the ENGD optimizer proposed by (Müller and Zeinhofer). The main contributions are:\n\n- In the context of PINNs, the authors view the ansatz/parametric model as a neural network followed by the application of the PDE and boundary operator. This means that the model is defined on some parameter space $\\Theta$, taking values in the product space $L^2(\\Omega) \\times L^2(\\partial\\Omega)$ via the intermediate space $\\mathcal H$ which is related to the PDE and boundary operators of the specific problem. This mathematical trick of changing what is the model and what is the loss function allows to view PINNs as a standard regression problem. This viewpoint motivates the authors to use an $L^2(\\Omega)\\times L^2(\\partial\\Omega)$ based natural gradient method, which agrees with ENGD for linear PDEs but performs better than ENGD for nonlinear PDEs\n\n- The second contribution is on the numerical linear algebra level. Instead of solving the normal equations of the regression, the authors propose to use an SVD directly for the Jacobian of the model. Here, when I write Jacobian I mean the discretized differential of the parametrization, which is denoted by $\\hat \\phi_{\\theta(t)}$ in the manuscript. This gives an improvement in computational complexity from $\\mathcal O(P^3)$ for solving the normal equations, to $\\mathcal O (\\min(N^2P, P^2N) )$, as the Jacobian is of shape $(N, P)$, where $P$ is the dimension of the parameter space $\\Theta$ and $N$ is the number of sampling points.", "questions": "- Can the authors add a reference for the computational complexity of the SVD for the rectangular solve?\n\n- Why is the proposed method performing better than ENGD for the linear equations? I would expect similar performance as it is the same algorithm (up to numerical linear algebra).\n\n- it is very common to include damping into natural gradient methods (adding a scaled identity term to the square Gramian). Can this be realized in this setting?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The viewpoint outlined above leads to a performant method, that works better than ENGD for nonlinear PDEs.\n\n- The improvement in complexity for the linear solve is helpful.", "weaknesses": "- The viewpoint of absorbing the differential operators into the model can also be understood as a Gauss-Newton method in function space compare to (Jnini et al, arXiv:2402.10680). Moreover, it leads to the standard Gauss-Newton algorithm in parameter space. The manuscript thus does not provide a novel algorithm, but a novel derivation of a known algorithm. While I like this idea and it is certainly a contribution, I think it is not significant enough for an exceptional venue such as ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)