Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks
Authors: Jules Berman
Benjamin Peherstorfer
Venue: NeurIPS 2023 spotlight (2023)
Keywords: numerical methods
deep networks
evolution equations
scientific computing
partial differential equations
model reduction
TL;DR: We propose Neural Galerkin schemes that update randomized sparse subsets of parameters. We are up to two orders of magnitude more accurate at a fixed budget and up to two orders of magnitude faster at a fixed accuracy than dense updates.

Abstract: Training neural networks sequentially in time to approximate solution fields of time-dependent partial differential equations can be beneficial for preserving causality and other physics properties; however, the sequential-in-time training is numerically challenging because training errors quickly accumulate and amplify over time. This work introduces Neural Galerkin schemes that update randomized sparse subsets of network parameters at each time step. The randomization avoids overfitting locally in time and so helps prevent the error from accumulating quickly over the sequential-in-time training, which is motivated by dropout that addresses a similar issue of overfitting due to neuron co-adaptation. The sparsity of the update reduces the computational costs of training without losing expressiveness because many of the network parameters are redundant locally at each time step. In numerical experiments with a wide range of evolution equations, the proposed scheme with randomized sparse updates is up to two orders of magnitude more accurate at a fixed computational budget and up to two orders of magnitude faster at a fixed accuracy than schemes with dense updates.

Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks
Jules Berman
Benjamin Peherstorfer
Courant Institute for Mathematical Sciences
Courant Institute for Mathematical Sciences
New York University
New York University
New York, NY 10012
New York, NY 10012
jmb1 gmb1174@nyu.en
pehersto@cims. wttpssttooiimmmmu...oomm
Abstract
Training neural networks sequentially in time approximate solution fields of time- dependent partial differential equations can be beneficial for preserving causality and other physics properties; however, the sequential-in -in-tiiil----eee training is numerically challenging because training errors quickly accumulate and amplify over time. This work introduces Neural Galerkin schemes that update randomized sparse subsets of network parameters at each time step. The randomization avoids overfitting locally in time and helps prevent the error from accumulating quickly over the sequential- intime training, which is motivated by dropout that addresses similar issue overfitting due to neuron co-adaptation. The sparsity of the update reduces the computational costs of training without losing expressiveness because many of the network parameters are redundant locally at each time step. numerical experiments with a wide range of evolution equations, the proposed scheme with randomized sparse updates is up to two orders magnitude more accurate at a fixed computational budget and up two orders of magnitude faster at fixed accuracy than schemes with dense updates.
Introduction
In science and engineering, partial differential equations (PDEs) are frequently employed to model the behavior of systems of interest. For many PDEs that model complicated processes, analytic solution remains elusive and so computational techniques are required to compute numerical solutions.
Global-in-time training There have been many developments using nonlinear parameterizations based on neural networks for numerically approximating PDE solutions. These include techniques such as Deep Galerkin Method [46), illysics-inforeed neural networks (PINNs) [41], and others well as early works such as [1] 42] In most of these methods, a neural network is used to represent the solution of (time-dependent) PDE over the whole space-time domain. For this reason they termed global-in-time methods in the following. To approximate the solution, the neural network is trained to minimize the PDE residual on collocation points sampled from the space- time domain, which requires solving large-scale optimization problem that can be computationally expensive. Additionally, the solutions learned by global-in- time methods can violate causality, which become an issue for complex problems that rely preserving physics (27]. We note that neural networks have been used for approximating PDE solutions in various other ways, such as learning
4 21 53 15:
specific component functions 23
31 43), finding closure models 2 25 550, de-noising [44] and
32 18 However, we are interested in this work in using neural networks for directly approximating PDB solutions.
for surrogate modeling 333
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

eeqquential-in-tine training with the Dirac-Frenker variational principle In this work, we follow the Strac-Frenkel variational principle, which has been used for numerical methods in the field of quantum dynamics for a long time [00 17 26 34 28 and for dynamic-low rank and related solvers [2
22 Instead of globally approximating a PDE solution in time, the Dirac-Frenkel variational principle allows a sequential-in can-ntiill----tmeee training that adapts a nonlinear parameterization, such as a neural network, over time. In contrast to classical numerical methods in vector spaces, the approximate solution in Dirac-Frenkel schemes is allowed to depend nonlinearly on its parameters and so to lie on a smooth manifold. The update to the nonlinear parameterization is calculated at each time step according the orthogonal projection of the dynamics onto the tangent space of the manifold induced by the nonlinear parameterization. The Dirac-Frenkel variational principle has been adapted for the nonlinear approximation of PDEs with neural networks. In particular [14] [ formulate a sequential-in- in-timmeeeeee method based on the Dirac-Frenkel variational principle.
The neural network represents the PDE solution at point in time. The time-dependence then arises by allowing the parameters- the weights and biases of the network-to to vary in time. The network parameters are then evolved forward according to the time dynamics which govern the PDE. This is in contrast to global-in-time methods, in which time enters the network as an additional input variable. By construction, approximate solution obtained with sequential-in-tie iimttimeeeeeeeeeeeeeeee--eeeee method is causal, in that the solution at future times depends only on the solution at the current time.
Although these methods have demonstrated success in solving various PDEs [4 14 54
16 g
30
there are open challenges: First, the crquential-in-timme training is prone to overfitting which can lead to a quick accumulation of the residual over time. Second, the local training step has to be repeated each time step, which can be computationally costly, especially with direct solvers that have costs increase quadratically with the number of network parameters. The work [16] proposes to address the two issues by using iterative solvers, instead of direct ones, and by re-training the network occasionally over the sequential-in-time training. We show with numerical experiments below that the re-training of the network can be computationally expensive. Additionally, the performance of iterative solvers depends on the condition of the problem, which can be poor in the context of sequential in--time training.
Our approach and contributions: Randomized sparse updates for schemes based on the Dirac- Frenkel variational principle We build on the previous work in sequential-in-time methods following similar set up as 7 based on the Dirac-Frenkel variational principle. Where all previous methods solve local training problems that update every parameter of the network at each time step, we propose modification such that only randomized sparse subsets of network parameters are updated at each time step:
(a) The randomization avoids overfitting locally in time and helps preventing the error from accumulating quickly over the sequential-in-time training, which is motivated by dropout that addresses similar issue of overfitting due to neuron co-adaptation.
(b) The sparsity of the updates reduces the computational costs of training without losing expressiveness because many of the network parameters are redundant locally at each time step.
Our numerical experiments indicate that the proposed scheme is up to two orders of magnitude more accurate at a fixed computational cost and up to two orders of magnitude faster at fixed accuracy.
We release our code implementation here: httt::///giccabbb.co/jjlleeeeere
2 Sequential-in-time training for solving PDEs
2.1 Evolution equations, Dirac-Frenkel variational principle, Neural Galerkin schemes
Given a spatial domain X' C Rd and time domain T= =
(0, C TR, we consider a solution field T X + IR so that u(t, -) : X + R is in a function space 21 at each time with dynamics
0.u(1, x) = f(x u) u(0, x) (=)
for (1,z) E T x X for x E X
(I)
where up l is the initial condition and f can include partial derivatives of u to represent PDEs. We focus in this work on Dirichlet and periodic boundary conditions but the following approach

---

f(0())
We propose Neural Galerkin schemes that update ran- domized sparse subsets of network parameters with the Dirac-Frenkel variational principle. Randomization avoids overfitting locally in time, which leads more accurate approxi- mations than dense updates. Sparsity reduces the computational costs of training without losing expressive- ness because many parameters are redundant locally in time.
Figure 1:
tangent space
{ (())
spanning set
Me
sparse subset
can be applied with, e.g., Neumann boundary conditions as well 77. One approach for imposing Dirichlet boundary conditions is by choosing parameterizations that satisfy the boundary conditions by definition [88.
Sequential-in-time training methods approximate u with nonlinear parameterization such neural network x o + IR, where the parameter vector (tt) C I2 depends on time f; the parameter ((t) has components and enters nonlinear in the second argument of û. The residual of (T) at time
r(x: (1), (()) Voñ(æ; (()) (()) - (ær, a(; 0())))
where we applied the chain rule to 0,6(-; (()) to formally obtain (tt). Methods based on the Dirac-Frre Frenk-l variational principle [10 17 34 seek O(t) such that the residual norm is minimized,
which leads to the least-squares problem
min Vou ((())))))))))))))))))))))))))))) ((a))))))a ((1)
in the (X)) norm || (1)) over X. The least-squares problem 3 gives a (t) such that the
residual is orthogonal the tangent space ( (()) of the manifold MM {u(: 0) 0 E θ} induced by the parameterization ú; see Figure 1 Schemes that solve (3) over time have also been
termed Neural Galerkin schemes because 3 can derived via Galerkin projection as well.
The tangent space at a(-; (t)) is spanned by the spanning set {0g, ú(:; 0(t)) which are the component functions of the gradient Vo(e) (()); it important to stress that (ög, ú(: (()) is not necessarily a basis of the tangent space because it can contain linearly dependent func- tions and be non-minimal. The least-squares problem (3) can be realized by assembling a ma- trix whose columns are the gradient sampled at n > P points ZI...., En E X resulting in (((()) (Vec O())),... Verc (xx; (()))² E Ræxx,, which a batch Jacobian ma- trix to which we refer to as Jacobian for convenience in the following. Additionally, we form the right-hand side vector (0(()) [f 0()), ... (m, (()))= E RR" and thus the least-squares problem
min (((()))) ((()))((()((((()))))) (((()) 0(t)
The choice of the points 31. ......... Fn critical so that solutions of 4 are good approximations of solutions of (3); however, the topic of selecting the points 21.... Tn goes beyond this work here and so we just note that methods for selecting the points exist 717 and that we assume in the following that we select sufficient points with n > ensure that solutions of (4) are good approximations of solutions of 3
2.2 Problem Formulation
Challenge I: Parameters are redundant locally in time. Typically, parameterizations based on deep neural networks lead to Jacobian matrices (((()) that low rank in the least-squares problem (4); see, e.g., (37) and Figure aa.. In our case, a low-rank matrix J(O(t)) means that components in (t) are redundant, because we assume that the samples æ1...., In are sufficiently rich. Even
if ((O(t) is low rank and thus the components in (t) are redundant, the problem (4) can still be

---

Time
Sequentil time
Spectrum j(((t))
10
Direct fit
Columns (((()
0.05
10-
Resiudal
1=4
10
10"
0.00
10
103
-005
0
1000
2000
3000
0
2n
Index
Time
Spatial Domain
(a) low-rankness of Jacobian
(b) residual growth
(c) co-adaptation of neurons
Figure 2: (a) Jacobians that have low rank locally in time imply that there are redundant parameters in the neural network, which motivates the proposed sparse updates that lead to speedups without losing expressiveness.
(b) The residual grows quickly with sequential-in-iime training (and dense updates). This is not due to a limitation with the expressiveness of the network because directly fitting the network to solutions indicates that there exist other parameters that can lead to lower residuals. (c) Sequential-in-time training (with dense updates) results in co-adapted neurons as indicated by the highly correlated columns of the J matrix. Plots for experiment with Allen-Cahn equation (Sec.
solved with standard linear algebra methods such as the singular value decomposition (SVD) because they compress the matrix J (t))) and regularize for, e.g., the minimal solution; however, the costs of performing the SVD to solve 44) scales as (npp), and thus is quadratic in the number of parameters p. This means that a redundancy in (()) of factor two leads to 4x increase in the computational costs. Note that the problem typically is poorly conditioned because J(O())) is low rank, which makes the direct application of iterative solvers challenging.
Challenge 2: Overfitting leads to high residual over time. The residual from solving rapidly increase over time which in turn increases the overall error. This indicates that the tangent space along the trajectory O(t) becomes ill suited for approximating the right- side vector (() in (4). We compare the residual of the least-squares problem 4 that is obtained along trajectory of training with the schemes above to the residual of (4) from a network
4 can
(t) from sequential-in-tiinn
that is fit to the true solution at each point in time; details in Appendix[A.]] As shown in Figure2Ib) a lower residual is achieved by the network that is fit to the true solution.
We aim to understand this phenomenon through the lens of overfitting: the sequential-in-ting training can be thought as successive fine-tuning, in the sense that at each time step we must make a small update to our parameters to match the solution at the next time step. However, fine-tuning is well known to be prone to over-fitting and model degeneration [3] In the setting considered in this work, overfitting means that the representation 6(= (t)) does not generalize well to the next time step. Not generalizing well means that a local change to ((t) is insufficient to move â(- (()) according to the desired update given by O(L) to match the right-hand side (0((), which implies that a large residual is incurred when solving 4 A common approach to prevent overfitting is dropout 47 especially when applied to fine-tuning tasks with dropout variants proposed in 3 29 while other approaches are formulated specifically around sparse updates [4] 52 Dropout motivated by the observation that dense updates to parameters in neural networks can cause overfitting by leading neurons to co-adapt. Typically, co-adaptation characterized by layer-wise outputs with high covariance ( In the case of sequential-in-tiin in-mme training with the schemes discussed above, co-adaptation implies the columns of the Jacobian matrix ((((t)) are correlated and thus close to linearly dependent. So as neurons co-adapt, component functions of the gradient become redundant and may be less suited for approximating (((()) causing the high residual for the least-squares problem; see Figure D... This could also be characterized by the ill conditioning issue pointed out in [66. We see empirical evidence of co-adaptation in Figure c), where we plot component functions of the gradient and see that they are strongly correlated at the end time T.

---

3 Randomized Sparse Neural Galerkin (RSNG) schemes
We introduce randomized sparse Neural Galerkin (RSNG) schemes that build on the Dirac-Frenkel variational principle to evolve network parameters O(t) sequentially over time t but update only sparse subsets of the components of (t) and randomize which components of (t) are updated. The
sparse updates reduce the computational costs of solving the least-squares problem 4 while taking
advantage of the low rank structure of (() which indicates components of the time derivative ((1) are redundant and can be ignored for updating (t)) without losing expressiveness. The randomization of which components of (t) are updated prevents the overfitting described above.
3.1 Randomized sketch of residual
To define the sketch matrix St, e Cp be the p-dimensional canonical unit vectors so that ep
has entry one at component and zeros at all other components. We then define independent and identically distributed random variables
(t) that depend on time t. The distribution of (.)t) is n, which is supported over the set of indices {l.....pp p}. The random matrix S, of size p X is then defined as St = (e) (e)] The corresponding sketched residual analogous to (2) is
(w: c (t), 02 (t) Voū(x; ((t)),,,))) f(x, a(: (()) T
(5)
where now (t) E R' is of dimension s < p.
3.2 Projections onto randomized approximations of tangent spaces
Using the sketch matrix S., obtain from the spanning set (dn, ú(: 0(1)) of component func- tions of (- ( (()) a subset {0o..cs i(: (());= with 8 functions. The set (0o "(: (()) spans at least approximately the tangent space at ú(.; (t)) of MMe but has only & p elements. The motivation is that the full spanning set (0g, á(= ((t))
1=1 contains many functions that are close to linearly dependent (Jacobian is low rank) and thus sub-sampling the component functions still gives reasonable tangent space approximations that preserves much of the expressiveness; see Figure] While the low rankness depends on the complexity of the problem and parametrization, we observe low rankness in all our examples; see Appendix [A. for further discussion.
We now introduce a least-squares problem based on the sparse spanning set (0 a(= ((t) that is analogous to the least-squares problem problem based on the full spanning set given in We seek (t) E R" with s components that solves
min
| ((1)),;(())))
(6)
0. (t)ER
To obtain ((t) to update (t), we set ((1) = (t). Thus, the components of (() that are selected by St are set to the corresponding value of the component of (L) and all other components are set to zero, which means that the corresponding components of O(1) are not updated. We can realize (6) the same way as the full least-squares problem in (4) by using the full Jacobian matrix and S, to define the sparse Jacobian matrix (f(t)) = (@ St),, and the right-hand side vector f((()))
analogous to Section to obtain the discrete least-squares problem
min (J,(O())) - (.(1)
(1)00(1)000)--0(0003)) (((()))
The choice of the distribution я is critical and depends on properties of the Jacobian matrix (())))) Distributions based on leverage scores provide tight bounds with regard to the number of columns one needs to sample in order for the submatrix to be close to an optimal low rank approximation of the full matrix with high probability [13]. But these distributions can be expensive to sample from. Instead, uniform sampling provides a fast alternative.
The number of columns will not grow too quickly if the full matrix is sufficiently incoherent [19. This means some columns do not carry a disproportionate amount of information relative to other columns. We numerically see that in our case the Jacobian matrix J(O())) is sufficiently incoherent. Thus we can choose uniform distribution over the set of indices (1,...,,) P) to get the benefits of low rank approximation in a computationally efficient way.
5

---

Algorithm Randomized Neural Galerkin scheme with sparse updates
Fit parameterization ii-; 0 (())) initial condition Wo to obtain ((0)
for ......,. do Draw realization of sketching matrix Sş described in Section 3.1
Solve for sparse update 20 ()11)
_ with least-qquares problem 9
Lift sparse update (0(k-1) St S:A0(-11>
Update 0 0 (k--1) + (k)
end for
3.3 Discretizing in time
We discretize the time interval T with K E regularly spaced time steps = to <h <
tx = with öt = t t for l = 1,...,Re K. At time to, we obtain ( E EP by fitting the initial condition uo. We then update
( ( (*-1)) StA0(k-1)
for 1: .,... K so that (b)) is the time-discrete approximation of ((tx) and thus ú(:; (())) approx- imates the solution u at time fx. The sparse update ( 1) approximates è (t and is obtain by the time-discrete counterpart of 7 which is given by
min 20
(0(k( 10
(((((-1))11)
if time is discretized with the forward Euler method. Other discretization schemes be used as well, which then lead to technically more involved problems (9) that remain conceptually similar though. The sparse update is lifted to (k-1 S,A0 so that the update (8) can be computed.
3.4 Computational procedure of RSNG
We describe the proposed RSNG procedure in algorithmic form in Algorithm I We iterate over the time steps = 1,..., K. At each time step, we first sketch the Jacobian matrix by creating a submatrix from randomly sampled columns. Notice that S₃ need not actually be assembled as its action on the Jacobian matrix can be accomplished by indexing. We then solve the squares
problem given in ( using our sketched Jacobian to obtain A (k-1, A direct solve of this system dominates the computational cost of making time step and scales in (mns²) time. The components of (k-1) corresponding to the indices that have not been selected are filled with zeros. We then update the parameter (k--1) to (k) via (k-1)
The whole integration process scales as 0 T ns²) in time.
57
4 Numerical experiments
We demonstrate RSNG on a wide range of evolution equations, where speedups of up to two orders of magnitude are achieved compared to comparable schemes with dense updates. We also compare to global-in-time methods, where we achieve up to two orders of magnitude higher accuracy.
4.1 Setup and equations
Examples We now describe the details of the PDEs that we use to evaluate our method. We choose these particular setups to test our method on a diverse set of challenges including problems with global and local dynamics and solutions with sharp gradients and fine grained details. For visualization of the solutions of these equations see the Appendix A..4
A..4
Reaction-diffasion problem modeled by Allen-Cahn (AC) equation: The Allen-Cahn equation models prototypical reaction diffusion phenomena and given as,
0,u(t,z) = cd.. x) + ult, x) u(t,x))3.
We choose 5 x 10 with periodic boundary condition X [0, 2n) and initial condition,

---

vo(x) (anh(2sin(zr)) exp( 2)) + exp( -.2)) + 5.4)
This initial condition results in challenging dynamics that are global over the spatial domain. Flows with sharp gradients described by Burgers' equation: The Burgers' equation given by, @cu(t, x) = u(t x)- u(t, x)),u(t,
We choose c = x 10-3, with periodic boundary condition X' = [-1, and initial condition,
uo(z) = (1 - z²) exp( + 0.5)²).
The corresponding solution field has sharp gradients that move in the spatial domain over time, which can be challenging to approximate.
Charged particles in electric fild:: The Vlasov equation describes the time evolution of collisionless charged particles under the influence of an electric field. The equation models the distribution of such particles in terms of their position and velocity. We consider the case of one position dimension and one velocity dimension, making our domain S R2. The equation is given by,
0,u(t, 2, v) -vû, (t, v) +
where x' is the position, is the velocity and Φ is the electric field. We consider the case with periodic boundary condition X' [0, 2x) x 1-6, 6) and initial condition,
1
un uo
exp(
2m
2
with fixed electric field 6(x) = cos(z). This particular setup evolves into distribution with fine grained details along a separatrix surrounding the potential well.
Setup We parameterize with a feed-forward multi-layer perceptron. All our networks use linear layers of width 25 followed by non- non-linear activation functions, except the last layer which has no activation and is of width so that ú(x (()) R. To vary the number of total parameters p, we vary the depth networks ranging from 3-7 layers. We use rational activation functions which in our experiments allowed for fitting initial conditions faster and more accurately than standard choice such as tanh or ReLU (5) To enforce periodic boundary conditions, we modify the first layer so that it outputs periodic embeddings as in 77] for details see Appendix [A.5] The periodic embedding ensures that the boundary conditions are enforced exactly. For additional details on enforcing other types of boundary conditions (e.g. Dirichlet and Neumann) exactly in neural networks see [12 48) We sample points from the domain on an equidistant grid. For time integration we use RK4 scheme with fixed time step size. The time step sizes are 5e-3, 1e-3, 5e-3 and we integrate up end time 4, 4, and for the Allen-Cahn, Burgers', and Vlasov equations, respectively. All error bars show +/- two standard errors over three random realizations which results in different sketching matrices at each time step. Relative errors are computed over the full space-time domain, unless the plot is explicitly over time.
All gradients and spatial derivatives are computed with automatic differentiation implemented in JAX (6) All computations are done in single precision arithmetic which is the default in JAX. All runtime statistics were computed on the same hardware, a Nvidia Tesla V100 w/ 32 GB memory. All additional hyperparameters are described in Appendix A.S
4.2 Results
RSNG achieves higher accuracy than schemes with dense updates at same computational costs In Figure 3 we plot the relative error over time. The curves corresponding to "dense updates" use layer network and integration is performed using dense updates. For RSNG, we use 7 layer network and integrate with sparse updates, setting the number of parameters we update, s, equal to the total number of parameters in the layer network and thus equal to the number of parameters that are updated by "dense updates." Thus, the comparison is at a fixed computational cost. The error achieved with RSNG one to two orders of magnitude below the error obtained with dense updates, across all examples that we consider. In Figure aa).
that we increase the network size,
7

---

10-3
10-2
updates
Relative Error
Relative Error
10-3
Relative Error
RSNG
10-4
10-3
10"4
10:3
10-4
Dense updates RSNG
10.5
Dense updates
RSNG
Time
Time
Time
(a) Allen-Cahn
(b) Burgers'
(c) Vlasov
Figure 3: We plot the relative eπrоr over time for RSNG versus dense updates at = 757. We see RSNG leads orders of magnitude lower errors than dense updates for the same costs.
Equation
Equation
Allen-Cahn
10-3
Allen-Cahn
Relative Error
Burgers
Relative Error
Burgers
1
Vlasov
Viasov
10-4
10-5
10-5
3500
3385 (dense)
(dense)
757 out of
S out of 3385
(a) fix sparsity 8, vary #network parameters p
(b) fix #network parameters p, vary sparsity s
Figure 4: (a) RSNG benefits from the additional expressiveness of larger networks (larger p) while only using fixed number of parameters (fixed s) at each time step. (b) As we decrease the number of parameters s in the sparse update, but keep the total number of parameters p of the network the same, we achieve lower errors than dense updates. Thus, RSNG outperforms dense updates while incurring lower computational costs. Error bars generated over random sketch matrices, St.
the relative error decreases as the sparse updates allow us to exploit the greater expressiveness of larger networks while incurring no additional computational cost computing (9). But note that
increasing the size of the full network will make computations of (()) and f(0) more expensive because of higher costs of computing gradients. However, for the network sizes that consider in this work, this effect is negligible compared to the cost of solving (9).
RSNG achieves speedups of up two orders magnitude In Figure 5ta), we compare runtime of RSNG to the runtime of scheme with dense updates that uses direct solver and to the runtime scheme with dense updates that uses an iterative solver as proposed in [16]. The time is computed for Burgers' equation and the sparsity of RSNG is chosen such that all methods reach comparable level error. We find that RSNG faster than direct solves with dense updates by two orders of magnitude and faster than the iterative solver by one order of magnitude.
The results show that while using an iterative solver as in 16 does speed up the method relative to direct solves with dense updates, it can still be quite slow for networks with many parameters Additionally, convergence the iterative method given [66] requires a number hyperparameters to be chosen correctly, which may require expensive search or priori knowledge about the solution. Note that our RSNG method does not preclude the use of iterative method to speed up the least-squares solves further.

---

39600
10.3
Method Iterative refits
Refits
RSNG
Runtime (s)
10
Relative Error
Refits 20
Refits 40
10a
1045
5= 125
10
s 200
600
800
Iterative
RSNG
1500
4500
Time
(a) runtime at 2 relative error with 125
(b) error versus runtime
Figure 5: RSNG has lower computational costs than dense updates with direct and iterative least-
squares solvers. Plots for numerical experiment with Burgers' equation
To address the overfitting problem, the work [16] refits the network to the current approximate solution from a random initialization periodically during time integration. In Figure[3]b] we show the relative enror versus the runtime for the iterative solver with various numbers of refits and for RSNG at different sparsity s. While refitting the network can reduce the relative error, incurs a high computational cost. By contrast, for appropriate sparsity s, RSNG outperforms the method given in 16 in both speed and accuracy.
Varying sparsity s at fixed number of total parameters p in network We now study the effect of varying the sparsity (i.e., number of parameters updated by sparse updates) for fixed network of total size in this case a 7 layer network. In Figure we see that for a network of fixed size, reduce the relative error by about 2-3x when compared dense updates. This is notable as the computational costs decrease quadratically with 8. Thus, the combination of sparsity deliver both improved performance and lower computational
sparse updates
and randomized updates in RSNG cost. We see that at the beginning, when the number is too small, the expressiveness sulfers and the error becomes large. This is because if is less than the rank of the dense Jacobian then the sparsified Jacobian will necessarily have less representational power. However, we stress that RSNG robust with respect in the sense that for wide range of s values the error is lower than for dense updates.
The high error when performing dense updates p in Figure (b) for Allen-Caie and Burgers' equation due to the overfitting problem described in Section 2.2 As updates become denser, the method is more likely overfit regions of the parameter space in which the Jacobian, [()), is ill suited for approximating the right-hand side f at future time steps (see Section 2 We can see this explicitly Figure where we plot the residual over time for sparse and dense updates on the Allen-Cahn equation. Initially, the dense updates lead to lower residual. This makes sense as they begin the same region of parameters space. But the methods navigate to different regions of parameters space, we see RSNG begins to incur a lower residual relative to dense updates. This indicates that RSNG ameliorates the problem of overfitting and so leads lower residual as shown in Figure b)).
Comparison with global-in-time methods We compare our method to global-in-time methods which aim globally minimize the PDE residual over the entire space-time domain. We compare to the original PINN formulation given in [11 Additionally we compare to a variant termed Causal PINNs, which impose a weak form of time dependence through the loss function [5T). We select this variant as it claims to have state of the art performance among PINNs on problems such as RSNG method achieves
Table I we see that our sequential-in-te higher accuracy by at least one order of magnitude compared to PINNs. Additionally, in terms of computational costs, RSNG outperforms both PINN variants, their global-in-time training expensive and requires many residual evaluations. We note that the training time of PINNs is directly
Allen-Cahn equation.

---

Figure 6: Plot (a) shows the residual of dense and sparse up- dates at early time steps. Ini- tially, dense updates must have lower residual as JS, spans subspace of the tangent space given by J. But in plot (b), we see that after a few time steps, dense updates overfit and the residual grows quicker than with sparse updates.
10
Residual
Residual
10
5/ 3585
S/ 3385
1200
1200
10
10-5
3385
0.0
0.2
Time
Time
(a) residual at early times
(b) residual over full time
PDE
Method
2 Relative Error
Time(s)
Allen-Cahn
PINN
6,85e 2
841
N/A
Allen-Cahn
Causal PINN
3.84c
3060
N/A
Allen-Cahn
RSNG
1.66c-5
776
800
Allen-Cahn
RSNG
5.34e-5
63
150
Burgers'
PINN
2.34e 3
3451
N/A
Burgers'
Causal PINN
5.19e
23027
N/A
Burgers'
RSNG
6.07e
2378
800
Burgers'
RSNG
2.05c
188
125
Table 1: The sequential-in-tee time training with RSNG achieves about one order magnitude higher accuracy than global-in -nttimmee methods in examples. Details training in Appendix AA.6)
dependent on the number of optimization iterations and thus they can be trained faster if one willing to tolerate even higher relative errors.
5 Conclusions, limitations, and future work
In this work, introduced RSNG that updates randomized sparse subsets of network parameters in sequential-in-tinn training with the Dirac- Freac-lrenkel variational principle to reduce computational costs while maintaining expressiveness. The randomized sparse updates are motivated by redundancy of the parameters and by the problem of overfitting. The randomized sparse updates have a low barrier of implementation in existing sequential-in-tine solvers. The proposed RSNG achieves speedups of up to two orders of magnitude compared to dense updates for computing approximate PDE solution with the same accuracy.
Current limitations leave several avenues for future research: first, as discussed in 3.2 uniform sampling is only appropriate when the Jacobian matrix is of low coherence. Future work may investigate more sophisticated sampling methods such as leverage score and pivoting elements of rank revealing QR. Second, there problems for which overfitting with dense updates is less of an issue; e.g., the charged particles example in our work. Note that due to the sparsity of the updates, RSNG still achieves speedup compared dense updates for the accuracy for this example though. However, more work is needed to better understand and mathematically characterize which properties of the problems influence the overfitting issue.
We make a general comment about using neural networks for numerically solving PDEs: The equations discussed in this paper are standard benchmark examples used in the machine-learning literature; however, for these equations, carefully designed classical methods can succeed and often have lower runtimes than methods based on nonlinear parameterizations [20] (3). While these equations provide an important testing ground to demonstrate methodological improvements, future work will extend these results to domains where classical linear methods struggle, e.g., high-dimensional problems and problems with slowly decaying Kolmogorov n-wddh [1]] 40
We do not expect that this work has negative socictal impacts.
10

---

Ackmowledgements The authors were partially supported by the National Science Foundation under Grant No. 2046521 and the Office of Naval Research under award N00014-22-1-2728. This work was also supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
References
[1] William Anderson and Mohammad Farazmand. Evolution of nonlinear reduced-order solutions for PDEs with conserved quantities. SIAM Journal on Scientific Computing, 44(1):A176-A197, 2022.
[2] Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P. Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116631):15344-1544 2019.
[3] Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. Regu- larization techniques for fine-tuning neural machine translation. CoRR, abs/1 2017.
[4] Jens Berg and Kaj Nyström. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing, 317:28 41, nov 2018.
[5] Nicolas Boulle, Yuji Nakatsukasa, and Alex Townsend. Rational neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14243-14253. Curran Associates, Inc., 2020.
[6] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. 2018.
[7] Bruna, B. Peherstorfer, and E. Vanden-Eijndenn Neural Galerkin scheme with active learning for high-dimensional evolution equations. Journal of Computational Physics, 2023. (accepted; preprint arXiv:2203.013660
[8] Peter Yichen Chen, Maurizio M. Chiaramonte, Eitan Grinspun, and Kevin Carlberg. Model reduction for the material point method via an implicit neural representation of the deformation map. Journal of Computational Physics, 478: 111111900 1908, 2023.
[9] Michael Cogswell, Faruk Ahmed, Ross B. Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[10] A. M. Dirac. Note exchange phenomena in the thomas atom. Mathematical Proceedings of the Cambridge Philosophical Society, 26(3):376-385, 1930.
[11] M. W. M. G. Dissanayake and N. Phan-Thien. solving partial differential equations. Communications in Numerical Methods in Engineering, 1953):195-011, 1994.
based approximations for
[12] Suchuan Dong and Naxian Ni. A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks. Journal of Computational Physics, 435:1 10242, jun 2021.
[13] Petros Drineas, Michael W. Mahoney, and S. Muthukrishnan. Relative-error CUR matrix decompositions. CoRR, abs/0708. 2007.
[14] Yifan Du and Tamer A. Zaki. Evolutional deep neural network. Physical Review E, 104(4), oct 2021.
[15] Weinan E, Jiequn Han, and Arnulf Jentzen. Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning. Nonlinearity, 35(1):278, dec 2021.

---

[16] Mare Anton Finzi, Andres Potapczynski, Matthew Choptuik, and Andrew Gordon Wilson. A stable and scalable method for solving initial value PDEs with neural networks, In The Eleventh International Conference on Learning Representations, 2023.
[17] J. Frenkel. Wave Mechanics, Advanced General Theory. Clarendon Press, Oxford, 1934.
[18] Stefania Fresca and Andrea Manzoni. POD-DL-ROM: Enhancing deep learning-based reduced order models for nonlinear parametrized PDEs by proper orthogonal decomposition. Computer Methods in Applied Mechanics and Engineering, 388: 14181, 2022.
[19] Alex Gittens. The spectral norm error of the naive Nystrom extension. arXiv, 11 10.5305, 2011.
[20] Tamara G. Grossmann, Urszula Julia Komorowska, Jonas Latz, and Carola-Bibian Schönlieb. Can physies-informed neural networks beat the finite element method? arXiv, 2302.04107, 2023.
[21] Jiequn Han, Amnulf Jentzen, and Weinan E. Solving high- -igeedimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 155(34)85005- 8510, 2018.
[22] Jan S. Hesthaven, Cecilia Pagliantini, and Gianluigi Rozza. Reduced basis methods for time- dependent problems. Acta Numerica, 31:265-344 345, 2022.
[23] Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving for high-dimensional committor functions using artificial neural networks. Research in the Mathematical Sciences, 6(1):1, Oct 2018.
[24] Othmar Koch and Christian Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications, 29(2):434-454, 2007.
(25) Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan computational fluid dynamics. Proceedings of the
Hoyer. Machine National Academy of Sciences, 1 118(21), 2021.
[26] Peter Kramer and Marcos Saraceno. Geometry of the time-dependent variational principle in quantum mechanics. In Lecture Notes in Physics, volume 140. Springer, 1981.
[27] Aditi S. Krishnapriyan, Amir Gholami, Shandian Zhe, Robert M. Kirby, and Michael W. Mahoney. Characterizing possible failure modes in physics-infeee inffsee-informed neural networks. CoRR, abs/2109.01050, 2021.
[28) Caroline Lasser and Christian Lubich. Computing quantum dynamics in the semiclassical regime. Acta Numerica, 29:229-401, 2020.
[29] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. CoRR, abs/1909, 12299, 2019.
[30] Kookjin Lee and Kevin T. Carlberg. Model reduction of dynamical systems on nonlinear mani- folds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973, 2020.
[31] Qianxiao Li, Bo Lin, and Weiqing Ren. Computing committor functions for the study of rare events using deep learning. The Journal of Chemical Physics, 151(5):05411,, 2019.
[32] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,, 2021.
[33] Lu Lu, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-2D) 2222 Mar 2021.
[34] Christian Lubich. From Quantum to Classical Molecular Dynamics: Reduced Models and Numerical Analysis. EMS Press, 2008.
12

---

[35] E. Musharbash, F. Nobile, and T. Zhou. Error analysis of the dynamically orthogonal approxima- tion of time dependent random PDEs. SIAM Journal on Scientific Computing, A781)1776-A810, 2015.
[36] K. Otness, A. Gjoka, J. Bruna, D. Panozzo, B. Peherstorfer, T. Schneider, and D. Zorin. An extensible benchmark suite for learning to simulate physical systems. In NeurIPS 2021 Track Datasets and Benchmarks, 2021.
[37] Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization, adap- tation and low-rank representation in neural networks. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 581-585, 2019.
[38] B. Peherstorfer. Model reduction for cansport-dominatee problems via online adaptive bases and adaptive sampling. SIAM Journal on Scientific Computing. 42:A2803-A2888 2020.
[39] B. Peherstorfer and K. Willcox. Online adaptive model reduction for nonlinear systems via low-rank updates. SIAM Journal on Scientific Computing, A2(3))22123- 2015.
[40] Benjamin Peherstorfer. Breaking the Kolmogorov barrier with nonlinear model reduction. Notices of the American Mathematical Society, 69(5)7725-733, May 2022.
[41] M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[42] R. Rico-Martinez, Krischer, G. Kevrekidis, M. C. Kube, and L. Hudson. Discrete- vs.
continuous-time nonlinear signal processing of cu electrodissolution data. Chemical Engineering Communications, 1 118(1): (18()):2544) 48, 1992.
[43] Grant M. Rotskoff, Andrew R. Mitchell, and Eric Vanden-Eijnden. Active importance sam- pling for variational objectives dominated by rare events: Consequences for optimization and generalization. arXiv, 2008.06334, 2021.
[44] Samuel H. Rudy, J. Nathan Kutz, and Steven L. Brunton. Deep learning of dynamics and signal-noise decomposition with time-stepping constraints. Journal of Computational Physics, 396:483-506, 2019.
[45] Themistoklis P. Sapsis and Pierre F.J. Lermusiaux. Dynamically orthogonal field equations for continuous stochastic dynamical systems. Physica D: Nonlinear Phenomena, 238(23):2347- 2360, 2009.
[46] Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 153511399164,, dee 2018.
[47] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 1556)):129-11588 2014.
[48] N. Sukumar and Ankit Srivastava. Exact imposition of boundary conditions with distance functions in thyyicc--nfformes deep neural networks. Computer Methods in Applied Mechanies and Engineering, 389:1 14333, 2022.
[49] Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. arXiv, 21 2111.09839, 2021.
[50] Qian Wang, Nicolò Ripamonti, and Jan S. Hesthaven. Recurrent neural network closure reduced-order models based the Mori-Zwanzig formalism. J. Comput. Phys., 410:109402, 2020.
of parametrie
[51] Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks. arXiv, 2203.07404 2022.
[52] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. BitFit: Simple parameter-efficient fine-tuning for transformer- based masked language-models. arXiv, 2106.10199, 2022.
13

---

[53] Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and Weinan E. Deep potential molecular dynamics: A scalable model with the accuracy of quantum mechanies. Phys. Rev. Lett., 120: 12:0143001, Apr 2018.
[54] Tianchen Zhao, Chuhao Sun, Asaf Cohen, James Stokes, and Shravan Veerapaneni. Quantum- inspired variational algorithms for partial differential equations: Application to financial deriva- tive pricing. arXiv, 2207. 10838, 2022.
14

---

A Appendix
A.1 Details Figure
For Figure 2 2 look at quantities generated by fitting network the true solution a point in time 1. This is done the same way we fit initial conditions described in A.5 but in this context the target function is taken to be the true solution time t. For Figure a), compute the Jacobian of the network fitted to the true solution at each point in time and then plot its spectrum. For Figure 2bb), we take the network fitted to the true solution and compute the residual from the least-squares problem 4 to give the data points in the "Direct fit" line.
In Figure we provide versions of Figure from the main text but for all the equations considered. Vlasov is a more complex problem that exhibits a less sharp decay but is still distinctly rank deficient.
A.2 Details on Speed-Up
Here we provide additional results on the speed-up provided by RSNG for different equations. In Figure [ we provide versions of FigureBla) from the main but for all the equations considered. In Figure | we show how the runtime of Neural Galerkin schemes scales with thus showing the quadratic speed-up provided by RSNG as we reduce 8.
A.3 Applications to High Dimensional Problems
Neural Galerkin schemes have been shown to be useful approach to high dimensional PDEs 7 In Figure 10 we demonstrate that RSNG may be applicable in these settings we well. We fit neural network to numerical solution of the Fokker-Planck equation in dimensions; see (7] Section 5.4. I] for description the setup. The results show that for sufficiently large network. the Jacobian has low-rank structure as in the examples in the paper and a steep decay in the singular values so that random sketching strategies will likely be successful.
A.4
Ground Truth
The ground truth for the Allen-Cahn and Burgers' equations were generated using a spectral method with fourth order integrator implemented in the spin solver part of the Chebfun package in Matlab. We used a spatial grid of 000 points and time step size of 1e-3.
The ground truth for the Vlasov equation was generated with 4th order finite difference scheme in space and an RK4 time integration scheme. We sample 10 points over the full 2D space domain and time step size of 1e-3 was used for time integration.
In Figure
we show plots of the ground truth solutions the beginning, middle, and end of integration time for the equations we examine. We can see that they display the characteristics described in Section
Spectrum of ((e(e))
Spectrum of ((O(t))
Spectrum of [(e(e))
10
10:2
1=4
10-1
10-4
10-4
10
10:4
10
10
10
3000
2000
Index
Index
Index
(a) Allen-Cahn
(b) Burgers'
(c) Vlasov
Figure 7: Decay of singular values of ((O())
15

---

39600
10
Runtime (s)
Runtime (
Runtime (s)
10*
10080
10a
1045
10a
2460
102
63
103
936
RSNG
(a) Allen-Cahn, error 50- 5, 8
150
(b) Burgers", 2e
125
(c) Vlasov, 2e 4, 800
Figure 8: Speedups of RSNG over dense updates with direct and iterative solver.
7500
5/2
40980
$/2
8100
speedup
Runtime (S)
4 spcedup
Runtime (s)
speeclpp
Runtime
1875
10245
2025-
1692
S
(a) Allen-Cahn
(b) Burgers'
(c) Vlasov
Figure 9: Speedups of RSNG scale quadratic with sparsity
Figure 10: Singular values of the
Depth 2
Singular Values ((O)
Jacobian of the network fit to a Depth Fokker-Plancl solution in five di- mension decays quickly too; pro- viding indication that our RSNG approach is applicable in these set- tings well.
Depth
10-*
Depth
Depth 10
10-1
1
0
100
% of Total Modes
M
Dit. 30
utti
a
2n
2n
Spatial Domain
Spatial Domain
Spatial Domain
(a) t
(b) f 2
(c) = 4
Figure 11: True solution u(t, x)) for Allen-Cahn
16

---

0.6-
0.6-
0.4
0.4
35 0.4
0.0
-1
0.0
-1
0
Spatial Domain
Spatial Domain
Spatial Domain
(a) f
(b)
(c) = 4
Figure 12: True solution u(t, x) for Burgers'
0.35
0.30
0.30
0.30
025
0.20
0.20
0.20
0.10
0.30
0.10
0.00
0.00
0.35
0.35
0.30
030
0.30
0.25
0.5
0.5
0.15
0.15
0.35
0.05
0.06
0.05
0.00
0.00
2n 0.00
(a) t =
(b) = 1.5
(c) = 3
Figure 13: True solution (top) vs RSNG solution (hottom) for Vlasov
A.5
Architecture and Hyperparameters
For an input E Id periodic embedding layer with period P is defined with the optimization parameters a, d, Rd as,
PeriodicEnbed (x)
Σ
cost
2T + Φ)) +
P
This operation is the repeated times for different parameters a, Φ. b where denotes the width of the layer resulting in output vector E R
To fit the initial condition, minimize the _ distance between the network and the initial condition as well as the L distance between the first derivative of the network and the first derivative of the initial condition with respect to the spatial domain X. We evaluate the loss function over 000 equispaced points for Allen-Cahi and Burgers' equation and 200 000 points for Vlasov. We fit our initial conditions with two nonlinear solvers. First run L-GFGS with 1000 iterations, then an Adam optimizer with the following hyperparameters,
iterations: 100
learning rate 1e-3
scheduler: cosine decay
17

---

decay steps: : 500
The number of iterations and the number of points we sample chosen fit the initial condition high accuracy to avoid polluting the results in our analysis with errors of fitting the initial condition.
To assemble the Jacobian matrix ((())), the gradient of ū is evaluated on samples generated from the spatial domain. not noted otherwise, we use 10 000 equidistant points for Allen-Cahn and Burgers' and 200 000 equidistant points for Vlasov. For the time-optimized RSNG results in Table [ (rows 4 and 8) we use 1000 equidistant points for Allen-Cahn and Burgers'. In the dense and sparse least-squares system
regularize the direct solver we set the rcond parameters numpy implementation of 1stsq. The values used are 1e-4, 1e-4, 4, 1e-5 for Allen- Cahn,, Burgers', and Vlasov respectively.
avoid numerical instability. For this
A.6
Global Methods Benchmark
Here we detail the training setups for our benchmarks of the global methods given in Table
For all PINN experiments we sampled data on a grid with 100 points in the time domain and 256 points in the spatial domain. All PINNs were trained with the following hyperparameters:
optimizer Adam then L-BFGS
spatial samples 256
time samples 100
activation tanh
For the plain PINN experiments our architecture is a MLP with layers sizes as follows: [2, 128, 128, 128, 128, I]. Boundary and initial conditions were enforced through penalty in the loss function.
For the Causal PINNs use the architecture described in the original paper, with periodic embedding and "modified MLP layers." The layer sizes were as follows: [periodic embedding. 128, 128, 128, 128, 1]. Our implementation uses much of the original code provided in [51]. We used only one line-window for training as this is what was chosen in [5]] for the Allen-Cahn equation. The tolerance hyperparameter, which controls the degree to which the loss funetion enforces causal training, was set to 100 and 50 for the Allen-Cahn and Burgers' equations respectively. Additionally the Aie parameter, which controls the loss functions' weighting on the initial condition, was set to 100 and for the Allen-Cahn and Burgers' equations respectively. These were both chosen via hyperparameter search. For details on the context and meaning these hyperparameters see the original paper [51]
The PINNs trained for Allen-Cahn used 1000 steps of Adam followed by 30000 steps of L-BFGS. For Burgers' equations we used 1000 steps of Adam followed by 60000 steps of L-BFGS. More steps are needed for Burgers' equation in order to sufficiently resolve the sharp gradient in the solution due to the low viscosity number. We similarly choose smaller timestep for Burgers' in the Neural Galerkin schemes. We note that fewer optimization iterations can be used for training PINNs but this resulted in much larger errors in our experiments. In any case, widely varying the number optimization iterations and other PINNs hyperparameters did not achieve errors in the range of what RSNG achieves in these examples.
18


Reviews:
[{"review_id": "NEknyJbK1p", "reviewer": "Reviewer_Gbtz", "summary": "The paper introduces Neural Galerkin scheme to sparsely update neural network over the sequential-in-time training. In particular, the proposed method randomly update sparse subsets of network parameters at each time step. With randomized sparse updates, overfitting problem of sequential-in-time methods can be mitigated, leading to better accuracy at test times. Moreover, the computational costs of training is reduced without losing expressiveness of the model, while accelerating the training process. Proposed method is validated on a wide range of evolution equations. ", "questions": "* The proposed method is inspired by dropout in neural networks. Thus, I am curious about comparison results with PINN and CausalPINN, with different rates of dropouts applied.", "limitations": "Limitations are addressed by the authors", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well organized and easy to follow. It also has a strong motivation inspired by the Neural Galerkin schemes and neural network dropouts.\n\n* The paper has a strong theoretical grounds and corresponding toy examples to support the claim. For instance, two challenges of previous sequential-in-time training are well addressed in the section 2.2 with adequate examples. \n\n* The paper presents a wide range of ablation studies and various experimental settings to show the effectiveness of the proposed method. Most of the claimed benefits of the proposed method are validated empirically, including lower error rates and faster running time. \n\n* The proposed method is considered a big contribution for the field of sequential-in-time training methods.", "weaknesses": "* The performance of the model is only validated in standard benchmark equations. Validation in high-dimensional PDE problems can benefit the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and detailed reading of our paper.\n\n> The proposed method is inspired by dropout in neural networks. Thus, I am curious about comparison results with PINN and CausalPINN, with different rates of dropouts applied.\n\nTo address the reviewer’s point, we performed numerical experiments with PINNs and dropout. We can report that for various levels of dropout, the error of PINNs increases compared to no dropout as in our experiments in the paper. We believe this is because for PINNs the train and test distribution are nearly identical, therefore it is not beneficial to increase training loss for lower test loss as is accomplished by dropout. \n\n> [...] Validation in high-dimensional PDE problems can benefit the paper.\n\nWhile not being a comprehensive numerical experiment with high-dimensional PDEs, we conducted the following proof of concept and added a figure (global response PDF, Fig 14): We fitted a neural network to a numerical solution of the Fokker-Planck equation in 5 dimensions. The results show that for a sufficiently large network, the Jacobian has a low-rank structure as in the examples in the paper and a steep decay in the singular values so that random sketching strategies will likely be successful. We will add details of this experiment in the appendix in the camera-ready version, if this paper gets accepted.\n"}, {"review_id": "aTc3g98skn", "reviewer": "Reviewer_LKD5", "summary": "In this paper, the authors propose a modified Neural Galerkin scheme for the solution of time-dependent PDEs, aiming at reducing computational cost and improving accuracy by avoiding overfitting.\n\nThe target solution $u(x,t)$ is approximated with a nonlinear parameterisation $\\theta(t)$, so that $u(x,t)\\approx \\hat{u}(x;\\theta(t))$ (this can be interpreted as a nonlinear form of reduced-order modelling). In practice, $\\theta(t)$ represents the parameters of a NN responsible for outputting $\\hat{u}(x;\\theta(t))$ for a given $x$. At each time-step $t$, the goal is to train the network to output the solution at the next time-step. As is classical with Galerkin methods (and derivative thereof), this is achieved by following a variational principle, whereby the norm of the residual of the PDE is minimised. This can be recast as a least-square minimisation problem in the unknown $\\dot\\theta$: once solved, the gradient of $\\theta$ is then used to guide the update of the NN.\n\nThe novel contribution of the paper lies in applying sketching to simplify the solution of this least-square problem. This is rooted in the observations that:\n- Often, the target matrix in the system $J(\\theta)$ (a collation of the gradients of $u$ wrt $\\theta$, sampled at specific collocation points) is low-rank, implying redundancy in the components of $\\theta$\n- Updating the whole $\\theta$ tends to overfit, which in turn leads to accumulation of errors and poor quality of the solution as time evolves.\nThis justifies reducing the space of parameters to update, and has the boosted effect of reducing computational time due to the effective shrinking of the least-square problem.\n\nThe performance of the novel method is verified on three benchmark PDEs (two 1D and one 2D), and expressed in terms of runtime and norm of residual. The proposed method outperforms both its dense (non-sketched) counterpart, and the global-in-time method (Causual) PINN.", "questions": "- How does Fig2a look like for Burgers’ and Vlasov? And if we vary the network size? I’m trying to understand whether the low-rankness observation is affected somehow by the type of problem/model considered, or if it is robust to these variables (Fig5a partially hints at this, but it would draw a fuller picture to include comparisons throughout the paper)\n- Line 257: can you report the actual value of the error in this case? Also, how does Fig4(a) look like for Allen-Cahn and Vlasov? This builds up on the previous question: I’m trying to understand how robust these findings are wrt problem chosen\n- Fig 4(b): how does time complexity scale with $s$? In my mind, this was supposed to be the bottleneck of the algorithm, and if you’re using a direct solver to solve the associated least-square system, complexity (and time) should scale quite heavily in $s$ (you mention \"quadratically\" a couple of times: what exactly are you using as solver?). But by eyeballing the datapoints in the plot, it looks almost linear? Am I missing something?\n- I liked Fig5(a): the fact that it’s U-shaped really fits the overfitting assumption (pun intended), and I think it really helps nailing down the validity of the method. I also like the fact that it shows that Vlasov kinda falls outside the trend - but this makes sense: it’s a more complex problem after all (if anything, because it’s 2D). Ultimately, I expect these graphs to be both problem- and NN- dependent: if you had chosen a NN with more parameters, likely at a certain point it would've started overfitting again, even for Vlasov. In any case, it’s extremely useful to know when a method might start breaking down, and it should be pointed out! \n\nMinor corrections\n- Fig3(d):\n    - For the x-label: I reckon $757$ is the fixed value of $s$? Just write $p (s=757)$ or something on those lines. Same for Fig5\n    - Please report an indication of scale (for example, by explicitly specifying a numerical value for the “Dense” tick)\n    - I reckon the relative error here refers to the end time $T=4$?\n    - Maybe they would look better if the figures shared the same scale for the y-axis? This way one could align them and compare better among them? Not sure if they’d get too squashed though\n- Line 237: I can see error bars only for Fig3(d). If for Fig(a-c) they’re too small to see, please state it explicitly. If you’re not reporting them, why not? Similarly for Fig5\n- Line 245: please clarify: “we plot the relative error [with respect to the underlying solution] over time”", "limitations": "The main limitations I could identify (particularly, dependency on properties of the target Jacobian) are addressed and made explicit in the corresponding section", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The sketching simplification at the core of the method is well-justified, also with experimental results. Moreover, I particularly appreciate the fact that the intuition behind using sketching is rooted both in algebraic considerations (pertaining rank of J), but is also interpreted through the lenses of regularisation techniques in NN (dropout). Both views are valid, and contribute to supporting the hypothesis\n- The paper is well written, neatly structured, with the right level of rigour and a good introduction to the topic", "weaknesses": "- Possibly the breadth of the contribution: after all, it represents a relatively small modification to an already existing method, and the modification itself takes inspiration from already-established techniques. Still, I appreciate this work, and the contribution seems valid and useful\n- Some experiments might be expanded to further substantiate the claims (see questions below), but this should be straightforward to address", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and detailed reading of our paper.\n\n> Some experiments might be expanded to further substantiate the claims (see questions below), but this should be straightforward to address\n\nWe provide additional figures/experiments that all further support the claims of the paper: \n- added figures in global response PDF that show the singular value decays of the Jacobians corresponding to all equations  (Global response PDF Fig 10)\n- added figures in global response PDF that show additional details of the runtime speed-ups of RSNG for all equations (Global response PDF Fig 11 and 12)\n- proof of concept for high-dimensional problem (Global response PDF Fig 14)\n\n> How does Fig2a look like for Burgers’ and Vlasov? And if we vary the network size? I’m trying to understand whether the low-rankness observation is affected somehow by the type of problem/model considered, or if it is robust to these variables (Fig5a partially hints at this, but it would draw a fuller picture to include comparisons throughout the paper)\n\t\nThe reviewer is completely right that the low-rankness is affected by the problem/model. We added Figure 10 (uploaded in the global response, and in camera-ready version if paper gets accepted) that show the decay of the singular values of the Jacobian for neural networks fit to Burgers' and Vlasov. For Burgers', the singular values decay similarly quickly as for the Allen-Cahn example. As the reviewer observes, and in agreement with our numerical experiments and Fig 5(a) in the paper, Vlasov is a more complex problem that exhibits a less sharp decay but is still distinctly rank deficient.\n\nWe additionally demonstrate what happens when we vary the network size in global response PDF Fig 14. Here we fit a neural network to a numerical solution of the Fokker-Planck equation in 5 dimensions. The results show that as we increase the size of the network, the Jacobian has a steeper decay in the singular values so that random sketching strategies will likely be successful. We will add details of this experiment in the appendix in the camera-ready version, if this paper gets accepted.\n\nThis agrees with and also addresses the reviewer's related comment:\n\n> that Vlasov kinda falls outside the trend - but this makes sense: it’s a more complex problem after all (if anything, because it’s 2D). \n\nIndeed, how sparse the updates can be, depends on the complexity of the problem, which is reasonable and expected and in alignment with the results of the additional figures mentioned in the previous paragraph. If the paper gets accepted, we will add this discussion to the appendix of the camera-ready version. We note, however, that low-rankness is only one part of what RSNG addresses. Additionally there is the overfitting problem, which we detail in the paper and which is addressed by the randomization. \n\n> Line 257: can you report the actual value of the error in this case? Also, how does Fig4(a) look like for Allen-Cahn and Vlasov? This builds up on the previous question: I’m trying to understand how robust these findings are wrt problem chosen\n\nThe actual value of the error on line 257 is 2E-4.  We uploaded a version of Figure 4(a) for each of the numerical examples above and we report the error values in the caption (global response PDF Fig 11). We will add these to the appendix of the camera-ready version if accepted.\n\n> Fig 4(b): how does time complexity scale with s? In my mind, this was supposed to be the bottleneck of the algorithm, and if you’re using a direct solver to solve the associated least-square system, complexity (and time) should scale quite heavily in s (you mention \"quadratically\" a couple of times: what exactly are you using as solver?). But by eyeballing the data points in the plot, it looks almost linear? Am I missing something?\n\nWe use a direct solver based on SVD for which costs scale quadratically in s. Please note that the aim of the method is to reduce s: Halving the number of parameters that need to be updated (halving s) with our RSNG approach vs dense Neural Galerkin leads to quadratic speedups of the solve step. We imply this on line 110, but we will make this more explicit in the camera-ready version, if accepted. Note that we also show speedups compared to dense Neural Galerkin with iterative least-squares solvers, which suffer from a condition problem, as we discuss in the paper. \n\nThe quadratic scaling of the speedup is not directly reflected in Figure 4b because of the low values of s and other effects (e.g., Jacobian computations, parallelization on GPUs). We will add comments to clarify this in the camera-ready version if accepted. Additionally, we added a figure (global response PDF Fig 12) where we plot directly s vs solver runtime for each of our equations. These figures show more precisely the quadratic scaling in s. We plan to add these to the appendix, if accepted.\n\n> Line 237: I can see error bars only for Fig3(d)...\n\nThe error bars are plotted but too small to be seen. We will state this explicitly in the camera-ready version, if accepted.\n\n> I reckon the relative error here refers to the end time \n\nThe relative error is computed over the whole space-time domain. We will make this formula explicit in the appendix, if accepted.\n\n> Minor corrections [...]\n\nWe plan to make all other minor corrections that the reviewer suggested.\n\n"}, {"review_id": "b1FIqqicKJ", "reviewer": "Reviewer_8WGb", "summary": "The authors suggest an alternative optimization scheme for Neural Galerkin methods - a family of methods in which parameters of a neural network are optimized so that the gradients of the solution obey a given PDE with the notable quality being that rather than solving over all of space-time, the parameters of the network evolve in time to match the system evolution equation - in which only a subset of the surrogate model's parameters are updated at each time step. The author's show that this approach seems to lead to improved accuracy at a fixed FLOP budget and improved speed to a fixed accuracy threshold. ", "questions": "- Were the PINN comparisons also performed with rational activation functions?\n- How do these methods compare against numerical methods?\n- Given the conclusion implies the comparison is unfavorable, it would be very valuable to see examples from the set of problems where numerical methods are expected to do poorly. ", "limitations": "Limitations are well covered and no ethical issues are foreseen. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and interesting.\n- Jargon is generally defined shortly after it is used.\n- As someone who is not very familiar with Neural Galerkin methods, it seemed that most questions that immediately came to mind after reading segments were answered quickly. \n- Overall, the paper was clear, compelling, and all claims were well-supported by the provided evidence. \n", "weaknesses": "The main weakness of the paper is that while the paper is clearly providing value to a current research topic, it could do a better job of explaining where Neural Galerkin methods are (or could be) practically useful. The PINN comparison is enough for me to be comfortable recommending approval, but given the method requires knowledge of $f$, it seems like true comparison would be numerical solvers which, as the authors mention in the conclusion, are still quite a bit ahead of the PINN comparisons on these problems. Without that, it's difficult to see whether the paper would be interesting to a wider audience.\n\nLine-item Issues:\n- Figure 2 is not very clear. a) makes sense, but (b) could use a better captioning so that it can be understood independently of the text and (c) is unclear even after reading the text. \n- Figures 3, 4 are similar in that the caption is currently insufficient to explain the figure.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and detailed reading of our paper.\n\n> Were the PINN comparisons also performed with rational activation functions?\n\nTanh activations are usually a standard choice in the PINN literature, which is what we used too. However, to ensure the choice of the activation function was not a key part of our advantage we reran our PINN experiments with rational activation functions. The results are comparable in terms of error, but for additional computation cost due the additional parameters. Allen Cahn: 6.97e-2 (cmp 6.42e-02 with tanh activations as in paper). Burgers: 7.04e-4 (cmp 1.32-e3 with tanh activations as in paper). Thus, regardless of the choice of activation function, RSNG still achieves a higher accuracy of 1 to 3 orders of magnitude, and lower runtime.\n\n> [...] while the paper is clearly providing value to a current research topic, it could do a better job of explaining where Neural Galerkin methods are (or could be) practically useful. [...] How do these methods compare against numerical methods? Given the conclusion implies the comparison is unfavorable, it would be very valuable to see examples from the set of problems where numerical methods are expected to do poorly.\n\nWe briefly mention the application scenarios in the conclusions but agree with the reviewer that this should be brought up more prominently: (a) high-dimensional PDEs, (b) model reduction of transport-dominated problems (slow decay of n-width), (c) domains with complex geometries, and others. In this paper, our main goal is to show superiority over the regular (dense) Neural Galerkin schemes for which we focus on simple benchmarks (see Limitations section) as we think these are sufficient for demonstrating the benefits of our RSNG. It is our hope that our RSNG approach is a reasonable step forward towards tackling current limitations of Neural Galerkin schemes to be a building block for fruitful approaches for these other applications, as we mention in Conclusions and Limitations sections.\n \nTo further address the reviewer’s concern, we added a proof-of-concept experiment that indicates that RSNG can be applied to high-dimensional PDEs as well: We fitted networks to solutions of five-dimensional Fokker-Planck equations and plot the decay of the singular values of the Jacobian (see Figure 14 in global response PDF and reply to comment by reviewer Gbtz). The decay is similarly fast as in our benchmark examples in the paper, which is an indication that RSNG can be successfully applied in such a high-dimensional setting as well. \n\n> [Caption of] Figure 2 is not very clear. [...]\n\nWe agree the captions could be made more clear such that they stand independent of the main text: For Fig 2(b) we will emphasize standard regression on ground truth data (fit) finds better regions in the parameter space than dense (in contrast to our RSNG) Neural Galerkin schemes. We will similarly clarify the other captions mentioned by the reviewer. "}, {"review_id": "PS1OHr0vUF", "reviewer": "Reviewer_mkqi", "summary": "The paper suggests a simple but effective way to train sequential-in-time models by randomly sampling weights to update. The method avoids overfitting and realizes speedup because only a small part of parameters is updated. The experiments show that the proposed model offers the best performance in terms of speed and accuracy.", "questions": "* Is the method faster than classical numerical solvers (e.g., finite difference method)? It could be important to compare with the classical methods.\n* Figure 5 (b) and (c) suggest that sampling could lead to instability. What is the reason, and any ideas to address this? The reviewer guesses that the updated weights are not smooth in time, i.e., the indices of the weight matrix are drastically changed at a time $t$ and the next time $t + \\Delta t$. Could it be interesting to introduce some \"momentum\" to make the change of indices moderate in time?", "limitations": "* The authors addressed limitations. Another limitation could be handling boundary conditions other than periodic (e.g., Dirichlet and Neumann).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Although the method is simple, the idea is based on the reasonable assumption that parameters are redundant locally in time. Due to its redundancy, the Jacobian matrix is prone to be singular or stiff and takes a huge computational budget for training with the existing sequential-in-time methods. The proposed method elegantly addresses these issues by randomly sampling weights to update.\n* The experiments clearly demonstrate that the proposed method works efficiently in terms of computation time and accuracy compared to the considered baseline methods.", "weaknesses": "* Figure 5 (b) and (c) suggest that sampling could lead to instability (see also Questions). The reviewer recommends that the authors show predicted solutions in figures to assess the smoothness of the prediction. Sometimes smoothness is key because some downstream tasks may perform additional differentiation on the prediction.\n\nMinor points:\n* In the first equation in Equation (1), the domain of t should be defined in the open set (see, e.g., Jürgen Jost \"Partial Differential Equations ThirdEdition\" (2012) Chapter 1) to make differentiation possible everywhere in the domain. That's why we need the initial condition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and detailed reading of our paper.\n\n> [...] reviewer recommends that authors show predicted solutions in figures to assess the smoothness of the prediction. Sometimes smoothness is key because some downstream tasks may perform additional differentiation on the prediction.\n\nWe do not observe that the random sampling is affecting the smoothness of the solution field. Changes in the weights are smoothed out by the network parametrization and its regularity. Furthermore, the sequential-in-time training is performed with a regular time integration scheme, which also leads to smooth updates in classical numerical solvers. To help readers assess the smoothness, we plot the predicted solution at various time steps (see global response PDF). We will add these figures to the appendix of the camera ready version, if accepted.\n\n> Figure 5 (b) and (c) suggest that sampling could lead to instability. What is the reason, and any ideas to address this? The reviewer guesses that the updated weights are not smooth in time, i.e., the indices of the weight matrix are drastically changed at a time t and the next time t + \\Delta t. Could it be interesting to introduce some \"momentum\" to make the change of indices moderate in time?\n\nThe reviewer raises an interesting point: The residual can be less smooth in time because at each time step a different random sketch of the Jacobian is used for approximations. We note two things: first, the residual is a nonlinear function of the solution and thus can be rough whereas the solution remains smooth in time (see comment to previous question). Second, we do not see empirically that the rougher residual is an issue for stability in our experiments. However, we certainly very much appreciate this really insightful comment. Indeed, it would be interesting how different sampling schemes (e.g., leverage score sampling to improve the random sketch) and adding momentum (as proposed by the reviewer) influences the residual. We will add a discussion in the camera-ready version (if accepted) that this is an important point to study.. \n\n> Is the method faster than classical numerical solvers (e.g., finite difference method)? It could be important to compare with the classical methods.\n\nIn this work, we focus on two pervasive challenges (high computational costs, overfitting) of Neural Galerkin schemes (and sequential-in-time methods in general) and demonstrate on simple benchmark problems the superiority of our RSNG approach over regular (dense) Neural Galerkin schemes. Because the benchmarks are standard problems (see the section on Limitations, where we make this very clear), it is unlikely that Neural Galerkin schemes achieve much speedup over classical numerical methods for these problems. However, there are several other settings where Neural Galerkin schemes are beneficial compared to traditional numerical solvers with linear parametrizations, e.g., (a) high-dimensional PDEs, (b) model reduction of transport-dominated problems (slow decay of n-width), and (c) domains with complex geometries. Yet, each of these areas poses its own unique set of challenges that are active research and go far beyond the scope of this paper, in our opinion. This is why we demonstrated superiority of our RSNG approach over regular (dense) Neural Galerkin schemes on simpler benchmark problems. \n\nTo further address the this, we added a proof-of-concept experiment that indicates that RSNG can be applied to high-dimensional PDEs, a problem domain where classical numerical methods would fail. We fitted networks to solutions of five-dimensional Fokker-Planck equations and plot the decay of the singular values of the Jacobian (see Figure 14 in global response PDF). The decay is similarly fast as in our benchmark examples in the paper, which is an indication that RSNG can be successfully applied in such a high-dimensional setting as well.\n\n> [...] handling boundary conditions other than periodic (e.g. Dirichlet and Neumann).\n\nWe will add comments in the camera-ready version (if accepted) to emphasize that (a) Dirichlet boundary conditions typically can be imposed in the network architecture and (b) Neumann boundary conditions (as well as Dirichlet) can be imposed via constraints, which is common when numerically solving PDEs with neural-network parametrizations. \n\n> [...] the domain of t should be defined in the open set [...]\n\nWe will make this correction. This is a good catch and improves the technical quality of the paper.\n"}, {"review_id": "8NFWIg0Vdo", "reviewer": "Reviewer_YG46", "summary": "The paper proposes a so-called randomized sparse neural galerkin (RSNG) algorithm to solve time-dependent PDEs. The basic idea is to\nconsider parameterized functions as ansatz spaces, with parameters that vary in time. Consequently, the method is of the sequential-in-time learning algorithms rather than global-in-time learning algorithms such as plain vanilla PINNs. Computing PDE residuals with this class of ansatz functions and collocating the loss (at every time instant) naturally leads to a least-squares type linear algebra problem which can be solved either directly or indirectly and has been previously considered. The author identify problems with this approach and propose a novel remedy by considering randomized sparse instantiations of the projection, loosely along lines of dropout, but in a more L2 sense. The algorithm is illustrated with some benchmark examples for PDEs.", "questions": "In addition to the questions above in the \"weaknesses\" section -- the following minor ones should be addressed:\n\n1. What is refit in Figure 2(b) -- it is also used in other figures ?\n\n2. What is computational cost of RNSG for 2-D Vlasov-Poisson ? How does it compare to PINNs, especially when they are optimized with L-BFGS. \n\n3. How do the results, particularly the choice of the sparsity parameter $s$ change with the viscosity $\\nu$ for viscous-Burgers' ? How low can $\\nu$ be to still allow for accurate approximation ? \n\n4. Please mention the value of $s$ in each of the figures (for instance in the option or the Appendix) where you have not mentioned the value.", "limitations": "It is a mostly a reiteration of the section on Weaknesses:\n\n-- No theoretical underpinning of the algorithm.\n\n-- Lack of a clear description of the limitations of the algorithm.\n\n--- How are the collocation points $x_i$ chosen ? The authors refer to a prior work but do not contextualize it here.\n\n--- More baselines need to be evaluated -- in particular, the runtime of traditional numerical methods has to be mentioned to provide the reader with an idea of what is the potential pay-off with their method, if any ? \n\nThe basic idea is interesting and I am happy to reconsider my score if the authors address my questions appropriately.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The premise of the paper is interesting. It is well-known that global-in-time methods such as PINNs are ill-conditioned and hard to train and the paper investigates a method of potentially avoiding some of the pitfalls of global-in-time methods. \n\n2. Randomized sparse algorithms can be efficient in the context described by the authors as long as certain conditions, such as incoherence that they loosely identify, are fulfilled. \n\n3. Numerical experiments clearly demonstrate that RNSG outperforms the dense least-squares method as well as baselines such as plain vanilla PINNs considerably.", "weaknesses": "1. Clarity: The paper could do with a rewrite where authors elaborate on the rather succinct notation, for instance in the appendix. In particular, to aid the reader, they could write out the neural networks in line 81, for instance for a simple one-hidden layer shallow neural network, as to how the time variable and the space variable exactly enter into Eqn (2) -- the reader will clearly see the distinct roles played by time here. Similarly, further rationale could be provided for the choice n >> p in deriving Eqn (4), one could for instance refer to it a quadrature for Eqn. (3). Caption of Figure 2 could be better explained, in particular what is refit in Figure 2(b) and what does co-adaptation mean and how does Figure 2 (c) shows this property. Quite a lot of heuristics is provided in a hand wavy manner that is potentially inscrutable to a reader -- for instance, what does the lines 173-176 mean ? \n\n2. Theory or rather the lack of it: The author do not attempt to provide any theoretical justification of their method. Some phrases are included here and there (e.g. line 162, lines 173-179 etc) but the reader feels that the authors know more about the theory then they present here. A discussion on these aspects could be illuminating. In particular, when is the method supposed to work and when not ? The lack of presentation of a theoretical perspective forces this reviewer to judge and criticize the paper on its empirical content as I proceed to do below. \n\n3. Empirical evaluation: The authors consider three time-dependent PDEs i.e., 1-d Allen-Cahn, 1-d viscous Burgers with a relatively low viscosity and a 2-d Vlasov-Poisson with a given potential and on a Cartesian domain. As such the problems are fairly well-studied benchmarks and the authors clearly show that their RNSG algorithm handily outperforms its dense foundation and an iterative version. However, the following limitations of the experiments can be identified: \n\n      a. The authors claim that it takes 18 mins for a PINN to obtain 6% errors for Allen-Cahn and almost 2 hours to obtain below 1% error for the simple viscous-Burgers example. This is unacceptably large and could very well be due to the use of ADAM as the optimizer. It is standard in PINNs literature to either use L-BFGS or start with ADAM and then switch to L-BFGS after say Order of 1000 iterations or so. This reviewer is curious if doing so reduces the run-time with PINNs ? In any case, the authors obtain significantly lower errors with RNSG -- so the timing is worth investigating. \n\n   b. Clearly, PINNs and even Casual PINNs (I am not convinced that it takes more than 6 and half hours to solve Burgers' equation with casual PINNs) are not the best conditioned to be the only baselines here. Rather I would suggest the following baselines: \n\n             b1) A method of lines version of PINNs (see the Raissi et al original PINNs paper) where the time is discretized with a RK4 solver and space with PINNs -- this is a fairer comparison as time is now iteratively updated with the PINNs and should be compared against your method, at least for some experiments. \n\n            b2) Alternatives to PINNs such as ODILs (Karnakov et al https://arxiv.org/abs/2205.04611) which directly collocate the PDE residual (with an implicit time-discretization) appear to be much better conditioned and could be a strong baseline for your method. \n\n         b3) The whole rationale would be out-compete standard numerical methods. Hence, it would be instructive to consider the computational cost of your ground truth generator. How much computational time does it take with this ground truth generator -- for all the experiments -- to reach comparable accuracy. Given that you require 13 mins to solve 1-D Allen-Cahn and 40 minutes to solve 1-D Burgers, traditional numerical methods would solve both these problems in a matter of secs, if not faster -- so where is the rationale for the use of your method ? One could argue that its rationale would be high-dimensional problems or complex geometries, which you don't consider empirically here. \n\n4. Scope: In its current version, it is unclear how this algorithm scales with spatial (or parametric) dimension -- is it fairly dimension independent ? If so, where is the evidence for that -- for instance, what is the computational time for the 2-D Vlasov-Poisson problem ? \n\nAnother issue pertaining to the scope is that the method relies on an explicit time-stepping integrator such as RK4 -- can it be used for stiff problems, with implicit time stepping ? Otherwise, you will require too many iterations in time, something that PINNs and other global-in-time methods don't have to. It would be useful to see if your algorithm can work with large time steps for such stiff problems. \n\n5. Comparison: The authors say that other works such as [1,7,14] have considered this approach and yet do not provide a clear comparison to these works. It is essential to distinguish between them and the current paper, in a related work section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and detailed reading of our paper.\n\n> [...] where authors elaborate on the rather succinct notation, for instance in the appendix. [...]\n\nThe reviewer mentions several good points on how to further improve the clarity of the paper. We will address these in the camera-ready version, if this paper gets accepted. In particular, we will carefully revisit the captions of Figure 2.\n\n> [...] The authors do not attempt to provide any theoretical justification of their method. [...] A discussion on these aspects could be illuminating. [...]\n\nWe provide several theoretical justifications in the paper that show that our approach is principled rather than heuristic: (a) One of the theoretic underpinnings of our approach is random matrix sketching, as we briefly mention on lines 173-179. There is a robust literature which we cite [13,19] on matrix sketching that provides tight bounds and when and why these randomized strategies will be successful. We mention some of the key findings for us in lines 173-179. (b) We also mention \"dropout\" and \"co-adaptation\" as building blocks for our approach, which have been studied by the ML community in theory and practice and we give several references to it [3,9,43,45]. We agree with the reviewer that a comprehensive theory would strengthen the paper and we consider this as future work.\n\n> The authors claim that it takes 18 mins [..] use L-BFGS or start with ADAM and then switch to L-BFGS [...]\n\nWe aim to provide a fair comparison by basing our experiments with PINNs and CausalPINNs on parameters that are used in the literature. For example, in case of CausalPINNs, we follow closely the original CausalPINNs paper (doi: https://doi.org/10.48550/arXiv.2203.07404) and use the implementation (link: github.com/PredictiveIntelligenceLab/CausalPINNs ) they provide which only uses the ADAM optimizer without L-BFGS. Our accuracy results are in alignment with what is reported in the original CausalPINNs paper using a similar number of ADAM iterations. \n\nWe ran the experiment suggested by the reviewer of training PINNs with 1000 ADAM iterations followed by L-BFGS. We got lower runtimes (e.g,  for Allen-Cahn 841s, for Burgers' 3451s) but with a slightly higher error. Our RSNG remained faster in terms of runtime and superior in terms of accuracy (2-3 orders of magnitude more accurate). Furthermore, there are also several steps to further tune the sparsity parameter s of RSNG to improve runtime performance without sacrificing too much accuracy of RSNG. We will incorporate these experiments and new benchmarks into the paper if accepted.\n\n> [...] I would suggest the following baselines [...]\n\nThe reviewer suggests we look at the method of line PINNs (MOL PINNs) as a baseline comparison. In order to accurately reproduce this method we take the implementation from the original paper given (https://github.com/maziarraissi/PINNs) and swap in our problem setups for Allen-Cahn and Burgers. \n\nFor Allen-Cahn the default hyperparameters result in higher accuracy (error 2.54e-03) but slower runtime (runtime 1247s) than our PINNs experiment. All results remain below the performance of RSNG (error 1.66e-05, runtime 776s) and thus support the overall conclusion of the paper. For Burgers, in our experiment, the method fails to converge (error 8.81e-01; runtime 1386s). This was consistent across a brief hyperparameter sweep. From our examination this could be due to our benchmark using a lower viscosity term than the problem setup in the original paper. \n\nIn terms of comparison to classical numerical solvers: the aim of the paper really is demonstrating superiority over dense Neural Galerkin schemes, for which we think the benchmark problems that we consider are sufficient. Settings where Neural Galerkin schemes outperform classical methods (e.g., high-dimensional PDEs) come with their own set of challenges that are addressed by other references (e.g., [7]) and go beyond the scope of this paper. Although we provide a proof-of-concept for high-dimensional PDEs (see below). We also clearly mention this in the Limitations section; see also response to reviewer mkqi.\n\n> What is the computational time for the 2-D Vlasov-Poisson problem?\n\nWe add plots showing the computational time for 2D Vlasov in our global response Fig 11. We obtain a speedup of 10x compared to the dense method.\n\n> ​​[How does the] algorithm scales with spatial (or parametric) dimension [...] ?\n\nThe runtime of RSNG is dominated by the number s of network weights that are updated in each time step. How small we can chose s is determined by the singular value decay of the Jacobian. To show this can be independent of spatial dimension, we fit neural network to the solution of a five-dimensional Fokker-Planck equation; see Fig 14 in the global response PDF. The singular values of the Jacobian decay quickly as well in this case; indicating that in high dimensions choosing s << p is valid. \n\n> [...] the method relies on an explicit time-stepping integrator such as RK4 -- can it be used for stiff problems, with implicit time stepping [...]\n\nNeural Galerkin schemes can be formulated with implicit time-stepping schemes, which is done, for example, in Reference [7]. So all what we propose applies just as well to implicit time-integration schemes.\n\n> Comparison: The authors say that other works such as [1,7,14] have considered this approach and yet do not provide a clear comparison to these works. [...]\n\nWe do provide comparisons: All of these works are performing dense updates. So we do provide numerical comparisons, which we label as ``dense’’ in our numerical experiments. We will further clarify this in the camera ready version, if accepted. \n\n> How are the collocation points chosen?\n\nWe detail in the appendix that we select our points on an equispaced grid in the spatial domain, to avoid sampling error polluting the overall results. \n"}]

Author rebuttal:
We thank the reviewers for their valuable comments and suggestions. Here is a summary of some of the key points in the response, including an uploaded PDF that includes additional figures:

- Additional numerical results (in response below and in figures in PDF) of ‘Methods of Line PINNs’ and other PINNs experiments that further support the claim of the paper.

- More detailed discussion of numerical experiments that show that the low-rankness of the Jacobian depends on the model/problem complexity, which is in agreement with the results shown in the paper. Additional plots that demonstrate speedups of 1-3 orders of magnitude of RSNG on all the equations considered.

- A proof-of-concept experiment that indicates that our RSNG approach will also be applicable on higher-dimensional PDE problems, where (dense) Neural Galerkin schemes have shown promising results. 

- We further clarify that our goal is to demonstrate that our RSNG approach can overcome two issues (costs, overfitting) that plague (dense) Neural Galerkin methods. Overcoming these issues is a critical step so that Neural Galerkin and related schemes can be successfully applied to classes of problems where traditional numerical solvers struggle, such as for high-dimensional PDEs, nonlinear model reduction and complex geometries (see Conclusions in paper).

We will incorporate the comments and the additional results in the camera-ready version, if accepted. We thank the reviewers again and hope the discussion below and the additional numerical results address the reviewers’ concerns.


Meta-review: This is a strong submission. Novelty and relevance to the NeurIPS community have been acknowledged by all reviewers. There were multiple suggestions regarding the positioning and clarity of the submission and we strongly encourage the authors to incorporate the feedback and especially the promised improvements in the final version of the paper.
Decision: Accept (spotlight)