Backpropagation-free training of neural PDE solvers for time-dependent problems

Authors: Chinmay Datar, Taniya Kapoor, Abhishek Chandra, Qing Sun, Erik Lien Bolager, Iryna Burak, Anna Veselovska, Massimo Fornasier, Felix Dietrich
Primary area: learning on time series and dynamical systems
Keywords: neural PDE solvers, time-dependent partial differential equations, random feature networks, backpropagation-free training

Abstract:
Approximating solutions to time-dependent Partial Differential Equations (PDEs) is one of the most important problems in computational science. Neural PDE solvers have shown promise recently because they are mesh-free and easy to implement. However, backpropagation-based training often leads to poor approximation accuracy and long training time. In particular, capturing high-frequency temporal dynamics and solving over long time spans pose significant challenges. To address these, we present an approach to training neural PDE solvers without backpropagation by integrating two key ideas: separation of space and time variables and random sampling of weights and biases of the hidden layers. We reformulate the PDE as an Ordinary Differential Equation (ODE) using a neural network ansatz, construct neural basis functions only in the spatial domain, and solve the ODE leveraging classical ODE solvers from scientific computing. We demonstrate that our backpropagation-free algorithm outperforms the iterative, gradient-based optimization of physics-informed neural networks with respect to training time and accuracy, often by 1 to 5 orders of magnitude using different complicated PDEs characterized by high-frequency temporal dynamics, long time span, complex spatial domain, non-linearities, shocks, and high dimensionality.

Decision: reject
Reviewer scores: 5, 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729344373500

thread_type:
official_review

<thread_object>
creator: reviewer_prxc

<content>
summary:
This paper proposes to use a hybrid framework consisting of a neural network ansatz and a classical ODE solver to solve typical time-dependent PDEs. Specifically, the neural network ansatz features separation of spatial and temporal variables, and the parameters of this network is randomly sampled rather than trained with back propagation. Numerical experiments are conducted to verify the high accuracy and reduced training time of the proposed method.

strengths:
- The proposed method is novel and provides a distinct method to solve time-dependent PDEs other than classical numerical methods and PINNs.
- The experiment results show that te proposed method outperforms PINNs by orders of magnitude of accuracy; the accuracy is even comparable to classical numerical solvers.
- The authors also provide techniques to satisfy boundary conditions and improve the condition number of the associated ODE.

weaknesses:
- The paper should consider add more backgrounds about the random-sampling methods of neural network weights. Without back-propagation, how does this random-sampling of weights influence the final solution of the proposed method? As can be seen in Table 2, the standard deviations of your proposed method is relatively larger than PINNs, although the accuracy is significantly better. 
- The paper should add some ablation studies to provide more insight about each component of the proposed method. For example, the necessity of the SVD layer, the influence of number of hidden neurons.
- It would add more practicabillity of the proposed method by providing more detailed comparisons between ELM-ODE and SWIM-ODE. Is one strategy better than another, or one should choose between these two strategies based on the PDE to tackle?

questions:
Is the proposed method able to handle PDEs with higher-order time derivatives?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730359688323

thread_type:
official_review

<thread_object>
creator: reviewer_ufbx

<content>
summary:
This paper proposes a backpropagation-free training algorithm for a neural partial differential equation solver, utilizing the Extreme Learning Machine (ELM) framework. The method reformulates the partial differential equation (PDE) as an ordinary differential equation (ODE) problem through the separation of variables, which is then solved using classical ODE solvers. Numerical experiments show that the proposed method outperforms traditional PINNs in both test accuracy and training speed.

strengths:
- Significantly lower relative error compared to PINNs
- Substantially faster training speed than PINNs
- Achieves both improvements without backpropagation while retaining a mesh-free approach

weaknesses:
- The experiments are insufficient to fully support the authors' claims.
- The paper lacks theoretical contributions.
- The proposed method has a limited range of applications, which restricts its overall contribution.

questions:
1. Experiments.
- The boundary conditions are approximated using a boundary-compliant layer. For instance, in the case of periodic BC, the authors approximate $\sin(kx)$ and $\cos(kx)$ by applying a linear transformation to the basis function. However, this raises the question: what advantage does the proposed method offer compared to just using $\sin(kx)$ and $\cos(kx)$ as basis functions, or P1, P2 basis functions in FEM? A numerical comparison in this scenario would be helpful.
- It appears that $C(t)$ is calculated by multiplying the pseudo inverse of feature matrix $[\Phi(X),1]$, where $X$ contains all the collocation points. In cases of high dimensionality $d>>1$ where $N>>1$ to cover the entire domain, there may be significant computational demands. Further discussion and experiments on the computational cost in high-dimensional settings would be needed.

2. Theoretical contributions
- Does ELM possess a universal approximation property? If so, can this be generalized to the neural PDE solver setting?

3. Limited applications
- As the authors mention, the method cannot be applied to grey-box or inverse problem settings. Given this, what advantage does the mesh-free nature provide?
- If the pseudo-inverse calculation for $[\Phi(X),1]$ becomes computationally expensive, especially in high-dimensional problems, what practical benefit does mesh-free implementation offer?
- Overall, what advantages does the proposed method offer over mesh-based approaches? In many cases presented in the paper, mesh-based methods achieve superior test accuracy with shorter training(computing) times.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730392587715

thread_type:
official_review

<thread_object>
creator: reviewer_znn8

<content>
summary:
The paper presents a method for training neural PDE solvers without backpropagation, which aims to improve efficiency in solving time-dependent partial differential equations (PDEs). The authors integrate two main ideas: separating space and time variables and randomly sampling weights and biases in hidden layers. By reformulating the PDE as an ordinary differential equation (ODE) using neural networks for spatial components, they leverage traditional ODE solvers for time evolution. The approach is benchmarked against standard backpropagation-based Physics-Informed Neural Networks (PINNs). It shows improvements in accuracy and speed on complex PDEs involving non-linearities, high-frequency temporal dynamics, and shocks.

strengths:
1. The authors propose a backpropagation-free method that leverages random sampling techniques like Extreme Learning Machines (ELM) and Sampling Where It Matters (SWIM) to address the inefficiencies of traditional backpropagation, especially for complex time-dependent PDEs.

2. The paper reports significant speed gains in training time, with improvements of up to 5 orders of magnitude over standard PINN approaches.

3. Specialized handling of boundary conditions and separation of variables for time-dependent PDEs are some of the contributions that could impact future neural PDE solvers.

4. The authors demonstrate extensive benchmarking across a range of PDEs with different challenges, showing superior performance in terms of speed and accuracy.

5. The paper is well-written and easy to follow.

weaknesses:
The authors have mentioned the limitations of their method and share possible directions to follow in future work.

questions:
Could the authors clarify the absence of experiments involving higher-dimensional PDEs? Given the introduction’s emphasis on the limitations of mesh-based methods—particularly their impracticality in complex domains and high-dimensional spaces—it would be valuable to see examples where the proposed method effectively addresses these challenges. Higher-dimensional cases are particularly relevant to machine learning applications, where scalability in complex domains is critical.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682947561

thread_type:
official_review

<thread_object>
creator: reviewer_jqay

<content>
summary:
The authors present a method of solving PDEs by parameterizing solutions fields with neural networks whose parameters depend on time. The integration scheme solves for the last layer cofficents. The basis functions, induced by the inner parameters, are generated via a data driven or data agnostic way.

strengths:
- The presentation is clear and the literature review is thorough and provides a good introduction.
- The method shows strong performance on the chosen benchmarks

weaknesses:
The motivation for the method is not totally clear. Introducing neural networks to solve PDEs where the parameterization evolves nonlinearly in time is motivated by breaking the kolmogorov n-width as in [1,2,3,4]. In this work the parameterization still evolves linearly in time. The neural network is only used to choose a basis. So it is unclear why one would pick this method over a traditional solver, which are extremely well understood in terms of convergence properties and accuracy. It seems to me the only reason would be to deal with complicated geometries? If so currently the paper does not devote enough attention to arguing and demonstrating this advantage. Additionally for these reason the comparison with PINNs is ill-chosen. The most appropriate comparison would be to traditional methods which also evolve linearly in time. While comparison is made to a finite-element method, this is not the best choice for some of the problems present. For the data-agnostic a reasonable spectral method should also be compared to and for the data-dependent method POD should be compared to.

It would be helpful to:

- make more explicit the advantages over traditional, show this advantages clearly in the experiments.
- add a comparison to a spectral methods for the data-agnostic case.
- add a comparison to POD for the data-dependent case.


[1] Evolutional Deep Neural Networks. Du et al.
[2] Randomized sparse neural galerkin schemes for solving evoluation equations with deep networks. Berman et al.
[3] Positional embeddings for solving PDEs with evolutional deep neural networks. Kast et al.
[4] Breaking the Kolmogorov Barrier with Nonlinear Model Reduction. B Peherstorfer.

questions:
What is the n-width of the problems considered (as given by the spectral decay of the snapshot matrix)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717342859

thread_type:
official_review

<thread_object>
creator: reviewer_62wa

<content>
summary:
In this paper, the authors propose training neural PDE solvers by variable separation and random sampling of neural network weights. The neural network ansatz is utilized for the spatial domain, and the system evolving in time is solved by classical ODE solvers. Extreme learning machines and adaptive sampling techniques (SWIM) are applied for better training efficiency. An SVD layer is introduced to improve the condition number of the associated ODE. It is claimed that the proposed method outperforms PINN by 1 to 5 orders of magnitude in time efficiency and accuracy, for PDEs including Advection, Euler-Bernoulli, Nonlinear diffusion, and Burgers'.

strengths:
- The writing is clear and detailed.
- The experiments are rich in problem types, specific difficulties, and baseline comparisons.

weaknesses:
- Meaning no offense, but I think researchers in AI4PDE with more AI background will think of this work as a huge step backward. The essence of deep neural networks is their surprisingly good performance in approximating high-dimensional functions, and the efficiency of backpropagation in implementing neural networks with huge amounts of parameters. Surely there are still issues even if we can obtain the gradients cheaply, but zeroth-order optimization, according to my personal judgment, cannot be the solution because it will only scale poorly.
- For the experiments, the spatial dimension is 1 or 2, and small in range. It would be interesting to see some results for problems huge in space.

questions:
I hope to confirm with the authors that if you claim supremacy in any metric of the proposed method compared to traditional FEM solvers?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a backpropagation-free method for training neural PDE solvers by separating spatial and temporal variables and leveraging classical ODE solvers for time evolution. The proposed approach offers potential speed and accuracy improvements over PINNs, demonstrating effectiveness across several PDE benchmarks with different challenges, including high-dimensional problems. Notably, the experiments report orders-of-magnitude gains in training efficiency and accuracy in many scenarios, particularly for problems with smooth solutions. The writing is clear, and the authors have conducted extensive benchmarking and provided thoughtful responses to reviewers’ concerns. The inclusion of high-dimensional examples, ablation studies, and additional comparisons with traditional methods strengthens the work.

Despite these contributions, the paper has several limitations that weigh heavily against acceptance. First, the method’s applicability is constrained to relatively simple or smooth problem settings, as it struggles with complex, high-dimensional PDEs with significant variability. While the authors addressed concerns about scalability, the experimental results do not convincingly demonstrate that the method generalizes effectively to challenging real-world scenarios. Additionally, the paper lacks significant theoretical contributions; key questions about the generalizability and universal approximation capabilities of the proposed framework remain unanswered. Furthermore, while the comparisons to PINNs and traditional methods are insightful, they are limited in scope and fail to address the broader challenges in AI4PDE applications, such as inverse problems and grey-box scenarios. Overall, while the paper introduces interesting ideas and offers promising directions, its limited applicability and lack of substantial theoretical grounding suggest it is not yet ready for acceptance.