Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper leverages modern scientific computing techniques, specifically the meshless radial basis function-finite difference (RBF-FD) method, to accelerate the training of PINNs. In detail, the paper replaces the computationally expensive automatic differentiation by RBF-FD with sparse-matrix vector multiplication, allowing a high efficiency and accuracy computation of the partial derivative terms, and a fp64 training on GPU. Several numerical experiments are carried out, including linear, non-linear and space-time problems, showing that the proposed method is significantly faster than fp32 vanilla-PINNs with comparable accuracy.
strengths_and_weaknesses: Strengths:
The idea of the paper is straight forward - replacing the automatic differentiation by RBF-FD and sparse matrix-vector multiplication. The theoretical grounding is explained clearly, several empirical evaluations are used to validate the proposed method. The novelty of the contribution is fair, although both the PINNs and RBF-FD are well developed techniques, this is the first work combining them together to accelerate the PINNs training. The topic is closely related to the NeurIPS community.

Weaknesses:
The weakness of the paper is mainly that the evaluation cases considered are somehow naive, which could not demonstrate the advantages of the proposed method sufficiently. The paper barely talks about the motivation of choosing fp64 over fp32 in the first place, and gives little reasons and evidences about why it would cause a training speed-up. And this leads to my suspicion on whether choosing fp64 would become a key point.
questions: 1) The paper claims in line 77-78 that the proposed method “can be trained without any loss in accuracy on irregular domains (just as autograd does)”, while all the evaluation cases are carried out on regular domains, i.e. a unit disk in 2D. Extra evaluation cases with irregular domains are recommended.

2) It is agreed that “the repeated computation of the partial derivative terms in the PINN loss functions via automatic differentiation during training is known to be computationally expensive, especially for higher-order derivatives”. Is the RBF-FD method capable to handle higher-order derivatives? If yes, it is recommended to add extra evaluation cases with higher order PDEs.

3) In section 2.2 the RBF method is explained, while it is not quite clear the computation power required for pre-computation of the RBF-FD weights. For example, in Figure1(b), the time for pre-computing the RBF-FD weights should also be considered to achieve a fair comparison between autograd and SpMV with RBF-FD.

4) Considering the RBF-FD weights, it is recommended to declare whether it is required to recompute and update during the training, especially for complex problems, e.g. strongly non-linear problem, shape deformation/optimization problem?
limitations: This method offers limited speed-up if the nonlinear terms not invoking differential operations dominates the calculation time.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper uses a linear combination of radial bias functions  (RBFs) with additive odd poly-harmonics to discretise partial differential equations in a meshless manner. A deep neural network is used as trial function to obtain the approximate solutions through training over the data points with the equation to be solved as soft constraints.  In the  RBF-FD (finite differences) method the calculation of derivatives is performed with precomputed stencils. Restricting the approximation to nearest neighbors reduces the computational load significantly. The paper displays a solutions of Poisson and Heat dissipation equations as examples of the strengths of the method. It is found that the using a 4-ayer deep neural network that is trained to fulfill the equations using  GPU fp64 is very efficient in solving the problem compared to the state of the art.  
strengths_and_weaknesses: The paper goes trough, in a clear way, the theory of the  physics informed neural networks (PINN= and then builds upon that to introduce RBF-DF in a way that is pretty easy to follow.  It described system addresses many of the problems that arise in  the work of solving PDEs for complicated (and practical) geometries. The meshless nature  of the method makes its able to handle complex cases. Also, as it lacks the usual meshing of the geometry it is  much faster to set up ready for the computation, in addition to being efficient in its calculations.

Of course, one would like that PyTorch would have all the  required autograd functions already implemented for even simpler scripting, but this is no way fault in the paper, but comment on paradigm shift where AI is providing new capabilities and better and better tools for problem solving. The appendix and submitted materials are valuable and greatly appreciated by those planning to use the solution method.

The figures do not have error bars, so one thinks the performance gains are reported from a single training run, albeit there is a good set of comparisons done to state of the art.  I would like to see results gathered from multiple runs for each of the points.                                                                                                                                   
questions: Question:
In the heat equation, the spatial data points are not in a regular mesh, but the mesh in time dimensions is. In the approximation (8) of the difference matrix in the implementation of the RBF-DT, the spatial stencil is based on nearest neighbors. How one is ensuring that the local Courant number is in the correct range (i.e CFL conditions are always fulfilled) despite the varying size of the spatial stencil? Is the global time step chosen in relation to the smallest spatial neighborhood in the irregular mesh? Or is the number of nodes in the spatial stencil made dependent of its local physical size? To me it seems that the RBF-DT with the full difference matrix is an implicit solver where this problem is not necessarily emerging, but making the approximation (8) makes the solver explicit as it will not depend on data further away of the nearest neighborhood.  Or is the fact that the temporal derivatives are calculated analytically with autograd using equation (15) automatically solving the issue?
Could this be clarified in the text for the final version?

Suggestion:
The reference to equations (3) and (2) on line 113 should be (2) and (1).
limitations: I do not think there are any potential negative societal impacts of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
The submitted work proposes the use of a discrete finite difference (FD) method for a faster computation of spatial derivatives (i.e. with respect to the signal domain) in Physics Informed Neural Networks (PINNs).
More specifically, the use of Radial-Basis-Function FD (RBF-FD) is proposed, which approximates an operator at a given position $x_i$ via a weighted linear combination of a position $x_i$ and its neighbors.
The weights can be learned by solving a dense block linear system, and an arbitrary point cloud may be used.
The underlying (implicit) interpolation between the points is a combination of RBFs and Legendre polynomials.
The linear operators are precomputed so that during the PINN training they can be directly evaluated on the collocation points.
Experiments show the application of RBF-FD for the linear and non-linear Poisson equation, as well as the two-dimensional heat equation. For the heat equation, the temporal derivatives are still computed using auto-grad, and only the spatial derivatives are approximated via RBF-FD.

Results show that the proposed method is able to accurately approximate the derivatives of Neural Networks with a significant speed-up compared to automatic differentiation. The speed-up depends on the complexity of the approximated operator.
Applied to PINNs, this results in faster training of the networks with comparable accuracy to vanilla PINNs. Furthermore, the use of RBF-FD required fewer training epochs than vanilla PINNs, however, the exact causes for this are not known.

Contributions:
- The use of RBF-FD for approximating differential operators in PINNs is proposed as an alternative to autograd
- It is empirically shown that, with a sufficiently high order, RBF-FD provides accurate and fast approximations to the Laplacian of a Neural Network
- On experiments for the (non-)linear Poisson equation and the Heat equation a  significant speed-up of PINNs is demonstrated. Fewer epochs were required for reaching a similar error as vanilla PINNs.
strengths_and_weaknesses: ### Significance:
- [High] The authors provide an efficient alternative to autograd for computing PDE losses in PINNs. 
- [Low] The proposed method results in fewer required epochs for attaining a similar loss compared to vanilla PINNs.
- [Low] The proposed method appears to be very general and orthogonal to existing methods for speeding up PINNs, allowing a seamless combination with existing work.

### Originality:
- The usage of RBF-FD as an alternative to autograd for PINNs is to the best of my knowledge novel.
- As far as I can judge, RBF-FD methods seem to be well-established within the numerical PDE literature but have not yet been explored much within Machine Learning.

### Quality:
- The submission and experimental settings appear to be sound to the best of my knowledge.
- While limited in their extent (in terms of different PDEs), the experiments serve as a valuable proof of concept.
- It is not explained, why the approximation is only used for spatial derivatives whereas autograd is used for temporal derivatives.
It is unclear, whether this serves as an example for mixed usage of autograd and RBF-FD, or is due to some limitation of the underlying method. 
- Well maintained and readable code is provided; The appendix provides implementation details for the autograd implementation of sparse matrix operations, that are required for the proposed method.
- No error bars are provided (Checklist 3.c). The authors argue that this is the norm for PINNs, which in my opinion does not warrant ignoring general best practices within the ML community. However, another argument made is the limited computational resources, which is reasonable for the PINN experiments. Experiments concerning Figure 1 (autograd vs RBF-FD) should not be very compute-intensive.


### Clarity:
- The paper is generally well written and argued for.
- (minor) The plots are at times a bit confusing:  
    - The x and y labels could be a bit bigger and more expressive (e.g. "network depth s" instead of just "s" in Figure 1, similar to N in Figures 2-4 and p in Figure 5)
    - It's hard to (quickly) distinguish the vanilla PINN from the proposed method. Maybe a different style or color coding would help.


To summarize,  the paper provides an interesting and feasible alternative for computing spatial derivatives in PINNs, bridging the gap between well-established mesh-free methods for PDEs and PINNs.
The proposed method is novel and to the best of my knowledge technically sound, experiments are reasonable and reproducible, and the paper is well written. 
questions: - Going back to my point in the "Quality" section, why are only spatial derivatives approximated with RBF-FD? 

- Is there any existing understanding or intuition regarding **why** the RBF-FD method requires fewer training epochs? 
limitations: - The authors highlight that the speedup depends on the complexity of the approximated operator, i.e. they may not offer much speedup if nonlinear terms not involving differential operators dominate training times.
- The approximations to the operators have to be precomputed.
review_title: null

Meta-review: metareview: 
All reviewers agreed that this paper has several strengths, such as a convincing motivation, a well structured and well-formulated model and solid theoretical grounding.
While two reviewers had a very positive general impression of the paper (emphasizing, in particular, the novelty and originality of this work), one reviewer raised some concerns about the application cases being too simplistic and not well suited for demonstrating potential strengths or weaknesses of the method. In my opinion, however, theses concerns (and further questions) could be addressed reasonably well in the rebuttal, and therefore, I recommend to accept this paper.


confidence: Certain
Decision: Accept