Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors presented NeuralPCG, a hybrid ML-numeric PDE solver. While prior ML PDE solvers
often do not have precision guarantees, their approach attains arbitrary precision (up to floating-point
error) by construction.
strength_and_weaknesses: Strength:  The authors propose a novel and efficient NeuralPCG  solver for linear second-order PDEs and present a framework for studying the performance of preconditioners in classic iterative solvers.

Weaknesses: The paper is well written and easy to follow with clear comparison to the literature.  The significance and novelty of this paper are limited.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow with clear comparison to the literature. 
summary_of_the_review: The significance and novelty of this paper are limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a so-called Neural-Network-Preconditioned Conjugate Gradient (NeuralPCG) method to solve partial differential equations. This method uses graph neural network to model the preconditioner of iterative solvers for PDE. It is noted that the NeuralPCG can solve PDE with a satisfying convergence condition and works faster than classic PDE solver. Several classic examples, including passion equation and heat conducted equation, are solved via the NeuralPCG, the comparing results with classic numerical PCG method show the generality and efficiency of the proposed method.
strength_and_weaknesses: Strength: 
The NeuralPCG combine the advantage of classic iterative solver and the machine learning based solver. It can have a good accurate solutions compared with neural network based PDE solver and work faster than classic solver.

Weaknesses: 
The approach is now limited to linear PDEs with symmetric-positive-definite matrix, and the preconditioner’s sparsity is constrained to be the lower triangular sparsity of A.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and the overall quality is good.

The investigation on deep learning based preconditioner of PDE solver can date back to many years ago, some more similar paper can be found, e.g. "Machine Learning-Aided Numerical Linear Algebra: Convolutional Neural Networks for the Efficient Preconditioner Generation" and “Deep Learning of Preconditioners for Conjugate Gradient Solvers in Urban Water Related Problems”. This paper does not clarify differences from other similar papers, except the type of neural network used.
summary_of_the_review: This is a well-written and easy to read paper. However, it did not review enough on the investigations of deep learning based precondition method. Some comments are given as follows:

Comment 1. Maybe the contour plot of Fig.1 and Fig.3 can have a bar of labeled field.

Comment 2: This paper shows the performance of NeuralPCG method based on the Conjungate Gradient method, can this paper give some discussions on the performance of general preconditioning method based on deep learning method?

Comment3: This paper only shows a little reviews on the investigations of deep learning based preconditioner method and does not clarify the distinguished difference of this paper with other similar paper.

Comment4: The accuracy of classic finite element method often shows dependence of the numbers of mesh, does this method also show this dependency?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes NeuralPCG, which learns a preconditioner using neural networks to improve the solution procedure of 
the linear solver for the resulting system of linear equations from discretizing partial differential equations (PDEs). The authors
tested the learned preconditioner for two- and three-dimensional simple PDEs.
strength_and_weaknesses: Strength: 

The paper is easy to read and it is straightforward to catch the basic idea of the paper.

Weaknesses:

1. Learning preconditioners using neural networks for improving solving systems of linear equations has been widely studied. However,
*the entire existing works are completely ignored by the authors*. I actually doubt if this paper has any novelty over existing works. By typing "Learning preconditioner" on the google search engine, we immediately get many related papers, e.g., the first two papers are: https://arxiv.org/abs/1906.06925 and https://arxiv.org/abs/2010.02866. Indeed, the study in the first paper is much more solid than the paper under review. Look at figure 5 of the the paper https://arxiv.org/abs/1906.06925, you will see the existing algebraic multi-grid (AMG) preconditioner is much better than the learned ones, Jacobi. and etc. In the paper under review, the authors never contrast the learned preconditioner with the existing state-of-the-art.


2. The authors claimed that "Classic numerical solvers provide unparalleled accuracy but often require extensive computation time.  Machine learning solvers are significantly faster but lack convergence and accuracy guarantees. We present Neural-Network- Preconditioned Conjugate Gradient, or NeuralPCG, a novel linear second-order PDE solver that combines the benefits of classic iterative solvers and machine learning approaches. Our key observation is that both neural-network PDE solvers and classic preconditioners excel at obtaining fast but inexact solutions. NeuralPCG proposes to use neural network models to precondition PDE systems in classic iterative solvers." --- I wonder if there is any convergence and accuracy guarantee of PDE solvers using the learned preconditioner? If not, then the bottleneck of existing learning-based PDE solvers is also a bottleneck of the proposed approach.'


3. The authors stated that "While this line of methods typically outperforms classical solvers in speed by a large margin, it struggles with converging into a highly precise solution (e.g., 1e − 12). A lack of theoretical analysis on  convergence and accuracy inhibits neural-network PDE solvers’ applications in mechanical engineering, structure  analysis, and aerodynamics, where precise PDE solutions have a higher priority than fast yet inexact results." --- My concern is whether we really need 1e-12 accuracy in real applications. Also, if we need such an accurate solution, do learning-based PDE solvers really faster than classical PDE solvers?


4. The authors stated that "NeuralPCG generalizes to physics parameters or meshes with moderate differences from the training data and is robust to outliers by design." --- Does the learned preconditioner generalizes to PDEs of different types and problems at different scales? AMG can be applied to problems of different sizes and even different types. I would like the authors to make some comparisons of their NeuralPCG with AMG and other state-of-the-art preconditioners.


5. Does NeuralPCG works well when to the linear PDE when A is x-dependent?


6. I wonder if NeuralPCG works for solving high-frequency wave equations?


7. The authors may compare the method with some fast methods for solving PDEs, e.g., fast Fourier transformer and fast multipole method. Rather than consider the method that is applicable to any type of PDEs, which is a highly unfair comparison as the learned 
preconditioner is problem-dependent.


8. Does the learned preconditioner reduce the condition number? I want to see some numerical evidence of reducing 
condition number of the underlying problems.


9. Does over-smoothing of graph neural network affect the performance of the learned preconditioner? The computational cost of 
graph neural networks can be non-negligible when the system of linear equations is massive.


10. In equation (6), the learned matrix L can be dense, significantly reducing the scalability of the PDE solver. This can be
a significant bottleneck of NeuralPCG. 


11. I think the authors should report the computational time, including the time for learning the preconditioner. In particular, for large-scale problems. It seems to me that NeuralPCG is much more expensive than existing SOTA algorithms. E.g. algebraic multi-grid.


12. The baseline preconditioning methods are very weak. In practice, we use much more efficient ones.
clarity,_quality,_novelty_and_reproducibility: Clarity: seems fine.


Quality: not high.


Novelty: very limited.

Reproducibility: Perhaps extremely difficult. The source code is not provided.
summary_of_the_review: The contribution of this paper is very limited. The paper ignores the entire existing research on learning preconditioners for solving systems of linear equations. Indeed, many existing works on deep learning of preconditioners are more solid in algorithm and validation than NeuralPCG. The experimental results are not convincing either.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a neural architecture for learning preconditioners of the form $LL^{\top}$ for sparse positive definite matrices and $L$ is sparse. They also put the right hand side into the neural network design, as in the MeshGraphNet. The main difference is that they generate the matrix $L$ rather than the solution itself.

The idea of the paper is simple and easy to follow, and potentially this is a useful approach.

However, a lot of weaknesses have been pointed out by the reviewers, which I can confirm based on my own reading. First of all, most of the work on sparse Cholesky factorization seems to be unknown to the authors. Incomplete Cholesky factorization (also known as IC(0)) is a classical but not the most efficient method, see for example, second-order LU method by Igor Kaporin from 1998 [1]
Sparsekit software (Y. Saad) has the implementation of so-called ILU(2) preconditioner, which is based on the higher-order sparsity structure [2]. It might be a good idea to extend the idea in the paper for different sparsity pattern.
Finally, the loss function (approximation of the matrix on the right-hand sides) is not the optimal one. The preconditioner should minimize the condition number of the matrix. I suggest to utilize the approach in the paper [3] as the loss function.

[1] Kaporin IE. High quality preconditioning of a general symmetric positive definite matrix based on its UTU+ UTR+ RTU‐decomposition. Numerical linear algebra with applications. 1998 Nov;5(6):483-509.
[2] Chow E, Saad Y. Experimental study of ILU preconditioners for indefinite matrices. Journal of computational and applied mathematics. 1997 Dec 10;86(2):387-414.
[3] Oseledets I, Fanaskov V. Direct optimization of BPX preconditioners. Journal of Computational and Applied Mathematics. 2022 Mar 1;402:113811.


justification_for_why_not_higher_score: The paper is weak on the comparison with state-of-the art sparse LU preconditioners. IC(0) is a very weak baseline.
Decision: Reject