Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors define the task of solving a family of differential equations as a task of gradient-based meta-learning generalizing the gradient-based model agnostic meta-learning to problems with differentiable solvers.
main_review: The problem is well formulated. Numerical integrators of differential equations can be sensitive to choices of parameters or initial conditions. At the same time numerical integration is a computationally complex task that can benefit from meta-learning. Iterative solvers and their surrogate meta-solvers can both be implemented by a neural networks which makes the problem approachable by gradient based meta learning.

The authors demonstrate their approach on a family of 1D Poisson equations and incompressible flow simulations. The authors successfully demonstrate the advantages of using gradient based meta solving with a neural network architecture for the meta-solver over a baseline learner and over regular supervised learning on this task.
summary_of_the_review: The presented work is solid. The main concern is with its limited audience in the scope of the conference and potential applications beyond those presented in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a framework for learning the parameters of computational algorithms in order to improve runtime and/or error. They study the specific case of linear systems that arise in PDE solvers, showing an objective whose solution is an initialization that decreases the number of Jacobi iterations required to solve the Poissson equation as well as empirical results for both Jacobi and SOR on several PDE systems.
main_review: Strengths:
1. The paper applies gradient-based methods to important problems of learning initializers for iterative methods in scientific computing.
2. The authors provide a guarantee for initializing the Jacobi iteration, albeit under what seems like a restrictive assumption on the model capacity.
3. The authors demonstrate improvement over simply learning one mapping (“supervised learning”) rather than back propagating through iterations.

Weaknesses:
1. While the improvement over the “supervised learning” setting is interesting, the evaluation largely seems to be in regimes where the error is far too high for practical applications. For example, in Table 1 the MSE of even the best approach seems quite high, although it is difficult for me to get a sense of what a good scale is. Would the advantage continue to hold and be significant-enough to be interesting if the methods were given sufficient number of iterations for practical purposes?
2. There is no demonstration of practical application utility, i.e. whether going through the trouble learning this initialization is actually useful. Is it more useful for me to spend the (likely substantial) amount of effort of back propagating through a lot of classical solves in order to get a better initialization, or just to use the classical solver to begin with. As an example, in the field of neural PDE solvers there is often a demonstration of end-to-end computational savings provided (c.f. Li et al., (2021)).
3. While the claimed framework is very general, it is only studied for linear system solving. The authors also do not compare their overall framework to the substantial work on data-driven algorithm design, which has been studying these problems both theoretically and empirically for quite some time (e.g. Hutter et al. (2011), Balcan (2020), Mitzenmacher & Vassilvitskii (2020)). 

References:  
Balcan. *Data-driven Algorithm Design*. In Roughgarden, *Beyond the Worst-Case Analysis of Algorithms,* 2020.  
Hutter et al. *Sequential model-based optimization for general algorithm configuration*. ICLLO 2011.  
Li et al. *Fourier Neural Operator for Parametric Partial Differential Equations*. ICLR 2021.  
Mitzenmacher & Vassilvitskii. *Algorithms with Prediction*. In Roughgarden, *Beyond the Worst-Case Analysis of Algorithms,* 2020.
summary_of_the_review: While the problem setup is reasonably well-motivated and some of the empirical results are interesting, it is not clear to me how practically relevant the empirical results are for the problems being studied. The very general framework is also only discussed in the restricted case of linear system solving for PDEs. As a result I tend to lean against acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes leveraging data from previous problem instances to improve efficiency of solving similar ones in the future. A general gradient-based method is proposed, which is applied to generating initial guesses to differential equation solutions. This problem is formulated as a meta-learning problem.

main_review: Strengths:
- The paper proposes a very general formulation of “meta-solving” numerical problems. Thorough theoretical foundation and justifications are provided.

Weaknesses:
- The only use-case that is thoroughly empirical validated is solving PDEs. As the paper mentions, other applications, such as root-finding, are applicable. Only evaluating the framework on one application does not showcase its general applicability.
- Data augmentations are required for the incompressible flow simulation experiment. Why isn’t it possible for the meta-solver to learn without these augmentations?
- The formulation of the dataset for the experiment in 3.2 seems arbitrary. Why are the two previous timesteps required? 

Are there stronger baselines that can be compared against? For example, are there problem-specific heuristic initial guesses that can be used that leverage domain knowledge about the particular problem?

Typos:
Section 2.2
- “is a algorithm”
- “to find a good initial weights”
- “for the meta-solving problems”
- “may not be an initial weights”
Section 2.3
- “\theta does not depend on task \tau” (paragraph 2)
- “\theta is weights of another” (paragraph 3)
- “In this work, meta-learning approach” (paragraph 4)
- “tested with multi steps of \Phi” (paragraph 4)
Algorithm 2
- “differntiable solver”
Section 3.1.2
- “tends to ignore high frequencies and more focus on low frequencies”  (paragraph 4)

summary_of_the_review: The paper proposes a general framework for efficiently finding solutions to numerical problems, but only evaluates the framework on PDE problems. Furthermore, additional tricks, such as data augmentations and using the previous two timesteps of the solution, are required to make the method work well empirically.

I’m not very familiar with meta-learning or PDE solvers, so I’m not very confident in my assessment.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a gradient-based algorithm GBMS to solve PDEs based on the solutions of other similar problems. In GBMS, a network is trained to produce good initial guess for the iterative solver of the PDE. Numerical experiments are performed to show the effectiveness of the method.
main_review: Strengths:
- The paper proposed to predict a good initial guess for traditional PDE solvers, so the PDE solver would converge fast. Also, by using traditional PDE solver, the obtained solution is usually more accuracy than other purely data-driven ML methods.

Weaknesses:
- The authors spent a lot of effort to create a new terminology “meta-solving”, which has a board meaning and many other algorithms can be formulated in this way. However, this is only a new terminology, but it is not a new idea or a new algorithm. From the paper, there is no clear evidence that why we would need this new terminology, or for what problems we have to use this new terminology.
- In fact, in the paper, it only tested the problem of generating good initial guess, which is not really a new idea.
- The paper only tested the algorithm on a 1D Poisson equation and 2D incompressible flow. Other challenging problems should be tested.
- There is no comparison between the proposed method and other methods in terms of inference speed and accuracy.
- The paper didn’t provide the details of networks used.

summary_of_the_review: The paper introduced a new terminology and is more like a perspective paper, instead of a comprehensive research article.
review_title: null

Meta-review: The authors define the task of solving a family of differential equations as a task of gradient-based meta-learning generalizing the gradient-based model agnostic meta-learning to problems with differentiable solvers. According to the reviews, there were some concerns regarding the practical value of the paper, for example, (1) the proposed technology is restricted to linear systems, and relatively easy problems (2) there is no demonstration of practical application utility (3) It lacks systematic comparison with other methods (4) some technical details are missing. There were quite a lot of discussions on the paper among the reviewers, and the consensus is that the paper is not solid enough for publication at ICLR in its current form (the reviewer who gave the highest score is less confident and does not want to champion the paper).
Decision: Reject