Gradient-based Meta-solving and Its Applications to Iterative Methods for Solving Differential Equations
Authors: Sohei Arisaka
Qianxiao Li
Venue: ICLR 2022 Submitted (2022)
Keywords: meta-learning
differential equation
iterative method

Abstract: In science and engineering applications, it is often required to solve similar computational problems repeatedly. In such cases, we can utilize the data from previously solved problem instances to improve efficiency of finding subsequent solutions. This offers a unique opportunity to combine machine learning (in particular, meta-learning) and scientific computing. To date, a variety of such domain-specific methods have been proposed in the literature, but a generic approach for designing these methods remains under-explored. In this paper, we tackle this issue by formulating a general framework to describe these problems, and propose a gradient-based algorithm to solve them in a unified way. As an illustration of this approach, we study the adaptive generation of initial guesses for iterative solvers to speed up the solution of differential equations. We demonstrate the performance and versatility of our method through theoretical analysis and numerical experiments.

Under review as a conference paper at ICLR 2022
GRADIENT-BASED META-SOLVINE AND ITS APPLICA- TIONS TO ITERATIVE METHODS FOR SOLVING DIFFER- ENTIAL EQUATIONS
Anonymous authors Paper under doublc-blind review
ABSTRACT
In science and engineering applications, it is often required to solve similar com- putational problems repeatedly. In such cases, we can utilize the data from pre- viously solved problem instances to improve efficiency of finding subsequent so- lutions. This offers a unique opportunity to combine machine learning (in partic- ular, meta-learning) and scientific computing. To date, variety of such domain- specific methods have been proposed in the literature, but a generic approach for designing these methods remains under-explored. In this paper, we tackle this issue by formulating a general framework to describe these problems, and pro- pose gradient-based algorithm solve them unified way. As illustration of this approach, we study the adaptive generation of initial guesses for iterative solvers to speed up the solution of differential equations. We demonstrate the per- formance and versatility of our method through theoretical analysis and numerical experiments.
INTRODUCTION
common and important in science and engineering solve similar problems repeatedly. For example, in material science, tremendous amount of physical and numerical experiments are con- ducted discover and characterize new materials (Schmidt al.] (2019)). For another example, in computational fluid dynamics, many methods involve solving a Poisson equation to compute the pressure field in every time step of the simulation (Ajuria Illarramendi et al. (2020)). these situa- tions, we can utilize the data of the previously solved problems to solve the next similar but unseen problems more efficiently, and machine learning is a natural and effective approach for this.
Thus, in recent years, many learning- baased methods have been proposed for repeated solutions of computational problems such as partial differential equations (PDEs) (Tang al.) (2017); Özbay 10 Tompson al. (2017): Hsich et al. (2088)Huang Huang et al. (2020)) For example, in Tang et al.
ct al. (017)) Ajuria Illarramendi et al. (020)) and Ozbay et al. (2021), I). convolutional neural net-
works are used predict the solution of Poisson equations, and Ajuria Illarramendi et al. (2020) and Özbay et al. (021) propose to use the predicted solution as an initial guess of traditional numerical methods. Hsich al. (2018) combines a neural network and an iterative solver accelerate it with maintaining the convergence guarantee. In addition to the regular supervised learning approaches, there are several works where meta-learning approaches are taken solve computational problems (Feliu-Fabà et al. (2020); Chen et al. 22000;; Psaros et al. (202T); Guo et al. (202T). ). Meta-learning, or learning leamn leverages previous learning experiences to improve future learning performance (Hospedales et al. (2.2T). which fits the motivation utilizing the data from previously solved equa- tions for the next one. For example, [Chnn et al.) (2020) use meta-learning to generate smoother of the Multi-grid Network for parametrized PDEs. Psaros et al.] (2021) propose meta-learning technique for offline discovery of physics-informed neural network loss functions.
Although many methods have been proposed in this direction, they are often problem-specifee In other words, there lacks both a unified framework to explain them and general design pattern to adapt them to new problem settings. On the other hand, in the machine learning literature, there ex- ists a general methodology gradient-based meta-learning (Finn et al. (2017) that covers a variety of meta-learning problem settings. In this paper, we generalize this approach to yield a framework,

---

Under review as a conference paper at ICLR 2022
which we call gradient-basee based meta-solving (GBMS), that encompasses both learning and computa- tional problems. This offers a general means to understand and develop learning-basee algorithms to speed up computation. As an illustration of our approach, we apply GBMS to accelerate the solution of differential equations with iterative methods through learning. We show the advantage of the proposed algorithm over the baseline classical and learning-based approaches through theo- retical analysis and numerical experiments. Finally, we incorporate the algorithm into practical application and demonstrate its versatility and performance.
2 GRADIENT-BASED META-SOLVING
In this section, we introduce our core idea of gradient-based meta-solving, First, we formulate class of problems, which we call meta-solving, that includes both ordinary meta-learning and comming-based computational problems. We then propose general gradient-base algorithm for solving them. Under our formulation, many methods proposed in related works can be regarded as special cases of the GBMS algorithm.
2.1 GENERAL FORMULATION OF META-SGLVING
Let us now introduce the general formulation of meta-solving. We first fix the required notations. A task r is tuple 7 (D,,4f, L.) consisting of dataset D,, solution space 41, and loss function L,. A solution space U, set of parametric candidate solutions, which is usually a subset of N for some N E N. A loss function L, is function from Ur to Rzo that measures the quality of solution candidates. To solve a task means to find an approximate solution á E Ur which minimizes (-). Meta-solving considers the solution of not one, but a distribution of tasks by a learnable solver. Thus, we consider a task space (T, as a probability space that consists of set of tasks T and a task distribution P, which is a probability measure defined a o-algebra on T. A solver is a function from T x θ to U, where U = =Urer UT- E θ is the parameter of Φ, and θ is its parameter space. Here, 0 may or may not be trainable, depending on the problem. Then, solving a task + E T by an algorithm & with a parameter 0 is denoted by Φ(7;0) 0) û. A meta-solver Vi is a function from T x Z to O, where I and Ω is its parameter space. A meta-solver V parametrized by E Ω is expected to generate an appropriate parameter 8- E θ for solving task E T with a solver 4, which is denoted by (T; = 0,. Then, by using the notations above, our meta-solving problem is defined as follows: Definition 1 (Meta-solving problem). For given task space (T, P). solver b and meta-solver V, find w E Ω which minimizes E,-P [L (T:W))
suitable
E Ω is a parameter of
We present some familiar examples, which can be regarded as special cases of the meta-solving problem. First, we can see that conventional meta-ina problems fall in this formulation.
Example (Few-shot learning). As an example of the meta-learning problem, we take the few- shot regression problem with MAML. (Finn et al. (017)). The components of the problem following. The task r is to learn a regression model from data. The dataset is D, = {(x,, y ) ). which satisfies = f- (ar) for an unknown function fr. D, is divided into the training set Dimin and validation set D The solution parameter space U, is weights space of a neural network (NN) that models f-. Note that U- = U, because the architecture of NN is shared across all tasks. The approximate solution â 2/ is the trained weights of NN, which is obtained by training on Dusin The loss function I., 21 - Ryo is the mean squared error (MSE) on D The task distribution (T, P) determined by the distribution of the target function fr and distribution of
the
samples (I, 8). The solver &b T x + ll is the single step gradient descent to minimize ( (u)
Diaa Σ D(u.y)eDui [|N](;; u) 2112. - Its parameter 0 E θ is initial weights (0) of NN, so θ u. ()) aV., v,, I.tain(
û. where is a learning rate. The meta-solver V. V:T. Ω - O considered as a constant function that returns its parameter w E Ω for any task Ω is expected to be appropriate initial weights for all 7 E T to be fine-tuned easily. Thus, Ω = θ = U, and (r; = w = 0 = (o). Note that the output of the meta-solver, the initial weights u (0)), does not depend on 7. Then, the few-shot learning problem is defined as meta-solving problem, which is
Thus, (T;;0)
E T. The parameter
min E [L, ( & (T:W)) = min E T~p
Σ (g.
|NN( aV train (w)) v112. (1)
TMP
2

---

Under review as a conference paper at ICLR 2022
In addition to the conventional learning problems, we regard other computational problems, such as solving a differential equation, as a task of the meta-solving problem. In constrast with MAML. the inner-loop learning is now replaced with family of iterative solvers for differential equations. This necessitates the distinction of the meta-solver parameter space Ω and the solution space 4. Moreover, the meta-solver has to produce task-specific parameter for the inner solver.
Example 2 (Solving differential equations). Suppose that need to repeatedly solve similar in- stances of a class of differential equations with a given numerical solver. The solver has a number of syper-parameters, which sensitively affect accuracy and efficiency depending on the problem in- stance. Thus, finding a strategy to optimally select solver hyper-parameters given a problem instance can be viewed as meta-solving problem. The components of the problem are the following. The task T is to solve a differential equation. In this example, suppose that the target equation is the Poisson equation ー fr with Dirichlet boundary conditions u gr. The dataset D, contains data of the differential equation, D, = (fr.gr) The solution parameter space U, is R. The loss function I,, 44 - 18z0 measures the accuracy of E Uz this example, the (--oorm of the residuals obtained by substituting the approximate solution û into the equation can be used. The task distribution (T, P)) is the joint distribution of fr and g.. The solver & T x θ - U is a numerical solver with parameter 0 E θ for the differential equation. In this example, suppose that & is the Jacobi method (Saad (2003)) and is its initial guess. The meta-solver V T x 52 - θ is a strategy characterized by w S2 select parameter, initial guess, 8, @ for each task T. Note that the output of the meta-solver depends on т, which is different from the case of Example I Then, finding the strategy select parameters of the numerical solver becomes meta-solving problem.
The above examples explain why we describe our problem as meta-solving instead of meta-learning. In Example|] 2. the task []] the task r is learning from data, and the solver & is gradient descent. In Example solving differential equation, and the solver & is iterative differential equation solver. Regardless of the type of task and algorithm, in both cases, the solver T solves the task T, and we do not distinguish whether & is a learning algorithm words, learning algorithms such as gradient descent are also a type of numerical solvers, and we regard learning as special case of solving. It also true for the outer learning algorithm to learn how to solve the task 7 with the solver 4, so learning to solve is a special case of solving to solve. In this sense, we call it solving..
other numerical solver. In other
2.2 GRADIENT-BASED META-SOLVING
In the previous section, we defined meta-solving problems generalization of meta-learning problems, but how can we solve them effectively? For meta-learning problems such as Example D general methodologies in the form of gradient-based meta-learning (Hospedales al. (2021)) e.g.
MAML (Finn et al. (2017)), have been proposed. In Example [ MAML solves the minimization problem I by updating w using the outer gradient descent algorithm + w - (&). MAML is an algorithm to find good initial weights of the neural network for conventional meta-learning problems, but it can be generalized to the meta-solving problems. In meta-solving problems, tasks may not be learning problems, & may not be gradient descent, and its parameter @ may not be initial weights of neural network. However, we can still employ the same update rule as MAML, as long as Lr, 4 and V/ are differentiable. Thus, for differentiable solvers & and V we propose gradient- based meta- voeta-solving algorithm (Algorithm T as a generalization of MAML. This is a form of data- driven algorithm design, and differs from previous works in this direction (e.g. Hutter et al. [2011). Mitzenmacher & Vassilvitskii (02TT) and Balcan 2021> ). These focus on discontinuous problems such combinatorial optimization, while our framework focuses on differentiable problems.
2.3 ORGANIZING RELATED WORKS
Owing to the general formulation presented above, can organize several related works on learning-based methods for scientific computing and describe them in a unified way. We highlight the advantages of our method using the examples in this section. The detailed description of how the task, the solver and the meta-solver are defined in each case are found in Appendix A
First, let review Feliu-Fabà al. (2020). This work proposes neural network architecture inspired by the nonstantard wavelet form with meta-learning approach to solve the equations con- taining partial differential or integral operators. It can be regarded special case of GBMS, where
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Gradient-based meta-solving algorithm
Require: (P: task space, Φ V: differentiable solver, S: stopping criterion, a: learning rate
Result: w
initialize wt
while S is not satisfied do
Sample t T P;
Generate parameter of & using meta-solver: 0g (w) + V(T:w); Solve task using solver: i(w)= t ((T;
Update meta-solver parameter: w f- - (a
end
task is solving the equation, solver d is the forward computation of the trained neural network,
0 E ④ is its weights, and meta-solver V/ is the constant function that returns the weights 0 = w Ω. Note that 0 does not depend on the task т. We also note that other works where neural networks replace a whole or part of numerical methods (Tang et al. Ozbay et al. (2021); Tompson al. (2017); Hsich et al. (2018) can be organized in the same way. Thus, the meta-solving formulation includes many learning-base based methods where meta-learning techniques are not explicitly employed.
We take Psaros et al. (202T) as another example. In this work, meta-learning is used to learn loss function of the physics-informer neural network, shortly PINN (Raissi et al. (2199), for solving PDEs. This also can be considered
special case of GBMS, where is training the PINN, the solver + is the gradient descent for training the PINN, 0 E θ is the weights of another neural network used as the loss function in the training, and the meta-solver V is the constant function that returns the weights 0 = ~ E Ω. As in the previous example, A does not depend on the task т. We remark that this example and Example T are similar in the sense that & is gradient descent and w is the constant function returning neural network's ' weights in both examples, though the task r is different. The unified framework sheds light on a similarity in various methods for various tasks, which enables us to apply a technique developed for one problem to another easily.
Lastly, let us consider Chen al. (2020). In this work. meta-learning is used to generate a parameter of PDE-MgNet, neural network representing the multigrid method, for solving parametrized PDEs. This also can be regarded as special case GBMS, where 7 is solving a PDE, the solver PDE-MgNet, whose iterative function is implemented by neural network e with weights 0, and the meta-solver V is another neural network with weights w to generate depending on task т. Note that the meta-solver V/ is trained with single step of the solver & but tested with multiple steps of O. In other words, this work does not consider the solver & itself but instead its iterative function & in the training. We will show the importance of this difference in section 3.1
the
The above examples show the generality of our meta-solving formulation that organizes a variety of methods in the systematic way regardless of their type of algorithm. This overall approach allows us to easily design learning-basse based methods for new problem settings, and we will demonstrate it in the next section.
GBMS FOR ITERATIVE METHODS
In this section, we demonstrate how GBMS can be used to yield effective meta-solvers for new prob- lem settings. In particular, we consider the problem of accelerating iterative methods by adaptively choosing initial conditions. Iterative methods are powerful tools solve computational problems. For example, the Jacobi method and SOR method are used to solve PDEs (Saad (2003)). In iterative methods, a function Φ7. which depends on the task т. is iteratively applied to the current approxi- mate solution to update it closer to the true solution until it reaches criterion, such certain error tolerance or number of iterations. These methods require an initial guess as the starting point of the iterations, and the performance of the method depends this choice. Thus, adaptively choosing initial guesses for solvers is an effective way to speed up the computation process.
Here, apply GBMS to derive mota-solver that produces effective initial guesses for iterative solvers. The meta-solving problem here can be viewed as a generalized version of Example P beyond Jacobi solvers. The task is any computational problem which can be solved by iterative
4

---

Under review as a conference paper at ICLR 2022
methods. For example, is solving PDE described in Example 2 The task distribution (T, P) is defined according to each problem. The solver & T x + 2 Is an iterative method with an initial guess / = () E + iterative function Φ7, and the number of iterations k; so ((T: = o (u) (k) and
U. The meta-solver V T x Ω + is a function parametrized by E 22 which takes т E 'I as input and generates an initial guess 0, uy E ④ for the solver @. We implement v by a neural network, so w E 52 is its weights. Then, v/ is trained to minimize the expectation of L- (u())) by the gradient descent. Since both 4 and are implemented in a deep learning framework, VwL+(u()) can be computed by puok-propagation. The entire process of the algorithm is presented in Algorithm 2
Hereafter, we consider solving a PDE a task of the meta-solving problem, but the method is applicable to other tasks, such root finding. Note that although Huang et al., (2020), AAuuiaa II-] larramendi et al. (222)), and Ozbay et al. (2021) propose to use initial guesses generated by neural networks, these initial guesses are independent of the solvers. On the other hand, our initial guesses are optimized for each solver. We will show its advantage the following sections.
Algorithm 2: Gradient-based meta-solving algorithm for iterative solvers
Require: (P: task space, V: differentiable solver, S: stopping criterion, k: number of iterations of the iterative solver, ar: learning rate
Result: w
initialize wt
while S is not satisfied do
Sample task T ~ P:
Form iterative function o, depending on T; Generate initial guess using meta-solver: ut (w) + (TT:W); for i < 1 to & do
Update approximate solution by iterative function: 40 (w) (w) + (24 (w)))))))))
end
Update meta-solver parameter: <- a Tw (k) (w);
10 end
3.1 TOY PROBLEM: ID POISSON EQUATIONS
To study the property of the proposed algorithm, consider solving ID Poisson equations as toy problem. First, we show a theorem that guarantees the improvement of the proposed meta-solving approach for the Jacobi method and linear neural networks. Then, demonstrate that the theorem numerically holds for another iterative method and practical nonlinear neural network.
3.1.1 THEORETICAL ANALYSIS
Let us recall Example and set the domain of interest D = (0, 1). Then, the target equation becomes the following ID Poisson equation with Dirichlet boundary condition:
d²
2x2 u(x) f(x), 10,1
(0)) = a, b.
To solve this equation numerically, is discretized with finite difference scheme, and we rewrite it as the matrix equation Au = S. where the domain [0. 1] is discretized into N points, so A E NRN and u, f e RN Suppose that we solve the equation Au = with the Jacobi method under randomly sampled fr. Then, the meta-solving problem for solving ID Poisson equations is defined by the following. The task is solve Au fr. The dataset D- {fr 4.), where u, is the solution
of Au = fr. The solution parameter space is 24, = R N The loss function is L, (u) = ||u, 2 The task distribution PT to centered and normalized, i.c. the mean of f- is 0 and the covariance is the identity matrix. The solver &; θ 24 is the Jacobi method with & iterations starting at initial guess
P) is determined by the distribution of fr, denoted by P/. It is assumed
5

---

Under review as a conference paper at ICLR 2022
To summarize, 2% (x: ) = of (u( û. Note that D the identity map. The meta-solver V T S2 θ is linear neural network with weights E Q. It can be represented as matrix multiplication for some matrix W, so (T(:W) = f = u u Here, we assume rank(W)) < N to avoid the trivial solution W A-1, i... the meta-solver does not have the capacity to produce an exact solution for each task. Then, the meta-solving problem becomes
v = 8 E O. Its iterative function is Φr (u) = Mu++ 2 where M II - 24.
min E [I. (T; = min E Jo-P,
||,,
- -0 (W
WER T-P
W
As for this meta-solving problem, the following theorem holds. The proof is in Appendix B
Theorem (Guarantee of improvement by meta-solving). For any k 2 0, 3 has the unique mini- mizer Wg. Furthermore, if ki < k22 then for all k z 12.
E Se~P,
1lur (W
||u, -0r--o (W SumPyy
E
(4)
where the equality holds if and only if Wk, = Wkg.
Theorem | guarantees improvement of meta-solving for the considered setting. For example, sup- pose 11 = 0 (regular supervised learning, i.c. directly predicting the solution without considering the solver) and 4/2 = Then, the theorem implies that the meta-solver trained with 5 Jacobi itera- tions is expected to give an better initial guess than the meta-solver obtained by regular supervised learning for any number Jacobi iterations larger than 5. More generally, Theorem | shows that meta-learning with a higher number of Jacobi iterations in the inner solver improves the perfor- mance, provided one carries out at least a greater number of Jacobi iterations during inference. This shows the advantage of the meta-solving approach over regular supervised learning.
3.1.2 NUMERICAL EXAMPLES FOR MORE GENERAL CASES
The key insight from the previous theoretical analysis is that meta-solving leverages the properties of the solver and adapts the selection of initial conditions to it. Here, we show using numerical examples that this is the case for more complex scenarios, involving different task distributions, different iterative solvers, and a practical nonlinear neural network.
The meta-solving problem in this section is defined by the following. The task is the same as the previous section 3.1.1 Let the number of discretization points N be 512. The task distribution (T, is determined by the distribution of We consider two distributions P. and Ph. where the solution u consists of sine functions and hyperbolic tangent functions respectively. Their details listed in Appendix C.I For each distribution, we prepare 30,000 tasks for training, 10,000 for validation, and 10,000 for test. To solve the tasks, we use the Jacobi method and the Red-Black ordering SOR method (Saad (2003)) with iterations starting an initial guess (0) 0 O, denoted by L & and & respectively. Note that they are implemented by convolutional layers. We consider two meta-solvers. One is heuristic initial guess generator BL that takes T as an input and gives the heuristic initial guess, which is the linear interpolation of the boundary condition. 4 BL does not have a parameter and used as a baseline. The other a variant of ID U-Net (Ronneberger et al. (2015) VN with weights E Ω which takes fr and the heuristic initial guess VBL(T) as inputs and generates an initial guess 0, E θ for the solver +.
The meta-solver NN is trained with solvers jac,k k SSOR, & with A: 0, 4, 16, 64 by using the Algo- rithm 2 For each setting, NN is trained six times with different random seeds. The details of the architecture and hyper-
of V NN are found in Appendix D.I The trained meta-sol ver V NN and baseline V VBL are tested with solvers Paackk PSOR,K with & = 0, 4, 16, 64. The performances of the meta-solvee
measured by the MSE on the test set and presented in Table Figure I shows
the comparison of the initial guesses (0) obtained by NN with Ja and Jac Figure is the convergence plot of NN trained with & on P..
The results show that the meta-solver NN is optimized for each corresponding solver 4. In Table I the best performance for a given test solver is achieved at the diagonal where the training and test solvers match. We can explicitly investigate this adaptive phenomena by visualizing the meta-solver outputs for a representative problem instance. Figure[[shone shows that the meta-solver trained with more iterations tends to ignore high frequencies and focus more low frequencies. This is because the

---

Under review as a conference paper at ICLR 2022
Jacobi method converges fast in high frequencies but slow in low frequencies (Saad (2003)), and the trained meta-solver compensates the weakness of the solver. These results show that the meta-solver actively adapts to the nature of the inner-loop solver.
Furthermore, we can observe that Thcorem T holds numerically for the results. Figure illustrates that meta-solver trained with tions, provided the number iterations during testing is large. In addition, at 10,000 iterations, the error of NN trained with $5OR,64 (MSE of 0.0054 and relative error of 0.013) remains approxi- mately 8% better than NN trained with $SOR,O. which shows that the GBMS keeps its advantage for a large number of iterations. These results support that Theorem holds for various practical settings with significant improvement.
iterations is always better than those trained with fewer itera-
Table I: Average MSE of GBMS for solving Poisson equations. Boldface indicates best perfor- mance for each column. Standard deviations are presented Table in Appendix
(a) MSE
Tested with
Trained with
Plac
Pac
Plac61
PSOR,4
$SOR.16
PSOR.64
M NN
P = 2su8,0
0.222
0.220
0.219
0.214
0.218
0.212
0.193
0.277
0.212
0.210
0.205
0.210
0.203
0,186
PJa,, 16
1.169
0.231
0.209
0.204
0.221
0,203
0.185
Jac.84
17.684
3.066
0.279
0.198
0.299
0.198
0.179
① SORR.
1.140
0.982
0.753
0.496
0.201
0.195
0.179
25.786
6.145
1.134
0.554
0.392
0.185
0.170
PSOR.64
22.451
8.306
2.459
0.744
2.536
0.224
0.169
BB.
17.033
12.184
9.482
7.926
9.083
7.611
6,606
(b) MSE
Pe.
Tested with
Trained with
1...0
4
Jac.16 16
Jac.66
$SOR.
$SOR,
PS08.64
W NN
Paac.0 = 4SnR.
1.049
1.041
1.030
1.005
1.028
0.994
0.922
1.059
1.023
1.006
0.976
1.002
0.964
0.888
16
1.202
1.049
1.006
0.971
1.000
0.958
0.885
Puc.
2.017
1.572
1.142
0.940
1.058
0.920
0.839
$OOK.4
1.632
1.479
1.383
1.333
0.984
0.938
0.862
15OR,
3.305
2.597
1.764
1,285
1.305
0.928
0.845
7.683
6.009
3.580
1.417
3.029
1.116
0.756
4 BL
10.761
10.744
10.695
10.539
10.681
10.463
9.908
10
10
1 s/
-3
-5
(1)
-15
-15
(9) generated by Vxx trained with 2 Jac.0
(b) (0) generated by vnN trained with de daccc.
Figure 1: Comparison of initial guesses ()) generated by un
3.2 APPLICATION: INCOMPRESSIBLE FLOW SIMULATIONS
In this section, we introduce an application GBMS for iterative solvers. GBMS is developed for repeatedly solving task sampled from given task distribution P. One of the typical situations where GBMS is effective that similar differential equations are repeatedly solved as step of solving time-dependent problem. For example, in many methods of incompressible flow simula-

---

Under review as a conference paper at ICLR 2022
107
4xw Pocku
Ras trained with Pica.
Pau trmined with Pack
Da troonel with
1UP 10
10P
10a
307
number Iterations
Figure 2: Convergence plot of V trained with $SOR, on P.- Vertical dotted lines are h: 4, 16, 64.
tions, Poisson equations are solved compute the pressure field at every time step, and this process occupies a large part of the computation time for fluid simulations (Ajuria Illarramenii et al. (2020).
Problem definition. Let now describe the incompressible flow simulation setup. We consider 2D channel flow with a fluctuating inflow. The velocity field v and pressure field p follow the Navier-Stokes equations in the form:
Ov
1 (v V)u - Vp vÂv
(5)
Ap = -vV ((v.V)))
(6)
where p is the density, and y is the viscosity. The equation (6) is called the pressure Poisson equation.
Let = 1, // = 0.01. Let the domain of interest 2 be (0, 1) x (0.1) and the time t E [0, 1]. Let y 0 and y 1 be the non-slip wall and fluctuating inflow Vin defined finite difference scheme, we discretize the equations into 128 by 128 spacial grids and 1,000 time steps. Then, and p are computed alternately by the discretized equations. Details of the numerical scheme for solving Navier-Stokes equations follow Barba & Forsyth (2018). GBMS for iterative method is used for solving the pressure Poisson equation (6).
0. By using the
Let us define the meta-solving problem for the above setting. The task J is to solve a pressure Poisson equation. The dataset D, is Pfr.Pr. Pr P:2) where fr is the right hand side and pr is the solution of the pressure Poisson equation. fr-i, Pr-i are those at ith previous time step. Although p- determined by fr theoretically, additional features of previous timesteps can provide useful information to determine a good guess for pr- The solution parameter space U, R 1288x128 The loss function is the relative eerrro The task distribution (T, P) is determined by the distribution of Vin and the computation process of the simulation. Let Pin be the distribution of Vin, whose detail is found Appendix C.2 We prepare inflows for training, 10 for validation, and 10 for test, which are denoted by Vnam Dual and Veeu respectively. Then, generate datasets under each inflow by solving the Navier-Stokes equations with the SOR method without GBMS, where the relative error tolerance is 10-9. The solver &SOR. is the SOR method with k iterations starting at an initial guess E θ. We consider two meta-solvers. One a baseline WBL that takes as an input and gives the heuristic initial guess, which is the solution of the previous step Poisson equation -1- This choice is based on the domain knowledge of fluid simulations and standard in the literature (Fezziger & Perie (002)) The meta-solver NN variant of 2D U-Net with weights Ω which takes D, (p-} as an imput and generates initial guess 0, E θ for the solver $SOR, Note that NN trained with @SOR. o (i.e.
indepedddd initial guess) similar to the method in Ajuuia Illarramendi et al. (2020) and is considered as a data-driven baseline in this paper. We also note that the choice of VNN arbittrrr, and any other problem-specife neural network architectures can be used NN.
Training. We train the meta-solver VN with the SOR solver & with = 0, 4, 16, 64 by using the Algorithm 2 The details of the architecture and hyper-parameters of NN are found in Appendix D.2 In addition, we use data augmentation during training to prevent overfitting to the accuracy training data. During the training, we augment data by proceeding 10 time steps with $ & and V N being trained. Specifically. after computing the loss for Pr. we proceed one time step using &OOR & and NN, and compute the loss for next p1+1 using the computed previous step information.
8

---

Under review as a conference paper at ICLR 2022
Then, it is repeated 10 times for each mini batch. Without the data augmentation, the inputs of NN are from the prepared dataset and are always accurate during training. However, during inference, the inputs of NN are the outputs of the simulation previous time steps, which can be of slightly different distribution than encountered during training (i.e. distribution shift) due to numerical errors and generalization gaps. This then causes further accuracy problems for the simulation at the next step, and this issue compounds itself in time. This eventually degrades the performance if the shift in distribution is not dealt with. The data augmentation is used prevent it. We remark that this training process does not require the whole of the simulation to be differentiable. This is practically important because we can employ GBMS with established solvers by only replacing the Poisson solver and can utilize the other parts without any modification.
Evaluation. We incorporate the trained meta-solvers into simulations and evaluate them by the relative e-error of the velocity field on test inflows under a fixed number of the SOR iterations. More specifically, the performance metric is I Ev 900 Eimo ||v- where v, is the [e
ground truth of ith step velocity under the inflow Unn which is obtained by using VBL and &sOR. with a large enough & to achieve the relative error tolerance of 10-9, and 6 is the ith step velocity obtained by using the trained NN and &SOR,E with a fixed k. To stabilize the simulation, the first 100 steps are excluded, and the simulation with the trained NN starts with 101 steps.
The results presented in demonssettttt the advantage of GBMS. Although all NN diverge for PSOR.0 because of error accumulation, for the other solvers, the best performance with significant improvement is achieved at the diagonal where the training and test solvers match. For example, for the test solver $SOR,4, the accuracy of NN trained with $SOR.4 is approximately 50 times better than regular supervised learning ( NN trained with $ and 130 times better than the classical baseline VBr. Furthermore, GBMS keeps its advantage for a larger number of iterations than the number which it is trained with. For example, for P5OR,64, the meta-solver trained with @ has the best performance, followed by one trained with 15OR,16. PSOR, and PSOR.O in that order. As for computation time, NN trained with PSOR, 1 and tested with $SOR,1 (GBMS approach) takes 11 sec to simulate the flow for one second, while V tested with DSOR, 128 takes 105 sec to achieve similar accuracy to the GBMS approach. Also, NN trained with #SOn,o and tested with $ takes 73 sec to achieve the similar accuracy. Since it takes approximately 7 hours to train NN with a GeForce RTX 3090, our approach is effective if we simulate the flow for more than 268 seconds, which can be easily satisfied in practical settings. To summarize, GBMS performs best for fixed number of iterations and can achieve high accuracy within a smaller number of iterations and shorter computation time than the classical baseline and regular supervised learning.
Table 2: Relative --error of GBMS incompressible flow simulations.
Tested with
V VNN
Trained with
PSOR.O
PSOR,
PSOR,
PSOR 64
&SSOR.D
NaN
0.054861
0.008543
0.001284
SPSOR
NaN 0.001171
0.001625
0.000864
PSOR,
NaN 0,017513
0.000829
0.000723
PSOR.04
NaN
NaN
0.005567
0.000312
BB
1.988177
0,152626
0.014112
0.002941
4 CONCLUSION
In this paper, we proposed a formulation of meta-solving and general gradient-based algorithm (GBMS) to solve this class of problems. In the proposed framework, many related works that used neural networks for solving differential equations were organized in a unified way and regarded as variants of GBMS Thus, the GBMS approach offers general design pattern to develop solu- tion algorithms that blends machine learning and scientific computing. As a concrete illustration, we applied GBMS to iterative methods for solving differential equations and showed its advantage over both classical numerical methods and regular supervised learning. In particular, the proposed method was tested in the incompressible flow simulation and the accuracy was improved by approx- imately 50 times compared to regular supervised learning and 130 times compared to classical baseline for a fixed number of iterative solver iterations. We will study applications of GBMS to other types of problems such as nonlinear equations in future work.

---

Under review as a conference paper at ICLR 2022
5 REPRODUCIBILITY STATEMENT
To ensure the reproducibility, we provide the detailed proof Theorem T in Appendix B As for the experiments, we provide the details of dataset generation in Appendix C and the details of network architecture and training hyper-parameters in Appendix D In addition, we conducted experiments with different random seeds to obtain robust results. Finally, we will make our source code public at a later date. Source code with limited documentation is available upon request.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ekhi Ajuria Illarrmenddi, Antonio Alguacil, Michaël Bauerheim, Antony Misdariis, Benedicte Cuenot, and Emmanuel Benazera. Towards an hybrid computational strategy based on Deep Learning for incompressible flows. In AIAA AVIATION 2020 FORUM, AIAA AVIATION Fo- rum. American Institute of Aeronautics and Astronautics, June 2020.
Maria-Florina Balcan. Data-Driven Algorithm Design. In Beyond the Worst-Case Analysis of Algo- rithms, pp. 626-645. Cambridge University Press, January 2021.
Lorena Barba and Gilbert Forsyth. CFD Python: the 12 steps to Navier-Stokes equations. Journal of Open Source Education, 1(9):21, November 2018.
Yuyan Chen, Bin Dong, and Jinchao Xu. Meta-MgNet: Meta Multigrid Networks for Solving Parameterized Partial Differential Equations. arXiv preprint arXiv:2010. /4088, October 2020.
Jordi Feliu-Fabà, Yuwei Fan, and Lexing Ying. Meta-learning pscudo-diffcrential operators with deep neural networks. Journal of computational physics, 408:109309, May 2020.
Joel H Ferziger and Milovan Perić. Computational Methods for Fluid Dynamics. Springer, Berlin, Heidelberg, 2002.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1 1126-112 1135. JMLR.org, August 2017.
Yue Guo, Felix Dietrich, Tom Bertalan, Danimir T Doncevic, Manuel Dahmen, Ioannis G Kevrekidis, and Qianxiao Li. Personalized Algorithm Generation: Case Study in Meta- Learning ODE Integrators. arXiv preprint 100.0133,, May 2021.
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J Storkey, Meta-Learning in Neural Networks: Survey. IEEE transactions on pattern analysis and machine intelligence, PP, May 2021.
Jun-Ting Hsieh, Shengiia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learning Neural PDE Solvers with Convergence Guarantees. International Conference on Learning Representations, September 2018.
Jianguo Huang, Haoqin Wang, and Haizhao Yang. Int-Deep: A deep learning initialized iterative method for nonlinear problems. Journal of computational physics, 419:109675, October 2020.
Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. Sequential Model-Based Optimization for General Algorithm Configuration. In Learning and Intelligent Optimization, pp. 507-523. Springer Berlin Heidelberg, 2011.
Michael Mitzenmacher and Sergei Vassilvitskii. Algorithms with Predictions. In Beyond the Worst-
Case Analysis of Algorithms, pp. 646-662. Cambridge University Press, January 2021.
Ali Girayhan Özbay, Arash Hamzehloo, Sylvain Laizet, Panagiotis Tzirakis, Georgios Rizos, and Bjöm Schuller. Poisson CNN: Convolutional neural networks for the solution of the Poisson equation on Cartesian mesh.
Engineering. 2, 2021.
Apostolos F Psaros, Kenji Kawaguchi, and George Em Karniadakis. Meta-learning PINN loss functions. arXiv preprint arXiv:2107.05544, July 2021.
M Raissi, P Perdikaris, and G E Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of computational physics, 378:686-707, February 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomed- ical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp. 241. Springer International Publishing, 2015.
Yousef Saad. Iterative Methods for Sparse Linear Systems: Second Edition. Other Titles in Applied Mathematics. SIAM, April 2003.
11

---

Under review as a conference paper at ICLR 2022
Jonathan Schmidt, Mário R G Marques, Silvana Botti, and Miguel A L Marques. Recent advances and applications of machine learning in solid-state materials science. npj Computational Materi- als, 5(1):1-36, August 2019.
Wei Tang, Tao Shan, Xunwang Dang, Maokun Li, Fan Yang. Shenheng Xu, and Ji Wu. Study Poisson's equation solver based on deep learning technique. In 2017 IEEE Electrical Design of Advanced Packaging and Systems Symposium (EDAPS), pp. 1-3, December 2017.
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating eulerian fluid simulation with convolutional networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 3424-3433, JMLR.org, August 2017.
12

---

Under review as a conference paper at ICLR 2022
A DETAILS OF EXAMPLES
Example Feeliu-Fab et al. (020)). Feliu-Fabà et al. (2020) propose the neural network archi- tecture with meta-learning approach that solves the equations in the form L,,(() f(z) with appropriate boundary conditions, where Ln is partial differential or integral operator parametrized by parameter function n(z). This work can be described follows:
Task 7: The task 7 is to solve L,,(x) f(x) for 1] T)r
Dataset D,: The dataset D, is D, = (1), S ,, where fr. ", E TR- are the D. right hand side, and solution respectively. The reference solution u, obtained by [] Solution parameter space U, The solution parameter space U, is a subset of R for
N E N.
-
Loss function I, The loss function L, 4 -> I2 is the mean squared emor with the reference solution, i.e. (ū) |0, ul|?
Task space P): The task distribution (T, P) is determined by the distribution of and fr.
Solver Φ: The solver + T - Ur implemented by neural network imi- tating the wavelet transform, which is composed by three modules with weights 0 (01, 02 0z). In detail, the three modules, Φ1(:;01. d2(::8)). and d3(:;0g) repre- sent forward wavelet transform, mapping 1 coefficients matrix of the wavelet trans- form, and inverse wavelet transform respectively. Then, + is represented by o(T; = dh ((12((12 (2)01 (fr; 0g) ú.
Meta-solver V: The meta-solver V T Ω2 θ is the constant function that returns its parameter w, so (T(:w) = w = and Ω = θ. Note that does not depend on 7 this example.
Then, the weights ~ is optimized by gradient descent algorithm as described in Algorithm D Example (Chen et al. (2020)). The target equation in Chen et al. (020) is linear systems of obtained by discretizing parameterized steady-state PDEs, where u, f E R
equations A, and Ag E N is determined by n) parameter of the original equation. This work can be described as follows:
Task The task is to solve linear system A, = f for n=7+:
Dataset D;: The dataset D, is (Mr, fr).
Solution parameter space U, The solution parameter space U, is RV
Loss function I., The loss function I,, U + R Rg0 is an unsupervised loss based on the residual of the equation, (ū) [| A,, all /1
Task space PT The task distribution (T determined by the distribution of n and fr.
Solver 4: The solver tb x + U is iterations of function d, (=:0) 2 + U that represents an update step of the multigrid method. o, is implemented using convolutional neural network and its parameter 0 is the weights corresponding to the smoother of the multigrid method. Note that weights of Φ, other than 0 are naturally determined by 7 and the discretization scheme. In addition, Φ, takes f, as part of its input at every step, but we write these dependencies as d, for simplicity. To summarize, 4(T; 0) (u 0) a, where & is the number of iterations of the multigrid method and (0) is initial guess, which is 0 in the paper.
Meta-solver V: The meta-solver T Ω θ is implemented by neural network with weights w, which takes A,, as its input and returns weights 0, that is used for the smoother inspired by the subspace correction method.
Then, w optimized by a gradient decent algorithm with the number of multigrid iteration & 1 as described in Algorithm]]
13

---

part of the parameter 0. In the Under review as a conference paper at ICLR 2022
Example 5 (Psaros al. (2021)) ) In Psaros et al. (2021) meta-learning is used for learning loss function of the physics-informet informed neural network, shortly PINN (Raissi et al. (2019) ). The target equations the following:
-, [0, x Bx [x -0, E [0, 0D x(0,z) - (2.A(2), ED,
(a)
(h)
(c)
M is the solution, Fx is nonlinear operator containing differential operators, Bx is a operator representing the boundary condition, Mo,A D + RR. represents the initial condition, and À is parameter of the equations.
where D C RM, is bounded domain, u [0. x D
R
Task 7: The task T is to solve differential equation by PINN:
Dataset D,: The dataset D, is the
points x) E [0, D and the values of u at the points if applicable. In detail, D- Dj.- Db. U U Du.T. where Dj.r: Ds,T. and D are sets of points corresponding the equation (a). (b). and (c) respectively. Du,r is the set of points (t) and observed values u(t, x) at the points. In addition, each dataset D.r . is divided into training set Dirain and validation set D
Solution parameter space Ur: The solution parameter space Ur is the weights space of PINN.
Loss function L,: The loss function L, U + Ryo is based on the evaluations at the points D In detail,
(,( = (1) = L.) + L (0) + val (ū), ua.
where
Wf
Σ
((F(()((,,))))
[Dir
W/b
Σ
(Bxla](t,
Do (t,
Wao
(0) 4o,,
Dupi7
(o.z)
and l: NRN, x N as l.
R20 is function. In the paper, the mean squared error used
Task space (T, P): The task distribution (T, P)) is determined by the distribution of X.
Solver 4: The solver & x + Ur is the gradient decent for training the PINN. The parameter 0 E θ controls the objective of the gradient decent, L (û; 0) [un () + I.rain (ũ; 0) Liman (f; 8) + Lirain (ū; where the difference from L is that parametrized loss lo is used in Liman instead of the MSE in L Note that the loss weights wy, Wh, Wao W,, in Liman are also considered paper, two designs of Eg are studied. One is using a neural network, and the other using a learned adaptive loss function. In the former design, is the weights of the neural network, and in the latter design, 0 the parameter in the adaptive loss function.
Meta-solver Wr The meta-solver V T x Ω - θ is the constant function that returns its parameter w, V/ (T;W) example.
and Ω O. Note that does not depend on in this
Then, the parameter 0 is optimized by gradient decent algorithm described in Algorithm) 1
14

---

Under review as a conference paper at ICLR 2022
B PROOF OF THEOREM 1
Proof of Theorem
1
Equation
3
can be represented as follows:
min W f, Py
ur (W
min E W So-Py
|| (u, (
(7)
min E W St-Py
(A W
Since the mean of fr is 0 and the covariance the identity matrix, W is the minimizer of ( if and only if W is the minimizer of the following:
min (M W)ll
Let X; and 02 (i (i= 1,2...., N) be eigenvalues and eigenvectors of M. Note that X; = cos ir
N+T and
> A > A2 > .... > AN -1. Since A = 2.M, eigenvalues and eigenvectors of are
2 - 2X; and U,. By the eigenvalue decomposition, M and A can be written as
M = VAVT
(10)
A = V(21 2A)V"
(11)
where A diag(X;) and V (11, U2 0...... UN). By the decompositions,
(A-1 Will = (V12- -w)l
(12)
-|
(13)
Thus, the minimizer Wa is
W6 Σ (2 - ( ( 1=1
(14)
where rank(W&)) N, and i(j, k) is the index i where i (1.2..... NJ. Then, the following lemma holds.
takes the top jth value in
2-2x.
Lemma 2. 1f < k2, then for all &s hg
||M WoDll 2 ||M W
(15)
where the equality holds if and only if WA, = Wk..
Since g and 9 are equivalent, Lemma holds, then Theorem holds.
Proof of Lemma 2 Let
:= (1,2, NJ (1(,A) 1, 1,2, ...,,,.. .).
(16)
x
1s is the set of N indices corresponding the least N values of 2-2x.
We have
|| - Wk, 112
M * (A-1 - Wkr)
(17)
A
Σ
Σ
(18)
2
selt,
sElk2
Σ
(19)
2 2
2 2
SElk, Vlz
VEley V8n
15

---

Under review as a conference paper at ICLR 2022
If Wk Wey then 19 equals 0. Assume Wa, # Wk2, Iks # Ixz- Note that Is
(min Ig. /e. I4 + 1,.... min L + (N - r) - = max }.) and for any i E Iks we have
Ad $ |mmax 1,1- In addition, for l < k2, we have max k, max Lg. Let
X
11. = arg min
(20)
ETT Veg
2 2X:
X
4b == arg max
(21)
i 1tn
2 - 2\
C = |1k
Ikzl
= 11kz \ IEal-
(22)
Then, we have
X
19
C
C
3kg
(23)
2
2\imn
2
2Xina
k2 (1-----a 122)
kz (1o-))))))
Aic,
Aiez
=0
(24)
2 2din
2
2drso
>0.
(25)
The last inequality
25
is because for any i E Ikg7
X
(26)
2dins
2
and
Diin
Mmmax loz
2 1eel.
(27)
This completes the proof of Lemma
2
Lemma holds. Thus, Theorem Tholds.
C DETAILS OF TASK DISTRIBUTIONS
C.1 DISTRIBUTIONS OF ID POISSON EQUATION
In Ps, u is represented by
20
u(x) = Σ 1=1
a sin (a, ci),
(28)
where
0, NO, 1)
(29)
b, Unif(0, 128)
(30)
Unif(0, 1).
(31)
In Ph. u is represented by
20 m(x)) Σ 1=1
Lanh(Irz( c;)),
(32)
where
0; \0 10)
(33)
b ~ Unif(0, 30)
(34)
C: Unif(0, 1).
(35)
16

---

Under review as a conference paper at ICLR 2022
C.2 DISTRIBUTION OF INFLOW
In Pa. Vin = (vi, U2) is represented by
"1 (y, 1) w(y, 1) cos (1)
(36)
(2( = w(y, 1) sin =(1,
(37)
where
wy. () = 0.5 + s.n bt sin yo sin tm
(38)
<tt) sin(ct ct d)m
(39)
and
a ~ Unif 10)
(40)
b~ Unif(-5,5)
(41)
em Unif(0,5) 5)
(42)
d~ Uni
(43)
Unif(0, m
5
(44)
D DETAILS OF NETWORK ARCHITECTURE AND EYPER-PARAMETERS
D.1 DETAILS OF SECTION
In section 3.1.2 NN variant of ID U-Net with residual connection leverage the heuristic initial guess consists of four stages with halved resolutions. Each stage has two convolutional layers with kernel size 11 and the activation function tanh. In the first stage, the number of channels is 8, and it is doubled as the resolution is halved. By utilizing the linearity of the Poisson equation, inputs are normalized by || before feeding them into NNN. and the final output ú is denormalized by Il The model is trained for 2000 epochs by Adam with the learning rate 0.0005 and the batch size 512. The learning rate is decreased by 1/5 times when the validation loss does not improve for 200 epochs, and the training is terminated when the validation loss does not improve for 250 epochs. The model with the best validation loss is used for evaluation.
(T), so (T) = NN un) = + IDUNet(ff, u,), where 1DUNet
D.2 DETAILS SECTION 3.2
In section 3.2 NN is variant of 2D U-Net. The differences from NN in section 3.1.2 are its dimension, kernel size 5, and the starting number of channels 16. The model trained for 50 epochs by Adam with the learning rate 0.0005 and the batch-size 64, and the learning rate is decreased by 1/5 at 30 epoch. Note that the meta-solver is trained for time steps for each mini batch by data augmentation. The model with the best validation loss is used for evaluation.
E DETAILS OF EXPERIMENT RESULTS
17

---

Under review as a conference paper at ICLR 2022
Table 3: Standard deviation of MSE of GBMS for solving Poisson equations.
(a) Standard deviation MSE on P.
Tested with
Trained with
Phe
4ac 16
P 64
PSOR
PSOR,
@SOR, 64
NN
@sux,0
0.0052
0.0052
0.0052
0.0050
0.0051
0.0049
0.0043
2uc.4
0.0097
0.0101
0.0098
0.0093
0.0098
0.0091
0.0074
1
0.5548
0.0038
0.0036
0,0033
0.0095
0.0024
0.0020
Ja
4.8458
0.7034
0.0167
0.0037
0.0381
0.0040
0.0029
$SOR,
0.1211
0.1005
0.0739
0.0734
0.0085
0.0080
0.0067
SOR,1G
10.4207
1.9210
0.2923
0.2396
0.0917
0.0072
0.0056
2 S0R.64
4.8430
0.9012
0.3301
0,2788
0.1937
0.0095
0.0078
(b) Standard deviation MSE on Ph.
Tested with
Trained with
Pnc.o
Pac
Pisc
PSOR.4
PSCK,
4sok.64
NN
Ple,0
4SOROO
0.0194
0.0192
0.0189
0.0184
0.0187
0.0181
0.0186
0.0273
0.0261
0.0258
0.0251
0.0257
0.0248
0.0223
16
0.0588
0.0212
0.0195
0.0208
0.0198
0.0213
0.0240
0,2278
0.1431
0.0513
0.0279
0.0369
0.0281
0.0288
$SOR.4
0.6228
0.6871
0.7175
0.7154
0.0286
0.0308
0.0324
PSOR,16
0.3757
0.3590
0.6168
0.7796
0.0309
0.0218
0.0224
SOR.64
1.2871
0.9696
0.4896
0.0872
0.3847
0.0492
0.0309
18


Reviews:
[{"review_id": "iDK9NAl6r8U", "reviewer": "Reviewer_ExFG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper applies gradient-based methods to important problems of learning initializers for iterative methods in scientific computing.\n2. The authors provide a guarantee for initializing the Jacobi iteration, albeit under what seems like a restrictive assumption on the model capacity.\n3. The authors demonstrate improvement over simply learning one mapping (“supervised learning”) rather than back propagating through iterations.\n\nWeaknesses:\n1. While the improvement over the “supervised learning” setting is interesting, the evaluation largely seems to be in regimes where the error is far too high for practical applications. For example, in Table 1 the MSE of even the best approach seems quite high, although it is difficult for me to get a sense of what a good scale is. Would the advantage continue to hold and be significant-enough to be interesting if the methods were given sufficient number of iterations for practical purposes?\n2. There is no demonstration of practical application utility, i.e. whether going through the trouble learning this initialization is actually useful. Is it more useful for me to spend the (likely substantial) amount of effort of back propagating through a lot of classical solves in order to get a better initialization, or just to use the classical solver to begin with. As an example, in the field of neural PDE solvers there is often a demonstration of end-to-end computational savings provided (c.f. Li et al., (2021)).\n3. While the claimed framework is very general, it is only studied for linear system solving. The authors also do not compare their overall framework to the substantial work on data-driven algorithm design, which has been studying these problems both theoretically and empirically for quite some time (e.g. Hutter et al. (2011), Balcan (2020), Mitzenmacher & Vassilvitskii (2020)). \n\nReferences:  \nBalcan. *Data-driven Algorithm Design*. In Roughgarden, *Beyond the Worst-Case Analysis of Algorithms,* 2020.  \nHutter et al. *Sequential model-based optimization for general algorithm configuration*. ICLLO 2011.  \nLi et al. *Fourier Neural Operator for Parametric Partial Differential Equations*. ICLR 2021.  \nMitzenmacher & Vassilvitskii. *Algorithms with Prediction*. In Roughgarden, *Beyond the Worst-Case Analysis of Algorithms,* 2020.", "summary_of_the_paper": "This paper introduces a framework for learning the parameters of computational algorithms in order to improve runtime and/or error. They study the specific case of linear systems that arise in PDE solvers, showing an objective whose solution is an initialization that decreases the number of Jacobi iterations required to solve the Poissson equation as well as empirical results for both Jacobi and SOR on several PDE systems.", "summary_of_the_review": "While the problem setup is reasonably well-motivated and some of the empirical results are interesting, it is not clear to me how practically relevant the empirical results are for the problems being studied. The very general framework is also only discussed in the restricted case of linear system solving for PDEs. As a result I tend to lean against acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LYA27w2oZTm", "reviewer": "Reviewer_yRZK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem is well formulated. Numerical integrators of differential equations can be sensitive to choices of parameters or initial conditions. At the same time numerical integration is a computationally complex task that can benefit from meta-learning. Iterative solvers and their surrogate meta-solvers can both be implemented by a neural networks which makes the problem approachable by gradient based meta learning.\n\nThe authors demonstrate their approach on a family of 1D Poisson equations and incompressible flow simulations. The authors successfully demonstrate the advantages of using gradient based meta solving with a neural network architecture for the meta-solver over a baseline learner and over regular supervised learning on this task.", "summary_of_the_paper": "The authors define the task of solving a family of differential equations as a task of gradient-based meta-learning generalizing the gradient-based model agnostic meta-learning to problems with differentiable solvers.", "summary_of_the_review": "The presented work is solid. The main concern is with its limited audience in the scope of the conference and potential applications beyond those presented in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9Z0mh92tOC", "reviewer": "Reviewer_Wihm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper proposes a very general formulation of “meta-solving” numerical problems. Thorough theoretical foundation and justifications are provided.\n\nWeaknesses:\n- The only use-case that is thoroughly empirical validated is solving PDEs. As the paper mentions, other applications, such as root-finding, are applicable. Only evaluating the framework on one application does not showcase its general applicability.\n- Data augmentations are required for the incompressible flow simulation experiment. Why isn’t it possible for the meta-solver to learn without these augmentations?\n- The formulation of the dataset for the experiment in 3.2 seems arbitrary. Why are the two previous timesteps required? \n\nAre there stronger baselines that can be compared against? For example, are there problem-specific heuristic initial guesses that can be used that leverage domain knowledge about the particular problem?\n\nTypos:\nSection 2.2\n- “is a algorithm”\n- “to find a good initial weights”\n- “for the meta-solving problems”\n- “may not be an initial weights”\nSection 2.3\n- “\\theta does not depend on task \\tau” (paragraph 2)\n- “\\theta is weights of another” (paragraph 3)\n- “In this work, meta-learning approach” (paragraph 4)\n- “tested with multi steps of \\Phi” (paragraph 4)\nAlgorithm 2\n- “differntiable solver”\nSection 3.1.2\n- “tends to ignore high frequencies and more focus on low frequencies”  (paragraph 4)\n", "summary_of_the_paper": "This paper proposes leveraging data from previous problem instances to improve efficiency of solving similar ones in the future. A general gradient-based method is proposed, which is applied to generating initial guesses to differential equation solutions. This problem is formulated as a meta-learning problem.\n", "summary_of_the_review": "The paper proposes a general framework for efficiently finding solutions to numerical problems, but only evaluates the framework on PDE problems. Furthermore, additional tricks, such as data augmentations and using the previous two timesteps of the solution, are required to make the method work well empirically.\n\nI’m not very familiar with meta-learning or PDE solvers, so I’m not very confident in my assessment.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6re0lxCK3LK", "reviewer": "Reviewer_ay6G", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper proposed to predict a good initial guess for traditional PDE solvers, so the PDE solver would converge fast. Also, by using traditional PDE solver, the obtained solution is usually more accuracy than other purely data-driven ML methods.\n\nWeaknesses:\n- The authors spent a lot of effort to create a new terminology “meta-solving”, which has a board meaning and many other algorithms can be formulated in this way. However, this is only a new terminology, but it is not a new idea or a new algorithm. From the paper, there is no clear evidence that why we would need this new terminology, or for what problems we have to use this new terminology.\n- In fact, in the paper, it only tested the problem of generating good initial guess, which is not really a new idea.\n- The paper only tested the algorithm on a 1D Poisson equation and 2D incompressible flow. Other challenging problems should be tested.\n- There is no comparison between the proposed method and other methods in terms of inference speed and accuracy.\n- The paper didn’t provide the details of networks used.\n", "summary_of_the_paper": "The paper proposed a gradient-based algorithm GBMS to solve PDEs based on the solutions of other similar problems. In GBMS, a network is trained to produce good initial guess for the iterative solver of the PDE. Numerical experiments are performed to show the effectiveness of the method.", "summary_of_the_review": "The paper introduced a new terminology and is more like a perspective paper, instead of a comprehensive research article.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors define the task of solving a family of differential equations as a task of gradient-based meta-learning generalizing the gradient-based model agnostic meta-learning to problems with differentiable solvers. According to the reviews, there were some concerns regarding the practical value of the paper, for example, (1) the proposed technology is restricted to linear systems, and relatively easy problems (2) there is no demonstration of practical application utility (3) It lacks systematic comparison with other methods (4) some technical details are missing. There were quite a lot of discussions on the paper among the reviewers, and the consensus is that the paper is not solid enough for publication at ICLR in its current form (the reviewer who gave the highest score is less confident and does not want to champion the paper).
Decision: Reject