Gradient-based Meta-solving and Its Applications to Iterative Methods for Solving Differential Equations

1 INTRODUCTION . It is common and important in science and engineering to solve similar problems repeatedly . For example , in material science , a tremendous amount of physical and numerical experiments are conducted to discover and characterize new materials ( Schmidt et al . ( 2019 ) ) . For another example , in computational fluid dynamics , many methods involve solving a Poisson equation to compute the pressure field in every time step of the simulation ( Ajuria Illarramendi et al . ( 2020 ) ) . In these situations , we can utilize the data of the previously solved problems to solve the next similar but unseen problems more efficiently , and machine learning is a natural and effective approach for this . Thus , in recent years , many learning-based methods have been proposed for repeated solutions of computational problems such as partial differential equations ( PDEs ) ( Tang et al . ( 2017 ) ; Özbay et al . ( 2021 ) ; Tompson et al . ( 2017 ) ; Hsieh et al . ( 2018 ) ; Huang et al . ( 2020 ) ) . For example , in Tang et al . ( 2017 ) , Ajuria Illarramendi et al . ( 2020 ) , and Özbay et al . ( 2021 ) , convolutional neural networks are used to predict the solution of Poisson equations , and Ajuria Illarramendi et al . ( 2020 ) and Özbay et al . ( 2021 ) propose to use the predicted solution as an initial guess of traditional numerical methods . Hsieh et al . ( 2018 ) combines a neural network and an iterative solver to accelerate it with maintaining the convergence guarantee . In addition to the regular supervised learning approaches , there are several works where meta-learning approaches are taken to solve computational problems ( Feliu-Fabà et al . ( 2020 ) ; Chen et al . ( 2020 ) ; Psaros et al . ( 2021 ) ; Guo et al . ( 2021 ) ) . Meta-learning , or learning to learn , leverages previous learning experiences to improve future learning performance ( Hospedales et al . ( 2021 ) ) , which fits the motivation utilizing the data from previously solved equations for the next one . For example , Chen et al . ( 2020 ) use meta-learning to generate a smoother of the Multi-grid Network for parametrized PDEs . Psaros et al . ( 2021 ) propose a meta-learning technique for offline discovery of physics-informed neural network loss functions . Although many methods have been proposed in this direction , they are often problem-specific . In other words , there lacks both a unified framework to explain them and a general design pattern to adapt them to new problem settings . On the other hand , in the machine learning literature , there exists a general methodology - gradient-based meta-learning ( Finn et al . ( 2017 ) ) - that covers a variety of meta-learning problem settings . In this paper , we generalize this approach to yield a framework , which we call gradient-based meta-solving ( GBMS ) , that encompasses both learning and computational problems . This offers a general means to understand and develop learning-based algorithms to speed up computation . As an illustration of our approach , we apply GBMS to accelerate the solution of differential equations with iterative methods through learning . We show the advantage of the proposed algorithm over the baseline classical and learning-based approaches through theoretical analysis and numerical experiments . Finally , we incorporate the algorithm into a practical application and demonstrate its versatility and performance . 2 GRADIENT-BASED META-SOLVING . In this section , we introduce our core idea of gradient-based meta-solving . First , we formulate a class of problems , which we call meta-solving , that includes both ordinary meta-learning and learning-based computational problems . We then propose a general gradient-based algorithm for solving them . Under our formulation , many methods proposed in related works can be regarded as special cases of the GBMS algorithm . 2.1 GENERAL FORMULATION OF META-SOLVING . Let us now introduce the general formulation of meta-solving . We first fix the required notations . A task τ is a tuple τ = ( Dτ , Uτ , Lτ ) consisting of a dataset Dτ , a solution space Uτ , and a loss function Lτ . A solution space Uτ is a set of parametric candidate solutions , which is usually a subset of RN for some N ∈ N. A loss function Lτ is a function from Uτ to R≥0 that measures the quality of solution candidates . To solve a task τ means to find an approximate solution û ∈ Uτ which minimizes Lτ ( · ) . Meta-solving considers the solution of not one , but a distribution of tasks by a learnable solver . Thus , we consider a task space ( T , P ) as a probability space that consists of a set of tasks T and a task distribution P , which is a probability measure defined on a suitable σ-algebra on T . A solver Φ is a function from T × Θ to U , where U = ⋃ τ∈T Uτ . θ ∈ Θ is the parameter of Φ , and Θ is its parameter space . Here , θ may or may not be trainable , depending on the problem . Then , solving a task τ ∈ T by an algorithm Φ with a parameter θ is denoted by Φ ( τ ; θ ) = û . A meta-solver Ψ is a function from T × Ω to Θ , where ω ∈ Ω is a parameter of Ψ and Ω is its parameter space . A meta-solver Ψ parametrized by ω ∈ Ω is expected to generate an appropriate parameter θτ ∈ Θ for solving a task τ ∈ T with a solver Φ , which is denoted by Ψ ( τ ; ω ) = θτ . Then , by using the notations above , our meta-solving problem is defined as follows : Definition 1 ( Meta-solving problem ) . For a given task space ( T , P ) , solver Φ , and meta-solver Ψ , find ω ∈ Ω which minimizes Eτ∼P [ Lτ ( Φ ( τ ; Ψ ( τ ; ω ) ) ] . We present some familiar examples , which can be regarded as special cases of the meta-solving problem . First , we can see that conventional meta-learning problems fall in this formulation . Example 1 ( Few-shot learning ) . As an example of the meta-learning problem , we take the fewshot regression problem with MAML ( Finn et al . ( 2017 ) ) . The components of the problem are the following . The task τ is to learn a regression model from data . The dataset is Dτ = { ( xi , yi ) Ki=1 } , which satisfies y = fτ ( x ) for an unknown function fτ . Dτ is divided into the training set Dtrainτ and validation set Dvalτ . The solution parameter space Uτ is a weights space of a neural network ( NN ) that models fτ . Note that U = Uτ because the architecture of NN is shared across all tasks . The approximate solution û ∈ U is the trained weights of NN , which is obtained by training on Dtrainτ . The loss function Lτ : U → R≥0 is the mean squared error ( MSE ) on Dvalτ . The task distribution ( T , P ) is determined by the distribution of the target function fτ and distribution of samples ( x , y ) . The solver Φ : T ×Θ→ U is the single step gradient descent to minimizeLtrainτ ( u ) = 1 |Dtrainτ | ∑ ( x , y ) ∈Dtrainτ ‖NN ( x ; u ) − y‖2 . Its parameter θ ∈ Θ is initial weights u ( 0 ) of NN , so Θ = U . Thus , Φ ( τ ; θ ) = u ( 0 ) − α∇u ( 0 ) Ltrainτ ( u ( 0 ) ) = û , where α is a learning rate . The meta-solver Ψ : T × Ω → Θ is considered as a constant function that returns its parameter ω ∈ Ω for any task τ ∈ T . The parameter ω ∈ Ω is expected to be an appropriate initial weights for all τ ∈ T to be fine-tuned easily . Thus , Ω = Θ = U , and Ψ ( τ ; ω ) = ω = θ = u ( 0 ) . Note that the output of the meta-solver , the initial weights u ( 0 ) , does not depend on τ . Then , the few-shot learning problem is defined as meta-solving problem , which is min ω E τ∼P [ Lτ ( Φ ( τ ; Ψ ( τ ; ω ) ) ] = min ω E τ∼P ∑ ( x , y ) ∈Dvalτ ∥∥NN ( x ; ω − α∇ωLtrainτ ( ω ) ) − y∥∥2 . ( 1 ) In addition to the conventional learning problems , we can regard other computational problems , such as solving a differential equation , as a task of the meta-solving problem . In constrast with MAML , the inner-loop learning is now replaced with a family of iterative solvers for differential equations . This necessitates the distinction of the meta-solver parameter space Ω and the solution space U . Moreover , the meta-solver has to produce a task-specific parameter for the inner solver . Example 2 ( Solving differential equations ) . Suppose that we need to repeatedly solve similar instances of a class of differential equations with a given numerical solver . The solver has a number of hyper-parameters , which sensitively affect accuracy and efficiency depending on the problem instance . Thus , finding a strategy to optimally select solver hyper-parameters given a problem instance can be viewed as a meta-solving problem . The components of the problem are the following . The task τ is to solve a differential equation . In this example , suppose that the target equation is the Poisson equation −∆u = fτ with Dirichlet boundary conditions u = gτ . The dataset Dτ contains data of the differential equation , Dτ = { fτ , gτ } . The solution parameter space Uτ is RNτ . The loss function Lτ : Uτ → R≥0 measures the accuracy of û ∈ Uτ . In this example , the ` 2-norm of the residuals obtained by substituting the approximate solution û into the equation can be used . The task distribution ( T , P ) is the joint distribution of fτ and gτ . The solver Φ : T ×Θ→ U is a numerical solver with a parameter θ ∈ Θ for the differential equation . In this example , suppose that Φ is the Jacobi method ( Saad ( 2003 ) ) and θ is its initial guess . The meta-solver Ψ : T ×Ω→ Θ is a strategy characterized by ω ∈ Ω to select a parameter , an initial guess , θτ ∈ Θ for each task τ ∈ T . Note that the output of the meta-solver depends on τ , which is different from the case of Example 1 . Then , finding the strategy to select parameters of the numerical solver becomes meta-solving problem . The above examples explain why we describe our problem as meta-solving instead of meta-learning . In Example 1 , the task τ is learning from data , and the solver Φ is gradient descent . In Example 2 , the task τ is solving a differential equation , and the solver Φ is an iterative differential equation solver . Regardless of the type of task and algorithm , in both cases , the solver Φ solves the task τ , and we do not distinguish whether Φ is a learning algorithm or other numerical solver . In other words , learning algorithms such as gradient descent are also a type of numerical solvers , and we regard learning as a special case of solving . It is also true for the outer learning algorithm to learn how to solve the task τ with the solver Φ , so learning to solve is a special case of solving to solve . In this sense , we call it meta-solving .

Summary: This paper proposes leveraging data from previous problem instances to improve efficiency of solving similar ones in the future. A general gradient-based method is proposed, which is applied to generating initial guesses to differential equation solutions. This problem is formulated as a meta-learning problem.