MetaP: How to Transfer Your Knowledge on Learning Hidden Physics
Authors: Lu Zhang
Huaiqian You
Tian Gao
Mo Yu
Chung-Hao Lee
Yue Yu
Venue: Submitted to ICLR 2023 (2023)
Keywords: meta-learning
neural operator
parametric PDEs

Abstract: Gradient-based meta-learning methods have primarily focused on classical machine learning tasks such as image classification and function regression, where they were found to perform well by recovering the underlying common representation among a set of given tasks. Recently, PDE-solving deep learning methods, such as neural operators, are starting to make an important impact on learning and predicting the response of a complex physical system directly from observational data. Since the data acquisition in this context is commonly challenging and costly, the call of utilization and transfer of existing knowledge to new and unseen physical systems is even more acute.

Herein, we propose a novel meta-learnt approach for transfer-learning knowledge between neural operators, which can be seen as transferring the knowledge of solution operators between governing (unknown) PDEs with varying parameter fields. With the key theoretical observation that the underlying parameter field can be captured in the first layer of the neural operator model, in contrast to typical final-layer transfer in existing meta-learning methods, our approach is a provably universal solution operator for multiple PDE solving tasks. As applications, we demonstrate the efficacy of our proposed approach on heterogeneous material modeling tasks, which shows that our method can handle complex and nonlinear physical response learning tasks while greatly improving the sampling efficiency in new and unseen materials.

Under review as a conference paper at ICLR 2023
METAP: HOW TO TRANSFER YOUR KNOWLEDGE ON LEARNING HIDDEN PHYSICS
Anonymous authors Paper under double-blind review
ABSTRACT
Gradient-based meta-learning methods have primarily focused on classical ma- chine learning tasks such as image classification and function regression, where they were found to perform well by recovering the underlying common represen- tation among a set of given tasks. Recently, PDE-solving deep learning methods, such neural operators, and predicting the response of a complex physical system directly from observa- tional data. Since the data acquisition in this context is commonly challenging and costly, the call of utilization and transfer of existing knowledge to new and unseen physical systems even more acute.
starting make important impact learning
Herein, we propose novel meta-learnt approach for learning knowledge between neural operators, which can be seen as transferring the knowledge of solution operators between governing (unknown) PDEs with varying parameter fields. With the key theoretical observation that the underlying parameter field can be captured in the first layer of the neural operator model, in contrast to typical final-layer transfer in existing meta-learning methods, our approach a provably universal solution operator for multiple PDE solving tasks. As applications, we demonstrate the efficacy of our proposed approach on PDE-based datasets and a real-world material modeling problem, demonstrating that our method can handle complex and nonlinear physical response learning tasks while greatly improving the sampling efficiency in new and unseen tasks.
INTRODUCTION
Few-shot learning is an important problem in machine learning, where new tasks are learned with
very limited number of labelled datapoints (Wang al. [2020). In recent years, significant progress
has been made on few-shot learning using meta-learning approaches (Koch et al., 2015
Vinyals
cet al. 2016 Snell et al. 2017: Finn et al.) 2017 Santoro et al. 2016 Antoniou et al. 2018
Ravi &
Larochelle, 2016 Nichol & Schulman 2018 Raghu et al. 2019 Tripuraneni et al. 2021 Collins
ctt al. 2022). Broadly speaking, given a family of tasks, some of which are used for training and
others for testing, meta-learning approaches aim learn shared multi-task representation that can generalize across the different training tasks, and result in fast adaptation to new and unseen testing tasks. Although most of meta-learning learning developments focus on conventional machine learning problems such as image classification, function regression, and reinforcement learning, studies on few-shot learning approaches for complex physical system modeling problems have been limited. The call of developing few-shot learning approach for complex physical system modeling problems is just as acute, while the typical understanding of how multi-task learning should be applied on this scenario is still nascent.
As motivating example, we consider the scenario of new material discovery in the lab environment, where the material model is built based on experimental measurements of its responses subject to different loadings. Since the physical properties (such as the mechanical and structural parame- ters) in different material specimens vary, the model learnt from experimental measurements one specimen would have a large generalization error on future specimens. That means, the data-driven model has to be trained repeatedly with a large number of material specimens, which makes the learning process inefficient. Further, experimental measurement acquisition of these specimens is often challenging and expensive. In some problems, large amount of measurements not even

---

Under review as a conference paper at ICLR 2023
feasible. For example, in the design and testing of biosynthetic tissues, performing repeated load- ing would potentially induces the gross-linking and permanent set phenomenon, which notoriously alter the tissue durability (Zhang & Sacks 2017). As result, it is critical to learn the physical re- sponse model of a new specimen with samples sizes as small as possible. Furthermore, since many characterization methods to obtain underlying material mechanistic and structural properties would require the use of destructive methods (Misfeld & Sievers] 2007] Rieppo et al. 2008). in practice many physical properties are not measured and can only be treated as hidden and unknown variables. We likely only have limited access to the measurements on the complex system responses caused by the change of these physical properties.
Supervised operator learning methods are typically used to address this class of problems. They take a number of observations on the loading field as input, and try to predict the corresponding physical system response field as output, corresponding to one underlying PDE (as one task). Herein, we consider the meta-learning of multiple complex physical systems (as tasks), such that all these tasks are governed by a common PDE with different (hidden) physical property or parameter fields. Formally, assume that we have a distribution p(7) over tasks, each task 7 ~ p(7) corresponds to a hidden physical property field b²(x) (((%)) that contains the task-specific mechanistic and structural information in our material modeling example. On task T", we have a number of observations on the loading field (x) A(Trd)) and the corresponding physical system response field u" (x) E ((R..)) according to hidden parameter field bx(x). Here, is the sample index, B, A and U are Banach spaces of function taking values in Rd Rd and Rd,, respectively. For task T", our modeling goal is to learn the solution operator g" : A + U, such that the learnt model can predict the corresponding physical response field u(x) for any loading field g(x). Without transfer learning, one needs to learn a surrogate solution operator for each task only based on the data pairs on this task, and repeat the training for every task. The learning procedure would require a relatively large amount of observation pairs and training time for each task. Therefore, this physical-basee modeling scenario raises a key question: Given knowledge on number of parametric PDE solving tasks with different unknown parameters, how can one efficiently learn the best surrogate solution operator for a new and unknown parameter, with only small set of training data pairs
To address this question, we introduce MetaP, novel meta-learnt approach for transfer-learning knowledge between neural operators, which can be seen as transferring the knowledge of solution operators between governing (unknown) PDEs with varying hidden parameter fields. Our main contributions are:
MetaP the first neural-operator- based approach for multi-task learning, which not only preserves the generalizability to different resolutions and input functions from its interral neural operator architecture, but also improves sampling efficiency on new tasks - for com- parable accuracy, MetaP saves the required number of measurements by ~90%.
With rigorous operator approximation analysis, we made the key observation that the hid- den parameter field can be captured by adapting the first layer of the neural operator model, in contrast to typical final-layer transfer in existing meta-learning methods. By construc- tion, MetaP serves as a provably universal solution operator for multiple PDE solving tasks.
From synthetic, benchmark, to a real-world biological tissue datasets, the proposed method consistently outperforms existing baseline gradient-based meta-learning methods.
BACKGROUND AND RELATED WORK
In this section we introduce the relevant materials on hidden physics learning, neural operators, and gradient-based meta-learning methods, which will later complement the definition of our method.
2.1 HIDDEN PHYSICS LEARNING AND NEURAL OPERATORS
For many decades, physics-based PDEs have been commonly employed for predicting and moni- toring complex system responses, then traditional numerical methods were employed to solve these
In some meta-learnin learning literature (e.g.. (Xu et al. 2020)). these small sets of labelled data pairs on a new task (or any task) is also called the context, and the learnt model will be evaluated on additional set of unlabelled data pairs, i.e., the target.
2

---

Under review as a conference paper at ICLR 2023
PDEs and provide predictions for desired system responses. However, three fundamental challenges present. First, the choice of governing PDE laws often determined a priori and free parameters often tuned to obtain agreement with experimental data. This fact makes the rigorous calibration and validation process challenging. Second, traditional numerical methods are solved for specific boundary and initial conditions, well as loading source terms. Therefore, they not gen- eralizable for other operating conditions and hence not effective for real-time prediction. Third, complex PDE systems such turbulence flows and heterogeneous materials modeling problems usually require a very fine discretization, and therefore very time-consuming for traditional solvers.
To provide an efficient surrogate model for physical responses, machine learning methods may hold the key. Recently, there has been significant progress in the development of deep neural networks
(NNs), focusing on learning the hidden physics of complex system (Ghaboussi et al. 1998
1991
Carleo et al., 2019, Karniadakis et al., 202T Zhang et al. 2018 Cai et al.) 2022 Pfau et al. 2020
He et al. 202 Besnard et al. (006). Among these methods, the neural operators show particular promises in resolving the above challenges. Neural operators aim to learn maps between inputs of dynamical system and its state, so that the network can serve as a surrogate for a solution operator
(Li et al. 2020a blc. You et al. 2022a Ong et al., 2022 Gupta et al. 2021 Lu al. 2019 2021b
Goswami et al. 2022a).
Comparing with the classical NNs, the most notable advantages of neural operators are resolu- tion independence and generalizability to different input instances. Moreover, comparing with the classical PDE modeling approaches, neural operators require only data with no knowledge of the underlying PDE. All these advantages make neural operators promising tools to PDE learning tasks, Examples include modeling the unknown physics law of real-world problems (Yin al. 2022a
Goswami et al., 2022a; Yin et al., 20225). and providing efficient solution operator for PDEs (Li jtt al.) 2020a c
al. 2021c a). On the other hand, data in scientific applications are often scarce and incomplete. Utlizationn of other relevant data sources could alleviate such problem, yet no existing work have addressed the transferability of neural operators. Through the meta-learning techniques, our work fulfills the demand of such a transfer setting, with the same type of PDE system but different (hidden) physical properties.
2.2 GRADIENT- BASED META- LEARNING METHODS
One highly successful weta-learning algorithm has been Model Agnostic Meta-Learning (MAML) (Finn et al. (017). which led to the development of a series of related gradient- -based meta-learning
(GBML) methods (Raghu al. 2019 Nichol & Schulman 2018 Antonnon et al. 2018 Hospedales ct al. 2020). Almost-No-Inant -onnet----nne-oooop Loop algorithm (ANIL) (Raghu et al., (219)) modifies MAML by freezing the final layer representation during local adaptation. Recently, theoretical analysis (Collins)
ct al. 2022) found that the driving force causing MAML and ANIL to recover the general repre- sentation is the adaptation of the final layer of their models, which harnesses the underlying task diversity to improve the representation in all directions of interest.
Although MAMI and the general meta-learning approaches have achieved impressive performance in some machine-learning applications such as the image classification and reinforcement learning scenarios, a few work has studied the hidden physics learning under meta (Mai et al. 2021 Zhang
eet al., 2022 Yin et al. 2021; Waag et al., 2021) or even transfer setting (Kailkhura et al.,
2019
Goswami et al., 2022b). . Among these meta- works, (Mai et al.)
|02T| et al. 2022
are designed for specific physical applications, while (Yin et al. [022]] 1 Wang et 2021) focus on on dynamics forecasting by learning the temporal evolution information directly (Yin et al. 2021)
or learning time-invariant features (Wang et al.] 2021). Hence, none of these works have provided generic approach nor theoretical understanding on how to transfer the multi-task knowledge between series of complex physical systems, such that all these tasks are governed by common parametric PDE with different physical parameters.
META-LEARNT NEURAL OPERATOR
3.1 INTEGRAL NEURAL OPERATORS
Here, we first state the base model of this work without the meta aspect. The integral neural oper- ators, first proposed in (Li et al. 2020a) and further developed in (Li et al. 2020ble You et al.)
3

---

Under review as a conference paper at ICLR 2023
Each tsak has eifferent (hioclen) physical parameters
Inout function
Litting layer
Iterativv Fourler layers
Projection layer
Outpul function
x. [
Po
loop for L times
Lxxx, ldr)
Jej
Qa
u(x)
xx [xx)
Pop
Task-wise layers
Common layers shared across all tasks
Figure 1: The architecture of MetaP based on integral neural operator model.
2022a c) comprises of three building blocks. First, the imput function, g(x) A. is lifted to a higher dimensional representation via h 0) (x]](x) := p(x)[x.g(x))' + p(().. Here, P(x) E R(++d,)xd, and p(x) E de define an affine pointwise mapping, which are often taken as constant parameters, i.e., P(x) = P and p(x) = p. Then, the feature vector func- tion h(x, 0) goes through an iterative layer block where the layer update is defined via the ac- tion of the sum of a local linear operator, a nonlocal integral kernel operator, and a bias function: h(, (I+1) = Ji-1 [h(. (At)], for 0, O..... L- 1. Here, ((.,AAt,, O.... L := T/At, is a sequence of functions representing the values of the network at each hidden layer, taking values in Rd JT,.. Iu are the nonlinear operator layers, defined by the particular choice of networks. In this work, we employ the implicit Fourier neural operator (IFNO) as the base model, because of its theoretical universal approximation property in PDE solving tasks You al. 2022c) and robustness in complex physical response modeling tasks (You et al. 2022b 2 In this case, the iterative layers JL J, where
are taken as J1
h(x, (1 + 1) = (At(
(1) Here, and F-1 denote the Fourier transform and its inverse, respectively. In practice, F and F-1 are computed using the FFT and its inverse to each component of separately. Also, c E id defines constant bias, W E Idd is the weight matrix, and F]][::]] ;= R is circulant matrix that depends on the convolution kernel K. a is an activation functioo, which is oftenly taken to be the popular rectified linear unit (ReLU) function. Finally, the output u(-) E 24 is obtained through a projection layer. In particular, we project the last hidden layer representation h(:, (") onto U as: u(x) Q[h(- (x) == (2 7) + q (x)) q2(x). Here, Q (x) E RRd (x( E Rdd xdo, q (x) E Rd and (x) E Rdd are the appropriately sized matrices and vectors that are part of the parameter set that we aim to learn. Similarly as for the lifting layer, (x),, (x)., qı (x) and (x) are also often taken as constant parameters, which will be denoted as Q1, Q2, qi and q2, respectively. In the following, we denote the set of trainable parameters in the lifting layer as 8p, the set from the iterative layer block as 1,, and the set in the projection layer as fQ.
1hhx,111 + Alo (Wh(x, (A() + FF((F(((::))) -F)(, + c(())
The neural operator can be employed to learn an approximation for the solution operator, G. Given D (gg, 1. labelled (context) set of observations, where the input {g:} C A is set of independent and identically distributed (i.i.d)) random fields from known probability distribution on A, and (x) E U, possibly noisy, is the observed corresponding solution, let Ω C R be the domain of interests, we assume that all observations can be modeled with parametric PDE form
Kb(x) [u: = (x), X Ω.
(2)
Here, N'b is the operator representing the possibly unknown governing law, e.g., balance laws. Then, the system response can be learnt by constructing surrogate solution operator of equation 2 Gig; (x) ;= Qs (Je,) ) Poy [x] se u(x), where the parameter set 0 [0p, 0,, 0²] is obtained by solving the optimization problem:
N g: 0). u.)].
min (+) = min Er~ [Clg 8). [gg)) 2s min #
Lic(
(3)
#
Here C' denotes a properly defined cost functional which is often taken as the the mean square error.
We also point out that the proposed multi-task strategy is generic and hence also applicable any other
integral neural operators (Li 2020a ble You al. 2022a
4

---

Under review as a conference paper at ICLR 2023
3.2 BASE META MODEL WITH MAML AND ANIL
To transfer the multi-task knowledge between a series of complex systems governed by different hid- den physical parameters, we proposed to leverage the integral neural operator with a meta-learning setting. Herein, assume that for each training task 7 ~ p(T) we have a set of observations of i=1 and each task can be modeled
loading field/respond field data pairs D == {(g (x), (x)(x)) L
with parametric PDE form
<b<(o|u][(x)) - (x), x x
(4)
where (x) is the hidden task- specifpe physical parameter field for the common governing
law. Given a new and unseen test task, Thett, and a context set of labelled samples Diest :=
(g!!" (x), - (x))) N on it, our goal is to obtain the approximated solution operator model on the test task as G[g; (les).
A straightforward approach would be simply apply MAML and ANIL a neural operator archi- tecture, which will be treated as the baselines of our studies. Here we formally state our implemen- tation of ANIL and MAML for the problem described above.
MAMI. The MAML algorithm proposed in (Finn et al., 2017) aims to find an initialization, 0, across all tasks, so that new tasks can be learnt with very few examples. First, we draw a hatch (T*) |= of H tasks from p(T). For each task 7 split the available of loading field/response field data pairs D to support set of samples, S", which will be used for inner loop updates, and target set of samples, Z", for outer loop updates. Then, for the inner loop we let (%%.0 :- and 0% be the task-wise parameter after gradient updates. During each inner loop update, we compute
0 = gn.i-1 a V 1 (0%.))) for = 1, H,
(5)
where Lsv (6n.-11 is the loss on the support set of the 17-th task, and a is the step size. After m inner loop updates, we update the initial parameter 0 with fixed step size B:
0 0 3 (0), where the meta-loss Lmeta (0) = Σ (z)(0 1
(6)
Then, on the test task, Tiss, we perform inner loop adaptation based on few labelled samples Diest until convergence, and obtain the approximated solution operator model on the test task as 9 [g; (lea).
ANIL. In (Raghu et al. 2019), , ANIL was proposed as a modified version of MAML with inner loop updates only for the final layer. The inner loop update formulation equation 5 is modified as
0
g a a Ls- (0%
for 1, 1,.., H,
where 8%.4 is the task-wise parameter on the projection layer after i gradient updates. Then, we perform the same outer loop updates following equation 6
3.3 METAP: A NOVEL META LETA-LEAA LEARNT NEURAL OPERATOR ARCHITECTURE
We now propose MetaP, which applies task-wise adaptation only to the first layer, iee., the lift- ing layer, with the full algorithm outlined in Algorithm [ Similar as in other meta-learning ap- proaches (Yoon et al. 2018 Vanschoren, 2018 Yang & Kwok 2022 Kalais & Chatzis) 2022). the algorithm consists of two phases: (I) meta-train phase which learns shared iterative layers pa- rameters 0 and projection layer parameters Op from existing tasks; (2) a meta-test phase which transfers the learned knowledge and rapidly learning surrogate solution operators for unseen tasks with unknown physical parameter field, where only a few test samples are required.
To see the inspiration of the proposed architecture, without loss of generality, assume that the underlying task parameter field (x)),, modeling the physical property field, is normalized and sat-
isfying (>() b(x) (22 S for all 77 E (1...., , H). where b == ET Denoting Fu[b] == AG, [u] as function from physical parameter fields B to loading fields A, we can take the Fréchet derivative of F with respect to b b and obtain:
Kb [u] = F. + D.F (b] b) + o[| - (obb-bblzgg))) )
5

---

Under review as a conference paper at ICLR 2023
Meta-Train Phase:
1 Require: a batch (T === of known tasks and available data pairs D = {(g) (x)))
each task
Output: common parameters 07 and 02 across all tasks
1. randomly initialize 12, lq. and {
2. solve the optimization problem:
{07, f/g {07-)
1 (6 (0))) 1
argmin
Σ
CDn Coo 8,, 02))
Meta-Test Phase:
Require: test task T and few labelled data pairs Dies (g) test (x). u [x
Output: the task-wise parameter opsi* and the corresponding surrogate PDE solution operator
Glg: 10p". 07 (x)](x) ) for the test task
3. solve for the lift layer parameter from the optimization problem:
0
argminG (10')' 07, 02])
Algorithm 1: MetaP for Few-Shot I Learning of New PDE Solver with Hidden Physical Parameters
Substituting the above formulation into equation
obtain
F b + DF u? [b" - -b) = g/
Denoting F1 [b²] == [1 D and F2(u == [Fug b). DFug [b]
we can actually reformulated
equation 4 into more generic form:
Fi [x) (2(u = (x). E
(8)
We point out that this parametric PDE form is indeed very general and finds applications in many science and engineering applications besides our motivating example on material modeling, ex- amples also include the monitoring of tissue degeneration problems (Zhang & Sacks 2017), the detection of subsurface flows (Dejam et al., 2017). the nondestructive inspection in aviation (Fallah) al. 2 and the prediction of concrete structures deterioration (Wei al. (02T). etc.
In the following, we show that MetaP are universal solution finding operators for the multi-task PDE solving problem in equation in the sense that they can approximate fixed point method to desired accuracy. For simplicity, we consider a 1D domain P2 and scalar functions F1 [.]" ]. F [2("]] These functions are assumed to be sufficiently smooth and measured at uniformly distributed nodes
= (x1, X2, ...,, XM) }. with Fi [x,] A 0 for all v, and j. Then, equation 8 can be formulated as an implicit system of equations:
Fx(4)((x1) (((xx))F((bb'((x
HUU G?))
0,
F2(u - g (xM) (b>]
(x1) (x2) (XM) is the solution we seek, G (x1) [xi]( gg (x2) F (x2), 8. (XM) (xM))FFF [b*[(()(xx)r) is the reparameterized loading vector, and G == g (x)) (x))..... g." (x is the original loading vector. Here, we notice that all task-specific information are encoded in G? and can be captured in the lifting layer parameter. Therefore, when seeing equation 9 as an implicit problem of UT and G it is actually independent of the task parameter field b i.e., this problem is task-independent. In the later contents refer to equation 9 without the task index, as H(U*; G) for notation simplicity.
where U
To solve for U from the nonlinear system H(U"; Ć) 0, a popular approach would be to fixed-point iteration methods such as the Newton-Raphson method. In particular, with an initial guess of the solution (denoted as U9), the process is repeated produce successively better approx- imations to the roots of equation from the solution of iteration i (denoted as U`) to the solution of iteration + (denoted as Ul+1]
as:
U = U
= RRU
(10)

---

Under review as a conference paper at ICLR 2023
until sufficiently precise value is reached. In the following, show that long as Assumptions and 2 hold, i.e., there exists a converging fixed point method, then MetaP can be seen as an re- semblance of the fixed point method in equation 10 and hence acts as universal approximator of the solution operator for equation g Assumptions [ and 2 ensure the hidden PDEs to be numer- ically solvable with converging iterative solver, which is required condition of most numerical PDE solving problems. Then, taking U ;= _', XM] as the initial guess, we aim to show that for any desired accuracy
one can find a sufficiently large L > 0 and sets of parame- ters 6 = {6p) fI, fq). J. such that the resultant MetaP model acts as a fixed point method and its prediction satisfies | o (Jo,)² o Poz, (LU), G U
S E for all tasks and samples.
12(aM)
Assumption 1.
There exists a fixed point equation, U = U R(U, G) for the implicit problem + RMM is a continuous function satisfying R(U, G) = and
equation such that R RRM. |RR(U, RRÜ, G) S mlI - Ulle(s for any two vectors U, E RM Here, m 0 constant independent of G.
Assumption 2. With the initial guess == R(U', G) (1 - 0,I,...) converges, i.e., for any given O, there exists integer L such that any
[x1,-.. ..
xMI), the fixed-poine iteration U'-I = U' +
||U UM"la(( S. >>> L,
for all possible input instances & R.M and their corresponding solutions U
Then, we have our universal approximation theorem as below, with proof provided in Appendix A:
Theorem (Universal approximation). Given Assumptions 2 let the activation function a for all iterative kernel integration layers be the ReLU function, and the activation function in the projection layer be the identity function. Then for any >> O, there exist sufficiently large layer number L > O and feature dimension number da. O, such that one can find a parameter set for the multi-task problem, ( = 10'p. 11, 0q), such that the corresponding MetaP model satisfies
Ono o (Jor)* o Poz((U', "O"0.))))-""" Un -U))) se,
AG E RRM.
for all tasks.
EXPERIMENTS
In this section, we demonstrate the empirical effectiveness of the proposed MetaP approach. Specif- ically, we conduct experiments on a synthetic dataset from a nonlinear PDE solving problem, a benchmark dataset of heterogeneous materials subject large deformation, and a real-world dataset from biological tissue mechanical testing, and compare the proposed method against competitive GBML baselines. All of the experiments are tested using PyTorch with Adam optimizer, with de- tailed settings provided in the Appendix B. In all tests we considered the averaged relative error, [|ud, u, ( /ls.://|lllla/00 as the error metric (lower means better). We have repeated each experiment for 5 times, and reported the averaged errors and their standard errors.
4.1 SYNTHETIC DATA SETS AND ABLATION STUDY
We first consider the PDE solution finding problem of the Holzapfel-Gasse--ddee (HGO) model (Holzaptel al. 2000), which describes the deformation of hyperelastic, anisotropic, and fiber-reifforeed materials. Different tasks correspond to different material parameter sets, (ki kg. E.v.a.) a), where k and 12 are fiber modulus and the exponential coefficient, respectively, E is the Young's 's modulus, v is the Poisson ratio, and is the fiber angle direction from the reference di- rection. In this example the physical response of interests is the displacement field u [o, 1]2 + R² subject different traction loadings applied the top edge of this material. Therefore, we take the input function g( gxx) as the padded traction loading field, and the output function as the corresponding displacement field.
To investigate the performance of MetaP in few-shot learning, we generate 60 tasks for training, val- idation, and in-distribution (ID) test task by sampling the physical parameters ky, ky ~ 0. 1], E (0,55, 1.5). v/ ~ 10 0.49], and a ~ 2[[=110, π /2.. Here 2/ stands for uniform distribution. To further evaluate the generalizability when the physical parameters in test tasks are outside the
7

---

Under review as a conference paper at ICLR 2023
10
MetaP
10
Relative Test Error
MetaPs
Metalaast Single
Relativee Test Error
In Distribution Test Task
Out-of-Ditribution
10
Out-of-Distribution Test Task?
Single+
MAML
10"
10
ANIL
10 2
10
300
100
300
N Nest
N test
Figure 2: Results on a synthetic data set. Left: The ablation study comparison on test errors in the in-distribution test. Right: The relative error of MetaP in in-distribution and out-of distribution tests.
training regime, we also generate 2 out-of-distribution (OOD) test tasks with physical parameters from different distributions. For the first OOD task we sample E ~ 2(115, 2]. while for the sec- ond OOD task E 14[0.3, 0.5]. As show in Figure 5 of Appendix B. the first OOD task (denoted as "OOD Task corresponds to stiffer material sample and smaller deformation for each given loading, while the second OOD task (denoted as "OOD Task2") generates a softer material sample and larger deformation. For each training task, we generate 500 data pairs 'Di == {(g," u²") } 5.00 by sampling the vertical traction loading from a Gaussian random field. Then, the corresponding ground-truth displacement field is obtained using the finite element method implemented in FEniCS [Alnas et al. 2015). For the test tasks, we train with Viest = (2, 4, 8, 12, 20, 100, 300} numbers of labelled data pairs (the context set), and evaluate the resultant model 200 data pairs (the target set). An 8-layer IFNO is employed as the base model.
=
additional dataset with
Ablation Study. We first conduct an ablation study with three settings. 1) Follow the meta-train and meta-test phases as in Algorithm|] [ with task-wise adaptation only to the lifting layer in both phases (denotes as "MetaP"). 2) After MetaP, perform an additional fine-tuning step to all parameters in the meta-test phase (denotes as "MetaP+"). With this test, we aim to investigate if our algorithm has successfully identified all the common features in the iterative and projection layers. 3) Apply task-wise adaptation only to the projection layer in both meta-train and meta-test phases (denoted as "MetaLast"). With this test, we study if the successful "adapting last layers" strategy in image classification problems would also apply for our PDE solving problem. Besides these three settings, we also report the few-shot learning results with four baseline methods: 1) Learn a neural operator model only based on the context data set on the test task (denoted as "Single"), 2) Pretrain a single neural operator model based on all training task data sets, then fine-tune it based on the context test task data set (denoted as "Single+"), 3) MAML, and 4) ANIL. As shown in the left plot of Figure MetaP and MetaP+ are both able to quickly adapt with few data pairs - to achieve a test error below 5%. "Single" and "Siingl+" require 100 data pairs, while MetaP and MetaP+ requires only 4 data pairs. On the other hand, MetaLast, MAML and ANIL have similar performance. They all require 100 data pairs achieve < 5% test error. This observation verifies our theoretical analysis: on the multi-task parametric PDE solution operator learning problem, one should adapt the first layer, not the last ones. Moreover, when comparing MetaP and MetaP+ we can see that the additional fine- tune step barely improves the performance, especially in the few-sample regime. This fact verifies the efficacy of MetaP, and indicates that our method has successfully capture the underlying task diversity by adapting the first layer, so no further fine-tuning is required.
in-Ditrribution and Dfittri-itttittttinn Tests. On the right plot of Figure 2 we demonstrate the relative test eтror of MetaP in both ID and OOD tasks. We can see that these three test errors are both in similar scale as the error on training tasks. The error from OOD task1 is slightly smaller than the ID test task error, while the error from OOD task2 is much larger, probably due to the fact that the solutions in OOD taskl generally have smaller magnitude and hence its solution operator lies more in linear regime, which makes the solution operator learning task casier. These results validates the good generalization performance of MetaP.
4.2 BENCHMARK MECHNICAL MNIST DATASETS
To further test the capability of MetaP on benchmark datasets, we test MetaP and four baseline methods on Mechanical MNIST (Lejeune 2020). Mechanical MNIST is a dataset of heterogeneous material undergoing large deformation. It contains 70,000 heterogeneous material specimens, and each specimen is governed by the Neo-Hookean material with a varying modulus converted from
8

---

Under review as a conference paper at ICLR 2023
10 0
10
Relative Test Error
MetaP Single Single+
Relative Test Error
MetaP
Single
Singlo+
MAML
MAMLL
ANIL
10
ANIL
10
-2
10
4
8
12
20
100
300
N lest
N lest
Figure 3: Comparison of MetaP and four baseline methods on the benchmark dataset (MechnicalM-
NIST, left plot) and the real-world dataset (heart valve tissue, right plot).
the MNIST bitmap images. On each specimen, 32 loading/response data pairs are provided Here in, randomly select 101 specimens for training and validation, and a new and unseen specimen as the test task. On the meta-test phase, we reserve 20 data pairs on the test task as the target set for evaluation, then train each model under the few-shot learning setting with Nust {2, 4, 8, 12) labelled data pairs as the context set. All approaches are developed based on an 8-layer IFNO model.
We present the results in the left plot of Figure The neural operator model learned by MetaP again outperforms the state-of-the-art GBML models. Our MetaP model achieves 1% when using only 2 labelled data pair on the test task, while the "Single" model has around 100% error even using labelled data pairs, due to overfitting. This fact highlights the importance of learning across multi- tasks in engineering applications when the total number of measurements on each specimen is notice that in this
limited, it is necessary to transfer the knowledge across specimens. Moreover, example ANIL is the least effective GBML method, which even less efficient than the pretrained model ("Single+"), probably due to the inefficacy of the adapting last layers strategy.
4.3 APPLICATION ON REAL-WORLD DATA SETS
We now take a step further to demonstrate the performance of our method on real-world physical response dataset which is NOT generated by sovling PDEs. We consider the problem of learning the mechanical response of multiple biological tissue specimens from DIC displacement tracking measurements. As demonstrated in Figure | we measure the biaxial loading of tricuspid valve ante- rior leaflet (TVAL) specimens from porcine heart, such that each specimen (as task) corresponds to a different region of the leaflet. Due to the material heterogeneity of biological tissues, these specimens are with different mechanical and structural properties.
In this task, we aim to model the tissue response by learning a neural operator mapping the boundary displacement loading to the interior displacement field on each tissue specimen. On each specimen, we have 500 available data pairs. Due to the challenges of obtaining the experimental tissue, only 14 specimens are available total. This example also stands for common challenging setting in real-worke applications: we not only have the few-shot learning challenge, but also suffer from the difficulty from limited available training tasks. With a 4-layer IFNO as the base model, we train each model based on Niest E [2, 300) samples. then evaluate the performance
another 200 samples. The results are provided on the right plot of Figure 3 MetaP performs the best with low data samples among all the methods, and still beat our MAML and ANIL variants when Nest 300. Interestingly, MAML and ANIL did not even beat the "Single+" method, possibly due to the low efficacy of the adapting last layers strategy and the small number of training tasks.
CONCLUSION
In this paper we propose MetaP, the first neural-operator-basse based meta-learning approach that are de- signed to achieve good transferability in learning complex physical system responses with significant improvement sample efficiency. The first layer adaption used by our method is theoretically mo- tivated and shown to be the universal solution operator for multiple parametric PDE solving tasks. We demonstrate the effectiveness of our proposed MetaP algorithm on various synthetic and real- world datasets, showing promises over baseline methods. For future work, we will investigate the applicability of the proposed approach to other neural operators.
We have excluded small deformation samples with the maximum displacement magnitude S 0.1.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Alnæs, Jan Blechta, Johan Hake, August Johansson, Benjamin Kehlet, Anders Logg, Chris Richardson, Johannes Ring, Marie E Rognes, and Garth N Wells. The fenics project version 1.5. Archive of Numerical Software, 3(100), 2015.
Antreas Antoniou. Harrison Edwards, and Amos Storkey, How train your maml. arXiv preprint arXiv:1810.09502. 2018.
Gilles Besnard, François Hild, and Stéphane Roux. "finite-clement" displacement fields analysis from digital images: application to portevin-le châtelier bands. Experimental mechanics, 46(6): 789-803, 2006.
Ilka B Bischofs and UIrich S Schwarz. Effect of poisson ratio on cellular structure formation. Physical review letters, 95(6):068 102))0068022202000 2005.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics- informed neural networks (PINNs) for fluid mechanics: A review. Acta Mechanica Sinica, pp- 1-12, 2022.
Giuseppe Carlco, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Re- views of Modern Physics, 91(4):045002, 2019.
Liam Collins, Aryan Mokhtari, Sewoong Oh, and Sanjay Shakkottai. Maml and anil provably learn representations. arXiv preprint arXiv:2202.03448. 2022.
Morteza Dejam, Hassan Hassanzadeh, and Zhangxin Chen. Pre-darcy flow in porous media. Water Resources Research, 53(10)81177-8200, 2017.
F Fallah, Mazdak Ghajari, and Yasser Safa. Computational modelling of dynamic delamination in morphing composite blades and wings. The International Journal of Multiphysies, 13(4):393- 430, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, volume 70, pp. 1126-1133 PMLR, 2017.
J Ghaboussi, JH Garrett Jr, and Xiping Wu. Knowledge-based modeling of material behavior with neural networks. Journal of engineering mechanics. 117(1):132-153, 1991.
Jamshid Ghaboussi, David A Pecknold, Mingfu Zhang, and Rami M Haj-Ali. Autoprogressive training of neural network constitutive models. International Journal for Numerical Methods in Engineering, 42(1):105-126, 1998.
Somdatta Goswami, Aniruddha Bora, Yue Yu, and George Em Karniadakis, Physics-informed neu- ral operators. 2022 arXiv preprint WrXiv:2207.05748. 2022a.
Somdatta Goswami, Katiana Kontolati, Michael D Shields, and George Em Karniadakis. Deep transfer learning for partial differential equations under conditional shift with deeponet. arXiv preprint #rXiv:2204.098100 2022b.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for dif- ferential equations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL littps://openreviee net./foru Forum d=1.2 CCGL.
Qizhi He, Devin W Laurence, Chung-Hao Lee, and Jiun-Shyan Chen. Manifold learning based data-driven modeling for soft biological tissues. Journal of Biomechanics, 117: 10124, 2021.
Gerhard A Holzapfel, Thomas C Gasser, and Ray W Ogden. A new constitutive framework for arterial wall mechanics and a comparative study of material models. Journal of Elasticity and the Physical Science of Solids, 61(1): 2000.
10

---

Under review as a conference paper at ICLR 2023
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey. 2020 arXiv preprint arXiv:2004.05433. 2020.
Bhavya Kailkhura, Brian Gallagher, Sookyung Kim, Anna Hiszpanski, and T Yong-Jin Han. Reli- able and explainable lional Materials, 1-1):1-9, 2019.
leanning methods for accelerated material discovery. upj Computa-
Konstantinos Kalais and Sotirios Chatzis. Stochastic deep networks with linear competing units for model-agnostic meta-learning. In International Conference on Machine Learning, pp. 10586- 10597. PMLR, 2022.
George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422-440, 2021.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, pp. 0. Lille, 2015.
Annika Lang and Jürgen Potthoff. Fast simulation of gaussian random fields. Monte Carlo Methods 1.0099 URL https://
and Applications, 17(3):1952211 2011. doi: doi aegl.org/13 /l. 1515 /wt///mma...mmm 2011.00100000 .009
Devin Laurence, Colton Ross, Samuel Jett, Cortland Johns, Allyson Echols, Ryan Baumwart, Rheal Towner, Jun Liao, Pietro Bajona, Yi Wu, et al. An investigation of regional variations in the biaxial mechanical properties and stress relaxation behaviors porcine atrioventricular heart valve leaflets. Journal of Biomechanics, 83:16-27, 2019.
Emma Lejeune. Mechanical mnist: benchmark dataset for mechanical metamodels. Extreme Mechanics Letters, 36:100659, 2020.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations. arXiv preprint wrXiv:2003.03483 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhat- tacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differ- ential equations. Advances Neural Information Processing Systems, 33:NeurIPS 2020, 2020b.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier NeuralOperator for Parametric Partial Differential Equa- tions. In International Conference on Learning Representations, 2020c.
Giacomo Lionello and Luca Cristofolini. A practical approach to optimizing the preparation of speckle patterns for digital-image correlation. Measurement Science and Technology. 25(10): 107001, 2014.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for iden- tifying differential equations based on the universal approximation theorem of operators. arXiv preprint 1900:/990.03///// 2019.
Lu Lu, Haiyang He, Priya Kasimbeg, Rishikesh Ranade, and Jay Pathak. One-shot learning for solution operators of partial differential equations. arXiv preprint 144:v:2104.05512, 2021a.
Lu Lu, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence 3(3):218-229, 2021b.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. arXiv preprint arXiv:21 11.25110..512...11 2021c
Haoxin Mai, Tu C Le. Takashi Hisatomi, Dehong Chen, Kazunari Domen, David A Winkler, and Rachel A Caruso. Use of meta models for rapid discovery of narrow bandgap oxide photocata- lysts. iScience, pp. 103068, 2021.
11

---

Under review as a conference paper at ICLR 2023
Martin Misfeld and Hans-Himrich Sievers. Heart valve macro-and microstructure. Philosophical Transactions of the Royal Sociery B: Biological Sciences, 362(1464):42111446 2007.
Alex Nichol and John Schulman. arXiv:1803.02299,, 2(3):4, 2018.
Reptile: scalable metalearning algorithm. arXiv preprint
Yong Zheng Ong, Zuowei Shen, and Haizhao Yang. IAE-NET: Integral autoencoders for discretizationninvarian learning. 03 2022. doi: 10.
Marco Palanca, Gianluca Tozzi, and Luca Cristofolini. The of digital image correlation in the biomechanical area: review. International Biomechanics, 3(1):1-21, 2016.
David Pfau, James S Spencer, Alexander GDG Matthews, and W Matthew Foulkes, Ab initio solution of the many-electron schrödinger equation with deep neural networks. Physical Review Research, 2(3):033429, 2020.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.0915: 091099999999,,,,,,...111 2019.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. 2016.
Jarno Rieppo, Jarmo Hallikainen, Jukka S Jurvelin, Ilkka Kiviranta, Heikki J Helminen, and Mika M Hyttinen. Practical considerations in the use of polarized light microscopy in the analysis of the collagen network in articular cartilage. Microscopy research and technique, 71 71 2008
Colton Ross, Devin W Laurence, Jacob Richardson, Anju R Babu, Lauren E Evans, Fan G Beyer, Rachel C Childers, Yi Wu, Rheal Towner, Kar-Ming Fung, Arshid Mir, Harold M. Burkhart, Gerhard Holzapfel, and Chung-Hao Lee. An investigation of the glycosaminoglycan contribu- tion to biaxial mechanical behaviours of porcine atrioventricular heart valve leaflets. Journal of The Royal Society Interface, 16(156):2010069) 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentee neural networks. In International conference on machine learn- ing. pp. 1842-1850, PMLR, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Ad- vances in neural information processing systems, 30, 2017.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations.
In International Conference on Machine Learning, pp. 10434-10443, PMLR, 2021.
Joaquin Vanschoren. Meta-learning: survey. 2018 arXiv preprint vrXiv:1810.03544 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference. arXiv preprint 10Xiv:2102.10277 2021.
Yaqing Wang, Quamming Yao, James Kwok, and Lionel Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
Congjie Wei, Charles S Wojnar, and Chenglin Wu. Hydro-chemo-meedee tion for corrosion induced cracking in reinforced concrete. Cement and Concrete Research, 144: 106404, 2021.
phase field formula-
Jin Xu, learning with iterative functional updates. International Conference on Machine Learning, pp- 10617-10627. PMLR, 2020.
Ton, Hyunjik Kim, Adam Kosiorek, and Yee Whye Teh. Metafun: Meta-
Hansi Yang and James Kwok. Efficient variance reduction for meta-learning. In International Conference Machine Learning, pp. 25070-25095. PML.R, 2022.
12

---

Under review as a conference paper at ICLR 2023
Minglang Yin, Ehsan Ban, Bruno V Rego, Enrui Zhang, Cristina Cavinato, Jay D Humphrey, and George Em Karniadakis. Simulating progressive intramural damage leading to aortic dissection using DeepONet: peerator-regresee neural network. Journal of the Royal Society Interface, 19(187):0010670) 2022a.
Minglang Yin, Enrui Zhang, Yue Yu, and George Em Karniadakis. Interfacing finite elements with deep neural operators for fast multiscale modeling of mechanics problems. Computer Methods in Applied Mechanics and Engineering. in press, pp. 115027, 2022b.
Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. Advances in Neural Informa- fion Processing Systems, 34:7561-77 -5561, 2021.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS 2018), pp. 7343-7353, 2018.
Huaiqian You, Yue Yu, Marta D'Elia, Tian Gao, and Stewart Silling. Nonlocal kernel network (NKN): A stable and resolution-independenn deep neural network. Journal of Computational Physics, pp. arXiv preprint arXiv:2201 Stt.i::2201002217,,. 2022a.
Huaiqian You, Quinn Zhang, Colton Ross, Chung-Hao Lee, Ming-Chen Hsu, and Yue Yu. A physics-guided neural operator learning approach to model biological tissues from digital image correlation measurements. Journal of Biomechanical Engineering, accepted for publication, pp- arXiv preprint arXiv:2204.00200. 2022b.
Huaiqian You, Quinn Zhang, Colton Ross, Chung-Hao Lee, and Yue Yu. Learning deep implicit fourier neural operators (IFNOs) with applications to heterogeneous material modeling. Computer Methods in Applied Mechanics and Engineering. 398:1 15296, 2022c. doi: https://doi.org// org//10
10166/ 202..1169...
Dong Sheng Zhang and Dwayne D Arola. Applications of digital image correlation to biological tissues. Journal of Biomedical Optics, 9(4))691 2004.
Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and E Weinan. Deep potential molecular dynamics: scalable model with the accuracy of quantum mechanics. Physical Review Letters, 120(14):14300, 2018.
Lu Zhang, Huaiqian You, and Yue Yu. Metanor: A meta-learnt nonlocal operator regression ap- proach for metamaterial modeling. arXiv preprint arXiv:2206.0228 020000 2022.
Will Zhang and Michael Sacks. Modeling the response of exogenously crosslinked tissue to cyclic loading: The effects of permanent set. Journal of the Mechanical Behavior of Biomedical Materials, 75:336-350, 2017.
13

---

Under review as a conference paper at ICLR 2023
A PROOF OF THEOREM
In this section we provide the detailed proof for Theorem 1, based on Assumptions ! and 2 Intu- itively, that means the underlying implicit problem is solved with converging fixed point method. This condition is basic requirement by numerical PDEs, and it generally holds true in many appli- cations governed by nonlinear and complex PDEs, such as in our three experiments.
Here, prove that the MetaP is universal, i.c., give fixed point method satisfying Assumptions 12 one can find parameter sets 8 whose output approximates U to a desired accuracy, > O, for all 1) = 1. H tasks. For the task-wise parameters, with slight abuse of notation, denote p (2daMMx(dz++
as the collection of the pointwise weight matrices at each discretization point in x for the 1y-th task, and p" E IR dd for the bias in the lifting layer. Then, for the parameters shared among all tasks, in the iterative layer we denote C [c(x1)..... c(xx)) collection of pointwise bias vectors c(x,), W E Rd xdv for the local linear transformation, and R = F v)] Xdax E Cd,.dda X xM for the Fourier coefficients of the kernel For simplicity, here we have assumed that the Fourier coefficient not truncated, and all available frequencies are used. Then, for the projection layer we seek Q1 E Igd Xxd, Q2 Rdd x qi E Rd and q2 E Idd,, For the simplicity of notation, in this section we organize the feature vector H E d in a way such that the components corresponding to each discretization point are adjacent, i.e., H = (xx))... ... H(xx)) and Hxx;) E Rdh
Id as the
We point out that under this cireumstance, we have the (discretized) iterative layer can be written as
JHH() = (IAtt) + Ato (WH((A)))) + Re(F (Rr + c)
HIAI)) Alo (VH(IA)) + C),
with
M-1 Σ Ru41 W a=0
M
R exp( Zindan
Ro+1 expl (M 1))-m
M-1
Σ R141 exp( 2in
Σ Ru +
Σ Ro41 exp( (M (MM)))Am 2)
V Re
n=0
M-1 Σ
M-1
M-1
Ru+1 exp( M 1)A2n
Σ Ra+1 exp( 2in(M (M- Amm
Rat1 W Σ
n=0
Here, R E C XM xd with R. E Cd xds being the component associated with each discretization
point x, E X, V E Mdd. C E Rd W = W + W @ .... @ W is a M x d, M block diagonal matrix formed by W xds, FAn and F-1 denote the discrete Fourier transform and its inverse, respectively. By further taking R2 = ... = R& = W 0, d, x d, matrix with all its elements being zero, it suffices to show the universal approximation property for an iterative layer as follows:
I ((H(A))) = H(II)) + Ato (VH(IAI) + C)
where V- = MM @ V with V E IRd xd, and 1 [mmin being an m by n all-ones matrix.
To be more precise, will prove the following theorem:
(x))) u² (Xx1) be the ground- 2 the activation function a for all iterative kernel integration layers be the ReLU function, and the activation function in the projection layer be the identity function. Then for any >> 0, there exist sufficiently large layer number L > 0 and feature dimension number dy, > O, such that one can find parameter set for the multi-task problem, gn = 10% G,. (o)) with the corresponding MetaP model satisfies
Theorem 1 (Universal approximation). Let U
truth solution of n-th task that satisfies Assumptions
Un |naa o (Jo,)² o Pop ".]T) - Se.
VG E R
For the proof of this main theorem, we need the following approximation property of a shallow neural network, with its detailed proof provided in You et al. (202cc):
14

---

Under review as a conference paper at ICLR 2023
Lemma 1. Given continuous function T RRM + II and non-potynomial and continuous activation function o, for any constant E 0 there exists shallow neural network model T So (BX + AA such that
IT(()) - ( E AX E E
for sufficiently large feature dimension d > 0. Here, MxdM, B R2Mx2M and R are matrices/vectors which are independent of X.
We now proceed to the proof of Theorem
Proof. Since all U satisfies Assumptions for any E > O, first pick sufficiently large integer L such that the L-th layer iteration result of this fixed point formulation satisfies |U= Ulllrar $ / for all tasks. By taking é 2(1mm)= in Lemma | there exists sufficiently large feature dimension d and one can find S IR MxAM ,B RdMx2M and RdM such that R(U, G3) == So (B(",, GTIT + A) satisfies
|R(UU" RU'", (ll ||R(U" G") G")"++)|l((MM))
ms
2(1 + m)
where m is the contraction parameter of R, as defined in Assumption T By this construction, we know that S has independent rows. Denoting d == d + 1 > O, there exists the right inverse of S, which we denote S+ E (g(d-1)AxxM) such that
Ss+ -MM. S+S : ((3-1)M
where IM is the M by M identity matrix, f(ā-1)a is (d 1)M by (d 1)M block matrix with each of its element being either or O. Hence, for any vector Z E Red - 1)M, we have a (((ā Z) (Z) Moreover, we note that S has very special structure: from the ((i - () 1) + 1)-th to the (i(d - 1))-th column of S, all nonzero elements are on its i-th row. Correspondingly. we can also choose 5 to have special structure: from the ((i (ii-)(d- 1) 1)-th its i-th column. Hence, when multiplying
to the (i(d-1)) row of St, all nonzero elements S+ with U, there will be no entanglement between different components of U. That means, S can be seen as a pointwise weight function.
We now construct the MetaP as follows. In this construction, we choose the feature dimension as dn dM. With the input [U" G*] E R2M for the lift layer set
pr = 1 C
S
0
[+
0
S
S
0
T
E Add.Mx2M,
D
0
D
D
D
repeated
and pd : 0 E Rdd.M. M, Here, D = (ia][l/F/(b*](x))))) 1/FF, [/F][b"|(xx) As such, the initial
layer of feature is then given by
(()) P" GTIT) [ @ SS+ DTG'TT = 1.. @ U दhITT RdM
Here, we point out that Pa and p" can be seen as pointwise weight and bias functions, respectively.
Next we construct the shared iterative layer J. by setting
[(a 11M B/M
S/At O
O /M/At
V == &V and C == 1...1 (d--) 1as Aaa-nas/AA A/Ai 0
V =
0
Note that V is independent of n, and falls into the formulation of V. by letting R₁ V and RM W 0. For the l + 1-th layer of feature vector, we then arrive
Rg
R2
at
H((! + 1)At) H(AAt) Ato (TH(AA)+C)
[M & =((A))++
[sts 0 (((((M)))))))
[B/M] ( (aM &
S 0 H(lIt) + 1 &
[1]
Is
O IM
15

---

Under review as a conference paper at ICLR 2023
where HIItt [h hn ...... haah A denotes the (spatially discretized) hidden layer feature at the 1-th iterative layer of the IFNO. Subsequently, we note that the second part of the feature vector, h/4 E TRA, satisfies
((+1)AA = h h2
h2 - =1= 0,.... L 1 -i 1,..,, .M
Hence, the first part of the feature vector, h E [B(d-1)M satisfies the following iterative rule: ((41)A4 2j-1 - S So GTJT + A), =1= 0, ... - 1, = ..... M,
and
h ((+1)) = hg (1+1) = =...= h (1+1)Ar
2M-1
Finally, for the projection layer Q. set the activation function in the projection layer as the identity function, Q1 == L (the identity matrix of size M). Q2 == [S,0] E R MM
q := 0 E Rd..M and q2 == 0 E RR,, Denoting the output U =- 2xg (Je,) ) o (U",
we now show that U can approximate U with a desired accuracy e:
||U U
110" 0² Ilagee, + ||U
U (barer))
||Sh| r. (Ra + 2
(by Assumption 2
||Sti (1-1))) 1) U (1 + [[R(SSi
G) R(U"-) -(Ul"r,))lla +
S ||Sh (L- DAl1 ()A1
11~11110RR + ( Gb) R(S (1))) (bb)ll
+ |[(S 1)A Cb) - -R(U--1 RRU""I,CU))1 (cb)lliaCRM) + 2
(1 + m||S
(L- 1) -2- (1 + 2(1 m)
ms
+ 8
(by Lemma ① and Assumption 7
2
$ 2(1 + m)
me
(1 (1 m) + (1 + mm) + + (1 m) 1-1) )
8 =
B DATA GENERATION AND TRAINING DETAILS
In the following briefly describe the empirical process of generating datasets, and the settings employed for each algorithm, we tune the hyperparameters, including the learning rate from (0.1,0.01 0.001, 0.0001, 0.00001, 0.000001], J. the decay rate from 00., decay parameter from {0.01, 0.001, 0.0001, 0.00001, 0.000001), and the inner loop learning rate for MAML and ANIL from {0.01, 0.001, 0.0001, 0.00001, 0. 0.000001}, }. to minimize the error on separate validation dataset. In all experiments we decrease the learning rate with a ratio of learning rate decay rate every 100 epochs. The code and the processed datasets will be publicly released at github for readers to reproduce the experimental results.
running of each algorithm. For a fair comparison,
the weight
B. EXAMPLE 1: SYNTHETIC DATA SETS
B.1.1 DATA GENERATION
In the synthetic data example, consider the modeling problem of hyperelastic, anisotropic, fiber-reinforced material, and seek to find its displacement field u : [0, 1]² - R2² under different boundary loadings. In this problem, the specimen is assumed be subject (x) on the top edge (see Figure a)). To generate training and test samples, the Holzapfel-Gasser- Odgen (HGO) model (Holzapfel al. (0000) was employed to describe the constitutive behavior of the material in this example, with its strain energy density function given as:
uniaxial tension
E
E In(J)
4(1 +D) (7, 2) ka
2(1 +
E
J
(xxp()((5(0)) exp (kz (} - -2) 6(1
In InJ)
2h
2v)
16

---

Under review as a conference paper at ICLR 2023
(a)
(b)
T (xx)
0.08
0.06
0.04
T₂K0
0.02
0.00
Lx =
0.00
0.25
0.50
0.75
1.00
(C)
Validation U,
0.100
In Distribution U,
0.100
Out Distribution d
0.10
Out Distribution a,
0.005
0.005
0.005
0.005
0.000
0.000
0.000
0.000
005
-0.005
0.0005
-0.00
000
Validation by
In Distribution u,
Out Distribution 1 dy
Out Distribution
0.02
0.02
0.02
0.02
0.02
0.01
-0.01
-0.01
-0.01
-0.01
Figure 4: Problem setup of example 1: the synthetic data sets. (a) A unit square specimen subject to uniaxial tension with Neumann- boundary condition. (b) & (c) Visualization of an instances of the loading field TT, and the corresponding ground-trute truth solutions u*(x) from the in-distribution and out-of-distribution tasks, showing the solution diversity across different tasks, due to the change of underlying hidden material parameter
17

---

Under review as a conference paper at ICLR 2023
Here, () denotes the Macaulay bracket, and the fiber strain of the two fiber groups is defined as:
S(a) Ta(a)= - 1+ [T(aa) -11
where k and l2 are fiber modulus and the exponential coefficient, respectively, ci0 is the moduli for the non-fibrous ground matrix, E is the Young's modulus, and v is the Poisson ratio. Moreover, 7, = (r(C) the is the first invariant of the right Cauchy-Grees gradient, and J/ is related with F such that J = det F. For the fiber group with angle direction from the reference direction, Tạ(a) n Cn(a) is the fourth invariant of the right Cauchy- -rrnn tensor C, where n(a) (cos(a)) (z). sin( To generate samples for different specimens, specimens (tasks) correspond to different material parameter sets, {kr, k2, E,, 14, ar.. For the training tasks, the validation task, and the in-distribution (ID) test task, their physical parameters are sampled from: 11, k2 ~ 14 1]. E ~ (4.555, 1.5], V ~ (0.0.11 0.49) and a ~ 1100 n/ /2). For the two out-of-distribution (OOD) test tasks, we sample their parameters following k1. k2 4[11 1.9), E ~ 2[115, [ 0.55], v ~ [00001, 0.49 and a ~ 1| 127 3Z/ / /4 T/ /0] To generate the high- fidelity (ground-truth) dataset, we sampled 500 different vertical traction conditions (x) the top edge from a random field, following the algorithm in Lang & Potthoff (2011): Yin et al. (2022b). In
tensor C FTF. . F is the deformation
particular, (x) is taken as the restriction of a 2D random field, ((x) =- (2((F(( (x), on the
top edge. Here, [(x) is a Gaussian white noise random field on .R2,-== (wf + m represents a correlation function, and W1, W22 are the wave numbers on x and y directions, respectively. Then, for each sampled traction loading, we solved the displacement field on the entire domain by minimizing potential energy using the finite element method implemented in FEniCS (Alnæs et al. 2015). In particular, the displacement filed was approximated by continuous piecewise linear finite elements with triangular mesh, and the grid size was taken as 0.025. Then, the finite element solution was interpolated onto X. structured 41 x 41 grid which will be employed as the discretization in our neural operators.
To visualize the domain characteristies for tasks, the distribution of each parameter for training, validation and test tasks are demonstrated in Figure 5) and the corresponding solution fields are plotted in Figure c)) showing the diversity across different tasks due to the change of underlying hidden material parameter set, {ki, K2, E, D, a}. From Figures 5 and 4fc. one can see that OOD 1 corresponds a stiffer material (with large Young's 's modulus E) and hence smaller deformation subject to the same loading T,(x), On the other hand, OOD Task2 corresponds a softer material (with small Young's modulus E' and larger deformation. Therefore, the material response of OOD 1ask1 specimen is more likely to lie in a linear region, which is easier to learn and explains the relatively small test error on this task. On the other hand, the material response of OOD Task2 is more nonlinear and hence complex due to larger deformation, as shown in Figure 4(c), and results in the relatively larger test error in Figure
1.1.2 ALGORITHM SETTINGS
Base model: As the base model for all algorithms, we construct an architecture for IFNO (You et al. 2022c) as follows. First, the input loading field instance g(x) E A is lifted to higher dimensional representation via lift layer Pxg](x), which is parameterized as I-layer feed for- ward linear layer with width (3,32). Then for the iterative layer in equation [ we implement F=([[((:vv]] Flb(,,/At]]] with 2D fast Fourier transform (FFT) with input channel and output channel widths both set as 32 and the truncated Fourier modes set 8. The local lin- ear transformation parameter, W. is parameterized as a 1-layer feed forward network with width (32,32). In the projection layer, 2-layer feed forward network with width (32,128,) 2) is employed. To accelerate the training procedure, we apply the shallow-to-deep training technique to initialize the optimization problem. In particular, we start from the NN model with depth L. = train until the loss function reaches a plateau, then use the resultant parameters to initialize the parameters for the next depth, with L 2, L and L 8. In the synthetic experiments, we set the layer depth as L = 8.
MetaP: We split the total 60 training tasks to two groups: 59 tasks for the purpose of training and task for the purpose of validation. During the meta-train phase, we train for the task-wise parameters
* 'err we sample both ID and OOD tasks from the same range of v. due to the fact that [0.01, 0.49) is the range of Poisson ratio for common materials (Bischofs & Schwarz 2005
18

---

Under review as a conference paper at ICLR 2023
In-distribution
In-distribution
Out di-distribttion
of-diistibbution
Out of distributtio test task2
Out of-distrilution test task2
Validation task
Validation task
20.0%
Training tasks
Distributtin of Tacks
Training tasks
Distributtoon of
15.0%
8.0%
20.0%
6.0%
4.0%
2.0%
0.0%
0.6
0.8
1.0
1.2
1.4
1.6
0.0%
0.0
0.1
0.2
0.3
0.4
0.5
Young's Modelus
16.0%
In-distribution test task
16.0%
In-distributten test
Out ai-iistribbttoon tast task]
Dut dudd----diitittttonn tast task3
dottaf-diittrbuution tast task2
14.0%
Out didittitiitiiioo test task2
12.0%
Validation task
Validation
Training
Training
10.0%
8.0%
Distributtonn
8.0%
6.0%
4.0%
2.0%
0.0%
0.4
1.4
1.5
0.0%
0.25
0.50
0.75
1.00
1.50
1.75
Fiber Modulus
Exponential Coefficient
18.00%
In distribution test task
In- In-distribution test task
16.0%%
distribution task)
distribution task]
14.0%
Validation
Validition
Distribution Taadee
Training tasks
Diverbuttion Tenks
Training Tasks
12.0%
10.0%
30.0%
8.0%
6.0%
4.0%
0.0700002
Direction
Figure 5: Distribution of physical parameters of different tasks, and the resultant magnitude of material response, (x))(x)ll on an exemplar loading instance shown in Figure (b.
19

---

Under review as a conference paper at ICLR 2023
8% and the common parameters 81 and 0q on all 59 tasks, with the context set of 500 samples on each task. After meta-train phase, we load 85 and fo and the averaged 0% among all 59 tasks as initialization, then tune the hyperparameterss based on the validation task. In particular, the 500 samples on the validation task is split into two parts: 300 samples are reserved for the purpose of training (as the context set) and the rest 200 samples are used for evaluation (as the target set). Then we train for the lift layer on the validation task, and tune the learning rate, the decay rate, and the weight decay parameter for different context set sizes (Nies)) minimize the loss on the target set. Based on the chosen hyperparameters, we perform the test on the test task by training for the lift layer on different numbers of samples on its context set, then evaluate and report the performance based on its target set. We repeat the procedure on the test task with selected hyperparemeters with different random seeds, and calculate means and standard errors for the resultant test errors on target set.
&ANI For MAML and ANIL, we use the same architecture as the base model, and also split the training tasks for the purpose of training and validation as in MetaP. During the meta- train phase, for each task we randomly split the available 500 samples to two sets: 250 samples in the support set used for inner loop updates, and the rest in the target set for outer loop updates. During the inner loop update, we train for the task-wise parameter with one epoch, following the standard settings of MAML and ANIL (Finn al., 2017 Raghu et al., (019). Then, the model hyperparameters, including the learning rate, weight decay, decay rate, and inner loop learning rate, are tuned. In the meta-test phase, we load the initial parameter and train for all parameters (in MAML) or the last-layer parameters (in ANIL) until the optimization algorithm converges. Similar as in MetaP, we first tune the hyperparameters on the validation task, then evaluate the performance on the test task.
B.2 EXAMPLE 2: MECUIICAL MNIST
B.2.1 DATA SETTINGS
Mechanical MNIST is a benchmark dataset of heterogeneous material undergoing large deformation, modeld by the Neo- Hookean material with a varying modulus converted from the MNIST bitmap images (Lejeune 2020). In this example, we randomly select 102 specimens corresponding to the hand-writter number r, On each specimen, we have 32 loading/response data pairs on a structured 27 by 27 grid, under the uniaxial extension, shear, equibiaxial extension, and confined compression load scenarios, respectively. All 102 specimens are splitted into three groups: 100 specimens for the purpose of training in the meta-train stage, 1 specimen for validation, and specimen for test. On the validation and test tasks, reserve target set consisting of 20 data pairs for the purpose of evaluation, then use the rest as the context set.
B.2.2 ALGORITHM SETTINGS
Base model: As the base model for all algorithms, we construct two IFNO architectures, for the prediction of un and uy, the displacement fields in the r- and y-directions, respectively. On each architecture, the input loading field instance g(x) E A is mapped to higher dimensional repre- sentation via a lifting layer (xg] parameterized as a 1-layer feed forward linear layer with width (4,64). Then for the iterative layer in equation [ we set the number of truncated Fourier mode as 13, and parameterize the local linear transformation parameter, W, as a I-layer feed forward network with width (64,64). In the projection layer, a 2-layer feed forward network with width (64, (44,12811) 1) is employed. In this example we also apply the shallow-to-dens technique to accelerate the training, and set the layer depth as L 8.
During the meta-train phase, we train for the task parameters 0% and the common parameters f and 0g on all 100 training tasks, with the context set of 32 samples on each task. After the meta-train phase, we load f and 0q and the averaged 0% among all 100 tasks as initialization, then train for Op on the validation task. In particular, the 32 samples on the validation task is split into two parts: 12 samples are reserved for the purpose of training (as the context set) and the rest 20 samples are used for the purpose of evaluation (as the target set). Then we train for the lift layer on the validation task, and tune the learning rate, the decay rate, and the weight decay parameter for different context set sizes (Nies), to minimize the loss on the target set. Based on the chosen hyperparameters, we perform the meta-test phase on the test task by training for the lift layer on
MetaP:
20

---

Under review as a conference paper at ICLR 2023
Training Task1, U.
Training Task2, U,
Validation Task,
Test Task, 4,
0.0015
000155
0.0010
0.0010
0.0010
0.0005
0.0005
00.005
0.0005
0.0000
0.0000
0.0000
0.0000
-0.0005
-0.0005
-0.0005
-0.0003
-000110
-0.0010
-0.0010
-0.0015
0.0020
Training Task1, 4
Training Task2, 4
Validation Task 4
Test Task,
0.0015
0.0015
0.0015
0.0015
0.0010
0.0005
0.0005
0.0005
0.0005
0.0000
0.0000
0.0000
0,0000
0.0005
000000
-0.0005
-0
0.0010
-0
-0.0010
-0.0010
-0.0010
-0.0015
Figure 6: Visualization of the processed dataset in example 3: learning the biological tissue re- sponses. Subject the same loading instance, different columns show the corresponding ground- truth solutions u ux(x) from different tasks, showing the solution diversity across different tasks due the change of underlying hidden material parameter field.
different numbers of samples on its context set, then evaluate and report the performance based on its target set.
MAML& For MAML and ANIL, we use the same architecture as the base model, and also split the training tasks for the purpose of training and validation as in MetaP, During the meta-train phase, for each task we randomly split the available 32 samples to two sets: 16 samples in the support set used for inner loop updates, and the rest in the target for outer loop updates. During the inner loop update, we also follow the standard settings of MAML and ANIL (Finn et al. [017]] Raghu et al. 2019). and tune the hyperparameters following the same procedure elaborated above for Example 1.
B.3 EXAMPLE 3: EXPERIMENTAL MEASUREMENTS ON BIOLOGICAL TISSUES
B.3.1 DATA GENERATION
now briefly provide the data generation procedure for tricuspid valve anterior leaflet (TVAL) response modeling example. In this problem, the constitutive equations and material microstructure are both unknown, and the dataset has unavoidable measurement noise. To generate the data, we firstly followed the established biaxial testing procedure, including acquisition of healthy porcine heart and retrieval of the TVAL [Ross et al. (209)); [aurence et al.) (2019). Then, we sectioned the leaflet tissue and applied a speckling pattern to the tissue surface using an airbrush and black paint Zhang & Arola (2004): Lionello & Cristofolini (2014); Palanca et al. (2016). The painted specimen was then mounted to biaxial testing device (Bio Tester, CellScale, Waterloo, ON, Canada). To generate samples for each specimen, we performed 7 protocols of wettlacmmeettccooroollloo
testing target various biaxial stresses: P11 P22 {1 1, 0.66, 1 0.33, 0.66 0.33 0.05 1, 0.1). Here, Pi and P12 denote the first Piola- MorlaKKirhhhoff stresses in the x- and y-directions, respectively. Fach stress ratio was performed for three loading/unloading cycles. Throughout the test, images of the specimen were captured by CCD camera, and the load cell readings and actuator displacements were recorded at 5 Hz. After testing, the acquired images were analyzed using the digital image correlation (DIC) module of the BioTTester's software. The pixel coordinate locations of the DIC-tracked grid were then exported and extrapolated to a 21 by 21 uniform grid.
In this example, have the DIC measurements 14 specimens, with 500 data pairs of loadings and material responses from the 7 protocols on each specimen. These specimens are divided into three groups: 12 for the purpose of meta-train, 1 for validation, and for test. To demonstrate the
21

---

Under review as a conference paper at ICLR 2023
diversity of these specimens due to the material heterogeneity in biological tissues, in Figure b plot the processed displacement field of two exemplar training specimens and the validation and test specimens.
B.3.2 ALGORITIM SETTINGS
Base model: As the base model, we first construct the lifting layer as a 1-layer feed forward linear layer with width (4,16). Then for the iterative layer in we keep truncated Fourier modes and pa- rameterize the local linear transformation parameter, W, I-layer feed forward network with width (16,16). In the projection layer, a 2-layer feed forward network with width (16,64,1) is employed. We construct two 4-layer IFNO architectures, for the prediction of ux and uy: the displacement fields in the x- and y-directions, respectively.
During the meta-train phase, train for the task-wise parameters 0% and the common parameters 0, and 0g on all 12 tasks, with the context set of 500 samples on each task. After meta- train phase, we load 0, and 0g and the averaged 0% among all 12 tasks as initialization, then tune the hyperparameters based on the validation task. In particular, the 500 samples on the validation task is splited into two parts: 300 samples are reserved for the purpose of training (as the context set) and the rest 200 samples are used for evaluation (as the target set). Based on the chosen hyperparametters we perform the test on the test task by training for the lift layer on different numbers of samples on its context set, then evaluate and report the performance based on its target set.
MetaP:
MAML&&NTIA For MAML and ANII,, we use the same architecture as base model, and also split the training tasks for the purpose of training and validation as in MetaP. During the meta-train phase, for each task we randomly split the available 500 samples to two sets: 250 samples the support set used for inner loop updates, and the rest in the target set for outer loop updates. During the inner loop update, we train for the task-wist parameter with one epoch, following the standard settings of MAML and ANIL (Finn et al. 2017 Raghu et al. 2019)
22


Reviews:
[{"review_id": "szTHm1Kamjm", "reviewer": "Reviewer_koC6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work developed a novel meta-learnt method to transfer learning knowledge between neural operators. Different from typical\nfinal-layer transfer in existing meta-learning methods, this paper adapted the first layer of the neural operator model to capture hidden\nparameter field. Evaluation on both synthetic and real world datasets demonstrate the efficacy of the proposed approach. ", "summary_of_the_review": "It is interesting and novel to transfer the knowledge between neural operators. It would be much better to explore more neural operators to further verify the effectiveness of the proposed method. ", "strength_and_weaknesses": "* Strength\n1. The idea of transferring knowledge between neural operators is interesting and novel\n2. It adapted the first layer of the neural operator model to capture hidden parameter field in contrast to the final-layer transfer in existing meta-learning methods\n3. Conduct extensive experiments on benchmark and real-world datasets to verify the effectiveness of the proposed approach \n\n* Limitations\n1. It only adopted Fourier Neural Operator to show the effectiveness of the method, and it would be better to study more neural operators to verify this idea.\n2. Please repeat the experiments multiple times and then report the average error and standard deviation.\n3. A couple of grammatical issues. For example, \"As an motivating example,\" on page 1; and \"ANIL In \" on page 5, I am curious if a reference is missing?\n4. Questions:\n[1] In fig 2, why the accuracy of OOD task is better than ID task? Please provide possible explanation on it in more detail.\n[2] I am curious where could we access the real-world dataset: biological tissue specimens from DIC displacement?\n[3] I am wondering if the proposed method could learn multi-tasks from different PDEs rather than one common PDE with different physical parameter?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qGdNiROSPZr", "reviewer": "Reviewer_dEyR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to use MAML-based meta learning to learn a neural operator that approximates complex PDE-based equations with different physical parameters. Instead of optimizing all parameters of the base model during inner optimization (like vanilla MAML) or only the last layer (like ANIL), this paper's main contribution was to show that the first layer of the neural operator should be optimized. Experiments were conducted both on synthetic data, benchmark data, and a real-world data that describes mechanical response of biological tissue specimens from DIC displacement tracking. Experiments were mainly concerned with baselines using the neural operator without the presented meta-learning formulati", "summary_of_the_review": "This paper applies a MAML variant to a neural operator for learning to approximate PDE-based physics for various parameters. The motivation is intuitive, and the point on having to adapt the first layer of the neural operator is interesting. Overall, however, this paper can be improved both in its novelty in relation to existing works, and stronger baseline/comparison methods.", "strength_and_weaknesses": "Strengths:\n\nTo be able to learn a neural operator for complex PDE-governed physics without varying parameters is important. The use of meta-learning is intuitive. \n\nThe experiments attempted various types of dataset beyond benchmark data, which is appreciated.\n\n\n\nWeakness:\n\n- A major concern is regarding the baseline. The primary baselines includes a \"single\" model trained on the meta-training data with multiple tasks, and three variants of meta-formulation. The comparison with the \"single\" model however is not fair. All the meta-models are further optimized to the context data (referred to as test data in the paper) at meta-test time; this was not accessible to the \"single model\". Two stronger baselines need to be added. One is the \"single\" model fine-tuned to the same meta-test support data (used to optimize the model for all the other meta-models during meta-testing). The other is the \"single\" model trained to the \"single\" task used in the meta-training set (and fine-tuned to the meta-test task depending on how different the tasks are). \n\n\n- Another major concern is regarding the comparison to alternative models used to approximate PDE-based physics -- there is currently a large set of literature that does this; some of them are designed to generalize across different parameter settings (examples below). These works should be included for comparisons.\n\nYin et al. LEADS: Learning Dynamical Systems that Generalize Across Environments\nWang et al. Meta-Learning Dynamics Forecasting Using Task Inference\n\n\n- The rationale for the choice of meta-learning formulations is not clear. There are many different frameworks for meta-learning, some of them do not require test-time optimization. Why this formulation versus alternatives?\n\n\n- The difference between ANIL and MetaLast is not clear -- both does inner optimization to the projection layer? Please clarify.\n\n- Intuitively, since MAML optimizes all parameters of the base model, it is not clear why it would be sub-optimal then optimizing only the first layer (except for computational overhead)? Comments on this would be appreciated.\n\n- The wording of this paper is in general somewhat confusing, since it did not utilize the standard terminology of context/target samples for meta-learning. Assuming that N_test is referring to the context set size (i.e., size of meta-test samples used for inner loop adaptation at meta-test time), what is the corresponding context set size during meta-training? Is it the same as what is used at test time? Fig 2 showed how the test error changes as N_test changes. Is this N_test only referring to the context set size used during meta_test, or is the same parameter changing during meta_testing. In all experiments, N_test is given but not the size used during meta_training. This needs to be clarified. If N_test is different from the context set size used in meta-training, the effect of the latter would be more interesting to see in ablation studies.\n\n\n\n- In synthetic experiments, it'd be good to more specifically describe exactly which parameters are used for meta-training, and which for meta-testing. i.e., it is good to have a sense of the distance between training and test task for in-distribution tasks.\n\n- In real-world data experiments, it was mentioned that \"we have 500 available data pairs\" for each specimen, but it was then stated that MetaP was trained based on but N test = 500 samples and then evaluated on another 200 samples? If only 500 pairs are available, how can we use all of them as the context set? Please clarify\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h34vXWNlD8", "reviewer": "Reviewer_GCvR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a (somewhat) novel method that combines meta-learning with a popular operator learning model (based on the Fourier-Neural Operator learning paradigm) and develop a few-shot learning framework to learn PDE based domains. Additionally, authors also introduce a novel modification to the few-shot inner-loop model update wherein the updates are only localized to the “lifting” layers in the FNO based operator framework. ", "summary_of_the_review": "The paper proposes a novel few-shot learning based neural network based operator learning framework. Although the results are convincing, the proposed model is significantly based on previously proposed research efforts and the main contribution of the paper is their insight about fine-tuning the \"lifting layer\" instead of the final layer as in traditional transfer learning paradigm. Overall, the novelty of the proposed method is somewhat lacking but the incrementally novel proposed model does produce effective few-shot transfer learning results.", "strength_and_weaknesses": "## Strengths: \n\n**[Novel Few-Shot Operator Transfer Learning]**: Authors address an important problem of few-shot learning in the context of knowledge transfer across PDE domains. Specifically, the proposed MetaP, MetaP+ methods learn to effectively transfer knowledge with only a few instances of data from the target task of interest. \n \n\n**[Demonstration on Synthetic and Real-world Data]**: The proposed framework not only demonstrates good results on a set of synthetic PDE based domains but also on real-world task of learning to estimate mechanical response of biological tissue specimen from displacement tracking measurements. \n \n\n**[Effective Few-shot Performance]**: The MetaP, MetaP+ models demonstrate very effective few-shot generalization performance with as little as 2 instances required to achieve <5% prediction errors on synthetic data with similar results also on real-world datasets.  \n \n\n**[Baseline Comparison]**: Authors compare with popular meta learning models MAML [2], ANIL [3] and a version of the base model [4] without meta-learning as ablation baselines. The proposed models outperform all baselines under few-shot learning settings.   \n\n \n\n## Weaknesses: \n\n**[Some Baselines Missing]**: A related few-shot operator learning baseline model [5] based on one-shot-learning with deep operator networks is missing. \n \n\n**[Out-of-Distribution Results Require Further Clarity]**: Some of the out-of-distribution based experiments seem to produce results that are better than in-distribution test results. Without qualitative visualizations of training and testing regimes (out-of-distribution case), it is hard to comprehend the degree of challenge posed by out of distribution data regime. \n \n\n**[Limited Novelty]**: Although the problem is well motivated and results of the proposed pipeline are somewhat convincing, the proposed framework itself is not significantly novel. Specifically, the sole contribution of the authors may be considered to be the insight of transfer learning by fine-tuning the “lifting layer” in the neural operator (backed by theoretical and quantitative results) instead of the last layer as in traditional transfer-learning paradigms. Other parts of the model (I.e., based model , meta-learning framework) are directly employed from previously published work. \n \n**[Typos]**: Paper has a few typos (e.g., Eq. 7 is referenced in multiple places in reference to the solution operator `G` , but Eq. 7 contains no such symbol. \n\n## Questions for Authors: \n\nHow has model architectures (number of layers, activations) selected? What is the validation set size and how was it selected?  \n \n\nPlease provide an explanation in terms of a performance breakdown for why the out-of-distribution test error of the proposed MetaP model is lower than in-distribution test error? This seems surprising considering the challenge of the task?  \n \n\nCould you please provide justification about the difference in the domain characteristics in the training and testing regimes for out-of-distribution tasks? For example, please comment on what about the test regimes makes (e.g., more stiff) make them a true test of out-of-distribution performance?  \n \n\nHow were out-of-distribution ranges selected for synthetic datasets in section 4.1? \n\n \n\n## References: \n\n1. Li Z, Kovachki N, Azizzadenesheli K, Liu B, Bhattacharya K, Stuart A, Anandkumar A. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895. 2020 Oct 18. \n \n\n2. Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning 2017 Jul 17 (pp. 1126-1135). PMLR. \n \n\n3. Raghu A, Raghu M, Bengio S, Vinyals O. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157. 2019 Sep 19. \n \n\n4. Li Z, Kovachki N, Azizzadenesheli K, Liu B, Bhattacharya K, Stuart A, Anandkumar A. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485. 2020 Mar 7. \n \n\n5. Lu L, He H, Kasimbeg P, Ranade R, Pathak J. One-shot learning for solution operators of partial differential equations. arXiv preprint arXiv:2104.05512. 2021 Apr 6. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LgJVAWyszhY", "reviewer": "Reviewer_evBd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a MAML-like approach for neural operators aimed at transferring learning across slightly varying physical systems. It builds on recent work on implicit Fourier neural operators and proposes a meta-learning method that uses MAML for the (first) lifting layer, while the remaining layers (including the Fourier layers) are meta-learned but not adapted at meta-test time. The proposed method is experimentally evaluated on simulated PDE data as well as a real-world physical tissue response setting.\n", "summary_of_the_review": "Overall, I think that the studied problem is relevant and the proposed method has its merits. Unfortunately, it is not very well presented which makes the paper hard to read. In addition, there are serious shortcomings in the experimental methodology and documentation. At the moment, I do not think the paper is ready for publication. Since I consider the proposed method relevant, I encourage the authors to seriously re-work and improve the writing of the paper based on my comments above. ", "strength_and_weaknesses": "Strengths:\n* Relevant problem with clear application\n* Real-world experiments\n* The first meta-learning approach based on neural operators\n\nWeaknesses:\n* The presentation of the work could be improved a lot\n* No standard errors are reported in the experiments. Were experiments even repeated with multiple seeds\n* Not enough experiment details to reproduce experiments\n* Assumptions in Section 3.3. are not motivated\n\nMinor typos:\n* 3rd paragraph on page 3: ‘neural operators requires’ -> ‘neural operators require’\n* Page 4, ‘we denote the set trainable parameters’ ->  ‘we denote the set of trainable parameters’\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject