Physics-aware Spatiotemporal Modules with Auxiliary Tasks for Meta-Learning
Authors: Sungyong Seo
Chuizheng Meng
Sirisha Rambhatla
Yan Liu
Venue:  (2021)
Keywords: physics-aware learning
spatiotemporal graph signals
few shot learning

Abstract: Modeling the dynamics of real-world physical systems is critical for spatiotemporal prediction tasks, but challenging when data is limited. The scarcity of real-world data and the difficulty in reproducing the data distribution hinder directly applying meta-learning techniques. Although the knowledge of governing partial differential equations (PDE) of the data can be helpful for the fast adaptation to few observations, it is mostly infeasible to exactly find the equation for observations in real-world physical systems. In this work, we propose a framework, physics-aware meta-learning with auxiliary tasks whose spatial modules incorporate PDE-independent knowledge and temporal modules utilize the generalized features from the spatial modules to be adapted to the limited data, respectively. The framework is inspired by a local conservation law expressed mathematically as a continuity equation and does not require the exact form of governing equation to model the spatiotemporal observations. The proposed method mitigates the need for a large number of real-world tasks for meta-learning by leveraging spatial information in simulated data to meta-initialize the spatial modules. We apply the proposed framework to both synthetic and real-world spatiotemporal prediction tasks and demonstrate its superior performance with limited observations.

Under review as a conference paper at ICLR 2021
PHYSICS-AWARE SPATIOTEMPORAL MODULES WITH AUXILIARY TASKS FOR META-LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Modeling the dynamics of real-world physical systems is critical for spatiotem- poral prediction tasks, but challenging when data is limited. The scarcity of real- world data and the difficulty in reproducing the data distribution hinder directly applying meta-learning techniques. Although the knowledge of governing partial differential equations (PDE) of the data can be helpful for the fast adaptation to few observations, it is mostly infeasible to exactly find the equation for obser- vations in real-world physical systems. physics-aware meta-learning with auxiliary tasks whose spatial modules incorpo- rate PDE-independent knowledge and temporal modules utilize the generalized features from the spatial modules to be adapted to the limited data, respectively, The framework is inspired by a local conservation law expressed mathematically continuity equation and does not require the exact form of governing equa- tion to model the spatiotemporal observations. The proposed method mitigates the need for a large number of real-world tasks for meta-learning by leveraging spatial information in simulated data to meta-initialize the spatial modules. We apply the proposed framework to both synthetic and real- world spatiotemporal prediction tasks and demonstrate its superior performance with limited observations.
this work, we propose a framework.
INTRODUCTION
Deep learning has recently shown promise to play a major role in devising new solutions to applica- tions with natural phenomena, such as climate change (Manepalli et al. [2099] Drgona et al. 2019).
ocean dynamics (Cosne et al. 2019)). air quality (Soh et 2018 Du et al. 20118 Lin et al. 2018
and so on. Deep learning techniques inherently require a large amount of data for effective represen- tation learning, so their performance is significantly degraded when there are only a limited number of observations. However, in many tasks in physical systems in the real-world we only have access to limited amount of data. One example is air quality monitoring (Berman 2017), in which the sensors are irregularly distributed over the space - many sensors are located in urban areas whereas there much fewer sensors in vast rural areas. Another example is extreme weather modeling and forecasting, i.c., temporally short events (c.g.. tropical cyclones (Racah et al.) (017b)) without suf- ficient observations over time. Moreover, inevitable missing values from sensors (Cao et al. 2018 Tang et al.) 2019) further reduce the number of operating sensors and shorten the length of fully- observed sequences. Thus, achieving robust performance from a few spatiotemporal observations in physical systems remains an essential but challenging problem.
Learning on limited amount of data from physical systems can be considered few shot learn- ing. While recently many meta-learning techniques (Schmidhuber) 1987; Andrychowizz et al. 2016 Ravi & Larochelle 2017 Santoro et al., 2016; Snell et al. 2017 Finn et al. (017)) have been de- veloped to address this few shot learning setting, there are still some challenges for the existing meta-learning methods to be applied in modeling natural phenomena. First, it is not easy to find a set of similar meta-tasks which provide shareable latent representations needed to understand tar- geted observations. For instance, while image-related tasks (object detection (He et al. 2017) or iisual-uusstion-assweiie
tasks (Andreas et al. 2016 Fukui al.] (2066) can take advantage of an image-feature extractor pre-trained by a large set of images (Deng et al. 2009) and well- designed ar- chitecture (Simonyan & Zisserman, 2014|He et al. 2016 Sandler et al. 2018 8), there is no such large data corpus that is widely applicable for understanding natural phenomena. Second, unlike computer vision natural language processing tasks where common object (images or words) is clearly de-

---

Under review as a conference paper at ICLR 2021
fined, it is not straightforward to find analogous objects in the spatiotemporal data. Finally, exact equations behind natural phenomena are usually unknown, leading to the difficulty in reproducing the similar dataset via simulation. For example, although there have been some works (de Bezenac et al., 2018 Lutter et al., 2019] Greydanus et al. 2019) improving data efficiency via explicitly in- corporating PDEs as neural network layers when modeling spatiotemporal dynamics, it is hard to generalize for modeling different or unknown dynamics, which is ubiquitous in real-world scenario.
In this work, we propose physics-aware modules designed for meta-learning to tackle the few shot learning challenges in physical observations. One of fundamental equations in physics describing the transport of physical quantity over space and time is a continuity equation:
3p V
where p is the amount of the target quantity (u) per unit volume, J is the flux of the quantity, and is source sink, respectively. This fundamental equation can be used to derive more specific transport equations such as the convection mann transport equation. Thus, the continuity equation is the starting point to model spatiotemporal (conservative) observations which are accessible from sensors. Based on the form of and J with respect partieular quantity u, Eq. [ can be generalized as:
equation, Navier-Stokes equations, and Boltz-
du Ōt
F(Vu, V u..,,
(2)
where the function F-) describes how the target u changed over time from its spatial derivatives. Inspired by the form of Eq. we propose two modules: spatial derivative modules (SDM) and time derivative modules (TDM). Since the spatial derivatives such as V, V., and V are commonly used across different PDEs, the spatial modules from synthetic data. Then, the PDE-specific temporal module is trained to learn the unknown func- tion F(-) from few observations in the real-world physical systems.
PDE-independent and they can be meta-initialized
This approach can effectively leverage large amount of simulated data to train the spatial modules the modules are PDE-independent and thus mitigating the need for a large amount of real-world tasks to extract shareable features. In addition, since the spatial modules are universally used in physics equations, the representations from the modules can be conveniently integrated with data- driven models for modeling natural phenomena. Based on the modularized PDEs, we introduce novel approach that marries physics knowledge in spatiotemporal prediction tasks with meta- learning by providing shareable modules across spatiotemporal observations in the real-world.
Our contributions are summarized below:
Modularized PDEs and auxiliary tasks: Inspired by forms of PDEs in physics, de- compose PDEs into shareable (spatial) and adaptation (temporal) parts. The shareable one is PDE-independent and specified by auxiliary tasks: supervision of spatial derivatives.
Physics-aware meta-learning: We provide a framework for physeis-awar meta-learning, which consists of DDFF-indeeendeet/speeiiie modules. The framework is flexible to be applied the modeling of different or unknown dynamics.
Synthetic data for shareable modules: We extract shareable parameters in the spatial modules from synthetic data, which can be generated from different dynamics easily.
MODULARIZED PDES AND META-LEARNING
In this section, we describe how the physics equations for conserved quantities are decomposable into two parts and how the meta-learning approach tackles the task by utilizing synthetic data when the data are limited.
2.1 DECOMPOSABILITY OF VARIANTS OF A CONTINUITY EQUATION
In physics, a continuity equation (Eq. [ describes how a locally conserved quantity such as temper- ature, fluid density, heat, and energy transported across space and time. This equation underlies
2

---

Under review as a conference paper at ICLR 2021
many specifie equations such the convection-diffusion equation and Navier-Stoker equations:
à - V (DVu) (wu) + R, = (u V)u vv Vw +
(Convection-Diffusion eqn.)
(Incompressible Navier-Stokes eqn.)
where the scalar and vector field are the variables of interest (e.g.. temperature, flow velocity, etc.). A dot over variable time derivative. The common feature in these equations that the forms of equations can be digested (Bar-Sinai et al. 2019 Zhuang al. 2020):
à == Myy Mazz Myy:...)
(3)
where the right-hand side denotes function of spatial derivatives. As the time derivative can be seen as Euler discretization (Chen et al., 2018), is notable that the next state is function of the current state and spatial derivatives. Thus, knowing spatial derivatives at time t is a key step for spatiotemporal prediction at time + for locally conserved quantities. According to Eq. 3 spatial derivatives are universally used in variants of Eq. [ and only the updating function F(-) is specifically defined for a particular equation. This property implies that PDEs for physical quantities can be decomposable into two modules: spatial and temporal derivative modules.
2.2 SPATIAL DERIVATIVE MODULES: PDE-INDEPENDENT MODULES
Finite difference method (FDM) is widely used to discretize a d-order derivative as linear combi- nation of function values on a nt-ointt stencil.
jou Ord
a,u(1;);
where > d. According to FDM, it is independent for form of PDE to compute spatial deriva- tives, which are input components of F(-) in Eq. Thus, we can modularize spatial derivatives as PDE-independent modules. The modules that can be learnable coefficients (a;)) have been proposed recently (Bar-Sina et al., 2019; Seo et al.] 2020). The data- driven coefficients particularly useful when the discretization in the n-point stencil is irregular and low-resolution where the fixed coefficients cause substantial numerical errors.
data-driven manner to infer the
2.3 TIME DERIVATIVE MODULE: PDE-SPECIFIC MODULE
Once upto d-order derivatives are modularized by learnable parameters, the approximated spatial derivatives from the spatial modules are fed into an additional module to learn the function F(-) in 3 This module is PDF-specific as the function F describes how the spatiotemporal observations change. Since the exact form of ground truth PDE is not given, the time derivative module is data- driven and will be adapted to observations instead.
2.4 META-LEARNING WITH PDE-NDDPENDENTT-SPCCII MODULES
Recently, Raghu al. (2019) investigate the effectiveness of model agnostic meta-learning (MAML, Finn et al. (2017) and it is found that the outer loop of MAML is more likely to learn parameters for reusable features rather than rapid adaptation. The finding that feature reuse is the predominant reason for efficient learning of MAML allows us to use additional information which beneficial for learning better representations. Previously, the objective considered to be matched with one in meta-test as the purpose of meta-lcarning is to learn good initial parameters applicable across similar tasks (e.g., image classification to image classification). We are now able to incorporate auxiliary tasks under meta-learning setting to reinforce reusable features for a main task. As described in Sec. 2.I the spatial modules are reusable across differ- ent observations, and thus, we can meta-initialize the spatial modules first with spatial derivatives provided by synthetic datasets. Then, we can integrate the spatial modules with the task- specifcc temporal module during meta-test to help adaptation of TDM on few observations. Since the spa- tial modules are trained by readily available synthetic datasets, a large number of similar tasks for meta-traie meta-training is not required.
meta-training has been
3

---

Under review as a conference paper at ICLR 2021
Auxilary tasks
D V.
(V4,
V 2h
Pirc Hl
G Oue
Meta-training
of objective
Φ
P
Task- lark-indepennent modules
== B
Taek-specific module
Target tasks
= 1,-0 ,M
Forward
Φv,
Φv,
Backward pass
On
Meta- test
Φ
ult K.
1:9m)
u(t; M(t:gim)
Figure 1: Schematic overview of the physics-aware meta-learning (PiMetal.).
3 PHYSICS-AWARE META-LEARNING WITH AUXILIARY TASKS
In this section, we develop a physics-aware meta-learning framework for the modularized PDEs.
Fig. deseribes the proposed framework and its computational process.
3.1 SPATIAL DERIVATIVE MODULE
Algorithm Spatial derivative module (SDM)
Input: Graph signals u, and edge features Cij = 2j 2; on & where is coordinate of node i.
Output: Spatial derivatives {ūx. E V and E [K} where K = (Va, Vy, V2 V
Require: Spatial derivative modules {dh h: K
1: for k E K do
2: (ak,i, (2,(4.)) (1,j) E E and E K)= du ((u), (e).9)
3: f V do
4: üR = a + Ei,)eR 6x.0.9)
5: end for
6:
end for
As we focus on the modeling and prediction of sensor-based observations, where the available data points are inherently on a spatially sparse irregular grid, we use graph networks for each module Φx to learn the finite difference coefficients (Bar-Sinal al. 2019). Given graph G = (V. E) where V -I,...,,N' and = {(i,j) i,j WJ. a node denotes a physical location x; = (x,, y:) u(2;, yi) is observed. Then, the graph signals with positional relative
where a function value displacement as edge features are fed into the spatial modules to approximate spatial derivatives by Alg. The coefficients (a,, b(e,)) each node i and edge (i,j) are output of and they linearly combined with the function values u; and Wj. K denotes a set of finite difference operators. For example, if we set K- [Vz. V, V2 7 we have modules which approximate first/second order of spatial derivatives in 2-dimension, respectively.
3.2 TIME DERIVATIVE MODULE
Once spatial derivatives are approxi- mated, another learnable module re- quired to combine them for target task. The form of line 2 in Alg. 2 comes from Eq. 3 and TDM is adapted to learn the unknown function F(-) in the equation. As target task is the regression of graph signals, we use recurrent graph network for TDM.
Algorithm Time derivative module (TDM)
Input: Graph signals and approximated spatial derivatives ûg where & E IK on G. Time interval At Output: Prediction of signals at next time step i(t) Require: Time derivative module
1: 0e TDM un,e l V and k K))
2: @(l)) = u(t - 1) + ut-1 1 A/
4

---

Under review as a conference paper at ICLR 2021
Figure 2: Examples of generated spatial function values and graph signals. Node and edge features (function value and relative displacement, respectively) are used to approximate spatial derivatives (arrows). We can adjust the number nodes (spatial resolution), the number of edges (discretiza- tion), and the degree of fluctuation (scale of derivatives) differentiate meta-train tasks.
3.3 META-LEARNING WITH AUXILIARY OBJECTIVE
As discussed in Sec. 2.1 is important to know spatial derivatives at time to predict next signals at time t + 1 for locally conserved physical quantities, however, it is impractical to access the spa- tial derivatives in the censor-based observations as they are highly discretized over space. In this section, we propose physics-aware meta-learning framework to meta-initialize spatial module by leveraging synthetic dataset with auxiliary tasks to provide reusable features for the main tasks: prediction spatiotemporal observations in the real-world.
The meta-initialization with the auxiliary tasks from synthetic datasets particularly important. First, the spatial modules can be universal feature extractors for modeling observations following un- known physics-based based PDEs. Unlike other domains such as computer vision, has been considered that there is particular shareable architecture for learning spatiotemporal dynamics from physical systems. We propose that the PDE-independent spatial modules can be applicable as feature extrac- tors across different dynamics long as the dynamics follow a local form of conservation laws. Second, we can utilize synthetic data to meta-train the spatial modules as they are PDE-agnostic. This property allows by numerical methods regardless of the exact form of PDE for targeted observations. Finally, we can provide stronger inductive bias which beneficial for modeling real-world observations but not available in the observations explicitly.
utilize large amount of synthetic datasets which are readily generated
Algorithm Meta ietaainitializztionn with auxiliary tasks: Supervision spatial derivatives
Input: A set of meta-train task datasets D = (DI,.... DB} where D, = (Df", D).). D, {(u,, etyy y/ (a1.b) ....... // (ax.b)) i Wo. (i. ) E Es) where y! (ax.) is an k-th auxiliary task label for the i-th node, given node/edge feature 2 and e², respectively. Learning rate a and B. Output: Meta-initialized spatial modules +.
Initialize auxiliary modules (d1..... δK)
2: while not converged do
3: for D, in D do
4: /
V EK DRux (D) (b:)
5: end for
6: x- V EB
Σ L (D} (h.k)
7: end while
Alg. B describes how the spatial modules are meta-initialian by MAML under the supervision of K different spatial derivatives. First, we generate values and spatial derivatives on a 2D regular grid from an analytical function. Then, we sample finite number points from the regular grid to represent discretized nodes and build graph from the sampled nodes. Each graph signal and its discretization becomes imput feature a meta-train task and corresponding spatial derivatives are the auxiliary task labels. Fig. visualizes graph signals and spatial derivatives for meta-initin neeta-iitiializatin.
Once the spatial modules are initialized throughout meta- trainin,,, we reuse the modules for meta- test where the temporal module (the head of the network) adapted few observations from real-world sensors (Alg. 4.. Although the standard MAML updates the body of the network (the spatial modules) well, we only adapt the head layer (0) like nimost-no-inner-loop method
5

---

Under review as a conference paper at ICLR 2021
Algorithm 4 Adaptation meta-test tasks
Input: A set of meta-test task datasets D (Pi...., DM3 where D, (D D Meta-initialized SDM (Φ). Learning rate a.
Output: Adapted TDM 6n on the m-th task.
1: Initialize temporal modules
Ows)
2: for Dm in D do
3: 6'm = 8m V L(D mi 4, 0m)
4: end for
in Raghu et al. 2019). The task at test time is graph signal prediction and the temporal modules (0) are adapted by regression loss function L = E'= llu(()) - a(()]l² on length T sequence (Di)
and evaluated on held-out (t >> T)) sequence (Dir) with the adapted parameters.
SPATIAL DERIVATIVE MODULES: REUSABLE MODULES
We have claimed that the spatial modules provide reusable features associated with spatial deriva- tives such as V ", V ut and V across different dynamics or PDEs. While it has been shown that the data-driven approximation of spatial derivatives is more precise than that of finite difference method (Seo et al.) 2020 Bar-Sinai et al.) 2019). it is not guaranteed that the modules effectively provide transferrable parameters for different spatial resolution, discretization, and fluctuation of function values. We explore whether the proposed spatial derivative modules based graph net- works can be used as a feature provider for different spatial functions and discretization.
We perform two sets of experiments: evaluate few-shot learning performance (1) when SDM trained from scratch; (2) when SDM is meta- initialized. Fig. [2) shows how the graph signal and its discretization is changed over the differ- ent settings. the number of nodes is large, it can provide spatially high-resolution and thus, the spatial derivatives can be more precisely ap-
Table I: Parameters for synthetic dataset.
Meta-train
Meta-test
# nodes (N)
(256, 625]
(450, 800)
# edges node (F))
[4, 8)
(3. 6. 10)
Initial frequency (E) 5).
{3.
proximated. Table [ shows the parameters we used generate synthetic datasets. Note that meta- test data is designed to evaluate interpolatinneextrapllaoo properties. Initial frequency decides the degree of fluctuation (In Fig. 2 the middle one has higher than that of the left one.). For each parameter combination, generate 100 different snapshots from the following form in Long et al. 20177:
04
Σ IIRISE
Ax, cos(kx; + ly:) + 7K. sin(kr, + ly),, Ak,s Tke1 ~ N (0, 0.02),
where the index denotes the i-th node whose coordinate is (I;, vi) in the 2D space ([0,2z) x [0, 2m]) and k, 1 are randomly sampled integers. From the synthetic data, the first- and second-order derivatives are analytically given and SDM is trained to approximate them.
The prediction results for spatial derivatives are shown Table 2 The results show that the pro- posed module (SDM) is efficiently adaptable to different configuration few samples from meta- initialized parameters compared to learning from scratch. The finding implies that the parameters for spatial derivatives be generally applicable across different spatial resolution, discretization, and function fluctuation.
5 EXPERIMENTAL EVALUATION
PRELIMINARY: WHICH SYNTHETIC DYNAMICS NEED TO BE GENERATED?
While Table 2 demonstrates that the IDE-independent representations reusable across different configurations, it is still an open question: which topological configuration needs to be used to construct the synthetic dynamies? According Table 2 the most important factor affecting error is

---

Under review as a conference paper at ICLR 2021
Table 2: Prediction error (MAE) of the first (top) and second (bottom) order spatial derivatives.
(N, B, F)
(450,3,3)
(450,3,7)
(450,6,3)
(450,6,7)
(4,0,10,3)
(450,10,7)
1.337:10.044
7.111110.148
1.152:10.043
7.206.10.118
1.112:10.036
7.529-10.241
SDM (from scratch)
7.228
51.544.10.148
5.997-10.083
47.527
5.353
47.356.10.560
1.075:10.005
5.528110.010
0.836110.002
5.354-10.001
0.782.1
5.550
SDM (pretrained)
6,482:10.207
46.254:10,262
5.251+0.224
42.243=0.420
4.728=:0.244
42.75410.442
EN
(800,3,3)
(800,3,7)
(800,6.3)
(800,6,7)
(800,10,3)
(800,10,7)
1.022:+0.030
5.699:L0.242
0.789-+0.021
5.179:+0.069
0.718
5.517+0.110
SDM (from scratch)
7.196++0.155
49.602 0..61
5.386+0.113
42.509- 1.080
4.536++0.204
39.642 19.642+11173
0.927--0.006
0.415++0.01
0.656+0.000
-.977-0.025
0.570+
4.107+00011
SDM (pretrained)
6.553-+0.193
44.591+-0.002
4.960++0.266
37.629 0.760
4.213
35.849+00947
Figure 3: Visualization of the first 5 frames of one extended sequence in the extreme weather dataset. Dots represent the sampled points. Greenish (purplish) area is higher (lower) surface temperature.
an initial frequency (F), which determines min/max scales and fluctuation of function values, and it implies that the synthetic dynamics should be similarly scaled target dynamics. We use the same topological configuration in Table to generate synthetic datasets for a task Section 5.2 and
adapted configuration for task in Section 5.3
We describe more details in Appendix
5.2 MUL.TI-TTEP GRAPH SIGNAL GENERATION
Task: We adopt set of multi-step spatiotemporal sequence generation tasks to evaluate our pro- posed framework. In each task, the data is sequence of L. frames, where each frame set of observations on N nodes in space. Then, we train an auto-regressive model with the first T frames (T-shot) and generate the following L T frames repeatedly from given initial input (T-th frame) to evaluate its performance.
Datasets: For ex- periments, we generate meta-train tasks with the parameters described in Table D and the target observations 2 real- world datasets: (1) AQI- CO: national air quality
Table 3: Multi-step prediction results (MSE) and standard deviations on the two world datasets.
T-shot
Method
AQI-CO
ExtremeWeather
FDM+RGN (scratch)
0.0291-10.0033
0.9883
5-shot
PA-DGN (scratch)
0.0363+40.0090
0.9653-101138
PiMetal. (meta-init)
0.0053
0.9167
index (AQI) observa-
FDM+RGN (scratch)
0.0258+0.0023
0.7626+0.0602
tions (Berman 2017):
7-shot
PA-DGN (scratch)
0.0225++00001
0.7478-0.0199
(2) Extreme Weather:
PiMetal. (meta-init)
0.0182+000019
0.7274+0.0089
the extreme weather
FDM+RGN (scratch)
0.0213+:000013
0.7090=0.0030
dataset (Racah et al. 2017b). For AQI-CO dataset, construct 12 meta-test tasks with the carbon monoxide (CO) ppm records from the first week of each month in 2015 at land-based stations. For the extreme weather dataset, we select the top-10 extreme weather events with the longest lasting time from the year 1984 and construct meta-test task from each event with the
10-shot
PA-DGN (scratch)
0.0146++0.000
0.4156=0.0145
PiMetaL (meta-init)
0.0115.10.0044
0.4066110.227

---

Under review as a conference paper at ICLR 2021
Table 4: Graph signal regression results (MSE, 10-5) and standard deviations on the two regions of weather stations.
s'otot (Region)
GCN
GAT
GraphSAGE
GN
PA-DGN
PiMetaL
5-shot (USA)
2.742-40.120
2.549:+0.115
2. 2.128
2.252++0.131
1.950=
1.794:10.133
10-shot (USA)
2.37110.095
2.178:0.066
1.848:0.206
1.949=10.115
1687=0.104
1.567:10.10
5-shot (EU)
1.218+00228
1.161+0.234
1.165++0.28
1.181+0.210
0.914=0.167
0.781+
10-shot (EUU
1.186+00.076
1.142+00.00
1.044+0.210
1.116++0.177
0.831=0.058
0.773+-0.014
observed surface temperatures randomly sampled locations. Since each event lasts fewer than 20 frames, each task has a very limited amount of available data. In both datasets, graph signals are univariate. Note that all quantities have fluidic properties such as diffusive and convection. Fig. shows the spatiotemporal dynamics of the extreme weather observations and sampled points. More details are in the supplementary material.
Baselines: We evaluate the performance of a physics-aware architecture (PA-DGN) (Seo et al. 2020), which also consists of spatial derivative modules and recurrent graph networks (RGN), to see how the additional spatial information affects prediction performance for same architecture. Note that PA-DGN has same modules in PiMetaL and the difference is that PiMetaL utilizes meta- initialized spatial modules and PA-DGN is randomly initialized for learning from scratch on meta- test tasks. Additionally, the spatial modules PA-DGN is replaced by finite difference method (FDM+RGN) to see if the numerical method provides better PDE-agnostic representations. The baselines and PiMetal. are trained on the meta-test support set only to demonstrate how the addi- tional spatial information is beneficial for few-shot learning tasks.
Discussion:
the multi- prediction performance our proposed framework against the baselines on real-world datasets. Overall, PA-DGN and PiMetaL show similar trend such that the prediction error is decreased as longer series are available for few-shot adaptation. There are two important findings: first, with the similar expressive power in terms of the number of learn- able parameters, the meta-initialized spatial modules provide high quality representations which are casily adaptable across different spatiotemporal dynamics in the real-world. This performance gap demonstrates that we can get a stronger inductive bias from synthetic datasets without knowing PDE-specific information. Second, the contribution of the meta-initialization is more significant when the length of available sequence is shorter (T' = 5) and this demonstrates when the meta- initialization particularly effective. Finally, the finite difference method provides proxies of exact spatial derivatives and the representations are useful particularly when T = 5 but its performance is rapidly saturated and it comes from the gap between the learnable spatial modules and fixed numer- ical coefficients. The results provide a new point of view on how to utilize synthetic or simulated datasets to handle challenges caused by limited datasets.
Table
5.3 GRAPH SIGNAL REGRESSION
Task, datasets, and baselines:
Defferrard et al. (2019) conducted graph signal regression task:
predict the temperature 2, from the temperature on the previous 5 days (#1-5 xt-1)). We split the GHCN datase spatially into two regions: (1) the USA (1,705 stations) and (2) Europe (EU) (703 stations) where there are many weather stations full functioning. In this task, the number of shots is defined the number of input and output pairs to train a model. As the input length is fixed, more variants of graph neural networks are considered as baselines. We concatenate the 5- step signals and feed into Graph convolutional networks (GCN) (Kipf & Welling 2017), Graph attention networks (GAT) (Veličković et al. 2018 GraphSAGE (Hamilton et al. 2017). and Graph networks (GN) (Battagia et al.) 2018) to predict next signals across all nodes.
Table shows the results of the graph signal regression task across different baselines and the proposed method. There are two patterns in the results. First, although in general we observe an improvement in performance for all methods when we move from the 5-shot setting to the 10-shot setting, PiMetaL's performance yields the smallest error. Second, for the EU dataset, while 5-shot seems enough to achieve stable performance, it demonstrates that the PDE-independent
Discussion:
'Glohal Historical Climatology Network (GHCN) provided by National Oceanic and Atmospheric Admin- istration (NOAA). hww.://wwww......oo/ognniaiiiiaa
8

---

Under review as a conference paper at ICLR 2021
representations make the regression error converge to lower level. Overall, the experimental results prove that the learned spatial representations from simulated dynamics are beneficial for learning on limited data.
RELATED WORK
Physies-informed learning Since physics- neural networks are introduced in Raissi al. (219)), which find that a solution of a PDE can be discovered by neural networks, physical knowl- edge has been used
inductive bias for deep neural networks. Advection-diffusion equation is incorporated with deep neural networks for sea-surface temperature dynamics (de Bezenac et al.,
mechanics 2018 Lutter al. (2019); Greydanus et al. (2019) show that LagrangianHHamltonne
can be imposed to learn the equations of motion of a mechanical system and Seoo & Liu (2019)
regularizes graph neural network with a specific physics equation. Rather than using explicitly given equations, physics-inspiree
inductive bias is also used for reasoning dynamics of discrete ob- jects (Battaglia et al. 2016 Chang et al.) 2016) and continuous quantities (Seo et al. 2020p. Long et al. (2017] 2019) propose a numeric-symbolic hybrid deep neural network designed to discover PDEs from observed dynamic data. While there are many physics-involved works, to the best of our knowledge, we are the first to provide a framework to use the physics-insping inductive bias under the meta-learning settings to tackle the limited data issue which is pretty common for real-world data such extreme weather events (Racah et al. 2017b).
Meta-learning The aim of meta-learning is to enable learning parameters which can be used for new tasks unknown the time of learning, leading agile models which adapt new task utiliz- ing only few samples (Schmidhuber, 1987 Naik & Mammone, 1992. Thrun & Pratt 1998). Based on how the knowledge from the related tasks is used, meta-learning methods have been classified as optimization-basel (Andrychowicz et al. [0016] Ravi & Larochelle| 2017; Duan et al. 2017: Finn
cet al. 2017 Nichol et al. 2018| (1i8.AAntor Antoniou et al.
2018
Rusu et al.] 2018] Grant al. 201 8), model-
based (Santoro et al.) 2016) Munkhdalai & Yu
2017
Duan al. 2017; Mishra et al. 2018), and
metrie-based (Koch et al.) 2015 Vinyals et al.
2016 Snell et al., 2017). Recently, another branch
of meta-learning has been introduced more focus finding set of reusable modules compo- nents solution to a new task. Alet et al. 2018 2019) provide a framework, structured modular meta-learning, where finite number of modules introduced
aaskiindependent modules and an optimal structure combining the modules is found from limited number of data. [Cenn et al. 2019) introduces techniques to automatically discover sas--independennddeeedeenn modules based on Bayesian shrinkage to find more adaptable modules. To our knowledge, none of the above works provide solution to use meta- leata-l for modeling physics-related spatiotemporal dynamics where it is hard to generate enough tasks for meta-initialization.
7 CONCLUSION
In this paper. we propose a framework for physics-aware meta-learning with auxiliary tasks. By incorporating PDDE-independem//inaariig
knowledge (spatial derivatives) from simulated data, the framework provide reusable features to meta-test tasks with limited amount of data. Experiments show that auxiliary tasks and physics-aware meta-learning help construct reusable modules that improve the performance of spatiotemporal predictions in real-world tasks where data is limited. Although introducing auxiliary tasks based on synthetic datasets improves the prediction perfor- mance, they need to be chosen and constructed manually and intuitively. Designing and identifying the most useful auxiliary tasks and data will be the focus of our future work.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ferran Alet, Tomás Lozano-Pérez, and Leslie Kaelbling. arXiv: 10166, 2018.
Modular meta-learning.
arXiv preprint
Ferran Alet, Erica Weng. Tomás Lozano- Lérezo-Pérez, and Leslie Pack Kaelbling. Neural relational inference with fast modular meta- meta-learning In Advances in Neural Information Processing Systems, pp. 11804 11815, 2019.
Jacob Andreas, Marcus Rohrbach, Trevor Darelll and Dan Klein. Ncural module networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 39-48, 2016.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthw W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp- 3981-3989, 2016.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint arXiv:/810.09502. 2018.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116(31):15344 15349, 2019.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, al. Interaction networks for learning about objects, relations and physies. In Advances neural information processing systems, pp. 4502-4510, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzaler, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint / 2018.
Lex Berman. National agi observations (2014-05 to 2016-12). Harvard Dataverse, 2017. doi: 10.7910/DVN/ QDX6L8. htthttpss:// //doi. org/10. /910/DVN/ /DDVG/ QDX6L8
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, pp. 6775 6785, 2018.
Michael Chang, Tomer Ullman, Antonio Torralba, and Joshua Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint 16Xi::111..00341, 2016.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equa- tions. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Yutian Chen, Abram L Friesen, Feryal Behbahani, David Budden, Matthew W Hoffman, Araad Doucct, and Nando de Freitas. Modular meta-leaming with shrinkage. arXiv preprint arXiv: 1909.05557, 2019.
Gautier Cosne, Guillaume Maze, and Pierre Tandeo. Coupling oceanic observation systems study mesoscale ocean dynamics. arXiv preprint 19X::190000857 2019.
Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: Incorporating prior scientific knowledge. In International Conference on Learning Representations, 2018. URL https:
//openreview. net. fooemm//ommmmmm ddm EWAZ
Michaël Defferrard, Martino Milani, Frédérick Gusset, and Nathanaël Perraudin. Deepsphere: a graph-hased spherical cnn. In International Conference on Learning Representations, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, leee, 2009.
Ján Drgona, Lieve Helsen, and Draguna Vrabie. Stripping off the implementation complexity of physics-based based model predictive control for buildings via deep learning. https://www. nips. co/, 2019.
Shengdong Du, Tianrui Li, Yan Yang, and Shi-Jinn Homg. Deep air quality forecasting using hybrid deep learning framework. arXiv preprint 1eXXiv:8112.44777 2018
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, llya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. In Advances in neural information processing systems, pp. 1087-1098, 2017.
10

---

Under review as a conference paper at ICLR 2021
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1 1126- 1135. JMLR, org, 2017.
Akira Fukui, Dong Huk Park, Daylen Yang, Amna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Mul- timodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv: 018477 2016.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-hased meta-learning as hierarchical bayes. arXiv preprint arXiv:1801.0S8 2018.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pp. 15353-15363, 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Thomas N Kipf and Max Welling. Sumi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recog- nition. ICML deep learning workshop, volume 2. Lille, 2015.
Yijun Lin, Nikhit Mago. Yu Gao, Yaguang Li. Yao Yi Chiang. Cyrus Shahabi, and José Luis Ambite. Exploiting spatiotemporal patterns for accurate air quality forecasting using deep learning. In Proceedings of the 26th ACM SIGSPATIAL International Conference Advances in Geographic Information Systems, pp. 359-368, 2018.
Zichao Long. Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. arXiv preprint arXiv:1710.09668, 2017.
Zichao Long. Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with numeric-symbolic hybrid deep network. Journal of Computational Physics, 399:108925, 2019.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep lcarning. arXiv preprint arXiv: 1907.04490. 2019.
A Manepalli, A Albert, A Rhoades, D Feldman, and M Prabhat. Emulating numeric hydroclimate models with physics-informed conditional generative adversarial networks. Environmetrics, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Chen, and Pieter Abbeel. A simple neural attentive meta-leamnrr In International Conference on Learning Representations, 2018. URL /uBR///ttpp://pppeeeewwe..e Forum? i d=B11=111U23
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2554-2563. JMLR. org, 2017.
Devang K Naik and Richard Mammone. Meta-neural networks that learn by leaming. /Proceedings 1992] IJCNN International Joint Conference on Neural Networks, volume I, pp. 437-442. IEEE, 1992.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 029.9, 2018.
Evan Racah, Christopher Beckham, Tegan Maharaj, Samira Kahou, Mr. Prabhat, and Chris Pal. Ex- tremeweathr: A large-scale climate dataset for semi-supervised detection. localization, and under- standing of extreme weather events. I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fer- gus, Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3405-3416. Curran Associates, Inc., 2017a. URL http::://appe...ee cc/ppprr/
6932 aww.rreeeeeeeee-aaaaee -arg----aarrssaaae le-cclll pdf
for semi supervi sed detection- izaiic
Evan Racah, Christopher Beckham, Tegan Maharaj. Samira Ebrahimi Kahou, Mr Prabhat, and Chris Pal. Ex- tremeweather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events. In Advances in Neural Information Processing Systems, pp. 3402-3413, 2017b.
11

---

Under review as a conference paper at ICLR 2021
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. arXiv preprint 091507 2019.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: deep learn- ing framework solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations, 2017.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals. Razvan Pascanu, Simon Osindero, and Raia Hadsell, Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05900 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-lcarning with eeemory-augmente neural networks, In Proceedings of The 33rd International Conference on Machine Learning, pp. 1842 1850, 2016.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how learn: The meta-meto- hook. Diplomarbeit, Technische Universität München, München, 1987.
Sungyong Seo and Liu. Differentiable thysics-infomenn graph networks. arXiv preprint arXiv:190..02950, 2019.
Sungyong Sco, Chuizheng Meng, and Yan Liu. Physics-aware difference graph networks for sparsely- observed dynamies. In International Conference Learning Representations, 2020. URL https:
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.
arXiv preprint arXiv: /409. 1556, 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077 4087, 2017.
Ping-Wei Wei Soh, fei Chang, and Wei Huang. Adaptive deep learning- based air quality prediction model
using most relevant spatial-temporal relations. leee Access, 6:38186-385 181999999,,,.. 2018.
Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. Joint modeling of local and global temporal dynamics multivariate time series forecasting with missing values. arXiv preprint arxiv: 102,03 2019.
Sebastian Thrun Lorien Pratt. Learning to learn: Introduction overview. In Learning to learn, pp. 3-17. Springer, 1998.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lid, and Yoshua Ben- gio. Graph attention networks. In International Conference on Learning Representations, 2018. URL net/
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Jiawei Zhuang, Dmitrii Kochkov, Yohai Bar-Sinai, Michael Brenner, and Stephan Hoyer. Learned discretiza- tions for passive scalar advection in a 2-d turbulent flow, arXiv preprint arXiv:2004.05477 2020.
12

---

Under review as a conference paper at ICLR 2021
TASK 1: MULTI-STEP GRAPH SIGNAL GENERATION A
A.1
META-TRAIN
Data: For experiments, we generate the data for meta-train tasks from sum of sinusoidal functions with different spatial frequencies (Eq. 6
u(,, v)
Σ Au. cos(ka + 1y) TA. sin(kx ly), AR., 7h.1 N (0, 0.02),
(6)
PAIPISP
where (x, y) in the 2D space 2x] x [0, 2x)) and k;, are randomly sampled integers, Once the spatially continuous function values are generated, uniformly sample different number of locations from all grid points observed nodes simulate the case where the observations irregularly distrihuted in space. We then construct k-Nearest Neighhor graph hased on the Euclidean distance the input of graph neural net- works. The combination of parameters to generate the synthetic dataset is given in Table We construct 100 snapshots combination of the parameters (N, E, F) using unique random seed. 75 snapshots each combination used for D and 25 snapshots for D
Table 5: Parameters for synthetic dataset
Meta-train
Meta-test
# nodes (N) # edges per node (F)
{256, 625)
(450, 800)
(4, 8)
(3. 6, 10)
Initial frequency (F)
53.
(3, 7}
Tasks: For each node, we have the first and second order derivatives. We meta-train the spatial derivative modules (Sec. 3.1) to predict the spatial derivatives by feeding node and edge features (function value at node and relative displacement, respectively) as input.
A.2 META-TEST
A.2.1
SYNTHETIC
We generate the synthetic meta-test data from Eq. but set different parameters simulate the realistic scenario where meta-train tasks and meta-test tasks do not share the same distribution.
Data:
Tasks: We reuse the spatial modules in A. to evaluate how the meta-initialized parameters are easily adaptable unseen graph signals with different spatial resolution, discretization, and degree of function fluctuation. We use 15 snapshots for the adaptation in meta-test and 75 snapshots are used to evaluate the proposed model.
A.2.2 REAL- WORLD DATASET
Figure 4: Sensor locations in the AQI-CO dataset. We show sensors as blue nodes and edges of k-NN graphs as red lines. Borders of provinces are shown in grey.
Data:
13

---

Under review as a conference paper at ICLR 2021
There multiple pollutants in the dataset and we choose carbon monoxide (CO) ppm as target pollutant in this paper. We select sensors located between latitude (26, 33) and longitude (115,125) (East region of China), In this region, we sample multiple multivariate time series whose length should be larger than 12 steps (12 hours) for multiple meta-tasks. There are around 60 working sensors and the exact number of the working sensors is varying over different tasks. Fig. 4 shows the locations of selected AQI sensors.
AQI-CO (Berman 2017)
ExtremeWeatherr We select the data in the year 1984 from extreme weather dataset in (Racah et all 2017a). The data is an array of shape (1460, 16, 768, 1152), containing 1460 frames (4 per day, 365 days in the year). 16 channels in each frame correspond to 16 spatiotemporal variables. Each channel has a size of 768 1152 corresponding to one measurement per 25 square km on earth. For each frame, the dataset provides fewer than equal 15 bounding hoxes, each of which labels the region affected by extreme weather event. one the four types of the extreme weather: (1) tropical depression, (2) tropical cyclone, (3) extratropical cyclone, (4) atmospheric river. In the single feature setting, we only utilize the channel of surface temperature (TS).
Tasks:
We select the first sequence of carbon monoxide (CO) ppm records from each month in the year 2015 at land- based stations, and set up the meta-test task on each sequence as the prediction of CO ppm. We construct 6-NN graph based on the geodesic distances among stations.
AQI-CO:
ExtremeWeatherr First, we aggregate all bounding boxes into multiple sequences. In each sequence, all bounding boxes (1) are in consecutive time steps, (2) are affected by the same type of extreme weather, and (3) have intersection
union (loU) ratio above 0.25 with the first bounding box in the sequence. Then we select the top-10 longest sequences. For each sequence, we consider its first bounding box as region affected by an extreme weather event, and extend it to a new sequence of 20 frames by cropping and appending the same region A from successive frames. For each region we uniformly sample 10% of available pixels as observed nodes to simulate irregularly spaced weather stations and build 4-NN graph based on the Euclidean distance. Fig. visualizes the first frames of one extended sequence. In the single feature experiment, we set up a meta- test task on each extended sequence as the prediction the surface temperature (TS) on all observed nodes with the initial TS given only.
A.3 EXPERIMENTAL DETAILS
A.3.1 BASELINES
PA-DGN (train from scratch) (Seo al. 2020
For each meta-test task, initialize one PA-DGN model randomly and train on the single task. The spatial derivative layer uses a message passing neural network (MPNN) with 2 GN blocks using 2-layer MLPs as update functions. The forward network part uses recurrent graph neural network with 2 recurrent GN blocks using 2-layer GRU cells as update functions. We set its hidden dimension to 64, in which case PA-DGN has similar number of parameters with RGN, The PA-DGN model has 384,653 learnable parameters.
A.3.2 OURS
PiMetaL: Meta-train the spatial derivative modules (SDM) with our proposed Alg. 3 on the meta-trat tasks generated [ Then for cach meta-test task, initialize one time derivative module (TDM) randomly and output of SDM is fed into the TDM to train it on the single task. The architecture for SDM and TDM are identical for the spatial derivative layer and the recurrent graph network in PA-DGN, respectively.
A.3.3 TRAINING SETTINGS
Training hyperparameters: For all meta-train and meta-test tasks, we use the Adam optimizer with the learn- ing rate le-3. In each training epoch, we sample task from all available tasks.
Environments: All experiments are implemented with Python3.6 and PyTorch 1.3.0, and are conducted with NVIDIA GTX 1080 Ti GPUs.
Runtime: The baselines RGN (train from scratch) and PA-DGN (train from scratch) will finish in 30 minutes. All other baselines will finish the meta-train stage in hours and the meta-test stage in 2 hours. The runtime is measured in environments described above.
14

---

Under review as a conference paper at ICLR 2021
Figure 5: (left) GHCN weather stations in the USA and (right) GHCN weather stations in Europe.
Figure 6: (first and third) Spatial distribution of the GHCN weather stations and synthetic nodes. (second and fourth) Function values of the GHCN records and synthetic function.
B TASK 2: GRAPH SIGNAL REGRESSION
B. META-TRAIN
Data:
For the graph signal regression task, we generate synthetic dynamics for meta-train tasks and the syn- thetic data is adapted to target dataset. Before setting the topological configuration for the synthetic dynamics, we first examine the target dataset to undersland its topological properties. Based on the number of stations and the scale of records, for USA records and (N, F) = (700, 1.5) for Europe records, respectively, and 100 different initial values are generated to define different tasks. Fig. 5 visualizes how the regional stations are distributed and Fig. demonstrates how the spatial distribution of synthetic nodes and scales of synthetic values are adapted to the corresponding target dynamics.
the topological configuration for the synthetic dataset. We
(N, F) (1700, 2)
B.2 META-TEST
Data: The GHCN-Dai Daily summaries from land surface stations across the globe provide daily climate records from numerous sources. As the records from 100,000 stations in 180 countries and territories, the distribution weather stations is spatially non-uniform.
sample sensors from two different regions (1) the USA and (2) Europe and construct graph structure from the regional stations hased on k-NN algorithm (k = 4) as described in Defferrard et al. (2019). There are 1,705 and 703 fully functioning sensors in the USA and Europe, respectively. We use 2010 year records and first few daily records for few-shot training (5 and 10) and next 100/150 days for validation and test. Note the number of learnable parameters significantly reduced compared to those of the previous task to minimize overfitting as well as be comparable to other variants of graph neural networks.
Table 6: The number of learnable parameters for baselines and PiMetal.
GCN
GAT
GraphSAGE
GN
PA-DGN
PiMetaL
# of parameters
10,801
11,203
21.401
20,385
33,795
33,795
B.3 EXPERIMENTAL DETAILS
B.3.1 BASELINES
Since the input length of the regression task is fixed (length=5), consider many variants of graph neural networks for the task. We concatenate the 5-step signals and feed it into Graph convolutional net-
15

---

Under review as a conference paper at ICLR 2021
Table 7: Regression error (MAE, 10 10-3) of different topology for synthetic dynamics (Furope)
FN, (700, 1.5)
(N,F) (1700,
(N, (128, 7)
5-shot
0.781:10.019 0.773:10.014
1.. 0.951=0.151
1.007=10096 0.932:10.058
10-shot
works (GCN) (Kipf Welling 2017 Graph attention networks (GAT) Velicković et al. 2018 Graph-
SAGE (Hamilton al. 2017). and Graph networks (GN) (Battaglia et al. 2018 to predict next signals across all nodes. For the baselines, commonly consider 3-hop neighbors of i-th node to predict of the i-th node and the number of Iearnalec parameters is similar provide similar expressive power.
C SENSITIVITY ANALYSIS OF SYNTHETIC DYNAMICS
is important to study how much the model's performance dependent on synthetic topology. this section, we conduct an ablation study to see whether different choices of the synthetic topology affects the performance significantly or not. According to Table g the regression error is fairly converged from a few samples for the Europe records. Thus, apply different synthetic topology for the data to see if the saturated regression error significantly changed. For this ablation study, reuse synthetic dynamics adapted the USA records and generate one more synthetic dynamics for spatially low-resolution cases.
Table 7 shows that the regression performance across different topology is stable regardless of the number of shois, however, it is significantly degraded when we change the synthetic topology from the adapted one ((N,)) F)= (700, 1.5)) for meta-training. When increase the spatial resolution (NV 700 1700), the meta-initialized spatial modules are adapted to learn spatial derivatives defined on spatially higher resolution. In such case, SDM likely assigns high weights to directly adjacent nodes as well as farther nodes (e.g.. 3-hop nodes) as all neighhor nodes are strongly associated with exact spatial derivatives. On the other hand, SDM is meta-initialized from lower resolution (N 700 - 128), further nodes are too much underestimated. Thus, important construct proper topology for transferring the PDE-independent representations from synthetic dynamics target dynamics.
16


Reviews:
[{"review_id": "v5rU1H2DmVx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework for physics-aware meta-learning to tackle the few-shot learning\nchallenges in physical observations. The authors claim that by incorporating PDE-independent\nknowledge from simulated data, the framework provides reusable features to meta-test tasks with\na limited amount of data. The experiments on synthetic and real-world spatiotemporal prediction\ntasks demonstrate the effectiveness of the framework. However, there still exist some limitations\nto the approach.\nPros:\n1. The idea of using meta-learning for modeling physics-related spatiotemporal dynamics is\nnovel.\n2. The paper is well written, and the idea is presented clearly.\nCons:\n1. How to choose and construct auxiliary tasks based on synthetic datasets is important for\nthe performance but there is no systematic approach to addressing such an issue.\n2. The authors can add more sensitivity analysis to show how different choices of auxiliary\ntasks can influence the experiment performance.\n3. The paper focuses on modeling and predicting sensor-based observation in 2-dimension.\nHowever, it is not clear if such an approach can be effective in 3 or higher dimension\nspace. Higher-dimension space and higher-order derivative can bring up challenges to\nconstructing useful auxiliary tasks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lLCjUwUvweS", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " The paper proposes to modular PDE solutions in networks into spatial and temporal modules, where spatial modules are rapidly adapted by meta learning.\n\nStrengths\n\nThis paper is clear and relatively easy to read.\n\nThe paper does a reasonable of citing related work and the approach seems novel  (although incremental) to my knowledge.\n\n\nWeaknesses\n\nThe overall quantitative benefits of the approach appear to be low. For example in table 2 or table 3, many of confidence intervals overlap. Furthermore, these results are shown in the best setting of the model -- when nearby tasks are drawn almost directly from the same training distribution. Furthermore\n\nThe overall technical novelty of approach seems to be low. Modularity with meta learning has been explored before, and the physical learning framework has also been learned before. This work seeks to combine both but does not seem have to have impressive performance. \n\nGiven that a central claim in the paper is that \"We extract shareable parameters in the spatial\nmodules from synthetic data, which can be generated from different dynamics easily.\" I think it would be good to empirically test this result and  synthetic data is generated more diversely. For example, in Table 1/2, what happens when the underlying uis have different functional forms? What if only the test task has a separate functional form?\n\nThere are only three evaluated datasets in the entire paper -- one being a toy problem to visualize the impact of the approach. Since PDEs occur in so many different disciplines, it would be good to evaluate on additional datasets to more throughly validate the utility of this approach. For example, it may also be interesting to test this on additional non graph based settings.\n\n### Post Rebuttal Update\n\nI thank the authors for their response. However, I still have some remaining concerns about novelty, since to me this paper reads as another application of MAML to domain X, to speed up performance in the particular domain. Furthermore, I think it would be good to evaluate generalization to larger extent, for example on different conformations of dynamics as opposed to a fixed set of parameters on the synthetic dataset.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lj6tRV0_ECN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The key contribution of the paper is not very known to reviewer. Is it employing the PDE-independent knowledge (spatial derivatives) from simulating the PDEs?\nCan the authors highlight some of the key contributions of the paper?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KIbI6_GLi_J", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes the approach to meta-learning of spatiotemporal predictions for sparse data with auxiliary spatial derivative modules. \n\nThe paper provides novel insights on how to perform physically informed meta-learning and can be useful to the community, but the questions below need to be answered to assure that it is indeed meta-initialisation which improves the few-shot learning performance (the review rating reflects these questions, especially question 2 on the motivation of pre-training, and could be improved based on the answers):\n1. It is said that *“While it has been shown that the data-driven approximation of spatial derivatives is more precise than that of finite difference method”*. At the same time, it is said that *'We perform two sets of experiments: evaluate few-shot learning performance (1) when SDM is trained from scratch; (2) when SDM is meta-initialized.’* The reviewer thinks that there might be a third scenario: while data-driven approximation might be more precise, the finite difference method provides for better features for PDE-agnostic methods. Has this scenario been accounted for or could it be used as an additional ablation study?\n\n2. The pre-training has been performed on the synthetic data, and the authors mention the limitations of the approach: *“\nAlthough introducing auxiliary tasks based on synthetic datasets improves the prediction performance, they need to be chosen and constructed manually and intuitively.”* In connection to that, there might be less scarce, non-extreme, events in the same dataset, which could be used for pretraining. For example, in the extreme weather dataset,  the top-10 extreme weather events since 1984 are selected; is it possible to use some of the remaining data for pretraining on non-extreme events? This could remove the bias of synthetic dataset hyper parameter tuning and could serve as  a baseline. Is it possible to say how would the method, pre-trained on such real non-extreme dataset, perform against the synthetic baseline in this case?\n\n3. In Table 3, the experimental results are reported, with an (anticipated) tendency of diminishing return in spatial meta-initialisation as the number of T-shots increase; is there any way to see whether, as the value of T is increased, there will be the value T when pre-training does not improve or even has worse performance? When does it happen, and does it ever get worse than no-pretraining baseline?", "extra_scores": {}, "extra_text": {}}, {"review_id": "72PkfEGOMbW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose methodology for sharing learned differencing coefficients for estimating spatial derivatives between multiple spatio-temporal modeling tasks. They show that increased number of tasks improves learning. Additionally, the authors propose a meta-initialization procedure by which the differencing coefficients are initialized to values obtained from synthetic data. They show that this initialization procedure improves performance. \n\nIt seems to me that the 'share-ability' of the spatial differencing coefficients requires that each task is operating on the same spatial manifold with same or similar measurement locations. Toward that end, the initial differencing coefficients appear to be chosen using a flat 2D topology. The appropriateness of this choice is probably responsible for the improvements offered by the proposed approach. However, if the initial topology is poorly chosen, or the different tasks have significantly different topologies one would expect poor performance or negative transfer. How to identify these degenerate cases seems to be an open problem.\n\nAside from my rehashing the perennial question of 'when does task sharing help?' I found this paper to be well written, and a solid contribution to the field of physics-informed machine learning. Certainly, many applications include multiple different types of measurements at common locations which might be expected to transfer.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by 5 experts in the field. The reviewers raised their concerns on lack of novelty, unconvincing experiment, and the presentation of this paper, While the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject