Physics-aware Spatiotemporal Modules with Auxiliary Tasks for Meta-Learning

1 INTRODUCTION . Deep learning has recently shown promise to play a major role in devising new solutions to applications with natural phenomena , such as climate change ( Manepalli et al. , 2019 ; Drgona et al. , 2019 ) , ocean dynamics ( Cosne et al. , 2019 ) , air quality ( Soh et al. , 2018 ; Du et al. , 2018 ; Lin et al. , 2018 ) , and so on . Deep learning techniques inherently require a large amount of data for effective representation learning , so their performance is significantly degraded when there are only a limited number of observations . However , in many tasks in physical systems in the real-world we only have access to a limited amount of data . One example is air quality monitoring ( Berman , 2017 ) , in which the sensors are irregularly distributed over the space – many sensors are located in urban areas whereas there are much fewer sensors in vast rural areas . Another example is extreme weather modeling and forecasting , i.e. , temporally short events ( e.g. , tropical cyclones ( Racah et al. , 2017b ) ) without sufficient observations over time . Moreover , inevitable missing values from sensors ( Cao et al. , 2018 ; Tang et al. , 2019 ) further reduce the number of operating sensors and shorten the length of fullyobserved sequences . Thus , achieving robust performance from a few spatiotemporal observations in physical systems remains an essential but challenging problem . Learning on a limited amount of data from physical systems can be considered as a few shot learning . While recently many meta-learning techniques ( Schmidhuber , 1987 ; Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2017 ; Santoro et al. , 2016 ; Snell et al. , 2017 ; Finn et al. , 2017 ) have been developed to address this few shot learning setting , there are still some challenges for the existing meta-learning methods to be applied in modeling natural phenomena . First , it is not easy to find a set of similar meta-tasks which provide shareable latent representations needed to understand targeted observations . For instance , while image-related tasks ( object detection ( He et al. , 2017 ) or visual-question-answering tasks ( Andreas et al. , 2016 ; Fukui et al. , 2016 ) ) can take advantage of an image-feature extractor pre-trained by a large set of images ( Deng et al. , 2009 ) and well-designed architecture ( Simonyan & Zisserman , 2014 ; He et al. , 2016 ; Sandler et al. , 2018 ) , there is no such large data corpus that is widely applicable for understanding natural phenomena . Second , unlike computer vision or natural language processing tasks where a common object ( images or words ) is clearly de- fined , it is not straightforward to find analogous objects in the spatiotemporal data . Finally , exact equations behind natural phenomena are usually unknown , leading to the difficulty in reproducing the similar dataset via simulation . For example , although there have been some works ( de Bezenac et al. , 2018 ; Lutter et al. , 2019 ; Greydanus et al. , 2019 ) improving data efficiency via explicitly incorporating PDEs as neural network layers when modeling spatiotemporal dynamics , it is hard to generalize for modeling different or unknown dynamics , which is ubiquitous in real-world scenario . In this work , we propose physics-aware modules designed for meta-learning to tackle the few shot learning challenges in physical observations . One of fundamental equations in physics describing the transport of physical quantity over space and time is a continuity equation : ∂ρ ∂t +∇ · J = σ , ( 1 ) where ρ is the amount of the target quantity ( u ) per unit volume , J is the flux of the quantity , and σ is a source or sink , respectively . This fundamental equation can be used to derive more specific transport equations such as the convection-diffusion equation , Navier-Stokes equations , and Boltzmann transport equation . Thus , the continuity equation is the starting point to model spatiotemporal ( conservative ) observations which are accessible from sensors . Based on the form of ρ and J with respect to a particular quantity u , Eq . 1 can be generalized as : ∂u ∂t = F ( ∇u , ∇2u , . . . ) , ( 2 ) where the function F ( · ) describes how the target u is changed over time from its spatial derivatives . Inspired by the form of Eq . 2 , we propose two modules : spatial derivative modules ( SDM ) and time derivative modules ( TDM ) . Since the spatial derivatives such as ∇ , ∇· , and ∇2 are commonly used across different PDEs , the spatial modules are PDE-independent and they can be meta-initialized from synthetic data . Then , the PDE-specific temporal module is trained to learn the unknown function F ( · ) from few observations in the real-world physical systems . This approach can effectively leverage a large amount of simulated data to train the spatial modules as the modules are PDE-independent and thus mitigating the need for a large amount of real-world tasks to extract shareable features . In addition , since the spatial modules are universally used in physics equations , the representations from the modules can be conveniently integrated with datadriven models for modeling natural phenomena . Based on the modularized PDEs , we introduce a novel approach that marries physics knowledge in spatiotemporal prediction tasks with metalearning by providing shareable modules across spatiotemporal observations in the real-world . Our contributions are summarized below : • Modularized PDEs and auxiliary tasks : Inspired by forms of PDEs in physics , we decompose PDEs into shareable ( spatial ) and adaptation ( temporal ) parts . The shareable one is PDE-independent and specified by auxiliary tasks : supervision of spatial derivatives . • Physics-aware meta-learning : We provide a framework for physcis-aware meta-learning , which consists of PDE-independent/-specific modules . The framework is flexible to be applied to the modeling of different or unknown dynamics . • Synthetic data for shareable modules : We extract shareable parameters in the spatial modules from synthetic data , which can be generated from different dynamics easily . 2 MODULARIZED PDES AND META-LEARNING . In this section , we describe how the physics equations for conserved quantities are decomposable into two parts and how the meta-learning approach tackles the task by utilizing synthetic data when the data are limited . 2.1 DECOMPOSABILITY OF VARIANTS OF A CONTINUITY EQUATION . In physics , a continuity equation ( Eq . 1 ) describes how a locally conserved quantity such as temperature , fluid density , heat , and energy is transported across space and time . This equation underlies many specific equations such as the convection-diffusion equation and Navier-Stokes equations : u̇ = ∇ · ( D∇u ) −∇ · ( vu ) +R , ( Convection-Diffusion eqn . ) u̇ = − ( u · ∇ ) u + ν∇2u−∇ω + g. ( Incompressible Navier-Stokes eqn . ) where the scalar u and vector field u are the variables of interest ( e.g. , temperature , flow velocity , etc. ) . A dot over a variable is time derivative . The common feature in these equations is that the forms of equations can be digested as ( Bar-Sinai et al. , 2019 ; Zhuang et al. , 2020 ) : u̇ = F ( ux , uy , uxx , uyy , . . . ) , ( 3 ) where the right-hand side denotes a function of spatial derivatives . As the time derivative can be seen as a Euler discretization ( Chen et al. , 2018 ) , it is notable that the next state is a function of the current state and spatial derivatives . Thus , knowing spatial derivatives at time t is a key step for spatiotemporal prediction at time t + 1 for locally conserved quantities . According to Eq . 3 , the spatial derivatives are universally used in variants of Eq . 1 and only the updating function F ( · ) is specifically defined for a particular equation . This property implies that PDEs for physical quantities can be decomposable into two modules : spatial and temporal derivative modules . 2.2 SPATIAL DERIVATIVE MODULES : PDE-INDEPENDENT MODULES . Finite difference method ( FDM ) is widely used to discretize a d-order derivative as a linear combination of function values on a n-point stencil . ∂du ∂xd ≈ n∑ i=1 αiu ( xi ) , ( 4 ) where n > d. According to FDM , it is independent for a form of PDE to compute spatial derivatives , which are input components of F ( · ) in Eq . 3 . Thus , we can modularize spatial derivatives as PDE-independent modules . The modules that can be learnable as a data-driven manner to infer the coefficients ( αi ) have been proposed recently ( Bar-Sinai et al. , 2019 ; Seo et al. , 2020 ) . The datadriven coefficients are particularly useful when the discretization in the n-point stencil is irregular and low-resolution where the fixed coefficients cause substantial numerical errors . 2.3 TIME DERIVATIVE MODULE : PDE-SPECIFIC MODULE . Once upto d-order derivatives are modularized by learnable parameters , the approximated spatial derivatives from the spatial modules are fed into an additional module to learn the function F ( · ) in Eq . 3 . This module is PDE-specific as the function F describes how the spatiotemporal observations change . Since the exact form of a ground truth PDE is not given , the time derivative module is datadriven and will be adapted to observations instead . 2.4 META-LEARNING WITH PDE-INDEPENDENT/-SPECIFIC MODULES . Recently , Raghu et al . ( 2019 ) investigate the effectiveness of model agnostic meta-learning ( MAML , Finn et al . ( 2017 ) ) and it is found that the outer loop of MAML is more likely to learn parameters for reusable features rather than rapid adaptation . The finding that feature reuse is the predominant reason for efficient learning of MAML allows us to use additional information which is beneficial for learning better representations . Previously , the objective in meta-training has been considered to be matched with one in meta-test as the purpose of meta-learning is to learn good initial parameters applicable across similar tasks ( e.g. , image classification to image classification ) . We are now able to incorporate auxiliary tasks under a meta-learning setting to reinforce reusable features for a main task . As described in Sec . 2.1 , the spatial modules are reusable across different observations , and thus , we can meta-initialize the spatial modules first with spatial derivatives provided by synthetic datasets . Then , we can integrate the spatial modules with the task-specific temporal module during meta-test to help adaptation of TDM on few observations . Since the spatial modules are trained by readily available synthetic datasets , a large number of similar tasks for meta-training is not required . 3 PHYSICS-AWARE META-LEARNING WITH AUXILIARY TASKS . In this section , we develop a physics-aware meta-learning framework for the modularized PDEs . Fig . 1 describes the proposed framework and its computational process . 3.1 SPATIAL DERIVATIVE MODULE . Algorithm 1 Spatial derivative module ( SDM ) Input : Graph signals ui and edge features eij = xj − xi on G where xi is a coordinate of node i . Output : Spatial derivatives { ûk , i | i ∈ V and k ∈ K } where K = { ∇x , ∇y , ∇2x , ∇2y } . Require : Spatial derivative modules { φk | k ∈ K } 1 : for k ∈ K do 2 : { ak , i , bk , ( i , j ) | ( i , j ) ∈ E and k ∈ K } = φk ( { u } , { e } , G ) 3 : for i ∈ V do 4 : ûk , i = ak , iui + ∑ ( j , i ) ∈E bk , ( j , i ) uj 5 : end for 6 : end for As we focus on the modeling and prediction of sensor-based observations , where the available data points are inherently on a spatially sparse irregular grid , we use graph networks for each module φk to learn the finite difference coefficients ( Bar-Sinai et al. , 2019 ) . Given a graph G = ( V , E ) where V = { 1 , . . . , N } and E = { ( i , j ) : i , j ∈ V } , a node i denotes a physical location xi = ( xi , yi ) where a function value ui = u ( xi , yi ) is observed . Then , the graph signals with positional relative displacement as edge features are fed into the spatial modules to approximate spatial derivatives by Alg . 1 . The coefficients ( ai , b ( i , j ) ) on each node i and edge ( i , j ) are output of φ and they are linearly combined with the function values ui and uj . K denotes a set of finite difference operators . For example , if we set K = { ∇x , ∇y , ∇2x , ∇2y } , we have 4 modules which approximate first/second order of spatial derivatives in 2-dimension , respectively .

Summary: The authors propose methodology for sharing learned differencing coefficients for estimating spatial derivatives between multiple spatio-temporal modeling tasks. They show that increased number of tasks improves learning. Additionally, the authors propose a meta-initialization procedure by which the differencing coefficients are initialized to values obtained from synthetic data. They show that this initialization procedure improves performance.