Meta-Learning Dynamics Forecasting Using Task Inference
Authors: Rui Wang
Robin Walters
Rose Yu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Meta-Learning
Dynamical Systems
Generalizability

Abstract: Current deep learning models for dynamics forecasting struggle with generalization. They can only forecast in a specific domain and fail when applied to systems with different parameters, external forces, or boundary conditions.  We propose a model-based meta-learning method called DyAd which can generalize across heterogeneous domains by partitioning them into different tasks.  DyAd has two parts: an encoder that infers the time-invariant hidden features of the task with weak supervision, and a forecaster which learns the shared dynamics of the entire domain. The encoder adapts and controls the forecaster during inference using adaptive instance normalization and adaptive padding.  Theoretically, we prove that the generalization error of such a procedure is related to the task relatedness in the source domain, as well as the domain differences between source and target. Experimentally, we demonstrate that our model outperforms state-of-the-art approaches on forecasting complex physical dynamics including turbulent flow, real-world sea surface temperature, and ocean currents.      

Meta-Learning Dynamics Forecasting Using Task Inference
Rui Wang*
Robin Walters* Northeastern University
Rose Yu UC San Diego
UC San Diego
Abstract
Current deep learning models for dynamics forecasting struggle with general- ization. They only forecast in specific domain and fail when applied to systems with different parameters, external forces, or boundary conditions. We propose model-based meta-learning method called DyÅd which can generalize across heterogeneous domains by partitioning them into different tasks. Ay has parts: encoder which infers the inveiinvriantt hidden features of the task with weak supervision, and a forecaster which learns the shared dynamics of the entire domain. The encoder adapts and controls the forecaster during inference using adaptive instance normalization and adaptive padding. We prove that the generalization error of such procedure is governed by task relatedness in the source domain and domain differences between source and target. Experimen- tally, demonstrate that our model outperforms state-of-the-art approaches forecasting complex physical dynamics including turbulent flow, real-world sea surface temperature and ocean currents. Our code is available open sourced https:/// http///tttttbb...//// com/ cwm/Rose-eeeoooooooo STL- Ltw.-lal/Dpppaa---adaaaaiiio---e
Introduction
Modeling dynamical systems with deep learning has shown great success in wide range of systems from climate science, Internet of Things (IoT) to infectious diseases [23] 64
33 42 22). However, main limitation of previous works limited generalizability. Most approaches only focus on specific system and train past data in order predict the future. Thus, new model must be trained to predict a system with different dynamics. Consider, for example, learning fluid dynamics; shown in Fig. M are two fluid flows with different degrees of turbulence. Although the flows governed by the same equations, the difference in buoyant forces would require two separate deep learning models forecast. Therefore, it is imperative to develop generalizable deep learning models for dynamical systems that can learn and predict well over a large heterogeneous domain.
Meta-leamnig 633 13,, or learning to learn, improves generalization by learning multiple tasks from the environment. Recent develop- ments in meta-learning have been successfully applied to few-shot classification [46] 662,,
time
tive learning [77), and reinforcement learning
18 20 However, meta-learning in the
text of forecasting high-dimensional physical dynamics has not been studied before. The chal- lenges with meta-learning dynamical systems are unique that (1) we need to efficiently infer the latent representation of the dynamical sys- tem given observed time series data, (2) we need account for changes in unknown initial and boundary conditions, and (3) we need to model the temporal dynamics across heterogeneous domains.
Figure 1: Meta-learning dynamic forecasting turbulent flow. The model needs to generalize to flow with very different buoyant force.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Our approach is inspired by the fact that similar dynamical systems may share time imme-invrriant hidden features. Even the slightest change these features may lead to vastly different phenomena. For example, in climate science, fluids are governed by set of differential equations called Navier- Stokes equations. Some features such as kinematic viscosity and external forces (e.g. gravity), are time-invariant and determine the flow characteristics. By inferring this latent representation, we can model diverse system behaviors from smoothly flowing water to atmospheric turbulence.
Inspired by neural style transfer [26. we propose a model-based meta-learning method, called DyAd, which can rapidly adapt to systems with varying dynamics. DyAd has two parts, an encoder g and a forecaster f. The encoder maps different dynamical systems to time- inneiinaaiaant hidden features representing constants of motion, boundary conditions, and external forces which characterize the system. The forecaster f then takes the learnt hidden features from the encoder and the past system states to forecast the future system state. Controlled by the time-invariant hidden features, the forecaster has the flexibility to adapt to wide range of systems with heterogeneous dynamics.
Unlike gradient-based meta-learning techniques such as MAML [13,, Dy automatically adapts during inference using an encoder and does not require retraining. Similar to model-based meta- learning methods such MetaNets [46], employ a two-part design with an adaptable learner which receives task-specific weights. However, for time-series forecasting, since input and output come from the same domain, a support set of labeled data is unnecessary define the task. The encoder can infer the task directly from the query input.
Our contributions include:
A novel model-based meta-learning method (DyAd) for dynamic forecasting in large heterogeneous domain.
An encoder capable of extracting the inve-invariant hidden features of a dynamical system using time-shift invariant model structure and weak supervision.
new adaptive padding layer (AdaPad), designed for adapting to boundary conditions.
Theoretical guarantees for DyAd on the generalization error of task inference in the source domain as well as domain adaptation to the target domain.
Improved generalization performance on heterogeneous domains such as fluid flow and sea temperature forecasting, even to new tasks outside the training distribution.
2 Methods
2.1 Meta-learning in dynamics forecasting
Let x E Rd be dddimensional state of a dynamical system governed by parameters v. The problem of dynamics forecasting is that given sequence of past states (x))...)x:). we want learn map if such that: f (XI...., x,) + (Xe+1.... .xt+k)) x)+
Here is the length of the input series, and h is the forecasting horizon in the output. Existing approaches for dynamics forecasting only predict future data for a specific system as a single task. Here task refers forecasting for specific system with given set of parameters. The resulting models often generalize poorly to different system dynamics. Thus new model must be trained to predict for each specific system.
To perform meta-lcarning, we identify each forecasting task by some parameters e C. eh, such as constants of motion, external forces, and boundary conditions. We learn multiple tasks simultancously and infer the task from data. Here we use e for a subset of system parameters v, because we usually do not have the full knowledge of the system dynamics. In the turbulent flow example, the state x, is the velocity field at time t. Parameters e can represent Reynolds number, mean vorticity, mean magnitude, or vector of all three.
Let μ be the data distribution over x y representing the function f: + )' where X Rdxt and y = d Our main assumption is that the domain X' can be partitioned into separate tasks X Uc, where X, is the domain for task e and C is the set of all tasks. The data in the same task share the same set of parameters. Let Me be the conditional distribution over X. x y for task

---

Forecaster Network
Forecasts Wi+1:t+k
Conv2D(3d)
Conv2D(3x3)
Input Wi:z
Encoder Network
ComaD (3xSx3)
Pooling
CamvB (bACKC)
Poalling ComoD
(BKEKO)
Pooling
FC
c
Ze
25
Figure 2: Overview of DyAd applied to two inputs of fluid turbulence, one with small external forces and one with larger external forces. The encoder infers the time-shift invariant characteristic variable 2 which is used to adapt the forecaster network.
During training, the model is presented with data drawn from subset tasks {(x; y) (x; y) ~ Me: C C}. Our goal is learn the function f: X' over the whole domain X' which can thus generalize across all tasks c E To do so, we need to learn the map g: + taking x E Ne to c in order to infer the task with minimal supervision.
2.2 DyAd: Dynamic Adaptation Network
We propose model-based meta-learning approach for dynamics forecasting, Given multiple forecast- ing tasks, we propose to learn function / two stages. That is, by first inferring the task from the input x, and then adapting to use a single deep neural network to directly model f in one step over the whole domain. But this requires the training set to have good and uniform coverage of the different tasks. If the data distribution z is highly heterogeneous or the training set not sampled i.i.d. from the whole domain, then single model may struggle with generalization.
specialized forecaster fe: Xe + ) for each task. An alternative
We hypothesize that by partitioning the domain into different tasks, the model would learn to pick up task- specific features without requiring uniform coverage of the training data. Furthermore, by separating task inference and forecasting into two stages, we allow the forecaster to rapidly adapt to new tasks that never appeared in the training set.
As shown in Fig. 2 our model consists of parts: encoder and forecaster f. We introduce ze as
hidden feature for task c. We assume that e depends linearly on the hidden feature for simplicity and easy interpretation. We design the encoder to infer the hidden feature &e given the input æ. We then use ze to adapt the forecaster f to the specific task, i.c., model as y = fxx zz). As the system dynamics are encoded in the input sequence x, we can feed the same input sequence to forecaster and generate predictions ÿ = fz()).
fa(
2.3 Encoder Network
The encoder maps the input x to the hidden features 2, that are time-invariant. To enforce this inductive bias, we encode inne-invariance both in the architecture and in the training objective.
Time-Invariant Encoder. The encoder is implemented using Conv 3D layers, each followed by BatchNorm, LeakyReLU, and max- max-pooling, Note that theoretically, max-pooling is not perfectly shift invariant since 2x2x2 max-pooling is equivariant to shifts of size and only approximately invariant to shifts of size 1. But standard convolutional architectures often include max- moo-iooling layers boost performance. We convolve both across spatial and temporal dimensions.
After that, we use global mean-pooling layer and a fully connected layer to estimate the hidden feature ε.. The task parameter depends linearly on the hidden feature. We use fully connected layer to compute the parameter estimate é.

---

Since convolutions are equivariant to shift (up to boundary frames) and mean pooling is invariant to shift, the encoder is shift-invarr invariant. In practice, shifting the time sequence one frame forward will add one new frame at the beginning and drop one frame at the end. This creates some change in output value of the encoder. Thus, practically, the encoder is only approximately shift-invariant.
Encoder Training. The encoder network g is trained first. To combat the loss of shift invariance from the change from the boundary frames, we train the encoder using time-invariant loss. Given two training samples (z(1), .)) and (z((), y())) and their task parameters c, we have loss
Lene
Σ
1le 110--c1 c/12 + Σ
| - =20 + 3 Σ
! mll
s.1.c
where 2(1) = g(x(1)) and ((ا) g(z(ه)) and 2(0) W - + b is an affine transformation of an.
The first term ||è - ell2 uses weak supervision of the task parameters whenever they are available. Such weak supervision helps guide the learning of hidden feature 2. for each task. While not all parameters of the dynamical system are known, we can compute approximate values in the datum cl0) based on our domain knowledge. For example, instead of the Reynolds number of the fluid flow, can the average vorticity surrogate for task parameters.
The second term || =!
|| is the time-shift invariance loss, which penalizes the changes in latent variables between samples from different time steps. Since the time-shift invariance of convolution is only approximate, this loss term drives the time-shift error even lower. The third term |||=(| m
(m is a positive value) prevents the encoder from generating small 2 due to time-shift invariance loss. It also helps the encoder learn more interesting z, even in the absence of weak supervision.
Hidden Features. The encoder learns time-invariant hidden features. These hidden features resemble the time-invariant dimensionless parameters (30) in physical modeling, such as Reynolds number in fluid mechanics.
The hidden features may also be viewed as partial disentanglement of the system state. As suggested by [37 47,, our disentanglement method is guided by inductive bias and training objectives. Unlike complete disentanglement, as in e.g. [39), in which the latent representation is factored into time- invariant and time-narying components, we focus only on
Nonetheless, the hidden features can control the forecaster which is useful for generalization.
2.4 Forecaster Network.
The forecaster incorporates the hidden feature ze from the encoder and adapts to the specific fore- casting task fe [(; ze). In what follows, we use = for ze. We use two specialized layers, adaptive instance normalization (AdaIN) and adaptive padding (AdaPad), AdaIN has been used in neural style transfer [26] (1) to control generative networks. Here. AdaIN may adapt for specific coefficients and external forces. We also introduce a new layer, AdaPad(x: 2). which designed for encoding the boundary conditions of dynamical systems. The backbone of the forecaster network can be any sequence prediction model. We use design that is similar to ResNet for spatiotemporal sequences.
AdaIN. We employ AdaIN to adapt the forecaster network. Denote the channels of input æ by x, and let ((z;) and (Ix) be the mean and standard deviation of channel i. For each AdaIN layer, particular style is computed s = (H, o.); = Az + b, where the linear map and bias are learned weights. Adaptive instance normalization is then defined as y: = o, p(=p( + Ns. essence, the channels are a(2,)
FC
renormalized to the style
For dynamics forecasting, the hidden feature z encodes data analogous to the coefficients of differential equation and external forces on the system. In numerical simulation of differential equation these coefficients enter as scalings of different terms the equation and the [24]. Thus in
external forces are added to the combined force equation our context AdaIN, which scales channels and adds a global vector, well-suited to injecting this information.
Figure 3: Illustration of the AdaPad operation.

---

AdaPad. To complement AdaIN, we introduce AdaPad, which en- codes the boundary conditions of each specific dynamical system. Gen- erally when predicting dynamical systems, enror is introduced along the boundaries since it is unknown how the dynamics interact with the boundary of the domain, and there may be unknown inflows or outflows. In our method, the inferred hidden feature may contain the boundary informa- tion. AdaPad uses the hidden features to compute the boundary conditions via a linear layer. Then applies the boundary conditions padding immediately outside the spatial domain in each layer, as shown in Fig.
Forecaster Training The forecaster is trained separately after the encoder. The kernels of the convolutions and the mappings of the AdaIN and AdaPad layers are all trained simultaneously as the forecaster network is trained. Denote the true state as y and the predicted state y. compute the loss per time step || w for each example. We accumulate the loss over different time steps and generate multi-step forecasts in an autoregressive fashion. In practice, observe separate training achieves better performances than training end-to-end, see experiments for details.
2.5 Theoretical Analysis
The high-level idea of our method is to learn good representation of the dynamics that generalizes well across heterogeneous domain, and then adapt this representation to make predictions on new tasks. Our model achieves this by learning on multiple tasks simultaneously and then adapting to new tasks with domain transfer. We prove that learning the tasks simultaneously as opposed to independently results in better generalization (Proposition B.3 in Appendix). We also provide theoretical decomposition for the generalization error.
Specifically, our hypothesis space has the form {z + fe(2, (x))] where 6 and are the weights of the encoder and forecaster respectively. Let ex be the error over the entire domain X' that is, for all
Let Eenc (g+) = E ( (g(x), (x)) be the encoder eπror where X' + C is the ground truth. C} be the encoder hypothesis space. Denote the empirical risk of gø by &anc (9a). W, denotes the Wasserstein distance between tasks. R(G) and R(F) represent the Rademacher complexities of encoder and forecaster. The following Proposition decomposes the generalization error in terms of forecaster error, encoder error, and distance between tasks.
Let G {g+:
Proposition 2.1. Assume the forecaster e > fo(,, c) is Lipschitz continuous with Lipschitz constant uniformly in 0 and 1 < 1/2. Let d minsez(e((J) + 1/K E teg (f).. For large and probability 2
K
(x(fo(-96)))) S (g0) +
1 K
Σ
(fo(.cuu) + Eow Wi
po:
Σ
Pex
+ Ad
+27 + + (7 + +1 (log(100)((2k)) + 2llog(108 (VI/n+ vI((nK))
This result helps to quantify the trade-offs with respect different terms in the error bound in our two-part architecture and the error of the model can be controlled by minimizing the empirical errors of the encoder and forecaster. See the Appendix Proposition B.6 for full proofs.
3 Related Work
Learning Dynamical Systems. Deep learning models are gaining popularity for learning dynamical 8 29
systems [60]
73
64 (1]. An emerging topic is physics-ifformed deep learning [53] 6
69 4
68 12
10 w which integrates inductive biases from physical systems to improve learning. incorporated Koopman theory into the architecture.
For example, 45
6
43 used deep neural
networks to solve PDEs with physical laws enforced in the loss functions.
[17 and [9] build models
upon Hamiltonian and Lagrangian mechanics that respect conservation laws. 68 proposed hybrid approach by marrying two well-established turbulent flow simulation techniques with deep learning to produce a better prediction of turbulence. [00 propose a physics-informed GAN architecture that uses physics knowledge to inform the learning of both the generator and discriminator models. However, these approaches deal with a specific system dynamics instead of large heterogeneous domain.
5

---

Multi-task learning and Meta-learning. Multi-task learning [65] focuses on learning shared representations from multiple related tasks. Architecture-basee MTL methods can be categorized into encoder-focusee [35] and decoder-focuset (74). There are also optimization-bal basod MTI. methods, such as task balancing methods [27]. But MTI. assumes tasks are known priori instead of inferring 63 is to leverage the shared representation to fast adapt to unseen tasks. Based on how the meta-level knowledge is extracted and used, meta-learning methods are classified into model-based [46] (0, [59] 78), metric-based
the task from data. On the other hand, the aim of meta-learning
667|6] and gradient-basee [13 55 16, 76.. Most meta-learning approaches are not designed for
forecasting with few exceptions. 72 proposed to train domain classifier with weak supervision to help domain adaptation but focused on low-dimensioal time series classification problems. [50] designed a residual architecture for time series forecasting with meta- meta-learning parallel. [ proposed a modular meta-learning approach for continuous control. But forecasting physical dynamics poses unique challenges meta-learning it requires encoding physical knowledge into model.
Style Transfer. Our approach is inspired by neural style transfer techniques. style transfer, a generative network is controlled by an external style vector through adaptive instance normalization between convolutional layers. Our hidden representation bears affinity with the "style" vector in style transfer techniques. Rather than aesthetic style in images, our hidden representation encodes time-invarian features. Style transfer initially appear in non-photorealistic rendering [1]. Recently, neural style transfer [25] has been applied to image synthesis [15] 36), videos generation [56], and language translation [52]. For dynamical systems, [58) adapts texture synthesis to transfer the style of turbulence for animation.
28 studies unsupervised generative modeling of turbulent flows but for super-resolution reconstruction rather than forecasting.
Video Prediction. Our work is also related to video prediction. Conditioning on the historic observed frames, video prediction models are trained to predict future frames, e.g., [4104] 75 66 49 [4] 70
31 40 34 There is also conditional video prediction [48] which achieves controlled synthesis. Many of these models are trained on natural videos from unknown physical processes. Our work is substantially different because we do not attempt to predict object or camera motions. However, our method can be potentially combined with video prediction models to improve generalization.
Experiments
Target
Unet-c
Weight-attn
DyAd
Target
Unet-c
Weight-attn
DyAd
t= 10
t
Figure 4: Target and predictions by Unet-c, Modular-wt and DyAd at time 1, 10 for turbulent flows with buoyancy factors 9 (left) and 21 (right) respectively. DyAd can easily generate predictions for various flows while basclines have trouble understanding and disentangling buoyancy factors.
4.1 Datasets
We experiment three datasets: synthetic turbulent flows, real-world sea surface temperature and ocean currents data. These are difficult to forecast using numerical methods due to unknown external forces and complex dynamics not fully captured by simplified mathematical models.
Turbulent Flow with Varying Buoyancy. We generate synthetic dataset of turbulent flows with numerical simulator, P It contains 64x64 velocity fields of turbulent flows in which we vary

---

Table 1: Prediction RMSE on the turbulent flow and surface temperature datasets. Prediction RMSE and ESE (energy spectrum errors) on the future and domain test sets of ocean currents dataset.
Turbulent Flows
Sea Temperature
Ocean Currents
Model
future
domain
future
domain
future
domain
ResNet
0.
0.65
0.73
0.71
9.44:11.55
10.9
9.65=
0.90::0.16
ResNet-c
0.88
0.64
0.70=
0.71
9.71=10.11
10.81
9.15=0.01
0..3
U-Net
0.92=10.22
0.68:10.2
0.57:10.05
0.63=0.05
7.64::0.05
10.83:10.02
7.11
0.86:10.03
Unet-c
0.86=0.07
0.68
0.47=0002
0,45=0.06
7.26=:0.01
10.94=00.2
7.50
0.87=00044
PredRNN
0.75
0.75
0.67=0.12
0.99=0.07
8.49
11.27:0002
8.99=0.03
1.69
VarSepNet
0.67=#0.05
0.63=#0.06
0.63=10.14
0.49=0009
9.36=#.02
10.63:+0.04
7.10=0.01
0.588
Mod-attn
0.63
0.92.10.03
0.89110.21
0.98
8.08110.07
10.76.10.11
8.31 0.19 0.88.0.14
Mod-wt
0.58
0.60.10.07
0.65 10.00
0.64
10. 10.12 1..9
8.11 1. 0.82.10.19
MetaNet
0.76110.3
0.761.0.08
0.84
0.82
10.9
1.15.10.18
11.2 1.116 1.0811.221
MAML
0.63=0001
0.68#0.02
0.90=0.17
0.67=0.04
10.1 #0.21 00.05
1079 0.99
DyAd-ResNet
0.42:10.01
0.502
0.42:10.03
0.44
7.28 I 0.58
7.04 0.54
DyAd+Unet
0.58
0.59
0.35
0.42
1..0 10.70.10.44
7.46 0. 0..00
the buoyant force acting on the fluid from 1 to 25. Each buoyant force corresponds to forecasting task and there are 25 tasks in total. We the mean vorticity of each task as partial supervision as we can directly calculate it from the data. Vorticity can characterize formation and circular motion of turbulent flows.
Sea Surface Temperature. We evaluate on a real-world sea surface temperature data generated by the NEMO ocean engine [8 We select an area from Pacific ocean range from 01/01/2018 to 12/31/2020. The corresponding latitude and longitude are (-150~-120, -20~-50). This area is then divided into 25 64 subregions, each is task since the mean temperature varies lot along longitude and latitude. For the encoder training, we use season as an additional supervision signal besides the mean temperature of each subregion. In other words, the encoder should be able infer the mean temperature of the subregion as well as to classify four seasons given the temperature series.
Ocean Currents. We also experiment with the velocity fields of ocean currents from the same region and use the same task division as the sea surface temperature data set. Similar to the turbulent flow data set, the mean vorticity of each subregion the weak signal.
4.2 Baselines
We include several SoTA basclines from meta-lcarning and dynamics forecasting.
ResNet 19]
widely adopted video prediction model [69]
U-net
55
Originally developed for biomedical image segmentation, adapted for dynamics
forecasting 111.
ResNet/Unet-c: Above ResNet and Unet with an additional final layer that generates task parameter and trained with weak-supervision and forecasting loss altogether.
PredRNN (70): A state-of-art RNNs-based spatiotemporal forecasting model.
VarSepNet 12 A convolutional dynamics forecasting model based on spatiotemporal disentan-
glement.
Mod-atta A modular meta-learning method which combines the outputs of modules to gener- alize to new tasks using attention.
Mod-wt: A modular meta-learning variant which uses attention weights to combine the parameters of the convolutional kernels in different modules for new tasks.
MetaNet [46]: A model-based meta-learning method which requires few samples from test tasks
support set to adapt.
MAML [13]: A optimization-basee meta-learning approach. We replaced the original classifier with a ResNet for regression.
"The data are available at https: http://rrssuucco..miieaeee marine. oat.rea...pppommiuo.///oooooooor
detailskproduct__t 1i- ANALYSIS BIAAECAAAT_PH__O00
7

---

Target
ResNet
U-net
PredRNN
DyAd
Model
future
domain
DyAr()urs) 0.00
0.51
No_enc 0.63+00.3
0.60+
No Ao_AdaPad 0.47
0.54
t= 10
Wrong_ent 0.66110.2
0.62.10.33
End2End
0.45
0.54
Table 2: Ablation study: prediction RMSE of DyÅd and its variations with different compo- nents removed from DyAd.
Figure 5: DyAd, ResNet, U-net, PredRNN velocity norm (vu² 6²) predictions ple in the future test set.
current
Note that both ResNet-c and Unet-c с have access to task parameters Modular-aa attn has convolutional encoder f that takes the same imput æ as each module M generate attention weights, exp expi f(x)())
2 expl (oe (A3) (x). Modular- ar-wlarwwt also has the same encoder but to generate weights for combining the convolution parameters of all modules. MetaNet requires samples from test tasks as support set and MAML needs adaptation retraining on test tasks, while other models do not need any information from the test domains. Thus, we use additional samples of to 20% of the test set from test tasks. MetaNet uses these as a support set. MAML is retrained on these samples for 10 epoch for adaptation. To demonstrate the generalizability of DyAd, experimented with ResNet and U-net as our forecaster.
EE=
4.3 Experiments Setup
For all datasets, we use a sliding-window approach to generate samples of sequences. We evaluate on two scenarios of generalization. For test-future, we train and test the same task but different time steps. For test-domain, we train and test on different tasks with an 80-20 split. All models are trained to make next step prediction given the history. We forecast in an autoregressive manner to generate multi-step ahead predictions. All results are averaged over 3 runs with random initialization.
Apart from the root mean square етror (RMSE), we also report the energy spectrum error (ESE) for ocean current prediction, which quantifies the physical consistency. ESE indicates whether the predictions preserve the correct statistical distribution and obey the energy conservation law, which is a critical metric for physical consistency, See details about energy spectrum and complete experiments details in Appendix A.T
Results
c
= 15
= 25
1= t=5
5.1 Prediction Performance.
Table [ shows the RMSE of multi-step predictions on Turbulent Flows (20 steps), Sea Surface Temperature (10 steps), and Ocean Currents (10 step) in two testing scenar- ios. We observe that DyAd makes the most accurate pre- dictions in both scenarios across all datasets. Comparing ResNet/Unet--e with DyAd, we observe the clear advan- tage of task inference with separate training. VarSepNet achieves competitive performances on Ocean Currents (second best) through spatiotemporal disentanglement but cannot adapt to future data. Table also reports ESEs real-world Ocean Currents. DyAd not only has small RMSE but also obtains the smallest ESE, suggesting it captures the statistical distribution of ocean currents well.
Figure 6: Outputs from DyAd while we vary encoder input but keep the forecaster input fixed. From left to right, the encoder is fed with flow with different buoyancy factor c = 5, 15, 25. the forecaster net- work input has fixed buoyancy c 15.
Figure 4 shows the target and the predicted velocity norm fields (Vu² + 6z) by Unet-c, Modular-wt and Ad at time step 1, 5, for Turbulent Flows with buoyancy factors and 21 respectively. We can see that Ay generate realistic flows with the corresponding characteristics while the

---

baselines have trouble understanding and disentangling the buoyaney factor. Figure S shows DyAd, ResNet, U-net, PredRNN predictions on an ocean current sample in the future test set. and we see the shape of predictions by DyAd is closest to the target. These results demonstrate that Dy not only forecasts well but also accurately captures the physical characteristics of the system.
5.2 Controllable Forecast.
DyÅd infers the hidden features from data, which allows direct control of the latent space in the forecaster. We vary the encoder input while keeping the forecaster input fixed. Figure [ right shows the forecasts from DyAd when the encoder is fed with flows having different buoyancy factors c 5, 15, 25. As expected, with higher buoyancy factors, the predictions from the forecaster become more turbulent. This demonstrates that the encoder can successfully disentangle the latent representation of difference tasks, and control the predictions of the forecaster.
5.3 Ablation Study.
We performed several ablation studies of DyAd on the turbulence dataset to understand the contribution of each model component.
Ablation of the Model and Training Strategy. We first performed ablation study of the model architecture, shown in Table [ We first remove the encoder from DyÅd while keeping the same forecaster network (No_enc). The resulting model degrades but still outperforms ResNet. This demonstrates the effectiveness of AdaIN and AdaPad for forecasting. We also tested Dy with AdaIN only (No_AdaPae AdaPad,, and the performance without AdaPad was slightly worse.
Another notable feature of our model is the ability infer tasks with weakly supervised signals It is important to have a that related to the task domain. As an ablative study, we fed the encoder in DyAd with random leading to trong_enc We can see that having the totally wrong supervision may slightly hurt the forecasting performance. We also trained the encoder and the forecaster in Ad altogether (End2End) but observed worse performance. This validates hypothesis about the significance of domain partitioning and separate training strategy.
Ablation of Encoder Training Losses. We performed an ablation study of three training loss terms for the encoder to show the necessity of each loss term, shown in Table tried training the encoder only with the supervision of task parameters (Sup_c). We also tested training with the first two terms but without the magnitude loss (Sup_ (Sup_cttime cttime__imm _iv). We can see that missing any of three loss terms would make the encoder fail to learn the correct tpsc-sp and ameeinnvriiat features.
Alternatives AdaIN. We also tested 5 different alternatives AdaIN for injecting the hidden feature 2, into the forecaster, and reported the results in Table We tested concatenating =e to the input of the forecaster, using 2, as the kernels of the forecaster, concatenating &e to the hidden states, adding &e to the hidden states, using & as the biases in the convolutional layers in the forecaster. AdalN worked better than any alternative we tested.
Model
future
domain
RMSE
AdaIN Conl
KGen
ConH
Sum
Bias
DyAd (ours) 0.42
0.51
future 0.42
1.09
0.78
0.75
0.84
0.84
Sup_c 0.53
0.56
domain 0.51
0.85
0.78
0.74
0.80
0.78
Sup_cttime_itt 0.51
0.54
Table 4: We tested concatenating ze the input of the forecaster (Conl), using ze as the kernels of the forecaster (KGen), concatenating 2e to the hidden states (ConH), adding to the hidden states(Sum), using 2e as the biases in the convolutional layers (Bias). AdaIN worked better than any alternative we tested.
Table 3: Ablation study of the encoder training loss terms. We tried training the encoder only with the weak super- vision (Sup_c), without the magnitude loss (Sup_ctt Ship__cttimm_mne inv).
Conclusion
We propose a model-based meta-learning method, DyAd to forecast physical dynamics. DyAd uses an encoder to infer the parameters the task and prediction network to adapt and forecast giving the inferred task. Our model can also leverage any weak supervision signals that can help distinguish

---

different tasks, allowing the incorporation of additional domain knowledge. On challenging turbulent flow prediction and real-world ocean temperature and currents forecasting tasks, we observe superior performance of our model across heterogeneous dynamics. Future work would consider non-grid data such as flows on graph or a sphere.
Acknowledgments and Disclosure of Funding
We are grateful to several anonymous reviewers for their comments which have helped us to improve this work. This work was supported in part by U.S. Department Of Energy, Office of Science grant DE-SC0022255, U. S. Army Research Office grant 101INF-20---03344 and NSF grants #2134274 and #214633. R. Walters NSF grants #2107256 and #2134178. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science the U.S. Department of Energy under Contract No. DE-AC02-05CHI using NERSC award DASCR-ERCAP002211
supported by the Roux Institute and the Harold Alfond Foundation and
References
[1] Ferran Alet, Tomas Lozano-Perez, and Kaelbling. Modular meta-learning. ArXiv, hls/1806.10166 2018.
[2] Rie Kubota Ando, Tong Zhang, and Peter Bartlet. framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(1 (), 2005.
[3] I. Ayed, Emmanuel de Bézenac, A. Pajot, J. Brajard, and P. Gallinari. Learning dynamical systems from partial observations. ArXiv, abs/1992. 11366, 2019.
[4] Ibrahim Ayed, Emmanuel De Bézenac, Arthur Pajot, and Patrick Gallinari. Learning partially observed PDE dynamics with neural networks, 2019.
[5] Omri Azencot, N. Erichson, V. Lin, and Michael W. Mahoney. Forecasting sequential data using consistent koopman autoencoders. In International Conference on Machine Learning, 2020.
[6] Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting se- quential data using consistent koopman autoencoders. In International Conference Machine Learning, pages 475-48$ PMLR, 2020.
[7] Jonathan Baxter. Theoretical models of learning to learn. In Learning to learn, pages 71-94. Springer, 1998.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary dif- ferential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6572-6583, 2018.
[9] M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, and S. Ho. Lagrangian neural networks, ArXiv, abbs22003.04630 2020.
[10] Arka Daw, M Maruf, and Anuj Karpatne. Pid-gan: A gan framework based on physics- informed diseriminator for uncertainty quantification with physics. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 237-247, 2021.
[11] Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical pro- cesses: Incorporating prior scientific knowledge. In International Conference on Learning Representations, 2018.
[12] Jérémie Donå, Jean- Yves Franceschi, sylvain lamprier, and patrick gallinari. (PDE) spatiotemporal disentanglement. In International Conference on Learning Representations, 2021.
10

---

[13] Chelsea Finn, P. Abbeel, and S. Levine. Model-agnosse meta-learning for fast adaptation of deep networks. In International Conference of Machine Learning. 2017.
[14] Chelsea Finn, Ian Goodfellow, and Sergey Leine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pages 64-72, 2016.
[15] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2422 2016.
[16] E. Grant, Chelsea Finn, S. Levine, Trevor Darrell, and T. Griffiths. Recasting gradient-basee meta-learning as hierarchical bayes. ArXiv Preprint,
2018.
[17] S. Greydanus, Misko Dzamba, and Yosinski. Hamiltonian neural networks. ArXiv, abs/1906.01563, 2019.
[18] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Systems, pages 5307-5316, 2018
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Yun Hua, Xiangfeng Wang. Bo Jin, Wenhao Li, Junchi Yan, Xiaofeng He, and Hongyuan Zha. Hmrl: Hyper-meta learning for sparse reward reinforcement learning problem. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pages 637-645, 2021.
[21] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1501-1510, 2017.
[22] Zijie Huang, Yizhou Sun, and Wei Wang. Coupled graph ode for learning interacting system dynamics. In 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2021, pages 705-715, 2021.
[23] Jessica Hwang, Paulo Orenstein, Judah Cohen, Karl Pfeiffer, and Lester Mackey. Improving subseasonal forecasting in the western us with machine learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2325-2335, 2019.
[24] J.C.Butcher. Applied Numerical Mathematics, volume 20. Elsevier B.V., 1996.
[25] Yongeheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neu- ral style transfer: A review. IEEE transactions on visualization and computer graphics, 26(11):3365-3385 2019.
[26] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[27] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1482-74991 2018
[28] Junhyuk Kim and Changhoon Lee. Deep unsupervised learning of turbulence for inflow generation at various reynolds numbers. arXiv: 10515, 2019.
[29] Zico Kolter and Gaurav Manek. Learning stable deep dynamics models. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pages 1112--1136, 2019.
[30] Josef Kunes. Dimensionless physical quantities in science and engineering. Elsevier, 2012.
11

---

[31] Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, and Tobias Isenberg. State of the" art": A taxonomy of artistic stylization techniques for images and video. IEEE transactions on visualization and computer graphics, 9955)86668883 2012.
[32] Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Computer Vision and Pattern Recognition (CVPR). 2020.
[33] Zongyi Li, Nikola Kovachki, Kamyar Arizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. International Conference on Learning Representations, 2021.
[34] Haoxing Lin, Rufan Bai, Weijia Jia, Xinyu Yang, and Yongjian You. Preserving dynamic attention for long-term spatial-tempont prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 36-46, 2020.
[35] Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR), pages 1871-1880, 2019.
[36] Xiao Liu, Spyridon Thermos, Alison Q. O'Neil, and Sotirios A. Tsaftaris. Semi-supervised meta-learning with disentanglement for domain- menain-generaalsed medical image segmentation. In MICCAI, 2021.
[37] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14 4124. PMLR, 2019.
[38] Gurvan Madec et al. NEMO ocean engine, 2015. Technical Note. Institut Pierre-Simon Laplace (IPSL), France. http::///ppacccccr.....////////s
vi0.s__60099ppp. pdd]
[39] Armand Comas Massague, Chi Zhang, Zlatan Feric, Octavia Camps, and Rose Yu. Learning disentangled representations of video with missing data. arXiv preprint arXiv:2006.1339, 2020.
[40] Armand Comas Massague, Chi Zhang, Zlatan Feric, Octavia Camps, and Rose Yu. Learning disentangled representations of video with missing data. arXiv preprinl arXiv: 233911
[41] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint a 1.05440, 2015.
[42] Yasuko Matsubara and Yasushi Sakurai. Dynamic modeling and forecasting of time-evolving data streams. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 458 468. 2019.
[43] George E Karniadakis Maziar Raissi, Paris Perdikaris. Physics-informed informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics. 378:686-707, 2019.
[44] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning.
MIT press, 2018.
[45] Jeremy Morton, Antony Jameson, Mykel J. Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. In Advances in Neural Information Processing Systems (NeurIPS), 2018
[46] Tsendsuren Munkhdalai and Hong Yu. Meta networks. Proceedings of machine learning research, 70:2554-2563, 2017.
[47] Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debhath, A. Patney, Ankit B. Patel, and Anima Anandkumar. Semi-supervised stylegan for disentanglement learning. In International Conference Machine Learning, 2020.
12

---

[48] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee. Richard Lewis, and Satinder Singh. Action- conditional video prediction using deep networks in atari games. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 2, pages 2863- 2871, 2015.
[49] Sergiu Oprea, P. Martinez-Goonzleez, A. Garcia-Garcia, John Alejandro Castro- Vargas, S. Orts- Escolano, J. Garcia- MorriiaRRddrigezz, and Antonis A. Argyros. A review deep learning techniques for video prediction. ArXiv, abs/2004.052144 2020.
[50] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2019.
[51] Tobias Pfaff, Meire Fortunato, Alvaro Gonbzlez,, and Peter Battaglia. Learning mesh- based simulation with graph networks. In International Conference on Learning Representations, 2021.
[52] Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 866-876, 2018.
[53] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learn- ing (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561 2017.
[54] levgen Redko, Amaury Habrard, and Mare Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 737-753. Springer, 2017.
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image compuling and computer-assisted intervention, pages 234-241. Springer, 2015.
[56] Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for videos. In German conference on pattern recognition, pages 26-36. Springer, 2016.
[57] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019.
[58] Syuhei Sato, Yoshinori Dobashi, Theodore Kim, and Tomoyuki Nishita. Example-bar based turbulence style transfer. ACM Transactions on Graphics (TOG), 37(4):1-9, 2018.
[59] Sungyong Seo, Chuizheng Meng, Sirisha Rambhatla, and Y. Liu, Physics-aware spatiotemporal modules with auxiliary tasks for meta-learning. arXiv preprint arXiv: 0883. 2020.
[60] Xingjian Shi, Zhihan Gao, Leonard I Lausen, Hao Wang, D. Yeung, W. Wong, and Wang chun Woo. Deep learning for precipitation nowcasting: benchmark and new model. In Advances in neural information processing systems, 2017.
[61] J. Snell, Kevin Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
[62] Qiuling Suo, Jingyuan Chou, Weida Zhong, and Aidong Zhang Tadanet: Task-adaptive network for graph-enriched meta-learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1789-1799, 2020.
[63] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pages 3-17. Springer, 1998.
[64] Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating Fulerian fluid simulation with convolutional networks. In JCML'17 17 Proceedings of the 34th International Conference Machine Learning, volume 70, pages 3424-3433, 2017.
13

---

[65] Simon Vandenhende. Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: survey, IEEE transactions on pattern analysis and machine intelligence, PP, 2021.
[66] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference on Learning Representations (ICLR), 2017.
[67] Oriol Vinyals, Charles Blundell, T. Lillicrap, K. Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
[68] Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1457-1466 2020.
[69] Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint 2002::2000.0306... 2020.
[70] Yunbo Wang, Mingsheng Long, Jianmin Wang. Zhifeng Gao, and Philip Yu. PredRNN: Recurrent neural networks for predictive learning using spatiotemporal LSTMs. In Advances in Neural Information Processing Systems, pages 879-888, 2017.
[71] Yunbo Wang. Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S Yu, and Ming- sheng Long. PredRNN: A recurrent neural network for spatiotemporal predictive learning, 2021.
[72] Garrett Wilson, Janardhan Rao Doppa, and Diane J Cook. Multi-source deep domain adaptation with weak supervision for time-series sensor data. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1778, 2020.
[73] You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow. ACM Transactions on Graphics (TOG), 37(4):95, 2018.
[74] Dan Xu, Wanli Ouyang, Xiaogang Wang, and N. Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing. 2018 1EEF/CVF Conference on Computer Vision and Pattern Recognition, pages 675-684, 2018.
[75] Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamies: Probabilistic future frame synthesis via cross convolutional networks. In Advances in neural information processing systems (NeurIPS). pages 91 199 2016.
[76] Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In International Conference on Machine Learning, pages 7045-7054. PMLR, 2019.
[77] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian monsl-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7343-7353, 2018.
[78] Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. In International Conference on Learning Representations, 2021.
14


Reviews:
[{"review_id": "qOM2dZf0cgm", "reviewer": "Reviewer_xxYt", "summary": "The paper proposes a method to predict dynamical systems when system parameters can differ between training and evaluation.\nFirst, a time invariant network is trained to predict the dynamical \"invariants\" which could be the number of vortices in a flow or system parameters.\nThen a prediction network is trained to forecast the system dynamics, taking the dynamic invariants as an input.\n", "questions": "1. How do meta-training and meta-testing sets differ? Can you quantify how large the shift is between training and testing dataset (e.g. a shift in the mean number of vortices)?  In particular, do you evaluate on values of the invariant that are not included in the training set?\n2. Feeding the invariant as an input to the prediction model is one way to incorporate the information. If one can compute invariants in a differentiable way one could also incorporate this as a loss where context and predictions should not differ in the invariant.\nHave you considered this aswell? It probably only works for some invariants, but may be interesting.", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Interesting and innovative idea. (I may not be aware of prior work using this approach)\n- Good experimental results, although maybe a bit limited datasets.\n- Good and clear presentation\n\n### Weaknesses\nOne potential weakness of the method is that one needs access to the invariants for training.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "icUqBmkSc_J", "reviewer": "Reviewer_Nf4r", "summary": "This paper proposes a meta-learning mechanism to address current generalization limitations in dynamics forecasting works, in which the majority of works learn deep learning models capable of capturing one dynamics at a time. To this end, they propose *DyAd*, a two-staged approach in which an encoder learns the time-invariant task-specific hidden features of a given observation sequence and influences a forecasting network that aims to learn the shared dynamics of the heterogeneous domain. A time-invariant loss function for the encoder is proposed to encourage time-task disentanglement between encoder and forecaster, leveraging weak-supervision from task-specific parameters. They leverage style-transfer adaptive instance normalization for forecaster adaptation and propose a novel encoder-influenced padding layer called *AdaPad* to address unknown boundary errors. Theoretical proofs are provided to quantify loss term trade-offs in the generalization error and show that there is a relationship between error and task relatedness for the source and target domains. Ablations are provided for each proposed model component and are evaluated on three physical dynamics tasks, including synthetic flows and real-world sea surface temperature and ocean current tasks.", "questions": "**Suggestions:**\n- Currently it is hard to interpret the differences in performance of each method through the metrics alone, given that only snapshots of time are shown within the figures. Adding in a figure in the Appendix showcasing full sequence predictions would help the reader understand the quantitative to qualitative mapping for RMSE and ESE.\n- The discussion on the Forecaster training and its loss function is a bit short-handed compared to other sections and it can be a bit difficult to interpret how it is exactly formulated. The Appendix highlights that multiple unrolled prediction steps are used in backpropagation while the Forecaster Training section highlights the loss is accumulated over 'different time steps'. Clarification on whether this refers to 1) each training sequence has multiple time windows evaluated on for its pure next step prediction and this is what is accumulated in loss for backpropagation or whether 2) for each window used, multiple steps are predicted out autoregressively and the multi-step sequence is used in the loss function.", "limitations": "The authors properly address limitations in the claims of their proofs and what information should be gleaned from them. Additionally, limitations in the metrics used to support evaluation are discussed in the Appendix. Potential negative societal impact are not discussed in the work and is denoted as such in the Author's Checklist. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- This work tackles a novel task for dynamic systems through the application of meta-learning to enable learning a set of dynamics functions with shared underlying mechanics under one model. It provides ample discussion on common failings of learning dynamical systems and proposes fixes to address them. It is placed well within relevant literature and indeed tackles an insofar uncommon setting.\n- The architecture choice of a task-feature encoder that adapts a forecasting network per task via inter-layer controlled padding is intuitive and the rigorous ablations to support the addition of each component strengthens the work.\n- Ample baselines are provided from classical video prediction models to applied meta-learning methods used as comparisons.\n- The provided code in the Supplementary Material is clean to run and has reproducible results regarding baselines and proposed model. \n- The writing and presentation is clear with little confusion on details concerning architecture or the training setting. The network components are explained well and effectively leverages visualizations in its presentation.\n\n**Minor Code Clean-up:**\n\nREADME.md:\n- A note regarding the base size of the dataset (>150GB) from 'data_generation.py' and how to generate smaller testing versions would help for more approachable evaluation of the provided code.\n- There are a variety of unused imports throughout all of the provided scripts which should be cleaned up to reduce unneeded dependencies.\n\ndata_generation.py: \n- Has mkdir errors given no path checking for output folders on subsequent runs\n- Requires a local module import for phi.geom Sphere not already included\n- Variable \"Resolution\" is undefined and needs removing\n\n**Minor Writing Clean-up:**\n- Line 623: 'backpropogation'\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8p3FCKU6_7a", "reviewer": "Reviewer_A5Th", "summary": "A physics informed meta-learning architecture is introduced to model different kinds of dynamics. An encoder generates a time-invariant latent state $\\hat{z}$ representing physical properties of the observed dynamics. In combination with the dynamics field, a decoder takes this latent state to condition its prediction of the next dynamic steps.", "questions": "- The input and output sequences have different indices compared to the text (lines 71 and 72). It might help readers if the indices were consistent, i.e., changing $w_{1:t}$ and $\\hat{w_{t+1:t+k}}$ into $w_{t-l+1:t}$ and $\\hat{w_{t+1:t+h}}$.\n- According to your reported changes in $\\hat{c}$ when shifting the input (over time), how much does $\\hat{c}$ actually change (in particular since you enforce it to stay constant with the second term in equation (1))? It would be great if you could show that $\\hat{c}$ is fairly constant for one time series encoded at different temporal samples. Certainly, though, $\\hat{c}$ should change meaningfully when another task is encoded.\n- Since the decoder conditions its prediction on the hidden features, could you also formulate $f_c = f(x|z_c)$ to emphasize this conditioning?\n- Could you use something different from $\\mu$ for the data distribution in line 84, since you also choose $\\mu$ when defining the mean of channels for AdaIN (line 155)?\n- In line 156, should it be \"For each AdaIN *channel*, a particular style is computed $s_i = $...\", or are the styles indeed layer-wise? From figure 2 I assume that you only use one AdaIN layer, though, meaning you only use one style?\n- Have you tested recurrent modules in the encoder and forecaster?\n- How do you explain the performance of the big U-Net as forecaster compared to the small one? And do you have an intuition why U-Nets seem better on the SST data while ResNet is superior in Turbulent Flows and Ocean Currents?", "limitations": "Limitations not addressed by the authors. I'd be curious to learn about situations where this model might have difficulties.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\nStrengths:\n- The AdaPad layer looks intriguing, as it allows the boundary condition (BC) to depend on $x$. In contrast to traditional padding methods (zero/constant or mirror), AdaPad thus seems able to account for dynamic BCs, such as Neumann and Cauchy. However, it might help if it additionally receives $x$ as input. Have the authors experimented with this?\n- Fairly simple but highly efficient and well designed architecture to model different kinds of dynamics with one and the same forecaster\n\n### Quality:\nStrengths:\n- Overall very detailed and well motivated model architecture, data description and evaluation\n- Clear demonstration of successful encoding of physically relevant factors (figure 6)\n- Exhaustive ablations showing the relevance of the different introduced network components\n\nWeaknesses:\n- Apart from the ablations in table 2 and in the appendix, it would be informative to see the effect of different choices of $m$\n\n### Clarity and significance:\nStrengths:\n- Superiority of the model shown on both synthetic and real-world datasets\n\nWeaknesses:\n- I'm not fully clear about the conclusion of the proof and would appreciate an intuition about the result and what it actually means.\n- AdaPad operator not explicitly evaluated on changing boundary conditions", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept