Meta-Learning Dynamics Forecasting Using Task Inference
Authors: Rui Wang
Robin Walters
Rose Yu
Venue: ICLR 2022 Submitted (2022)
Keywords: meta-learning
generalizability
dynamical systems

Abstract: Current deep learning models for dynamics forecasting struggle with generalization. They can only forecast in a specific domain and fail when applied to systems with different parameters, external forces, or boundary conditions.  We propose a model-based meta-learning method called DyAd which can generalize across heterogeneous domains by partitioning them into different tasks.  DyAd has two parts: an encoder which infers the time-invariant hidden features of the task with weak supervision, and a forecaster which learns the shared dynamics of the entire domain. The encoder adapts and controls the forecaster during inference using adaptive instance normalization and adaptive padding.  Theoretically, we prove that the generalization error of such procedure is related to the task relatedness in the source domain, as well as the domain differences between source and target. Experimentally, we demonstrate that our model outperforms state-of-the-art approaches on both turbulent flow and real-world ocean data forecasting tasks.     

Under review as a conference paper at ICLR 2022
META- DYNAMICS FORECASTING USING TASK INFERENCE
Anonymous authors Paper under double-bling review
ABSTRACT
Current deep learning models for dynamics forecasting struggle with generalization. They can only forecast in specific domain and fail when applied to systems with different parameters, external forces, or boundary conditions. We propose a model-based meta-learning method called Dyåd which can generalize across heterogencous domains by partitioning them into different tasks. DyAd has two parts: an encoder which infers the time-invariant hidden features of the task with weak supervision, and a forecaster which learns the shared dynamics of the entire domain. The encoder adapts and controls the forecaster during inference using adaptive instance normalization and adaptive padding. Theoretically, that the generalization error of such procedure related to the task relatedness the source domain, as well as the domain differences between source and target. Experimentally, we demonstrate that our model outperforms state-of- -fhe--f---h--tttt approaches on both turbulent flow and real-world ocean data forecasting tasks.
prove
INTRODUCTION
Modeling dynamical systems with deep learning has shown great success in wide range of systems from fluid mechanics to neural dynamics (Tompson et al. 2017 Chen et al. 2018 Kolter & Manek 2019 Zoltowski al. 2020 Li et al., 2021). However, the main limitation of previous works is very limited generalizability. Most approaches only focus on specific system and train on past data in order to predict the future. Thus new model must be trained predict a system with different dynamics. Consider, for example, learning fluid dynamics; shown in Fig. Ture two fluid flows with different degrees of turbulence. Even though the flows are governed by the same equations, the difference in buoyant forces would require two separate deep learning models to forecast. Therefore, it is imperative to develop generalizable deep learning models for dynamical systems that can learn and predict well over a large heterogeneous domain.
Meta-learning (Thrun & Pratt 1998 Baxter 1998; Finn al. 2017), or learning to learn, improves generalization by learning multiple tasks from the environment. Recent develop- ments in meta-learning have been successfully applied few-shot classification (Munkhdalai & Yu, 2017), active learning (Yoon et al. 2018) and reinforcement learning (Gupta et al. 2018
time
?
However, meta- leaannen in the context of fore- casting high- -igh-dimensional physical dynamics has not been studied before. The challenges with meta-learning dynamical systems are unique in that (1) need to efficiently infer the latent representation the dynamical system given observed time series data, (2) we need to account for changes in unknown initial and boundary conditions, and (3) we need model the temporal dynamics across heterogeneous domains.
Figure 1: Meta-learning dynamic forecasting on
turbulent flow. The model needs to generalize to flow with a very different buoyant force.
Our approach is inspired by the fact that similar dynamical systems may share time invariant hidden features. Even the slightest change these features may lead to vastly different phenomena. For example, in climate science, fluids governed by of differential equations called Navier-

---

Under review as a conference paper at ICLR 2022
Stokes equations. Some features such as kinematic viscosity and external forces (e.g. gravity), lime-invariant and determine the flow characteristics. By inferring this latent representation, we can model diverse system behaviors from smoothly flowing water to atmospheric turbulence.
Inspired by neural style transfer (Karras et al., 2019), we propose model-based meta-learning method, called DyAd, which rapidly adapt to systems with varying dynamies. DyAd has two parts, an encoder and a forecaster f- The encoder maps different dynamical systems to time- invariant hidden features representing constants of motion, boundary conditions, and external forces which characterize the system. The forecaster / then takes the hidden representations and the past system states to forecast the future system state. Controlled by the ameeinnarian hidden features, the forecaster has the flexibility to adapt to a wide range of systems with heterogeneous dynamics.
Unlike gradient- meta-learning techniques such MAML (Finn al. 2017). DyAd auto- matically adapts during inference using encoder and does not require any retraining. Similar to model-based meta-learning methods such as MetaNets (Munkhdalai & Yu, 2017), we employ
two-part design with an adaptable learner which receives task-specific weights. However, for time series forecasting, since input and output come from the same domain, a support set of labeled data unnecessary to define the task. The encoder can infer the task directly from query input.
Our contributions include:
A novel model based meta- meta-learning method (DyAdl for dynamic forecasting in large hetero- geneous domains.
boodel-based
An encoder capable of extracting the time-invariant hidden features of dynamical system using time-shift invariant model structure and weak supervision.
A new adaptive padding layer (AdaPad), designed for adapting to boundary conditions.
Theoretical guarantees for DyAd on the generalization error of task inference in the source domain as well domain adaptation the target domain.
Improved generalization performance on heterogeneous domains such as fluid flow and sea temperature forecasting, even to new tasks outside the training distribution.
2 METHODS
2.1 META-LEA IN DYNAMICS FORECASTING
Let x Rd d-dimensional state of a dynamical system governed by parameters th. The problem of dynamics forecasting is that given sequence of past states (X1-141...., x,), we want to learn map f such that:
(X.-I-1.... ..... x1) .x.+1....xx+)))
... X(+h).
Here / is the length of the input series, and h the forecasting horizon in the output.
Existing approaches for dynamics forecasting only predict future data for a specific system as single task. Here task refers to forecasting for specific system with given set of parameters. The resulting models often generalize poorly to different system dynamics. Thus new model must be trained predict for each specific system.
To perform meta-learning, we identify each forecasting task by some parameters e C. th, such as constants of motion, external forces, and boundary conditions. We learn multiple tasks simultaneously and infer the task from data. Here we use e for subset of system parameters v because we usually do not have the full knowledge of the system dynamics. the turbulent flow example, the state x; is the velocity field at time I. Parameters can represent Reynolds number, average vorticity, average magnitude, or a vector all three.
Formally, let be the data distribution over X' y representing the function f: X + y where X = Rd4xl and y = dx Our main assumption is that the domain X' can be partitioned into separate tasks X' = Ue where X. is the domain for task and C is the set of all tasks. The data in the same task share the same set of parameters. Let Pe be the conditional distribution over X. x y for task c.
During training, the model is presented with data drawn from a subset tasks {(z, y) (æ, y) ~ Pes C C}. Our goal is learn the function f: X over the whole domain X' which can thus
2

---

Under review as a conference paper at ICLR 2022
Forecaster Network
Forecasts
Wi-lirkk
AdaPad
Conv2D
AdaPad
Conv2D13x3)
AdaiN
Input
X N
Encoder Network
(xdxa Pooling
ComdD (Bx3x3)
Poaling Gonda (iccdd)
Pooling Dunad
25
Figure 2: Overview of DyAd applied to two inputs of fluid turbulence, one with small external forcing and one with larger external forces. The encoder infers the time-shift invariant characteristic variable z which used to adapt the forecaster network.
generalize across all tasks C. To do so, need learn the map g: X C taking E Re to in order to infer the task with minimal supervision.
2.2 DYAD: DYNAMIC ADAPTATION NETWORK
We propose model- -base meta-learning approach for dynamics forecasting. Given multiple fore- casting tasks, we propose to learn the function f in two stages, that is, by first inferring the task c from the input z, and then adapting to a specialized forecaster fe: X - y for each task.
An alternative is to use single deep neural network to directly model f in one step over the whole domain. But this requires the training the data distribution is highly heterogeneous or the training set is not sampled i.i.d. from the whole domain, then a single model may struggle with generalization.
have good and uniform coverage of the different tasks.
We hypothesize that by partitioning the domain into different tasks, the model would learn to pick up task-specific features without requiring uniform coverage of the training data. Furthermore, by separating task inference and forecasting into two stages, we allow the forecaster to rapidly adapt to new tasks that never appeared in the training set.
As shown in Fig. 2 our model consists of two parts: encoder and forecaster f. We introduce Ze as imme-invrriant hidden feature for task e. We assume that e depends linearly on the hidden feature for simplicity and easy interpretation. We design the encoder to infer the hidden feature &e given the input
æ. We then use & to adapt the forecaster f to the specific task, i.c., model = (z as y = (xx ze)- As the system dynamics encoded in the input sequence x, we feed the same input sequence x to a forecaster and generate predictions y = fc()).
2.3 ENCODER NETWORK
The encoder maps the input x to the hidden features 2. that are time-invariant. To enforce this inductive bias, we encode
both in the architecture and in the training objective.
Time-Invariant Encoder. The encoder implemented using Conv 3D layers, each followed by BatchNorm, Lea RyR,,,, and max-pooling. Note that theoretically, max-pooling is not perfectly shift invariant since 2x2x2 max-pooling equivariant to shifts of size and only approximately invariant to shifts of size 1. But standard convolutional architectures often include max-pooling layers to boost performance. We convolve across spatial and temporal dimensions.
After that, we use global mean-pooling layer and a fully connected layer to estimate the hidden feature &g. The task parameter depends linearly on the hidden feature. We use a fully connected layer equivariant shift (up to boundary
to compute the parameter estimate č. Since convolutions frames) and mean pooling is invariant to shift, the encoder is shift-invariant. In practice, shifting the time sequence frame forward will add one new frame the beginning and drop one frame the
3

---

Under review as a conference paper at ICLR 2022
end. This creates some change in output value of the encoder. Thus, practically, the encoder is only approximately inft-invariant.
Encoder Training. The encoder network g is trained first. To combat the loss of shift invariance from the change from the boundary frames, we train the encoder using a time-invarian loss. Given two training samples (xx(", ()) and (z(0), ())) and their task parameters c, we have loss
Lene
Σ 112-ccl cl2 anc
Σ 1 -=()==(0)' 1.j.c
Σ 1| mll
where 2(() =x(x(() and 2(ا) g( and e(0) = w + b is an affine transformation of Ze.
The first term ||è - e|I2 uses weak supervision of the task parameters whenever they are available. Such weak supervision helps guide the learning of hidden feature for each task. While not all parameters of the dynamical system are known, we can compute approximate values in the datum d(i) based our domain knowledge. For example, instead of the Reynolds number of the fluid flow, we can use the average velocity as a surrogate for task parameters.
The second term || 2
1|2 is the time-shift invariance loss, which penalizes the changes in latent variables between samples from different time steps. Since the time-shift invariance of convolution is only approximate, this loss term drives the time-shift error even lower. The third term |||=[0 m|/
(m is a positive value) prevents the encoder from generating small 2 due to time-shift invariance loss. It also helps the encoder to learn more interesting 2, even in the absence of weak supervision.
Hidden Features. The encoder learns time-invariant hidden features. These hidden features resemble the time-ine invariant dimensionless parameters (Kunes 2012) in physical modeling, such as Reynolds number fluid mechanics. The hidden features may also be viewed as partial disentanglement of the system state. As suggested by Locatello et al. (2019); Nie et al. (2020), our disentanglement method is guided by inductive bias and training objectives. Unlike complete disentanglement as in e.g. Massague et al. (2020), in which the latent representation factored into iime-invariant and time-varying components, we focus only on time-shift-ing inniit-eivvriance. Nonetheless, the hidden features can control the forecaster which useful for generalization.
2.4 FORECASTER NETWORK
The forecaster incorporates the hidden feature z. from the encoder and adapts to the specific forecasting task fc = =c). In what follows, we use z for Ze. We use two specialized layers, adaptive instance normalization (AdalN) and adaptive padding (AdaPad).
AdaIN has been used in neural style transfer (Karras et al. 2019 Huang & Belongie 2017) to control generative networks. Here, AdaIN may adapt for specific coefficients and external forces. We also introduce a new layer, AdaPad(z, 2), which is designed for encoding the boundary conditions of dynamical systems. In principle, the backbone of the forecaster network can be any sequence prediction model. We use a design that is similar to ResNet for spatiotemporal sequences.
We employ AdaIN to adapt the forecaster network. Denote the channels of input æ by x; and let p(x;)) and (xx) be the mean and standard deviation of channel i. For each AdaIN layer, particular style is computed s = (Mi o4); = Az + b, where the linear map and bias b are learned weights. Adaptive instance normalization is then defined as y: o, -i-p(z)) + Ns. essence, the channels are renormalized the style
AdaIN.
Figure 3: Illustration of the AdaPad operation.
a(z;)
For dynamics forecasting, the hidden feature encodes data analogous to the various coefficients of a differential equation and external forces on the system. In numerical simulation a differential equation these coefficients enter scalings of different terms in the equation and the externll forces are added to the combined force equation (J.ChButcher (196). Thus in our context AdaIN, which scales channels and adds global vector, is well-suited to injecting this information.
4

---

Under review as a conference paper at ICLR 2022
AdaPad. To complement AdaIN, introduce AdaPad, which encodes the boundary conditions of each specific dynamical system. Generally when predicting dynamical systems, error is introduced along the boundaries since it is unknown how the dynamics interact with the boundary of the domain, and there may be unknown inflows or outflows. In our method, the inferred hidden feature: z may contain the boundary information. AdaPad uses the hidden features to compute the boundary conditions via a linear layer. Then it applies the boundary conditions as padding immediately outside the spatial domain in each layer, as shown in Fig.
Forecaster Training. The forecaster network is trained separately after the encoder. The kernels of the convolutions and the mappings of the AdaIN and AdaPad layers are all trained simultaneously as the forecaster network is trained. Denote the true state as y and the predicted state as D. we compute the loss per time step |/ - yyl for each example. We accumulate the loss over different time steps and generate multi-step forecasts in autoregressive fashion.
THEORETICAL ANALYSIS
The high-level idea of our method is to learn good representation of the dynamics that generalizes well across a heterogeneous domain, and then adapt this representation to make predictions on new tasks. Our model achieves this by learning on multiple tasks simultaneously and then adapting to new tasks with domain transfer. We prove that learning the tasks simultaneously as opposed to independently results in better generalization. We also find theoretical support for training the encoder and forecaster separately. See Appendix for for a longer treatment with proofs.
Suppose we have K tasks {c K space C. Here are the parameters of the task, which can be inferred by the encoder. For each task Ck2 we have a collection of data fe., of size n, sampled from M4:, shorthand for Pex.
C, each of which is sampled from a continuous parameter
Multi-task Learning Error. Our model performs multi-tase task representation learning (Maurer et al. 2016) with joint risk = (1/K) En Ck. which is the average risk of each task c₁ defined separately. Denote the corresponding empirical risks é and éx. We can bound the true risk using the empirical risk and Rademacher complexity R(F) the hypothesis class F. The following theorem restates the main result in Ando et al. (2005) with simplified notations.
Theorem 3.1. (Ando et al. (2005) Assume the loss is bounded 1 & 1/2. Given n samples each from K different forecasting tasks f1."". Pkk, then with probability least the following inequality holds for each f FF in the hypothesis class:
<()) S {() + +R(F) + Vlgg (2/k)
The following inequality compares the performance for multi task learning to learning individual tasks. Let (F) be the Rademacher complexity for F over Pk- Lemma 3.2. The Rademacher complexity for multi-task learning is bounded as
K
R(F) < (1/K)
(F.
We can now compare Theorem 3.1 to the bound obtained by considering each task individually.
Proposition 3.3. Assume the loss is bounded 1 < 1/2, then the generalization bound given by considering each task individually
K
<(f) +(E(()+((U)
Rx) / Vlo (1/6)/(2nK).
(3)
The upper bound in Theorem 3.1 is strictly tighter than that of Proposition 3.3 as the first terms E(f)
are equal, the second term is smaller by Lemma 3.2 and the third is smaller since 1/ 1/V2nK' < 1/vnn. This helps explain why our multitask learning forecaster has better generalization than learning each task independently. The shared data tightens the generalization bound. Ultimately, though tighter upper bound suggests lower error, it does not strictly imply it. We further verify this theory in our experiments by comparison to baselines which learn each task independently.
5

---

Under review as a conference paper at ICLR 2022
Encoder versus Forecaster Error. Error from DyAd may result from either the encoder 9 the forecaster fo. Using (Redko et al. 2017p. we can bound the generalization error in terms of: (1) the empirical error of the encoder, (2) the empirical forecaster error on the source domains, and (3) the Wasserstein distance W between the source and target domains. We use Wasserstein distance since different tasks may have disjoint support and thus infinite KI divergence, however, samples from lasks c, c' close in C may be close in Wi (He. pe) ) (see Appendix Fig. [T]. Our two-stage training scheme is motivated by this decomposition for achieving generalization.
Our hypothesis space has the form {ar + fa (x())g(()) (o()) where & and 0 are the weights of the encoder and forecaster respectively. Let exx be the error over the entire domain X. that is, for all c. Let Cenc (gx) = (Ci(g(r), [g+: - C) be the task encoder hypothesis space. Denote the empirical risk of go by (9o)-
be the encoder error where g: X' + C is the ground truth. Let
Proposition 3.4. Assume c +++ fol:,c) is Lipschitz continuous with Lipschitz constant yy uniformly in 0 and 1 S 1/2. Let d mingerleelf)+1//5 + E Eax (f)). For large n and probability 2 1 1 8,
K
Lea (fo((, (k)) + E [wi Wi
K
cx(fo(;goo))) S Ranc (gx) +
1 K
Σ
Mes K
Σ
Dew
+ the
k=1
+27 + (7 (oo (2tp:///((22))//10((0// + log(1/)) (V1/n+ v1/(nk)).
Although this result does not settle either the question of end-to-end versus pre-training or encoder- forecaster versus monolithic model, it quantifies the trade-off's these choices depend on. We empiri- cally consider both questions in the experiments section.
4 RELATED WORK
Learning Dynamical Systems. Deep learning models are gaining popularity for learning dynamical
systems (Shi et al. [017]] Chen et al.
2018 Kolter& & Manek 2019 Azencot et al.) 2020a] Xie et al.,
2018: Tompson et al. [0117] Pfaff et al.
2021). An emerging topic is physics-iffomed deep learning
2020b de Bezenac et al.) 2018 Wang et al.) 2020b Ayed et al. 2019b al Li et al. [202]] Donà et al.
(Raissi al.] 2017]] Lutter al. 2018 Azencot al.
(2021) which integrates inductive biases from physical systems to improve learning. For example, Lutter et al. (2018) encoded Buler-Lagrange equation into the deep neural nets but focus on learning low-dimensional trajectories. Morton et al. (2018): ; Azencot et al. (2020b) incorporated Koopman theory into the architecture. |Mziarr Raissi (2019) used deep neural networks to solve PDEs with physical laws enforced in the loss functions. Wang et al. (2000a) proposed a hybrid approach by marrying two well-established turbulent flow simulation techniques with deep learning to produce better prediction of turbulence. However, these approaches deal with a specific system dynamics instead of the meta-learning problem in this work.
Multi-task learning and Meta-learning Multi Multi-task learning (Vandenhende et al. 2021 focuses on learning shared representations from multiple related tasks. Architecture- based MT methods can be categorized into mncoder-focused (Liu et al. 2019) and decoder-focused (Xu al. 2018). There also optimization-base based MTL methods, such as task balancing methods (Kendall et al. 2018). But MTL assumes tasks are known a priori instead of inferring the task from data. On the other hand, the aim of meta-learning (Thrun & Pratt 1998) is to leverage the shared representation to fast adapt to unseen tasks. Based on how the meta-level knowledge is extracted and used, meta-learning methods are classified into model-based (Munkhdalai & Yu 2017; Alet et al., 2018 Oreshkin et al. 2019
Seo al. 2020; Zhou al.) 2021 Li et al. 2017). metric- based Vinyals et al.) 2016 Snell et al. 2017) and gradient-based (Finn et al. 2017 Andrychowoi et al. 2016 Rusu et al. 2019 Grant et al. 2018 Yao et al. al. 2019). Most meta-learning approaches are not designed for forecasting with a few exceptions. Oreshkin et al. (2019) designed residual architecture for time series forecasting with meta-learning parallel. Alet et al. (2018) proposed a modular meta-learning approach for continuous control. But forecasting physical dynamics poses unique challenges to meta-learning we seek ways to encode physical knowledge into our model.
Style Transfer. Our approach is inspired by neural style transfer techniques. In style- transfer generative network is controlled by an external style vector though adaptive instance normalization between convolutional layers. Our hidden representation bears affinity with the "style" vector in

---

Under review as a conference paper at ICLR 2022
style transfer techniques. Rather than aesthetic style in images, our hidden representation encodes time-invariant features. Style transfer initially appear in contoooooeca rendering (Kyprianidis et al. 22012). Recently, neural style transfer (Jing al. 2019) has been applied to image synthesis (Gatys et al. 20166] Liu et al. 2021), videos generation (Ruder et al. 2016). and language translation (Prabhumoye et al. 2018). For dynamical systems, (Sato al. (2018) adapts texture synthesis to
transfer the style of turbulence for animation. Kim & Lee (2019) studies unsupervised generative modeling of turbulent flows but for super-resolution reconstruction rather than forecasting.
Video Prediction. Our work is also related to video prediction. Conditioning on the historic observed frames, video prediction models are trained to predict future frames, e.g.. (Mathieu et al. 2015 Finn
et al.
2016 Xue et al. 2016,] Villegas et al.) [2017; Oprea et al., 2020; Finn et al. [016]] Wang et al. 2021 Le Guen & Thome 2020] Massague et al.). There is also conditional video prediction
2017
(Oh et al., 2015) which achieves controlled synthesis. Many of these models are trained on natural
videos from unknown physical processes. Our work is substantially different because we do not
attempt to predict object or camera motions. However, our method can be potentially combined with video prediction models to improve generalization.
EXPERIMENTS
We compare our model with a series of baselines on the multi-step forecasting with different dynamics. We consider two lesting scenarios: (1) dynamics with different initial conditions (test-future) and (2) dynamics with different parameters such as external force (test-domain). The first scenario evaluates the models' ability to extrapolate into the future for the same task. The second scenario estimates the capability of the models to generalize across different tasks.
We experiment on three datasets: synthetic turbulent flows, real-world sea surface temperature and ocean currents data. These are difficult to forecast using numerical methods due to unknown external forces and complex dynamics not fully captured by simplified mathematical models. For the weak supervision c, we use the mean vorticity for turbulence and ocean currents, and mean temperature for sea surface temperature. We defer the details of the datasets and experiments to Appendix A.2
Table I: Prediction RMSE on the turbulent flow and sea surface temperature datasets. Prediction RMSE and ESE (energy spectrum errors) on the future and domain test sets of ocean currents dataset.
Turbulent Flows
Sea Temperature
Ocean Currents
Model
future
domain
future domain
future
domain
ResNet
0.94
0.65
0.73
0.71=0.16
9.44
1009
9.65 0.90=10.16
ResNet-c
0.88
0.64
0.70=
0.71=0uu6
9.71
10,81 +0.03
9.15=0001
0.73
U-Net
0.92=0.02
0.68
0.57
0.63=0.05
7.64
00.83
7.61
14 0.86:0003
Unet-c
0.86=0007
0.68
0.47=0002
0.45=0.06
7.26=10.11
10.94+10.2
7.51=0.03
0.87=4004
PredRNN
0.75=+0.02
0.75=
0.67=0.12
0.99=0.07
8.49
11.27=0002
8.99=0.03
1.69910.01
VarSepNet
0.67
0.63
0.63
0.49
9.36110.2
10.63
7.10=
0.58
Mod-attn
0.63
0.92.10.33
0.89
0.98
8.08 10.07 10.76.10.11
8.31 10.19 0.88
Mod-wt
0.58
0.60
0.65
0.64
10.1 1.19.10.2
0.11 0.82:10.19
MctaNet
0.76+0.13
0.76#0.08
0.84=0.16
0.82=0.09
10.9
11.15
11.2=0016
1.08=0221
MAME
0.63=+0.0
0.68=+002
0,90=+011
0.67=0.04
10. +0.21 00855+0.06
10.9=0.79
0.99:+0.14
DyAd+ResNet.
0.42:10.01
0.51
0.42:10.33
0.44
7.28
0.58
7.04 0.04 0.54 10.03
DyAd+Unet
0.58=00011
0.59+0001
0.35++0.03
0.42:10.05
7.38=+0.01
10.70+0..44
7.46=0002
0.70=
Baselines. We include several SoTA baselines from meta-learning, as well as common methods for dynamics forecasting.
ResNet (He al. 2016): widely adopted video prediction model (Wang et al. 2020b).
U-net (Ronneberger et al. 2015): Originally developed for biomedical image segmentation.
ResNet/Unet Above ResNet and Unet with an additional final layer that generates task
parameter e and trained with supersuuerviiioon and forecasting loss altogether.
PredRNN (Wang et al.) 2017): A state-of-the-ar RNN-based spatiotemporal forecasting model.
VarSepNet (Donà et al. 202T) A convolutional dynamics forecasting model based on spatiotem- poral disentanglement.
7

---

Under review as a conference paper at ICLR 2022
Target
Unet-c
Weight-attn
DyAd
Target
Unet-c
Weight-attn
DyAd
t= 10
t=
Figure 4: Target and predictions by Unet-c, Modu modular-wtt lar-wt and DyAd at time 5, 10 for turbulent flows with buoyancy factors (left) and 21 (right) respectively, DyAd can easily generate predictions for various flows while baselines have trouble understanding and disentangling buoyancy factors.
Mod-attn: A modular meta-learning method which combines modules to generalize to new tasks (Alet al. 2018) using attention.
Mod-wt: A modular meta-lcarning variant which uses attention weights to combine the parameters of the convolutional kernels for new tasks.
MetaNet (Munkhdalai & Yu 2017): : A model-based meta-learning method which requires few
labels from test tasks as support set to adapt.
MAML (Finn al. 2017) : A popular optimization-base
meta-learning approach. We replaced the classifier in the original model with a ResNet for regression.
Note that both ResNet-c and Unet-C have access to task parameters c, but cannot adapt. Mod-attn and Mod-wt also use a convolutional encoder to generate attention weights. MetaNet requires samples from test tasks as support set and MAML needs adaptation retraining on test tasks, while other models do not need any information from the test domains. VarSepNet employs sepa- ration of variables through different loss terms but it is difficult to find the optimal hyperparameters for these terms. To demonstrate the generality of DyAd, we experimented with both ResNet and U-net as our forecaster. See details about baselines in Appendix A.2
Experiment Setup. For all datasets, we use a sliding window approach to generate samples of sequences. For test-future, we train and test on the same tasks but different time steps. For test- domain, we train and test on different tasks with an 80-20 split. All models are trained to make next step prediction given the previous steps input. We forecast in an autoregressive manner to generate multi-step ahead predictions. All results are averaged over 3 runs with random initialization.
Apart from the root mean square error (RMSE), also report the energy spectrum error (ESE) for ocean current prediction which quantifies the physical consistency. ESE indicates whether the predictions preserve the correct statistical distribution and obey the energy conservation law, which is a critical metric for physical consistency. See details about energy spectrum in Appendix A.3
5.1 EXPERIMENT RESULTS
Prediction Performance. Table shows shows the RMSE of multi-step predictions on Turbulent Flows (20 steps), Sea Surface Temperature (10 steps), and Ocean Currents (10 step) two testing scenarios. We observe that DyAd makes the most accurate predictions in both scenarios across all datasets. Comparing ResNet/Unet:
with DyAd, we observe the clear advantage of task inference with separate training. VarSepNet achieves competitive performances through spatiotemporal disentanglement but cannot adapt to future tasks. Table [ Ocean Currents. DyAd not only has small RMSE but also obtains the smallest ESE, suggesting captures the statistical distribution of ocean currents well.
Ocean Currents (second best) also reports ESE on
Figure 4 shows the target and the predicted velocity norm (Vu² u2) by Unet-c, Modular-ww and D DyAd at time step 5, 10 for Turbulent Flows with buoyancy factors 9 and 21 respectively. We can see that DyAd can generate realistic flows with the corresponding characteristics while the baselines have trouble understanding and disentangling the buoyancy factor.
8

---

Under review as a conference paper at ICLR 2022
Target
ResNet
U-net
PredRNN
DyAd
Model
future
domain
DyAd()urs)
0.42
0.51
No_enc
0.63
0.60++0.02
No ad__dPPad 0.47=0001
0.54
t= 10
Wrong_ens 0.66+0002
enc
0.62++0.03
End2End
0.45
0.54
Table Ablation study: prediction RMSE of DyAd and its variations with different compo- nents removed from DyAd.
Figure 5: DyAd, ResNet, U-net, PRedRNN veloc- ity norm (vu² u²) predictions on an ocean current sample in the future test set.
Figure ( shows DyAd, ResNet, U-net, FredRNN predictions on an ocean current sample in the future set, and we see the shape of predictions by DyAd closest the target. These results demonstrate that DyAd not only forecasts well but also accurately captures the physical characteristics of the system. We also visualize the energy spectrum of target and predictions by ResNet, U-net and DyÅd on two test sets of ocean currents in Figure 10 with DyAd being the closest to the target.
Ablation Study. We performed an ablation study of DyAd on the turbulence dataset to understand the contri- bution of each component, shown in Table 2 We first remove the encoder from Dyåd while keeping the same forecaster network (No_enc). The resulting model de- grades but still outperforms ResNet. This demonstrates the effectiveness of AdaIN IN and AdaP ad for forecasting. We also tested DyAd with AdaIN only ANo_AdaPall and the performance without AdaPad was slightly worse.
c=
15
C 25
[=10 t=5
Another notable feature of our model is the ability infer tasks with weakly supervised signals It is important have c that is related the task domain. As an ablative study, we fed the encoder in DyAd with a random c, leading Wrong_ens
Figure 6: Outputs from DyAd while vary encoder input but keep the forecaster input fixed. From left to right, the encoder is fed with flow with different buoyancy factor 5, 15, 25. the forecaster net- work input has fixed buoyancy e 15.
We can see that having the wrong supervision may hurt the forecasting performance. also trained the encoder and the forecaster in DyAd altogether (End2End End) but observed worse performance. This validates our hypothesis about the significance of domain partitioning and separate training strategy. We also tested 5 different altermatives to AdaIN for injecting the hidden feature 2, into the forecaster, and reported the results in Table 5 of Appendix A.4
Forecast. DyAd infers the hidden features from data, which allows direct control of the latent space in the forecaster. We tried varying the encoder input while keeping the forecaster input fixed. Figure 6 shows the forecasts from DyAd when the encoder is fed with flows having different buoyancy factors c 5, 15, 25. As expected, with higher buoyancy factors, the predictions from the forecaster become more turbulent. This demonstrates that the encoder can successfully disentangle the latent representation of difference tasks, and control the predictions of the forecaster.
Controllable
CONCLUSION
We propose a model-based meta-learning method, DyAd to forecast physical dynamics. DyAd uses an encoder to infer the parameters of the task and a prediction network to adapt and forecast giving the inferred task. Our model can also leverage any weak supervision signals that can help distinguish different tasks, allowing the incorporation additional domain knowledge. On challenging turbulent flow prediction and real-world ocean temperature and currents forecasting tasks, observe superior performance of our model across heterogeneous dynamics. Future work would consider non-grid data such flows graph sphere.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Our model allows for generalizable predictions of dynamical systems. One example domain we apply it to is turbulent flow predictions. While there are many peaceful applications of turbulent flow prediction, our method could also potentially be helpful for researching explosive weapons or for designing aircraft or missiles.
REPRODUCIBILITY STATEMENT
The implementation code of DyAd and baselines are included the supplementary material. The readme file includes the instructions of downloading PhiFlow and generating the turbulent flow dataset. The ocean currents and sea temperature can be downloaded manually from the link provided the Appendix A.2 The full proofs of the theorems in Section can be found in Appendix
REFERENCES
Ferran Alet, Tomas Lozano-Perez, and L. Kaelbling. Modular meta-learning. ArXiv, abss/880..101e 2018.
Rie Kubota Ando, Tong Zhang, and Peter Bartlett. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(11), 2005.
Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, M. W. Hoffman, D. Pfau, T. Schaul, and N. D. Freitas Learning learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016.
I. Ayed, Emmanuel de Bézenac, A. Pajot, J. Brajard, and P. Gallinari. Learning dynamical systems from partial observations. ArXiv, abs/1902, 136, 2019a.
Ibrahim Ayed, Emmanuel De Bézenac, Arthur Pajot, and Patrick Gallinari. Learning partially observed PDE dynamics with neural networks, 2019b. URL UtL/httpss//ppeneeeieww.. net/
Omri Azencot, N. Erichson, V. Lin, and Michael W. Mahoney. Forecasting sequential data using consistent koopman autoencoders. In International Conference on Machine Learning, 2020a.
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent koopman autoencoders. In International Conference Machine Learning, pp. 475-485. PMLR, 2020b.
Jonathan Baxter. Theoretical models of learning learn. In Learning to learn, pp. 71-94. Springer, 1998.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6572-6583, 2018.
Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: Incorporating prior scientific knowledge. In International Conference on Learning Representations, 2018. https:
n forum? d=By4 Hs fWAZ.
Jérémie Donå, Jean- Yves Franceschi, sylvain lamprier, and patrick gallinari. (PDE)-driven spa- tiotemporal disentanglement. In International Conference on Learning Representations, 2021. URL
Chelsea Finn, Ian Goodfellow, and Sergey Leine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Chelsea Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
10

---

Under review as a conference paper at ICLR 2022
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2413, 2016.
E. Grant, Chelsea Finn, S. Levine, Trevor Darrell, and T. Griffiths. Recasting gradient- baasd meta- learning hierarchical bayes. ArXiv Preprint, abs/180 10//8001.000 301300, 2018.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Proceedings of the 32nd Inter- national Conference on Neural Information Processing Systems, pp. 5307-5316, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510, 2017.
B.CcBr Applied Numerical Mathematics, volume 20. Elsevier B. V., 1996.
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: review. IEEE transactions on visualization and computer graphics, 26(1 (6(11):33653336) 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1401-4410, 2019.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1182-7491, 2018.
Junhyuk Kim and Changhoon Lee. Deep unsupervised learning of turbulence for inflow generation at various reynolds numbers. arXiv: 10515, 2019.
J. Zico Kolter and Gaurav Manek. Learning stable deep dynamics models. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pp. 11128-11136, 2019.
Josef Kunes. Dimensionless physical quantities in science and engineering. Elsevier, 2012.
Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, and Tobias Isenberg. State of the" art": taxonomy of artistic stylization techniques for images and video. IEEE transactions on visualization and computer graphics, 19(5))866 08566888,111 2012.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Computer Vision and Pattern Recognition (CVPR). 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. arXiv preprint arXiv: 1707,09833 2017.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. International Conference Learning Representations, 2021.
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2019.
Xiao Liu, Spyridon Thermos, Alison Q O'Neil, and Sotirios A. Tsaftaris. Semi-supervised meta- learning with disentanglement for domain- 2021.
domain-generalised generalised medical image segmentation. In MICCAI,
11

---

Under review as a conference paper at ICLR 2022
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In international conference machine learning, pp. 4114-4124 PMLR, 2019.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations, 2018.
Gurvan Madec et al. NEMO ocean engine, 2015. Technical Note. Institut Pierre-Simon Laplace (IPSL), France, pdf
Armand Comas Massague, Chi Zhang, Zlatan Feric, Octavia I. Camps, and Rose Yu. Learning disentangled representations video with missing data. arXiv preprint arXiv: 2006. 1339/.
Armand Comas Massague, Chi Zhang. Zlatan Feric. Octavia Camps, and Rose Yu. Learning disentangled representations of video with missing data. arXiv preprint arXiv:2006. 13391, 2020.
Michael Mathicu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint 1rXiv:151..05444 2015.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredee The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1-32, 2016.
George E Karniadakis Maziar Raissi, Paris Perdikaris. Physics-informet neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differen- tial equations. Journal of Computational Physics, 378:686-707, 2019.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
Jeremy Morton, Antony Jameson, Mykel J. Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control unsteady fluid flows. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. Proceedings of machine learning research, 70:2554-2563, 2017.
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debhath, Patney, Ankit B. Patel, and Anima Anandkumar. Semi-supervised stylegan for disentanglement learning. In International Conference on Machine Learning. 2020.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 2, pp. 2863,2871, 2015.
Sergiu Oprea, P. Marzinez- A. Carcia-Garcia, John Alejandro Castro- Castro-Vargas, S. Orts- Escolano. J. Garcia- Rodrigue, and Antonis A. Argyros. A review on deep learning techniques for video prediction. ArXiv, abs/2004.052144 2020.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2019.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzaler and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021.
URL https:: wwt.rriwww.eeeeeeeeeeeeeeeeeeeee
wwt.:et///oooooo forum ro= XP
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 866-876, 2018.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10560 2017.
12

---

Under review as a conference paper at ICLR 2022
levgen Redko, Amaury Habrard, and Mare Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 737-753. Springer, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image compuling and computer- assisted intervention, pp. 234-241. Springer, 2015.
Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for videos. In German conference on pattern recognition, pp. 26-36. Springer, 2016.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Confer- ence on Learning Representations, 2019. URL BJgkIhAcK?
net forum?id
Syuhei Sato, Yoshinori Dobashi, Theodore Kim, and Tomoyuki Nishita. Example-bard based turbulence style transfer. ACM Transactions Graphics (TOG), 37(4):1-9, 2018.
Sungyong Seo, Chuizheng Meng, Sirisha Rambhatla, and Liu. Physics-aves aware spatiotemporal modules with auxiliary tasks for meta-learning. arXiv preprint arXiv: 2006.08831 2020.
Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, D. Yeung, W. Wong, and Wang chun Woo. Deep learning for precipitation nowcasting: A benchmark and new model. In Advances in neural information processing systems, 2017.
J. Snell, Kevin Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pp. 3-17. Springer, 1998.
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating Eulerian fluid simulation with convolutional networks. In 177 Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 3424 3433,, 2017.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions pattern analysis and machine intelligence, PP, 2021.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference on Learning Representations (ICLR), 2017.
Oriol Vinyals, Charles Blundell, Lillicrap, Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1457-1466, 2020a.
Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXi::2002.0306/ 2020b.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip Yu. PredRNN: Recurrent neural networks for predictive learning using spatiotemporal LSTMs. In Advances in Neural Information Processing Systems, pp. 879-888, 2017.
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S Yu, and Mingsheng Long. PredRNN: A recurrent neural network for spatiotemporal predictive learning, 2021.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow, ACM Transactions on Graphics (TOG), 37(4))95, 2018
13

---

Under review as a conference paper at ICLR 2022
Dan Xu, Wanli Ouyang, Xiaogang Wang, and N. Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 675 675-684, 2018.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances in neural information processing systems (NeurIPS), pp. 91-99. 2016.
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In International Conference Machine Learning, pp. 7045-7054, PMLR, 2019.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7343-7335555 2018.
Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparametteizzation In International Conference on Learning Representations, 2021. URL /
David Zoltowski, Jonathan Pillow, and Scott Linderman. general recurrent state space framework for modeling neural dynamics during decision-makinn In International Conference Machine Learning, pp. 1680-11691. PMLR, 2020.
14

---

Under review as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
A.1 MODEL DESIGN
The prediction network 9 f(x, =) is composed of 8 blocks. Each block operates hidden state 1,(0) of shape B H x W x Cin and yields new hidden state (1(+1) of the shape B H x W x Cout- The first input is ho a and the final output is computed from the final hidden state as = (o We define each block as
( (Conv2((AdaPad(/)) ,)))
( (Conv2D(AdaPad( (6) ,2)) (
h+41 AdaIN(bO),. 2)
as illustrated in Figure 3
X
Com 3x3xx
LastyBell
64694c128
ConvaD 3x3x3
Lat
Hadiaa
30006025
Conr3D
Leto
Hedue
10e200012
Conv3D 3x3333
Bascaborm
Global McanPool
To forecisster
Figure 7: Detail of the DyAd encoder. The conv3D layers are shift equivariant and global mean pooling is shift invariant. The network is approximately invariant to spatial and temporal shifts.
Adapad
ConyzD Exc
AdaPad
Cony2D Exd3
AdoiN
Figure 8: Detail of one block of the forecaster network.
15

---

Under review as a conference paper at ICLR 2022
A.2 EXPERIMENT DETAILS
A.2.1 DATASETS
Turbulent Flow with Varying Buoyancy. We generate synthetic dataset turbulent flows with numerical simulator, PhiFlow It contains 64x64 velocity fields of turbulent flows in which we vary the buoyant force acting on fluid from 1 to 25. Each buoyant force corresponds task and there are 25 tasks in total. We use the mean vorticity of each task as partial supervision as we can directly calculate from the data. Vorticity can characterize formation and circular motion of turbulent flows.
forecasting
Sea Surface Temperature. We evaluate on real-world sea surface temperature data generated by the NEMO ocean engine (Madec et al. 2015 We select area from Pacific ocean range from 01/01/2018 to 12/31/2020. The corresponding latitude and longitude are (-150~- 120, -20~-50), This area is then divided into 25 x4 subregions, each is task since the mean temperature varies lot along longitude and latitude. For the encoder training, we use season as an additional supervision signal besides the mean temperature of each subregion. In other words, the encoder should be able infer the mean temperature of the subregion as well as to classify four seasons given the temperature series.
Ocean Currents. We also experiment with the velocity fields of ocean currents from the same region and use the same task division as the sea surface temperature data set. Similar the turbulent flow data set, we use the mean vorticity of each subregion as the weak-supervision signal.
A.2.2 BASELINES.
For fair comparison, of parameters. Hyperparameters including learning rate, input length and the number of steps of accumulated loss for training are tuned on validation sets. Modular-ate attn has convolu- tional encoder f that takes the same input x as each module M to generate attention weights, exp(f(=)()) (XE) Mr Mz)... Modular-we also has the same encoder but generate weights for combining the convolution parameters of all modules. We use additional samples of up 20% of the test set from test tasks. Met aNet uses these as a support set. MAML is retrained on these samples for 10 epoch for adaptation.
these models to have equal capacity DyAd in terms of number
EE
77
A.2.3 HYPERPARAMETES TUNING.
We tuned learning rate (e-53~1e-5), batch size (16~64), the number of accumulated errors for backpropogation (2~5), and hidden size (64~512) Modular Networks and Meta-Nets. We fixed the number of historic input frames as 20. When we trained the encoder on turbulent flows and sea surface temperature, we used and 3 = For ocean currents, we used a 0.2 and B = 0.2. We performed all our experiments on 4 V100 GPUs.
A.2.4 MODEL CAPACITY.
Table displays the number of parameters of each tuned model on the turbulent flow dataset.
Table 3: The number of parameters of the best model of each architecture
ResNet
U-net
PredRNN
VarSepNetWod-atta Mod-wt
MetaNets
MAML
DyAd
20.32
9.69
27.83
9.85
13.19
13.19
9.63
20.32
15.60
https: / htt.bbbccwmmmuunnppp/hi/iioooo
The data available at httpps//reesurrrss.mrrriiee cswiview-dete
sutpott onncom
16

---

Under review as a conference paper at ICLR 2022
A.3 TURBULENCE KINETIC ENERGY SPECTRUM
The turbulence kinetic energy spectrum E(k) is related to the mean turbulence kinetic energy as
dkk) = ((u') + (22)
T Σ (u(1) - =))
(u))3 T 0=
where the the wavenumber and is the time step. Figure shows a theoretical turbulence kinetic energy spectrum plot. The spectrum can describe the transfer of energy from large scales of motion to the small scales and provides a representation of the dependence of energy on frequency. Thus, the Energy Spectrum Error indicate whether the predictions preserve the correct statistical distribution and ohey the energy conservation law. A trivial example that can illustrate why we need ESE is that if model simply outputs moving averages of input frames, the accumulated RMSE of predictions might not be high but the ESE would be really big because all the small or even medium eddies smoothed out.
Integral Scale
Inertial Subrange
log E(k)
Large Scale
3
Dissipation Scale
log k
Figure 9: Spectrum plot
A.4 ADDITIONAL RESULTS
DyAd
DyAd
18
ResNet
ResNet
Energy Spectrum
17:
U-net
Energy Spectrum
17
16
15
14
14
13
12
10
20
30
40
10
20
40
Wave Number
Wave Number
Figure 10: The energy spectrum of target and predictions by ResNet, U-net and DyAd on future test set (left) and domain test set (right) of ocean currents.
B THEORETICAL ANALYSIS
The high-level idea of our method is to learn a good representation of the underlying dynamics from multiple tasks, and then transfer this representation to a target task (domain adaptation).
17

---

Under review as a conference paper at ICLR 2022
Wasserstein distances between tasks
RMSE on five tasks in the domain-test set
0.70
0.65
17
22
RMSE
0.60
0.55
0.50
0.45
13
0.40
3
12
20
22
Buoyancy factor
Figure 11: Left: Pairwise RMSEs between the averaged samples of different tasks in the turbulent flow dataset. RMSE between the averaged samples is lower bound of Wasserstein distance between tasks. Right: DeAd+ResNet prediction RMSE breakdown on five tasks in the domain test set.
Table 4:
We test that our design results in time- inn-iinvriennee (low time-shift error). The scaled time-shift errors with and without time-shift invariant loss on the turbulent flow dataset are shown the table.
With time-shift loss
Without time-shift loss
Scaled Time Shift Error
6.67e-10
1.56e-05
Table 5:
We tested concatenating 2, to the input of the forecaster (concat input), using =. as the kernels of the forecaster (Kernel Gen), concatenating 2ę to the hidden states (concat hidden), adding 2e the hidden states(Sum), using 2e the biases in the convolutional layers (Bias). AdalN worked better than any alternative we tested.
RMSE
AdaIN
Concat Input
Kernel Gen
Concat Hidden
Sum
Bias
future
0.423
1.085 0.826
0.782 0.777
0.746 0.735
0.804 0.800
0.843
do
0.513
0.778
Definition (Forecasting task). Each forecasting lask Xx+1 = (xx:....) is to learn a conditional distribution μ over the system states p. p(xt-1 (t...... conditioned the sequence of previous states where p is probability measure.
In our setting, we have K tasks, each of which is sampled from continuous, finite space {ck} ~ C. Let μl be the corresponding conditional probability measure p(x,,... ..., X1 [Gx). For each task Ck. we have collection
series as realizations from the dynamical system X, = ((x,..... X1: cR) (+) } ( sampled from μa. The semicolon here represents the system behavior specific domain cp. Let
X U X be the union samples over all tasks.
In practice, often have some intuition of the variables that dictate the domain. Therefore, we have two possible scenarios for the role of c in dynamical systems:
1. fully distinguishes the task: the differences in Xr can be completely explained by the differences in CR:
2. partially distinguishes the task: more realistic scenario where only have partial knowledge of the domain. There exist latent variables that need to be inferred from raw data. Together
|c, 2"] describe the behavior of the system in domain.
18

---

Under review as a conference paper at ICLR 2022
We assume Scenario 1, which resembles the multi-task representation learning setting (Maurer et al. 2016) with joint true risk over all tasks e and individual task true risk e₁ defined respectively
cff) K
Σ (f))
cx(f) E 10 mpn
(xx)))))
x ))
and corresponding empirical risks
K
e(S, x)
K 1=1
where is loss function.
B. MULTI-TASE LEARNING ERROR
We want to bound the true loss using the empirical loss é and Rademacher complexity of the hypothesis class F. We can use the classic results from Ando et al. (2005). Define empirical Rademacher complexity for samples from all tasks as
K
(f(((())) (2)
(F = E.
sup FEF
k
(5)
i=1
E 11, The true Rademacher complexity is where ( are independent binary variables 0% then defined R(F) = Ex (F)).
The following theorem restates the main result from Ando al. (2005). We simplify the statement by using Rademacher complexity rather than the set cover number argument used in the original proof.
Theorem B.1. Ando et al. (2005) Given data from K different forecasting tasks f1..... l and f in hypothesis class F, for some constant C' with probability at least 1 6, the following inequality holds:
log 1/5
Σ aal)ss
Σ
CV
(6)
nK
If we assume the loss is bounded S 1/2, then we may take C = 1/v2.
Proof. Consider (xx") as independent random variables. For function that satisfies
[d(x("), ..... (1)) ..... (m)) - ()) ..... x.. (n)) S cs
by McDiarmid's inequality, we have
p p(o(x ... x()) - E(o] 2 I exp
212
Applying this inequality the max difference Q(X) sup ((f)- then with probability at least 1 - 6, we have
log 1/8 nk
Q(X) - Ex S CV
where C is a constant depending on the bounds c; If the loss / 5 1/2, then IQ] < 1/2 and so we can take c; 1 leading C' 1/v2. A standard computation (see Mohri al. (0018). Theorem 3.3) using the law of total expectation shows Ex [xx) < 2R(F), which finishes the proof.
We can use this to compare the generalization error of multi-task learning versus that of learning the individual tasks. The following inequality compares the Rademacher complexity for multi-task learning to that of individual task learning. Denote Rx, and RG the empirical and true Rademacher complexity for F over pt.
19

---

log 1/8 Under review as a conference paper at ICLR 2022
Lemma B.2. The Rademacher complexity for multi-tast learning is bounded R(F) (L/K) Ek= (F).).
Proof. We compute the empirical Rademacher complexity,
k
K
Rx(FF= = E
sup |EF
o
(s (xx+
(x4
E
sup
(1)
xx
(0)
nk
FEF
nk
k=1
B
sup
Σ ok
1
K
/EF
i=1
BØg
sup fEF
(4) %
(f
(xx
K
n
k=1
1 K
Rx, (F)
The first inequality follows from the sub-additivity the supremum function. The next equality is due to the fact positive scalars commute with supremum, and by the linearity of expectation. The expectation Eg reduces to the expectation Eg over only those Rademacher variables appearing inside the expectation. (J) is the Rademacher complexity of the function on the individual task k. Taking expectation over all samples gives the result.
It instructive to compare the bound from Theorem B.I with the generalization error bound obtained by considering each task individually.
Proposition B.3. Assume generalization bound given by considering each task individually is
n& for all tasks k and the loss is bounded S 1/2. then the
log 1/ō
( S S(S)
RR(FF)
+
K
2n
which is strictly looser than the bound from Theorem B.I Junder the same assumptions.
This result helps explain why our multitask learning framework has better generalization than learning each task independently. The shared data tightens the generalization bound.
Proof. Applying the classical analog of Theorem B. than - 6,
to single task, we find with probability greater
log 1/5
cx()) < (x) (F)
n
Averaging over all tasks yields
K
E
Σ ()) k=1
2
1
Σ (x(FF
C2
K
n
Since the loss / is bounded < 1/2, we can take C C₁ = 1/v2, giving the generalization upper bound for the joint error of [qquation 7
By Lemma B.2 and the fact 1/V2nK 1/V2n, the bound in Theorem B.1 is strictly tighter.
Gap Between Bounds. To quantify how much tighter the bound of Theorem B. is relative to
Proposition B.3 we compute the difference upper bounds. Assuming I 1/2, the gap between the bounds is.
((f) +2
1
K
(A(F))
+
log 1/8
((()) ((f))+ 2R(F) +
log 1/8 2nK
K
2n
K
log 1/8
=2
Rx(F) - R(FF)
+ 1/vK)
K
2n
1=1
20

---

Under review as a conference paper at ICLR 2022
Simplifying gives
K K
RR(F) - -(F)) +
VK VK
VK-1
V
log 1/8 2n
A=
The first term is positive by Lemma B.2 and second term is positive since v K 2 1.
B.2 DOMAIN ADAPTATION ERROR
Since we test on c ~ C' outside the training set {c*}, we incur error due to domain adaptation from the source domains per..... Hex to target domain Pe with being the true distribution. Denote the corresponding empirical distributions of n samples per task by täc mhe Σe= 5260 For different c and é' the domains f4. and jee may have largely disjoint support, leading to very high KL divergence. However, if and e' are close, samples X. ~ [e and X ~ Me' may be close in the domain X' with respect to the metric || 1lxx For example, if the external forces and e' are close, given X. ~> Pe there likely X Pe such that the distance between the velocity fields ||x_- -x,ll is small. This implies the distributions μrc and flr may be be close in Wasserstein distance Wi (i Her). The bound from Redko et (2017) applies well our setting as such:
Theorem B.4 (Redko et al. (211)). Theorem 2). Let A minser (colS) 1/K cc2 (n) There is N ((din())) such that for n >> for any hypothesis f. with probubility at least 1 8,
K
K
c.lf) S
ca (f) - WW,
Pes
Σ
Paa
K
K
1=1
k=1
+ log(1/))
V///++1//nn/) vI((K)) + Ae
Proof. We apply Redko et al. (017) Theorem 2 to target domain Pr = Me and joint source domain JAs = 11KK K/KY
Ek= >
k= Mey with empirical samples for = tea and As = 1/K Σ Bew
B.3 ENCODER VERSUS PREDICTION NETWORK ERROR
Our goal learn joint hypothesis h over the entire domain X' in two steps, first inferring the task e and then inferring T++1 conditioned on c. Error from DyÅd may result from either the encoder Io or the prediction network fo- Our hypothesis space has the form (ar - fo (æ, (xx)) where Φ and i are the weights of the encoder and prediction network respectively. Let 6xx be the error over the
entire domain X. that is, for all c. Let Cenc = xx (g(x), , go (x)) be the encoder error where g: X' + C is the ground truth. We state result that decomposes the final error into that attributable to the encoder and that to the prediction network.
Proposition B.S. Assume c + fol,, )) is Lipschitz continuous with Lipschitz constant y uniformly in Then we bound
ex(So(-.go ())) S YYanc (ennc(go) where the first term is the error due to the encoder incorrectly identifying the task and the second term is the error due the prediction network alone.
+ Eevc (eo(fo(xx c))]
(8)
The hypothesis in the second term consists of the prediction network combined with the ground truth task label + fo g(x)).
Proof. By the triangle inequality and linearity of expectation,
ex ex(fo(, Ee-e [E [[] (x))) fx))ll]]
E Egappe (xf((=gg(()) - fo(x, e)ll + Eone EEwn (1lfo(,.))))) [+]
By Lipschitz continuity, I E [Ezpre (x)) cllell + Eowe [Eey,, [ll c)ly lIf())l which, since g(z) and by linearity of expectation,
= =B [Eeeag. (x) [(x)]]e]]] Ence (Ezwn, [1l e)lb>> [[((])]}
21

---

Under review as a conference paper at ICLR 2022
and by definition of Cene and Ec.
Yenne (g6) + Bove (=)(f((z,c))
as desired.
By combining Theorem B.1
Proposition B.5 and Theorem B.4 we can bound the generalization
error in terms of the empirical error of the prediction network on the source domains, the Wasserstein distance between the source and target domains, and the empirical error of the encoder.
Let G [yo: X + C} be the task encoder hypothesis space. Denote the empirical risk of the encoder Do with respect to by (ge).
Proposition B.6. Assuming the hypotheses of Theorem B. Proposition
B.5 and Theorem B.4
K
1
(x(fo(::9)()) S (o)c
cas 27) 2
K
1og(1/5)
(r
1)y
v 21og(118)
(VI/n+ vI/(nK)
2nk
K
+ Bax
Wi
Pe: K
Σ
Pec
+ Ae
k=1
Proof. We start with the bound of Proposition B.5
(x(o((s
Yaan (gx) + Bewe [ea(So(s, c))]-
By Theorem B.1 or Mohri et al. 2018), Theorem 3.3, we can bound
(go) I Eanc (go) + 2R(G) log(1/8)
(10)
2nK
In order to apply Theorem B. to the risk and relate it to the empirical risk, we need to first relate the error on the target domain back to the source domain of our empirical samples. By Theorem B.4
K
K
cc(fo(;c)) <
Eou (fo(:.o))+W;
Pe.
Acz
+ 2oglo1(1/))
(VI/n+ vI/(nk) tho.
K
K
(11)
Applying Theorem B.I this is
log 1/6 c (2(())))))
K
+Wi
jes
jinn
+ 22og(11/)) (VI/n + v1/(kk) +hc
K
2nK
K
k=1
(12)
Substituting equation and
and equation 12 into equation glives
|og( /0)
cx(fo(:.o)))) ? (go) -2(g)) +
2nK
log 1/5
+ E
Eog (ffo(;c)))) +2(FF)
2nK
+ W,
Pe:
Pen
llg((1
(VI/n+ vI/(nK) + A
Finally using linearity of the expectation over c C, removing where there is no dependence on C. and rearranging terms gives the result.
22


Reviews:
[{"review_id": "S-z0UjvO0Vk", "reviewer": "Reviewer_cEq7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros :\n\n- This is an interesting problem which is quite timely given the development of the field of forecasting physical dynamics using neural networks. \n\n- The proposed solution seems sound and principled. Moreover, it is well motivated and the writing was quite clear.\n\n- The different additions made to the forecaster network are also quite interesting, I especially liked the AdaPad solution to deal with boundary conditions. Conducting an ablation study also considerably strengthens the paper. \n\nCons :\n\n- All experiments are conducted on somewhat similar datasets, which are based on fluid dynamics PDEs. It would be nice to see how the model deals with other families of dynamics. Especially given the fact that the contributions of this work seem geared towards practical considerations.\n\n- The setting of the experiments should be more precise and additional details should be given: how are the different datasets constructed, what supervision is there exactly regarding the different tasks, how many domains are there in each dataset and what are the differences, how is the balance between the different domains ect.", "summary_of_the_paper": "This work tackles the task of forecasting dynamics in different domains simultaneously. Using an encoder which is trained to determine the task, the inferred latent vector is then used to adapt a forecasting network to the task at hand. Experiments on three datasets linked to fluid dynamics are then conducted to assess the proposed model.", "summary_of_the_review": "This is a good work on a timely subject. The contribution is not groundbreaking but should be significant enough to warrant acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H34jsXYS5IO", "reviewer": "Reviewer_nhax", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The problem statement is well-motivated. Learning generalizable deep learning models across diverse settings is an important open problem.\n- Experiments use interesting and real-world problems.\n- Results are strong and appear reliable.\n- AdaPad is an interesting idea specialized to the case of physical complex systems, since it is designed to address boundary condition issues.\n- Visualizations show the model is behaving essentially as expected.\n- Although there are many design choices that go in to the model, each such design choice is well-motivated.\n- Aside from some aspects of the theory section, the exposition is generally quite clear and well-organized.\n- Assumptions are made clear.\n- The fact that the encoder can be trained first and independently of the forecaster should be very useful for further rapid developments.\n- Great to see ESE metric used as a complement to raw error.\n- Table in Appendix showing alternatives to AdaIn is very useful in increasing confidence in AdaIn for this application.\n\n\nWeaknesses:\n- The biggest concern is the theory section. The multi-task learning and domain adaptation results are general results that are not adequately connected back to the specific model and problem the paper is considering. Yes, it is widely accepted that multi-task learning and domain adaptation can work well, especially when tasks are related in some measurable way, and it can be a useful exercise to restate existing theory in the language of your framework, but what (if any) novel claims is the theory implying? Are there any predictions the theory makes about the particular approach which can be validated in experiments?\n- The theoretical bound on error that decomposes the error of the encoder and forecaster is similarly lacking in its interpretation. Yes, it can be a useful exercise to show that the error can be decomposed along the lines of the model, but does this bound somehow suggest that the decomposition results in lower error than a monolithic model? Or is it showing that you can work independently on improving either part of the model and improve the overall error? Where is there potential for practical value in this theorem?\n- For example, one place there could be potential to validate the theory is to check in experiments that task pairs with lower Wasserstein distance actually support better domain adaptation. However, in the Introduction of the paper it acknowledges that “Even the slightest change in these features may lead to vastly different phenomena”, but doesn’t that suggest that Wasserstein distance may not be a useful metric here for measuring task similarity? Couldn't turbulence limit the usefulness of such a metric?\n- Proposition 3.3 says the bound is “strictly looser” than the bound in Theorem 3.1. For clarity, it would be very helpful to combine the bounds into an inequality showing this strictly-looser property. It is not immediately apparent from the statement of the theorems since the inequalities contain different terms.\n- As is, the theory doesn’t really hurt the paper, but, for the amount of space dedicated to it, it doesn’t add much. The paper could be substantially improved by either (1) adding interpretation/predictions/validation of the theory that connect it back to the approach in the paper, or (2) removing some of the less useful parts of the theory from the main paper to free up space for more of the interesting analysis of what the model actually learns.\n- Also, it is interesting but a bit counter-intuitive that the theory section relies on results in multi-task learning and domain adaptation, instead of theoretical results from the meta-learning literature. As is, since the paper relies on multi-task learning so much, it is missing references to related work in multi-task learning (i.e., related work outside of modeling physical dynamical systems).\n- Similarly, it would be helpful to mention why there are no comparisons to multi-task learning or domain adaptation methods in the experiments. Why do they not apply here?\n- The three terms in the loss function of the encoder are well-motivated, but it is not clear how important each term is. Ablations on these terms would be very informative for the reader to understand what’s generally required to train an encoder.\n- In Section 5 it says “VarSepNet employs separation of variables through different loss terms”. What are these loss terms and how are they different from the ones in the paper?\n- In the ablations with no encoder, how do AdaIn and AdaPad work? Don’t they require some z? Where does this come from if not from the encoder?\n- U-Net does seem it could be at a qualitative disadvantage compared to DyAd in terms on number of parameters, especially since U-Net c is one of the more competitive baselines. It would be useful to see results for a larger U-Net c, or at least some evidence that the U-Net is not underfitting the training data.\n\n\nAdditional question of interest:\n\nOverall, this is a very important a potentially deep line of research. The most exciting promise of such work is the potential of revealing shared regularities across vastly disparate dynamic systems, that is, across complex physical processes. And it seems the approach in the paper could be particularly well-suited to such research. For example, the authors could train a single encoder+forecaster model across all the datasets in the paper, and analyze relationships in the learned encodings across datasets. Training models across highly diverse domains have been tried in multi-task learning (e.g., \"Pretrained Transformers as Universal Computation Engines\" arxiv 2021, \"The Traveling Observer Model\" ICLR 2021, \"Modular Universal Reparameterization\" NeurIPS 2019, \"One Model to Learn Them All\" arxiv 2017). Is such a generalization part of the longer term vision for this line of work?\n\n\nMinor comments:\n- In Section 2.4, some references would be useful in the sentence ending with “…the combined force equation.”\n- There are several inconsistencies in the use of parentheses in citations throughout the paper. Correcting these would improve readability.\n- In last sentence of first paragraph of Section 4, the word “task” could be changed to something like “problem”, since “task” has another meaning in the paper.\n- Should the 7.26 for U-Net-c on Ocean Currents future be bolded?\n- In the last paragraph of Section 5.1: “We tried to vary…” -> “We tried varying…” or “We varied…”.\n- Appendix A.2.1: footnote for PhiFlow is on the wrong page.\n- Appendix A.2.1: The last paragraph seems like it should be the first paragraph of A.2.2.\n- In proof of Proposition B.5, there is an extra or missing set of norm bars in the first inequality.", "summary_of_the_paper": "This paper is interested in learning general forecasting models for physical dynamical processes. The paper proposes a decomposition of such a model into an encoder that captures the innate properties of the system, and a forecaster that autoregressively makes predictions conditioned on the encoded properties. This is framed as a meta-learning approach, and is shown to substantially outperform single-task approaches and off-the-shell meta-learning approaches across multiple datasets. The paper provides some theoretical analysis, and qualitative analysis of what is learned. Overall, the paper shows that learning shared models across domains is an important and fruitful way forward for modeling physical processes with machine learning.", "summary_of_the_review": "Overall, this is very interesting and useful work. The problem is well-motivated, and the approach and experiments are carefully designed and generally convincing. If the concerns about the theory are addressed, I would be happy to increase my score. Adding the additional info and experiments requested could increase it further, and make this a particularly strong paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FirkqBw1CnL", "reviewer": "Reviewer_4kg2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As a caveat: I'm not an expert in the area, so my review remains on a superficial level consequently for which I apologize. I overall liked the paper quite a bit, the question discussed is relevant, the empirical evaluation is very good, the theoretical results seem as relevant as they would get and the related work discussed is crisply presented and relevant. \n\nOne question I would have is that results in Table 1 are overwhelmingly good with only UNET-c coming close. Do we know for these tasks what the \"theoretical\" upper bound (e.g. by the right PDE system) would be? Is it computationally even possible to compute this upper bound? I'm wondering how much of a gap there still is too close. \n\nIn a similar vein, what is the intuition behind DyAD + ResNet being better than DyAD + UNET mostly? Are there some complementary strengths between DyAD and ResNet that this combination can exploit better than DyAD + UNET?", "summary_of_the_paper": "The paper suggest a remediation for a common problem for dynamics forecasting which is the lack of generalization to other domains/tasks. The author suggest to tackle this with via a 2 component architecture, one for learning the task and one for forecasting. In empiricial experiments the authors show the practical feasibility of their approach.", "summary_of_the_review": "This is a good paper that I'd like to see accepted for its combination of theoretical results, empirical results and methodological novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4lPHv-aEFVF", "reviewer": "Reviewer_SrGv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+\n\n* This paper addresses a new and interesting generalization problem for dynamics forecasting \n* It proposes a model to address different changes in the dynamics. \n* Evaluation is done on relevant datasets with several baselines and some ablation studies.\n\n-\n\n* The applicability of the proposed approach is restricted to problems where relevant weak supervision from task parameters is available. This seems like an important limitation in real-world applications. How valid is this scenario? The question of choosing relevant parameters for weak supervision is important for applying this model to other datasets, yet the definition of these parameters is unclear; how robust is the model when chosen parameters are not useful ? The performance of Wrong_enc (Table 2) tends to say that this model will then fail. \n* It is unclear why the model can adapt to changing boundary conditions with AdaPad as it generates them from features $\\hat{z}_c$ extracted from data inside the domain and weakly supervised by quantities unrelated to the boundary condition (e.g. mean vorticity or season). \n* The theoretical analysis, inspired by existing work in multi-task learning / domain adaptation, has some limitations and does not add much value to the paper. I have some concerns with the domain adaptation upper-bound to the target error in Theorem 3.4 and Proposition 3.5. This upper-bound is not minimized thus the target risk can be high i.e. the model is not guaranteed to adapt well. Moreover, the validity of the theoretical analysis is unclear as several assumptions may not be verified e.g. bounded loss in Theorem 3.1, Proposition 3.3; lipschitz continuity in Proposition 3.5. Theorem 3.4 requires that the assumptions in Theorem 2 in Redko et al 2017 are verified, yet these assumptions are not mentioned in the paper.\n* Some ablation studies are missing: 1) the contribution of each term in equation (2) and 2) the dimensionality of $\\hat{z}_c$ which is fixed arbitrarily.\n\nOther questions: \n* It would be good to better explain how the experiments include changing boundary conditions between domains. The testing scenarios only mention different initial conditions or external forces.\n* Why do the baselines ResNet-c and Unet-c not adapt well despite having access to relevant weak supervision (p8)? This is the same information used by the proposed model to adapt.\n* How redundant is the time invariance term (3rd term in equation (2)) with the invariances enforced in the architecture of the encoder?\n", "summary_of_the_paper": "This paper addresses the problem of learning a deep learning model for dynamics forecasting which generalizes to changes in dynamics. These changes can be induced by different parameters, boundary conditions or external forces. The proposed model takes a meta-learning approach and proposes to partition data into different heterogeneous domains. It consists of two components: an encoder which infers time-invariant features given observed domain data and a forecaster which predicts the dynamics given these features. The paper evaluates the proposed approach on several datasets and provides some theoretical insights.", "summary_of_the_review": "This paper tackles a new generalization problem for dynamics forecasting and proposes a model supported by experimental results. However, this model can only be applied to problems with relevant weak supervision which may not always be available in practise. Moreover, the definition of relevant parameters is unclear and the robustness of the model to the choice of these parameters is not measured which may restrict its application to other datasets. There are also unclarities on the ability of the model to adapt to changing boundary conditions with AdaPad, some ablation studies are missing and I have concerns on the theoretical analysis which brings limited value to the paper. For this reason, I am giving this paper a weak reject.\n\n--- Post-Rebuttal comments ---\nI thank the authors for their response. After studying it, the theoretical results still have some major issues and feel disconnected from the model. In particular, key assumptions are not enforced in the model (e.g. lipschitz continuity) and the generalization error of the model in Th3.3 is uncontrolled as the upper-bound is not minimized by the model (the Wasserstein distance between domains is fixed and is high in all generality). Its use for the model is thus not very convincing. On practical aspects, the capability of handling boundary conditions should be better justified and evaluated. For this reason, I keep my score unchanged and recommend rejecting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of domain generalization for learning spatio-temporal dynamics. It proposes a solution where an encoder captures some characteristics of a given environment, and a forecaster autoregressively predicts future dynamics conditioned on the characteristics learned by the encoder. Said otherwise, the forecaster learns the general form of dynamics parameterized by an environment representation extracted by the encoder. The conditioning is implemented via an adaptive instance normalization mechanism. A form of padding is also introduced in order to take into account boundary conditions. The two components encoder and forecaster are trained sequentially. This approach is casted in a meta-learning framework. Theoretical results inspired by multi-task learning and domain adaptation are also demonstrated. The model is evaluated and compared to different baselines on three problems, and for two different settings: varying initial conditions with a given dynamics, and dynamics with varying parameters.

This is a borderline paper. It targets a timely and important problem of domain generalization for dynamic environments. The proposed solution is original and compares well experimentally to several baselines. It allows for better generalization performance for the two test settings considered. In the current version, the paper however suffers from different weaknesses. First there is the imprecision of the arguments and the description of the experiments. Some of the arguments and claims are vague and sometimes abusive, not backed up by evidence. For example, a central claim is that the encoder learns time invariant quantities characterizing the environment when the learned representations indeed change with a time shift in the input for any environment. The same goes for the argument developed for the padding construction. It is claimed to model boundary conditions, but this is not supported by any theoretical or empirical evidence.
As noted by the reviewers, the theoretical analysis is disconnected from the algorithmic and experimental developments and does not bring much additional value to the paper. What is more embarrassing is that some of the claims in this section are overstated and induce incorrect conclusions.  From Theorem 3.1 and proposition 3.3, the authors suggest that multitask learning leads to better generalization than learning independently, while this is not formally guaranteed by the results (this is acknowledged by the authors in a later comment). Besides, the conditions of validity are not discussed while they seem to only cover situations for which the train and the test distributions are the same. The same holds for the second theoretical results (theorem 3.4). It is claimed that this result supports the authors’ idea of training encoder and forecaster sequentially, while it does not. Besides, the bounds in this result cannot be controlled as noted by the reviewers and are not useful in practice.

Overall, the paper addresses an important topic and proposes new solutions. The results are promising and it is indeed an interesting contribution. However, inaccuracies and incorrect or exaggerated claims make it difficult to accept the current version of the article. The article would make a strong and innovative contribution if it were written as a purely experimental article with a detailed description of the experiments and comparisons.
Decision: Reject