Sequential Latent Variable Models for Few-Shot High-Dimensional Time-Series Forecasting
Authors: Xiajun Jiang
Ryan Missel
Zhiyuan Li
Linwei Wang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Time series
generative models
Bayesian meta-learning

Abstract: Modern applications increasingly require learning and forecasting latent dynamics from high-dimensional time-series. Compared to univariate time-series forecasting, this adds a new challenge of reasoning about the latent dynamics of an unobserved abstract state. Sequential latent variable models (LVMs) present an attractive solution, although existing works either struggle with long-term forecasting or have difficulty learning across diverse dynamics. In this paper, we first present a conceptual framework of sequential LVMs to unify existing works, contrast their fundamental limitations, and identify an intuitive solution to long-term forecasting for diverse dynamics via meta-learning. We then present the first framework of few-shot forecasting for high-dimensional time-series: instead of learning a single dynamic function, we leverage data of diverse dynamics and learn to adapt latent dynamic functions to few-shot support series. This is realized via Bayesian meta-learning underpinned by: 1) a latent dynamic function conditioned on knowledge derived from few-shot support series, and 2) a meta-model that learns to extract such dynamic-specific knowledge via feed-forward embedding of support set. We compared the presented framework with a comprehensive set of baseline models trained 1) globally on the large meta-training set with diverse dynamics, and 2) individually on single dynamics, both with and without fine-tuning to k-shot support series used by the meta-models. We demonstrated that the presented framework is agnostic to the latent dynamic function of choice and, at meta-test time, is able to forecast for new dynamics given variable-shot of support series.

Published as a conference paper at ICLR 2023
SEQUENTIAL LATENT VARIABLE MODELS FOR FEW-SHOT HIGH-DIMENSIONA, TIME-SERIES FORECASTING
Xiajun Jiang, Ryan Missel; Zhiyuan Li & Linwei Wang Golisano College of Computing and Information Sciences Rochester Institute of Technology
Rochester, NY 14623, USA
[x]7056, rxm r244 2 217 904, Linwei. Lin.ei.aang) Wang)
ABSTRACT
Modern applications increasingly require learning and forecasting latent dynamics from high-dimensional time-series. Compared to univariate time-series forecasting, this adds a new challenge reasoning about the latent dynamics of an unobserved abstract state. Sequential latent variable models (SLVMs) present an attractive solution, although existing works either struggle with long-term forecasting or have difficulty learning across diverse dynamics. In this paper, we first present con- ceptual framework of SLVMs to unify existing works, contrast their fundamental limitations, and identify an intuitive solution to long-term forecasting for diverse dynamics via meta- meta-learning. We then present few-shot forecasting framework for high- digensiimnii time-series: instead of learning a single dynamic function, we leverage data of diverse dynamics and learn to adapt latent dynamic functions to few-shot support series. This realized via Bayesian meta-learning underpinned by: 1) latent dynamic function conditioned on knowledge derived from few-shot support series, and 2) a meta-model that learns to extract such dynamic-specific knowledge via feed-forward embedding of support set. We compared the presented framework with a comprehensive set of baseline models 1) trained globally on the large meta- veta-training set with diverse dynamics, 2) trained individually dynamics with and without fine-tuning to k-shot support series, and 3) extended few-shot fotaa-oormulations We demonstrated that the presented framework is agnostic to the latent dynamic function of choice and, at meta-test time, is able to forecast for new dynamics given variable-shor of support scries
single
INTRODUCTION
In many applications, an ultimate goal is forecast the future states or trajectories dynamic system from its high-dimensional observations such as series of images. Compared to the relatively well-studied univariate time-series forecasting (Makridakis et al. 2018 Oreshkina et al. 2020
Salinas et al. 2020), high-dimensional time-series forecasting raises new challenges: it requires the extraction of the dynamics of an abstract latent state that is not directly observed (Botev et al., 2021).
Sequential latent variable models (SLVMs) provide an attractive solution that, unlike autoregressive models, abstracts latent dynamic function z; = f(Ze;; 0z) with state z; and parameter 0z, along with is emission to observations x, g(z,) (Chung et al. 2015). This pair of learned models can support long-term forecasting given only initial frames of observations, as well as controlled generation of new dynamics. Critical bottlenecks however remain in reaching these goals.
The earlier formulation of SLVMs relies on a natural extension of the static LVMs: as illustrated in Fig. [A the latent state z; is modeled the latent variable for the generation of x,, and a sequential encoder is used to facilitate the inference of z, from current and past observations Xgi (Chung et al.) 2015 Krishnan et al. 2017). Recent works have argued instead model and infer the parameter
Both authors contributed equally to this work Source code available at htttps:: / /g/ thub

---

Published as a conference paper at ICLR 2023
C) Presented few
varistble
meta-lerming
P
B2
Figure 1: Sequential datent-variabl models for forecasting eigh-dimensional sequences.
of the latent dynamic function, often modeled as time-varying linear coefficients O2.s (Karl et al. 2017 Fraccaro et al., 2017, Rangapuram et al., 2018 Klushyn et al. 202T). This results in an LVM formulation as illustrated in Fig. BI, where the latent variable 0 is inferred at each i from observations Xgi. While strong at time-series reconstructions and classifications, a fundamental limitation makes these SLVMs less suited for long-term forecasting: the latent dynamic function has a limited ability to forecast without near-term observations to support the inference of z, or 0z,4-
This limitation in the mainstream SLVMs raises natural question: are we able to relax the assumption of linear dynamic function and directly infer its 0.? Works adopting this idea have emerged: as
al. 2019 Botev et al. 2021) or stochastic (Yildiz et al. 2020) f(zzs: 02) can be asked to predict a time sequence using only an inferred initial state. This formulation has shown strong long- terettermm forecasting, although with its own fundamental limitation: it learns a single dynamic function global to all training sequences. This would not only require all training data share identical latent dynamics, but also has difficulty to forecast test sequences with dynamics different from or unknown to the training.
illustrated in Fig. B2, by modeling single 0- either deterministic (Rubanova
In this paper, we answer this important open question of long-term forecasting for diverse dynamics. We first present a conceptual framework of SLVMs to unify existing works, and identify an intuitive solution the underlying critical gap via meta-learning: instead of learning a single dynamic function, we can learn to pull knowledge across datasets of different dynamics and learn to adapt fynamic function to few-shot high- nigh-dimensional time-series. We then present a Bayesian meta-learning framework as illustrated in Fig. [ instead of being a single fixed function in Fig. [B2, we let the latent dynamic function be conditioned on knowledge derived from few-shot support time-series via a ferd-forward set-embedding meta-model; given k-shot time-series of specific dynamics, the model is asked to forecast for query time-series using only the initial frames, meta-learned across dynamics. We develop this framework to be agnostic to the latent dynamic functions of choice, and with the flexibility to forecast with a variable size of k.
We evaluated the presented framework in benchmark image sequences with mixed physics including bouncing balls (Fraccaro et al. 2017), pendulum (Botev et al. 202I). and mass-spring (Botev et al. 2021). We further applied it to forecasting complex physics of turbulence flow (Wang al.) 2021) and electrical dynamics over 3D geometrical meshes of the heart. We compared the presented work with SLVMs representative of each of the formulations in Fig. [A-B, along with recent autoregressive model designed to forecast for diverse dynamics (Donà et al. 2020). Each baseline model was trained on 1) the large meta-training set with diverse dynamics, and 2) each dynamics individually, both with and without tinettunng to k-shot support data. Representative SLVMs were further tested in their feed-forward or optimization-ban based meta-extensions. Results demonstrated clear margins of improvements by the presented work in forecasting diverse dynamics, with added ability to recognize clusters of distinct dynamics and allow controlled time-series generation given only initial conditions.
RELATED WORKS & BACKGROUND
Sequential LVMs: Among the first SLVMs is the variational recurrent neural networks (VRNN)
(Chung et al. 2015), followed by series of deep state-space models (SSMs) (Krishnan al. 2017
2

---

Published as a conference paper at ICLR 2023
Maddison et al., 2017 Li al. 2019) focused on modeling the dependence of the posterior and
transitional density of the latent state 2₁; on past latent states 2zk and observations x (Fig. (TA resembling the deep extensions of the classic Kalman filter (Krishnan et al. (017) and particle filter (Maddison et al. 2017). An alternative line of deep SSMs aims to infer the parameters of the latent dynamic function instead (Karl et al., 2017 Fraccaro al., 2017 Rangapuram al. 2018
Klushyn et al.) (021). Existing approaches along this line assumed linear latent dynamics, where the linear transition matrix each time frame k modeled as a linear combination of set of global matrices. The linear coefficients are modeled to be limeviarying and inferred from observations Xgk as illustrated in Fig. TBI. In both formulations, the latent dynamic function's 's reliance on inferred time-varying variables reduces its ability to forecast without near-term observations.
In parallel, set of models (Fig. 22 have been presented that aims to learn a latent dynamic function that forecasts sequence using only an inferred initial state, in stochastic (Rubanova et al. 2019 Yildiz et al. 2020) or deterministic forms (Botev et al. 202T). The resulting latent dynamic function strong at forecasting, albeit only single function is learned at time. We build on and advance this formulation of learning to learn a dynamic-specific function from few-shot observations.
Modeling switching dynamics in SLVMs. often based on the formulation in Fig. TA shares the presented idea of using context variables to control the latent dynamics (Becker- [Becker-Ehmck et al. [011 Linderman et al. 2017). They however are concerned with the switching of dynamics within time-series, whereas we are interested in learning to learn dynamics from k-shot support series.
Sequential neural processes (SNPs), based SLVM formulation in Fig. TA (Singh et al. 2019 Qin et al. 2019), are underlined by Bayesian meta-learning similar to the presented work. They are originally designed for supervised learning of regression function over time instead of forecasting. In this work, we will extend SNP to realize a meta-version of the SLVM formulation in Fig. A as counterpart to be compared with the presented meta-SLVM in Fig. ITC
Autoregressive dynamics: Autoregressive models are also popular for modeling and forecasting dynamics, especially for approximating physics-basee based simulations (Wang et al. 2020; Pfaff et al..
2020). Some recent works have focused on generalizing across dynamics by. for instance, disentan- gling spatial and temporal modeling (Donà et al. 2020) or learning dynamic-specinn functions in addition to a global dynamic function (Yin et al. 2021). A recent autoregressive model considered "meta-learning" dynamics by using task-embedding to condition the forecaster (Wang et al., 2021), although this task encoder is trained separately from the forecasting model via weak supervision and it infers the task from the observed frames of a forecasting series. Moreover, autoregressive models cannot support controlled generation of time-series
will demonstrate in Section [
General few-shot learning: Few-shot learning has seen substantial progress with static data, includ- ing weight initialization (Finn et al. 2017: Yoon et al., 2018), model optimizers (Ravi & Larochelle 2016), and feed-forward models to condition (Garnelo et al., 2018) or parameterize the primary networks (Bertinetto et al.) 2016 Sung et al.) 2018)) Among these, feed-forward meta- meta-models replace test-time optimization with simple feed-forward passes using support data. It also has an interesting high-level relation to Exemplar VAE (Norouzi et al. 2020) where the few-shot support samples can be viewed as the exemplar. It thus constitutes the basis of the presented few-shot forecasting methods.
Few-shot time-series forecasting: Meta-learning is well studied in univariate time-series forecasting (Montero-Manse el al. 2020) including recent deep-learning advances (Oreshkin et al.. 202T). Few- shot forecasting for high digh-dimeniional time-series, however, has not been attempted to our knowledge.
3 UNIFYING CONCEPTUAL FRAMEWORK FOR LEARNING LATENT DYNAMICS
We first describe an LVM framework that unifies existing works under two choices of probabilistic graphical models (PGMs). It includes dynamic function of latent zμ and its emission to data x;: g(zz), where 0 represents the parameter of the latent dynamic function.
2): = [(zz;; 0z),
System states as latent variables: One natural choice of the latent variable is the latent state zą underlying the observations x;- This gives rise to the PGM as illustrated in Fig. TA where the marginal likelihood of an observed sequence X0.T can be expressed as:
p(Xo:T)= = /
((Xolzo))(Z0)
II.
p(x:22, (olps)p((2//2)) Xei d22.7,
(1)
3

---

Published as a conference paper at ICLR 2023
where p(x; z;i describes emission and (/z, describes latent dynamics. To facilitate inference, a variational approximation of the posterior density ((Zorr) ((:or))xor) is often modeled as (Zorr = [T (zzi/zei Xgi). log (ogo()): 2 1=0
The evidence lower bound (ELBO) of Equation is:
E (xleeeee p(x, 2.)] (L Zei. x
(2)
Existing works adopting this PGM CChng et al. 2015 Krishnan et al, 2017 Li et al. 2019) differ primarily in how p(z, x<s) and g(z, Bis X X<s) are modeled. The first term above encourages reconstruction using the inferred q(z, Zz xga) at each time frame i: this weakens the latent dynamic function underlying p(z; Zci. x<i) that constrained only by the KL-divergence term. This leads to limited ability to forecast without near-term Xgi to support the inference of (2, Xgi).
System parameters as latent variables: An alternative choice of the latent variable is the parameters themselves of the LVM equation, especially 0g of the latent dynamic function. This gives rise to the PGM Fig. BB, where the marginal likelihood of Xo.T can now be expressed as:
p(Xurr) = [ p(xo (z)))z)))(z))d2))
/ II",, p(x;|zi)) 2. f(ac00 (P(O.)
(3)
where the observations are explained by an initial latent state Zo and parameter Og of the latent dynamic function. With variational approximation of the posterior density q(0,, 2o) and an assumption of their prior densities p(zo) and p(Oz), the ELBO of Equation 3 becomes:
(4) This covers different lines of existing works depending on how (0,)) and p(0.) are modeled. In series of works (Karl et al. 2017 Fraccaro et al. 2017 Rangapuram et al. 2018 Klushyn et al. 2021). 0 is modeled as cume-varyng system parameters 02.0TT. This involves intricate temporal modeling of g(02,,i (xxx) and p( (z over time as illustrated in Fig. [ Because the latent dynamic function relies on time-varying 0z,0:T, its forecasting again relies on near-term observations to support the inference of 2z. Alternatively, q(Oz) can be simply assumed to be global across observations and the dynamic function becomes a Bayesian neural network as presented by (Yildiz et al. 2020). As a more special case, 0g be deterministic which leads to the latent ODE model presented by (Rubanova et al. 2019). If we further assume Zo to be deterministic, we arrive at the set of deterministic encoding mecoding-eeooding network with latent dynamic functions examined by (Botev et al., 2021). This set of formulations, as summarized in Fig. 32, shares the advantage of strong long-term forecasting, albeit a fundamental limitation in learning single dynamic function time.
log ((x:r)) 2 =o) [log p(xor Zu. 0z)] KL (q(z)) (((z))))(()))))))))))))))))))))) (L (p))))))
In Section 5 we will include representative models from each PGM to provide empirical evidence for the identified limitations. With this basis, we derive an intuitive solution to the identified critical gaps by extending the PGM in Fig. [B2 to the presented PGM in Fig. [ig.[C: instead of learning single dynamic function, we will learn adapt a latent dynamic function to few-shot support time-series.
4 FEW-SHOT FORECASTING VIA BAYESIAN META-LEARNING
Consider dataset D of high-dimensional time-series with M similar but distinct underlying dynamies: D { For each D,, we consider disjoint few-shot support series Dj
x.2 {xi:), x.. x and query series DJ (x x2.2 Xurr: ..., x where h I. IIstead of
maximizing the marginal likelihood of xp.r for all Xp:T E as in Equation 3 we formulate a
meta-objective to learn maximize the marginal likelihood of x for all query series x D
when conditioned on support series Dj. for all dynamics j {1, ..., M):
p(x),r1D3)) (cx X Djj
(5)
where p(x).r /c), though similar to Equation is now conditioned on (thus adapted to) knowledge derived from support series of a specific dynamics. p(c(D;) is the meta-model describing how to extract such dynamic-specifie knowledge from few-shot support set Dj.
Set-conditioned latent dynamic functions: We model p p(x),rlc) based on Equation 3 as:
/f
p(x (c)
po, (Xo (an)))))p(ddddd II.
Po. (x: (2:) 12,= (0..
(6)
4

---

Published as a conference paper at ICLR 2023
where the latent dynamic function is parameterized by 0₂ but conditioned embedding from the support set. To focus on C, we assume 0 to be deterministic and global as in (Rubanova et al. 2019 Botev et al.] 202T). As example, we can describe z; 2 + Az, where Az; conditions gated recurrent units (GRUs) (Chung et al. 2014) on c, as detailed in Appendix B This conditioning can be generalized to other functional forms of f(-), which we will demonstrate in experiments.
Meta-model for amortized variational inference: We model Pc (clD;) with a meta-model param- eterized by in the form of feed-forward embedding of support set Dj. Specifically, each support sequence x6:r E DDj is first encoded through neural function he ho(xx:z)) (x with blocks of interlaced spatial convolution and temporal compression layers. To extract knowledge shared by the set, the embedding from all sequences in Dj is aggregated by an averaging function: Exp.r (x(x))) where ic is the size of the support set. The value of l can be fixed or variable in our framework. This set embedding parameterizes P. (c( ~ (Nee, r²) via separate linear layers.
To enable inference, we approximate the posterior density p(c|D), j. xb_r) as ge (cID: U x sharing the same meta set-embeddingg model by augmenting Dj with x The ELBO of Equation across all dynamics D = {D, can then be derived as:
Σ M
log logp(x ((x)(())))))
Σド= Exer 1e llog po, (x b c)] (7)
(L ( )ll((z)) KL (aş (c U x
where 9o (2)) 20 ~ ( 2 is parameterized by an encoder with 20 = 2 in all experiments.
p(zo) is assumed to be NN(0, I). The likelihood term is estimated with reparameterization trick (Kingma & Welling 2013), and the KL-divergence terms are calculated analytically.
The optimization of Equation (7) is realized via episodic training where, in each training episode, data in each dynamic set Dj is divided into disjoint support set Dj and query set Dj For each query series across all dynamics, starting with an initial latent state 2o (inferred from Z frames) and k-shot support embedding c, the latent dynamic function is asked propagate forward to forecast the entire sequence of Zo:r and their corresponding high-dimensional observations XO:T-
5 EXPERIMENTS ON BENCHMARK IMAGE SEQUENCES
We first considered benehmark images generated with controllable physics, including bouncing ball Fraccaro et al. (2017). Hamiltonian pendulum (Botev et al.) (02T), and Hamiltonian mass-spring systems (Botev et al. (021). Details of data generation are available in Appendix G To intentionally create data with diverse dynamics, we included 1) a bouncing ball dataset with 16 different directions of gravity, each with 3000 samples simulated using a combination of different initial positions and velocities (gravity-16)) and 2) mixed -hyysiiss dataset consisting of bouncing balls under 4 gravity directions, and pendulums and mass springs each with four different values of friction coefficients of O, 0.05, 0.1, 0.15 (mixed-physics). Each physics with a unique parameter includes 3000 samples.
Data:
We considered baseline models representative of each formulation outlined in Fig. This includes VRNN Chung et al. (2015) and DKF Krishnan et al. (2017) representing Fig. DAI. DVBF et al. (217)) and KVAE Fraccaro et al. (017)) representing Fig. IB1, and three models representing Fig. BB2 with latent dynamic functions as residual GRUs (GRU-res), neural ordinary differential equation (NODE), and residual Recurrent Generative Networks (RGN-res) (Botev et al. 2021). We also considered recent autoregressive model designed to tackle forecasting diverse dynamics (Donà et al.) 2020). All baseline models were 1) trained using the entire meta-training data consisting mixed dynamics, 2) trained in 1) and further fine-tuned to the meta-test k-shot support set (k- 15) (except for (Donà et al. (2000) as we were uncertain about proper approach of fine-tuning due to its specialized architecture), and 3) trained individually for each single dynamics, with and without fine-tuning to the meta-test shotot support set (A= 15).
Models:
For each of the global latent dynamic models (GRU-res, NODE, and RGN-res), we extended it into our shot framework. While shw- learning with the rest of the SLVMs is not yet reported in literature, we further selected DKF as a representative of the SLVM in Fig. TA and extended into a feed-forward meta- formuloon via variant of the SNP (meta-DKF). We also attempted
5

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of the presented meta-models with all baselines trained on the meta-training for gravity-16 16 data. The improvement of meta-GRU-res (best-performing) over its closest baseline is statistically significant in all metries (p 0.01, paired r-test).
PGM type
Model
MSF4
VPT-MSE1
Dist.
VPT-Distf
meta-GRU-res
1.44(0.34))-2
0.68(0.26)
2.88(1.45)
0.97(0.07)
Fig. 1
meta-NODE
1.60(0.26)e-2
0.58(0.22)
6.10(2.63)
0.80(0.12)
meta-RGN-res
1.59(0.24)
0.56(0.21)
6.97(3.08)
0.76(0.13)
GRU-res
1.63(0.21)e-2
0.50(0.17)
10.4(3.30)
0.61(..09)
GRU-res finetune
1.65(0.24)e-2
0.50(0.18)
9.35(3.33)
0.66(0.12)
NODE
1.69
0.48(0.16)
10.9(3.2))
0.59(0.08)
Fig. 32
NODE finetune
1.70
0,48(0.17)
10.4(3.23)
0.61(0.09)
RGN-res
1.70(0.17)e-2
0.47(0.16)
11.2(3.39)
0.58(0.09)
RGN-res finetune
1.72(0.19)e-2
0.47(0.17)
10.0(3.36)
0.62(0.11)
DVBF
2.32(14.4)e-2
0.02(0.10)
45.3(0.0)
0.00(0.00)
Fig. IBI
DVBF finetune
2.33(13.4)e-2
0,02(0.10)
45.3(0.00)
0.00(0.00)
KVAE
3.37(1.36)e-2
0.24(0.19)
4.81(3.61)
0.57(..29)
meta-DKF
3.80(0.59)e-2
0.10(0.11)
7.35(3.26)
0.70(0.25)
DKF
3.84(0.59)e-2
0.10(0.11)
7.39(3.21)
0.69(0.25)
Fig. AA
DKF finetune
3.85(0.58)e-2
0.10(0.11)
7.51(3.26)
0.69(0.25)
VRNN
1.78(10.9)e-2
0.24(0.11)
23.1(21.6)
0.51(0.07)
VRNN finetune
2.15(12.2)e-2
0.21(0.16)
8.31(11.6)
0.75(0.19)
Autoregressive
Donà et al
3.52(0.26)e-2
0.001(0.01)
13.7(3.05)
0.06(0.15)
optimization-basee meta-learning of MAML (Finn al., 2017) to the DKF and GRU-res models. although challenges of stability and convergence as noted in literature (Mehta et al. 2021 Antoniou al. 2018) were encountered, suggesting that MAML extensions SLVMs may not be trivial due to issues such as vanishing gradient issues over the complex computation graph.
All GRE-res, NODE, and RGN-res based models were trained to forecast for a sequence of frames using only the first 3 frames. We investigated k-shot forecasting when & is fixed at different values of l: 1, 5, 10, 15, or allowed to variable both meta-training and -test with 15 as the upper limit. For VRNN, DKF, DVBF, and KVAE, we used their public implementations for training and evaluation. Similar network components with the meta-models were scaled to have comparable parameter scales. Because of their reliance observed time frames to support prediction, observed frames were exposed to the encoder to reconstruct the 8 frames and forecast the additional 12 frames.
We considered four quantitative metrics meta-test series. We included the commonly used mean squared error (MSE) of forecasted images, and the prenttlyproposed metric of Valid Prediction Time (VPT) that measures how long the predicted object's 's trajectory remains close the ground truth trajectory based on the MSE (VPT-MSE) (Botev et al.) 2021). Because pixel-level MSE does necessary well capture the quality of the predicted dynamics due the small object size on the image, we further introduced two new metrics: distance (Dist) between the ground-truth and predicted location of the moving object; and VPT determined based this distance error (VPT-Dist).
Metrics:
Comparison with baseline models trained on full dynamics: gravity in meta-train veta-training. in meta-validation, and 4 in meta-testing. Table [ summarizes the quantitative test performance of the three k-shot meta- models obtained with k = I5, in comparison each of the baseline models trained from the full meta-training set. We include complete results across all models in Appendix [ with Table 4 Visual examples for these quantitative results are in Appendix [ with Fig. 7 (shaded blue): all the baseline models, including their fine-tuned and meta-versions, struggled with limited forecasting ability, especially evidenced by the error in predicting the movement of the ball over time (Dist and VPT-Dist). For DKF/VRNN/KN VAE and meta-DKF, there were strong reconstruction and near-term forecasting from partially observed frames (marked by red vertical lines), but incorrect forecasting further away from the observed frames. GRU-res/NOD//RGNN and their fine-tuned versions exhibited difficulty describe mixed gravity.
For gravity-16 16 data, we used 10

---

Published as a conference paper at ICLR 2023
A
Ball
Pendulum
Mass-sorigg
Dist
GRU-res
4.392.50)
0.890.13)
GRU
17411.99
0.960.10)
GRRU-Ies
0.18/8.99)
1,00.00
GRL--m
4.6(2.88)
0.890.14)
GRU-ve
3.57(3.86)
0.8700220
GRL-1R0
0.250.15)
1.000.00)
GRU res tinetune
4.3712.49)
0.900.13)
ORU me finetune
3.5544.05)
0.6700200
GRU--res linetune
0.16/0.88)
1.000.00)
meea DKF
7.28(3.55)
0.7210.20)
mDra m
3,W2996)
8.930.18)
IOSS DKF
0.54g0.29)
1.000.000
0.78(3.45)
0.3800.20)
6.55(4.1)
0.6600.35)
0.8620488
1.000.00)
9.870.57)
0.83(047))
1,000.00)
KVAE
0.01/0.01)
0,42/0.34)
KVAD
4.1244.26
0.700.34)
KVAE
0.52/0.5)
1,000.000
Donk et
13.8(2.97)
0.06/C.17
Doak et
15,8(2.1)
0.02(0.06)
Donk ct
5.602279)
0.61
Bouncing Ball
Pendulum
Ground truhh
meta-GRU-ve
Meta-DKF
KVAC
Figure 2: A: Comparison with baselines trained mixed-physies. B: Forecasting examples.
Table 2: Comparison with baselines trained on single dynamics in meta meta-training training data on gravity-
Model
Dynamics
MSEJ
VPT-MSE1
Dist
VPT-Distf
known
(.33(0.34)
0.68(0.26)
2.86(1.44)
0.97(0.06)
meta-GRU-res
unknown
1.45(0.33)e-2
0.67(0.25)
2.96(1.49)
0.97(0.07)
known 1.80(0.29)e-2
0.46(0.23)
6.86(3.95)
0.77(0.19)
GRU-res
unknown 1.99(0.277)e-2
0.37(0.18)
8.07(3.56)
0.69(0.17)
GRU-res finetune
unknown
2.03(0.27))-2
0.35(0.17)
8.51(3.42)
0.66(0.18)
known
3.81(0.59)e-2
0.10(0.11)
7.37(3.27)
0.70(0.25)
meta-DKF
unknown
3.80(0.59)e-2
0.10(0.11)
7.30(3.21)
0.70(0.25)
known 3.74(0.55)e-2
0.10(0.11)
8.37(3.79)
0.63(0.28)
DKF
unknown 3.79(0.52)e-2
0.09(0.10)
8.72(3.75)
0.60(0.27)
DKF finetune
unknown
3.82(0.52))-2
0.09(0.10)
8.77(3.77)
0.59(0.27)
known
3.42(1.30)e-2
0.39(0.34)
5.05(3.57)
0.5(0.34)
KVAE
unknown
3.46(1.36)e-2
0.22(0.19)
5.17(3.91)
0.53(0.28)
known
3.58(0.33)e-2
0.00(0.01)
13.7(3.36)
0.07(0.18)
Donà et al
unknown
3.56(0.34)e-2
0.00(0.01)
14.1(3.84)
0.08(0.19)
data, for each of the three physics, we included three dynamic settings in meta- training and left out one in meta-testing. Fig. 2A summarized the test results of the presented meta-GRU-res (with variable k) with representative baseline models. Visual examples are shown in Fig.2B. As shown, meta-DKF, DKF, and DVBF again demonstrated limited ability for long-term forecasting across all physics. KVAE, VRNN, and the finetuned global latent GRU-res were more successful with the mass spring and pendulum systems with relatively simpler dynamics, yet they struggled with the gravity system. The presented meta-GRU-re model consistently outperformed all the baselines across all dynamics, with a larger gain in more complex dynamics.
For
Comparison with baseline models trained on single dynamics: Table summarizes the perfor- mance of representative baseline models when trained on single gravity on gravity-16 data in comparison meta-GRU. As shown, both test dynamics known and unknown to the training, the meta-models outperformed the single-dynamic baselines, suggesting the added benefits of learning across dynamics. This margin of improvements remained even when the single-dynamics basclines were fine-tuned to the k-shot support series of unknown test dynamics. Visual examples of these baselines are also shown in Appendix D with Fig. (orange shade).
Ablation study: Table summarized the effect of & on k-shot forecasting using the meta-GRU-ae model. As expected, model performance improved as the size increased. Even with = however, the performance significantly better than all the base models summarized in Table T
7

---

Published as a conference paper at ICLR 2023
Table Performance metrics of meta-GRU-res models with fixed vs. variable & values
K
Mode
MSE
VPT-MSE1
Dist4
VPT-Dist
Fixed
1.8000221)-2
0.44(0.16)
10.6(3.40)
0.60(0. 10)
5
Fixed
1.53(0.36)e-2
0.61(0.25)
3.49(1.89)
0.94(0.10)
10
Fixed
1 1.46(0.34)e-2
0.65(0.26)
3.08(1.58)
0.96(0.08)
15
Fixed
1.44(0.34))-2 (.50(0.34)
0.68(0.26)
0.97(0.07) 2.88(
Variable
0.64()
3.44(1.80)
0.94(0.10)
A
1SKE plat support sinbeddings from stachastic model
cSNE plut of support embedding from detreminiatic model
Forecasting
Pendular
Mas Spring
Figure 3: A: t-SNE plot of support-set embedding e from stochastic (left) and deterministic (right) meta-models. B: Generated forecasting by sampling the distribution of given the 20-
Allowing k to be variable had no noticeable effect on model performance. This flexibility highlights the practicality of the presented framework to forecast with any given size of support series.
Latent embedding and generation of diverse dynamics: Fig. 3 shows the distribution of the latent embedding obtained from randomly-selectdd support set, comparison to a deterministic version of the presented meta-model on mixed-physics data. As shown, the presented framework was able to recognize and separate the three dynamics using the k-shot support set: given an initial Zo. was then able to generate different time-series within the same dynamics as well as across dynamics by sampling the distribution of Fig. B)). This was not possible with its deterministic counterpart.
6 EXPERIMENTS ON COMPLEX PHYSICS SIMULATIONS
We then considered learning and forecasting two complex physics-based dynamics: turbulent flow dynamics and cardiac electrical dynamics.
Turbulent flow dynamics: We customized the meta-GRU model to a dataset of turbulent flow dynamics, simulated with 25 varying buoyant forces acting on the fluid. Each dynamic contains 64 64 velocity fields of turbulent flows. We use 20 dynamics in meta-training and meta-validation with 80-20 split, and the rest 5 in meta-testing. We followed the experimental setup in (Wang et al. 202T) with an observed window (20 frames in theirs vs. 5 in ours) and a prediction roll-out of 20 frames. Despite using smaller number of observed frames, the presented meta- meta-GRU model obtained rooted MSE (RMSE) of 0.26 + 0.05 on seen dynamics and 0.49 + 0.05 on unseen dynamics, in comparison to respective RMSEs of 0.42 + 0.01 and 0.51 + 0.02 reported in (Wang et al. 2021), all reported on the 20 roll-out frames. We included trajectory visualizations Appendix F
Cardiac electrical dynamics: The propagation of electrical waves in the heart is governed by reaction-diffusion partial differential equations (PDEs) (Aliev & Panfilov, [996).. While direct PDE-based simulation holds clinical potential (e.g., for virtually testing treatment response), its patient-specifi parameters are difficult to estimate and its computational cost is high. Although neural approximations provide promising computationaall--ffiiiii how to personalize such a neural model remains an open challenge where existing models are typically trained for PDE with given parameter configurations. Here, apply the presented framework
alternative (Fresca et al.) 2021)
8

---

Published as a conference paper at ICLR 2023
A
Bround truth
Meta-Gra
1-5 GRU
Giobal GRU
S-D GRU
PV
B
Model
MSEJ
CC1
Meta-GRU S
2.61(1.88)e4
0.81(0.13)
Instance-speciln GRU
1.17(0.474)e.3
0.43(0.15)
Global GRU
5.70(2.79)c-
0.430.20)
Single-dynamies GRU
4.97(3.171e-4
0.64(0.28)
Personalized virtual heart (PVH)
4.92(5.51)e-4
0.64(0.25)
High
Figure 4: Visual examples (A) and performance metrics of meta-GRU versus other baselines trained on the meta-training set for forecasting electrical dynamics on the heart.
for few-shot learning of a personalized neural model that can be used to efficiently forecast how a pptient-specific heart may respond to electrical simulations at different locations.
We simulate electrical propagation originating from various locations in a 3D heart mesh, with 15 settings of PDE parameters representing 15 dynamics with different locations of injury to the heart muscle. We use dynamics in meta-training, time-series with different initial conditions (meta-traite training: 450; meta-test: 2,020). Each time series describes 3D+T propagation electrical wave with blocks at locations of musele injury specific to each dynamics (see example if Fig. 4A column 1). The quality of the forecast series is measured by its MSE and spatial correlation coefficient (CC) with the actual time-series.
meta-validation, and meta-testing, with disjoint
We adopted graph-CNN encoder/decoder and ODE-GRU latent dynamic function 2₁: [Zzck 02) similar to that described in (Jiang et al. 2021). We trained it with a global 0, (global GRU), an individual 0g for each PDE parameter -single-dynamics GRU), conditioned f(z,- 02) with encoded from individual training series (instance-speciff GRU), and the presented framework (meta GRU) with & varying between 1 and 5. We further added strong personalized virtual heart (PVH) baseline using the original PDE simulation, with the PDE parameter optimized by SOTA approach from shoshot support series as descrihed (Dhamala et 2018
As shown in Fig. 4 and additional examples in Appendix only meta-GRU was able to accurately forecast the propagation block while the other baseline models missed the correct locations of musele injury specific to subject (black circles in column 1): note that the single-dynamics GRU performed well on the training dynamics Fig- 4A, but fails on unknown dynamics (Appendix F). Unable to identify injury to patient-specing heart, the forecasting model will be of little value in clinical tasks such as personalized prediction and treatment planning. This gain in forecasting performance by meta-GRU is quantitatively summarized in Fig. 4 across all meta-test time-series. Note that meta- GRU exhibited a notable margin of improvement even versus the PVH: PVH takes on average minutes to forecast each series, versus 0.24 seconds by the meta-GRU; moreover, to optimize PDE parameters of the PVH on average required 100 calls to the PDEs (i.e., ~ 10 hours), versus 0.032 seconds for meta-GRU adapt to patient-specific dynamics. This substantial gain in efficiency without loss of accuracy holds significant value for clinical applications.
7 CONCLUSIONS AND DISCUSSION
In this paper, we present a sequential LVM framework to unify existing approaches to learning latent dynamics, identify their limitations associated with the underlying choices of PGMs, and provide empirical evidence for the identified limitations. We further identify meta-learning as an intuitive solution to the identified open gaps, present a framework for few-shot tigh-dimensional time-series forecasting, and demonstrate that its performance gain is agnostic to the underlying choice of latent dynamic functions. Limitations: An avenue of future work expand the latent dynamic functions in this framework, especially those integrating strong inductive biases based on physics such as Hamiltonian mechanies (Botev al. 2021)
ACKNOWLEDGMENTSS
This study supported by NIH National Heart, Lung, And Blood Institute (NHLBI) grant 145590, NIH National Institute of Nursing Research (NINR) grant ROINRO18301, and NSF OAC-2212548.

---

Published as a conference paper at ICLR 2023
REFERENCES
Rubin R Aliev and Alexander V Panfilov. A simple two-variable model of cardiac excitation. 7(3): 293-301, 1996.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint arXiv:/810.09550 2018.
Philip Becker-Ehmek, Jan Peters, and Patrick Van Der Smagt. Switching linear dynamics for variational bayes filtering. In International Conference on Machine Learning, pp. 553-562. PMLR,
2019.
Luca Bertinetto, Henriques, J Valmadre, P. H. S. Torr, and Vedaldi A. Learning feed-forward one-shot learners. In Neural Information Processing Systems, 2016.
Aleksandar Botev, Andrew Jaegle, Peter Wirnsberger, Daniel Hennes, and Irina Higgins. Which priors matter? benchmarking models for learning latent dynamics. In Advances in Neural Information Processing Systems, 2021.
Junyoung Chung. Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems, 2015.
Jwala Dhamala, Sandesh Ghimire, John L Sapp, B Milan Horáček, and Linwei Wang. High- dimensional bayesian optimization of personalized cardiac model parameters via an embedded generative model. In International Conference Medical Image Computing and Computer- Assisted Intervention, pp. 499-507, Springer, 2018.
Jérémie Donå, Jean-Yve Yves Franceschi, Sylvain Lamprier, and Patrick Gallinari. Pde--riven spatiotem- poral disentanglement. arXiv preprint arXiv:2008.01332, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. agdes-agmostic meta-learning for fast adaptation of deep networks. In International conference on machine learning. pp. 1126-1133 PMLR, 2017.
Marco Fraccaro, Simon Kamronn, Ulrich Paquetz, and Ole Winthery. A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances Neural Information Processing Systems, 2017.
Stefania Fresca, Andrea Manzoni, Luca Dedè, and Alfio Quarteroni. Pod-enhanced deep learning- based reduced order models for the real-time simulation of cardiac electrophysiology in the left atrium. Frontiers in physiology, pp. 1431, 2021.
M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. J. Rezende, and S. M. A. Eslami. Conditional neural processes. In International conference on machine learning. 2018.
Laurent Girin, Simon Leglaive, Xiaoyu Bic, Julien Diard, Thomas Hueber, and Xavier Alameda- Pineda. Dynamical variational autoencoders: A comprehensive review. Foundations and Trends@ in Machine Learning, 15(1-2):1-175, 2021. ISSN 1935-8237. doi: 10. 10.156//2200000089,
Xiajun Jiang, Ryan Missel, Maryam Toloubidokhti, Zhiyuan Li, Omar Gharbia, John I. Sapp, and Linwei Wang. Label-free physics-informed image sequence reconstruction with disentan- gled spatial-temporal modeling. In International Conference on Medical Image Computing and Cospiter-Assisted Intervention, pp. 361-371. Springer, 2021.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. In International Conference on Learning Representations, 2017.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.60 2013.
arXiv preprint
10

---

Published as a conference paper at ICLR 2023
Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cseke, and Patrick van der Smagt. Latent matters: Learning deep-state-spate models. In Advances in Neural Information Processing Systems, 2021.
Rahul G. Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Association for the Advancement of Artificial Intelligence, 2017.
Li, J Yan, Yang, and Y Jin. Learning interpretable deep state space model for probabilistic time series forecasting. In International Joint Conference Artificial Intelligence, 2019.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference recurrent switching linear dynamical systems. In Artificial Intelligence and Statistics, pp. 914-922. PMLR, 2017.
Chris J Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Amaud Doucet, and Yee Teh. Filtering variational objectives. Advances in Neural Information Processing Systems, 30, 2017.
S Makridakis, E Spiliotis, and V Assimakopoulos. S. Statistical and machine learning forecasting methods: Concerns and ways forward. PLoS ONE, 13, 2018.
Bhairav Mchta, Ankur Handa, Dieter Fox, and Fabio Ramos. user's 's guide to calibrating robotic simulators. In Conference Robot Learning, pp. 1326-1340 PMLR, 2021.
P. Montero-Manso, G. Athanasopoulos, R. J. Hyndman, and T. S Talagala. FFORMA: Feature-based forecast model averaging. International Journal of Forecasting, 36, 2020.
Sajad Norouzi, David Fleet, and Mohammad Norouzi. Exemplar vae: Linking generative models, nearest neighbor retrieval, and data augmentation. Advances in Neural Information Processing Systems, 33:8753-8766 2020.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-lcarning framework with applications to zero-shot time-series forecasting. In AAAI Conference on Artificial Intelligence, 2021.
B. N. Oreshkina, D. Carpov, N. Chapados, and Y. Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
Tobias Pfaff, Meire Fortunato, Alvaro
and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:2010.03400 2020.
Shenghao Qin, Jiacheng Zhu, Jimmy Qin, Wenshuo Wang, and Ding Zhao. Recurrent attentive neural process for sequential data. In arXiv, 2019.
Syama Sundar Rangapuram, Matthias Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Neural Information Processing Systems, 2018.
S. Ravi and H. Larochelle. Optimization as a model for few- shot learning. In International Conference Learning Representations (ICLR), 2016.
Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent odes for eeegularly-samplee time series. In Neural Information Processing Systems, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36: 1181-1191, 2020.
Gautam Singh, Jaesik Yoon, and Youngsung Son. Sequential neural processes. In Neural Information Processing Systems, 2019.
Flood Sung, Yongxin Yang. Li Zhang. Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Computer Vision and Pattern Recognition, 2018.
11

---

Published as a conference paper at ICLR 2023
Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv:2002.03061, 2020.
Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference. arXiv preprint arXiv:2102.1022 2021.
C. Yildiz, M. Heinonen, and H. Lahdesmaki. VAE Deep generative second order odes with bayesian neural networks. In Neural Information Processing Systems, 2020.
Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments, Advances in Neural Information Processing Systems, 34:7561-7573, 2021.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning Advances in neural information processing systems, 31, 2018.
A DERIVATION OF EQUATION 7
M
log p(x (D))
XOTED
Σ
log
pe, (x,12,1,)) s) log //// pe. 220 c) II
p(zo)
Pc Pc(cIDj)
(c(cl ; Ux!T)) 90 (000 Nbil (c(() U
96b 20 Xasii
yos go( x1:10)
Σ
logpo, (x c) log
p(zo)
EDy
(c UX
log
g0o (c(c) U (d.r)dzode
Pc(C(D3)
Σ Σ (40 Pogro, (x)0 12g. c)] ED
KL( x )lp(()) KL (
B GATED RECURRENT UNIT (GRU) SET-CONDITIONING
We condition the GRU cell of the transition function through Equation
(1)
1-1 = ELU 2- + 3,c + 71),
gi-1 = a W 2 - + bi)
z 1-1 (LU + 32c 72),
(2)
h ELU W 22 _1 + b2)
(8)
4-1 0r37,-1 + 3gc + 73:
Az, (1 g)) の (W W3 + b3) + - @ h
where 0, (W b; ar, Br, 72 are learnable parameters of the dynamic function.
C EXTRA CONSTRAINT TO META-MODEL
We considered adding an additional regularization to the set embedding Pc(c(D)) that it is constrained to reasonable range. Since the true posterior density Pc Pc(e)D)) is unknown, assume
12

---

Published as a conference paper at ICLR 2023
MSE
VPT
Dist
VPT-Dist
0.075
1.0
0.0
0.020
0.9
0.015
0.8
0.7
0.010
0.2
ho Food
(Zoomed) x-axis: K 1, 5,
K. Flndd
Figure 5: Comparison with models with and without constraint.
Ground Truth
With Constraint
No Constraint
Figure 6: Examples of models with and without constraint.
that it is bounded by standard Gaussian distribution NV(,, I). Therefore, the objective function of the model becomes:
arg min
Eg (x)))))) (el23)xx) log (x kz c)]
(Ts b. ( llp()))) KI. (a<(c)D) U (
KL (P((CI))))
We applied the extra constraint to the proposed meta-GRU-re model for both fixed and variable I different value of 1,5, 15 and evaluated gravity-16 dataset. Fig. 5 summaries the quantitative test performance of the two models trained with and without the constraint on the set embedding. The constraint generally had no noticeable effect model performance. Visual examples at = are also shown in Fig. 6 shows that when k is small, the model with the constraint had slightly better performance.
D ADDITIONAL GRAVITY- 16 RESULTS
Here provide the complete results of all baseline models trained gravity-16 in Table when split between the known dynamics during training and the unknown ones during testing. Similarly, provide the full visualization of all baselines within Fig. 7 We note that the VRNN fails to converge in any of the single dynamics cases, in contrast to its full dynamics training, which is likely due the lower data availability in these settings. The other baselines DKF and K similar decreases in performance compared to their full meta-training set performance.
E TURBULENT FLOW VISUALIZATIONS
Here we provide visualizations of predicted trajectories for both known and unknown buoyancy factors the turbulent flow dataset within Fig. and Fig.
F ADDITIONAL CARDIAC FIGURE
Here provide additional result the cardiac experiments in Figure 10, this time highlighting the performance of the model on unknown dynamic cardiac dynamics when compared against the baselines. We varying support set & size between to 5 samples. The proposed model manages
13

---

Published as a conference paper at ICLR 2023
Full dynamics
Single dynamics
Mett-Gil-rres
GRU res
GRU finetunc
Gil--res fineune
Meta-AODE
Meta-NDE
NODE
NODE
Meta-GGirrs
Meta-RGhires
RBN-rus
AGNres
RGN finetune
RōK res finztune
Meta-OXF
Mett-EKI
DAF
KVKE
CWE
DVBF
CVBI
Dvarr finetun
DVBF finetune
VINN
VRNN
VRNN Enotuns
ViDeN finecune
Cona
Doná at
Figure 7: Forecasting gravity- 16 under (blue) full dynamics and (orange) single dynamics training.
Ground Truth
Prediction
Time
Figure Example of forecasting known turbulent flow dynamies.
Ground Truth Prediction
Time
Figure 9: Example of forecasting unknown turbulent flow dynamics.
Unknown cardiac dynamics
Ground truth
Global GRU
Meta-GRU
I-S GRU
S-D GRU
PVH
Time
Low
High
Figure 10: Example of forecasting unknown electrical dynamics on the heart.
to effectively model around the scar tissue present in the ground truth. The baselines, besides the expensive personalized virtual heart (PVH), are unable to account for the specific dynamics this subject and propagate over it.
14

---

Published as a conference paper at ICLR 2023
Table 4: Comparison of the presented meta and base models known and unknown dynamics.
Model
Dynamics
MSEJ
VPT-MSE1
Dist.
VPT-Distf
known
1.43(0.34)e-2
0.68(0.26)
2.86(1.44)
0.97(0.06)
meta- meea-GRR-ress
unknown
1.45(0.33)e-2
0.67(0.25)
2.96(1.49)
0.97(0.07)
known
1.60(0.26)e-2
0.58(0.23)
6.05(2.65)
0.80(0.12)
meta-NODE
unknown
1.62(0.26)e-2
0.57(0.22)
6.23(2.54)
0.80(0.12)
known
1.59(0.25)e-2
0.57(0.21)
6.89(3.08)
0.76(0.13)
meta-RGN-res
unknown
1.60(0.23)e-2
0.56(0.20)
7.23(3.09)
0.75(0.13)
known 1.80(0.29)e-2
0.46(0.23)
6.86(3.95)
0.77(0.19)
GRU-res
unknown
1.99(0.277)e22
0.37(0. 18)
8.07(3.56)
0.69(0.17)
GRU-res finetune
unknown
2.03(0.27)e-2
0.35(0.17)
8.51(3.42)
0.66(0.18)
known 1.97(0.21)e-2
0.30(0.20)
10.6(4.13)
0.60(0.17)
NODE
unknown 2.06(0.22)e-2
0.27(0.17)
11.1(3.71)
0.57(0.16)
NODE finetune
unknown
2.07(0.21)e-2
0.27(0.17)
11.2(3.72)
0.55(0.16)
known 1.93(0.22)e-2
0.34(0.20)
10.1(4.11)
(.61
RGN-res
unknown 2.03(0.22)e-2
0.30(0.16)
10.8(372)
0.57(0.16)
RGN-res finetune
unknown
2.05(0.22)e-2
0,29(0.16)
10.7(3.62)
0.56(0.16)
known 3.81(0.59)e-2
0.10(0.11)
7.37(3.27)
0.70(0.25)
meta-DKF
unknown
3.80(0.59)e-2
0.10(0.11)
7.30(3.21)
0.70(0.25)
known 3.89(0.32))-2
0.08(0.06)
10.7(3.17)
0.46(0.20)
DKF
unknown
3.88(0.32)e-2
0.08(0.06)
10.7(3.23)
0.45(0.20)
DKF finetune
unknown
3.89(0.32)e-2
0.08(0.06)
10.8(3.2))
0.45(0.20)
known 2.32(14.6)e-2
0.02(0.11)
45.3(0.00)
0.00(0.00)
VRNN
unknown
2.32(14.6)e-2
0.01(0.08)
45.3(0.00)
0.00(0.00)
VRNN finetune
unknown
2.34(14.5)e-2
0.04(0.15)
45.3(0.00)
0.00(0.00)
known 2.32(14.3)e-2
0.01(0.02)
45.2(0.00)
0.00(0.00)
DVBF
unknown 2.43(14.1)e-2
0.01(0.07)
45.3(0.00)
0.00(0.00)
DVBF finetune
unknown
2.35(14.1)e-2
0.01(0.08)
45.3(0.00)
0.00(0.00)
known 3.42(1.30)e-2
0.39(0.34)
5.05(3.57)
0.5(0.34)
KVAE
unknown
3.46(1.36)e22
0.22(0.19)
5.17(3.91)
0.53(0.28)
known 3.58(0.33)e-2
0.00(0.01)
13.7(3.36)
0.07(0.18)
Donà et al
unknown
3.56(0.34)e-2
0.00(0.01)
14.1(3.84)
0.08(0.19)
G DATA DETAILS
In this section, we give the specific data sizes and splits used for training throughout the experiments, as well generation procedures and considerations for each.
For the bouncing balls, we leveraged the PyMunk Physics Engine (www. pymunkk org) to perform simulations under various gravity following (Faccarro al. 2017). For pendulum and mass spring systems, we leveraged the Hamiltonian Dynamics Suite presented in (Botev et al. (02T). The suite's 's default physical parameters were used and friction coefficients were introduced build non-energy conserving systems. For data consistency, we extracted the red color channel of pendulum and mass spring systems to generate gray-scale images.
For 16 experiments, we generated bouncing balls dynamic trajectories on 32 x 32 images. All dynamics are under 16 different gravitational constants, where g = 3 + e ~ U(0, 1) and the direction of each gravity is evenly distributed over the 2D space. We took the trajectories samples at every At = 0.2 intervals. The initial position of the ball is set to a 16 x 16 region centered in the image, and the initial velocity is randomly sampled from [0, 10] on both x and y directions. Each gravity setting has 3,000 samples total. For gravity- data, we used 10 gravity in meta mnta-training 2 in meta-validation, and 4 in meta-testing. We also left out samples from both meta-training and
15

---

Published as a conference paper at ICLR 2023
ineaa-validttion sets (~1500 for each gravity) to evaluate the performance of the model on the known dynamics.
For mixed mixed-physics experiments, generated a mixed--phsiie dataset consisting of bouncing balls under 4 gravity directions, and pendulums and mass springs each with four different values of friction coefficients of 0, 0.05, 0.1, 0.15. The bouncing ball dataset is similar gnavity- experiment except for the gravity directions. Both pendulums and mass springs took the trajectories samples at every At 0.2 intervals. In pendulums, the mass of the particle and the pivot length / = 1. The friction coefficients are chosen in 0, 0.05, 0.1, 0.15. In mass springs, the mass of the particle chosen in O, 0.05, 0.1, 0.15. For each of the three physics, we included three dynamic settings in meta-training and left out one in meta-festing.
0.5, the gravitational constant g -
0.5, the spring force coefficient 2. The friction coefficients are
For the turbulent flow dataset, we generated the tasks given by the instructions and scripts found within the official code repository from (Wang et al., [02T]. Rose- STL Lab/ haab/Dynamicc Adaptation Net Nork We split the seen and unseen buoyancy factors according to the same task split used within their work and directly compare RMSE values based on their implementation and magnitude coefficients.
//::i//gtttt com/
For cardiac electrical dynamics, we generated 3D electrical signal propagation in the heart simulated by the Aliev-Panfiloy model Aliev & Panfilov (1996) heart meshes and total of 12 different tissue parameters (4 on each heart) representing different injury to the heart muscle. This was treated 12 tasks in meta-learning. All 12 tasks appeared in meta-training and -testing, with disjoint time sequences resulting from different external stimulations (meta-training: 300; meta-test: 2,020).
All ball data
be found here: https::/drive..gonn
The
heart data be found here: https:
12857330KKMgbICXDDDZE Oroy
H IMPLEMENTATION DETAILS
In this section, give the specific syper-parameters each experiment over all models, well as resources and considerations for each. All experiments were run on NVIDIA Tesla T4s with 16 GB memory.
H.0.1 ARCHITECTURE FOR META MODELS
The implementation of our proposed meta models is here: https: /ttt::://gttuuu.ue com/ faa-xx-jiannmmtaa.
can be changed by trans_mmaat Specifically in the configuration file, the type of transition function and trans_arge in the model section. The size of the support set is controlled by k_shot in the data section, and the variable/fixed K is set up by changeabl le both training and evaluation section. During meta-testing, the paired support and query sets should be put under eval tags and pred tags in the data section. Detailed hyperparameter settings are shown below.
Meta Model Architecture on Mixed-Physice and Gravity-16
Domain Input: 20 observation timesteps of 32 32 dimensions
Initialization Input: 3 observation timesteps 32 x 32 dimensions
Optimizer: Adam, x 10-4 learning rate
Transition: Gated transition function (GRU-res) / Recurrent Generative Network (RGN-res) Neural Ordinary Differential Equation (NODE)
Batch size: 50
Number of epochs: 200
Training time: 1.5 5 hours
Latent Units: 8
16

---

Published as a conference paper at ICLR 2023
Transition Units: 100
Domain Encoder Filters: [8, 16, 8]
Domain Time Units: [10, 5, I]
Initial Encoder Filters: [8, 16, 8]
Emission Filters: [32, 16, 8, 1]
KL term initialization:
10-2
KI. term set-emhedding: X2 10-2
Meta Model Architecture on Turbulent Flow
Domain Input: observation timesteps of 64 x 64 dimensions
Initialization Input: observation timesteps 64 x 64 dimensions
Optimizer: Adam, x 10-4 learning rate
Transition: Gated transition function (GRU-res) / Recurrent Generative Network (RGN-res) Neural Ordinary Differential Equation (NODE)
Batch size: 20
Number of epochs: 500
Training time: 6 hours
Latent Units: 64
Transition Units: 100
Domain Encoder Filters: [32, 64. 128, 32]
Domain Time Units: [10, 5, 2, I]
Initial Encoder Filters: [64, 128, 256, 64]
Emission Filters: [256, 128, 64, 1]
KL term initialization: À 10-1
KL term set-embedding: A2 10-1
H.0.2 ARCHITECTURE FOR BASELINE MODELS
The implementation of baseline models is here: meta__sat _sam_ssmm The setting of transition function can be changed by trans s_dess__oeeeeeee_______e___eeeee and in the model section. The testing sets should be put under pred_tags in the data section. Detailed hyperparameter settings are shown below.
Baseline Model Architecture
Initialization Input: 3 observation timesteps of 32 32 dimensions
Optimizer: Adam, 5 x 10-4 learning rate
Transition: Gated transition (unction(GRU-res)) / Recurrent Generative Network (RGN-res) Neural Ordinary Differential Equation (NODE)
Batch size: 50
Number of epochs: 200
Training time: 1.3 hours
Latent Units: 8
Transition Units: 100
Initial Encoder Filters: [8, 16, 8]
Emission Filters: [32, 16, 8. 1]
KI. term initialization: X 10-2
17

---

Published as a conference paper at ICLR 2023
H.0.3 ARCHITECTURE FOR DKF
The DKF model (Krishnan et al. 2017) we used is based on this implementation: https:// github. www.cm/m/yy//e http://lloo/pttoooo httppyoorrc-dee-marre-ee ddep-markov-modee
Our implementation is provided in our code. modified the script to perform reconstruction of the observed part of input sequences and prediction of the unobserved parts. Specifically, the output of the correction function was used in decoder in the reconstruction phase, while the transition module was used for prediction. Detailed hyperparameter settings are shown below.
DKF Architecture
Input: observation and 12 prediction timesteps of 32 x 32 dimensions
Optimizer: Adam, 5 x 10 learning rate
Transition: Gated transition function
Batch size: 50
Number of epochs: 200
Training time: 1.2 hours
Encoder Units: [2048, 2048, 100]
RNN Units: 100
Correction Units: 100
Transition Units: 100
Emission Filters: [32, 16, 8, I]
H.0.4 ARCHITECTURE FOR DVBF
The PyTorch implementation of DVBF Karl et al. (017)) we used be found here: https ; //github. com/ gregorsemmles ez bytorcch dvbf Our implementation is provided in our code. We modified this script to perform initial state generation, as highlighted in DVBF's experiments (Karl et al., (017)). We found lacking convergence in more complex dynamics and lower data- availability scenarios, specifically bouncing ball settings. We detail the hyperparameter values chosen experiment with respect to the hyperparameters found in this repository.
DVBF Architecture on Gravity-16
Input: observation and 12 prediction timesteps of 32 x 32 dimensions
Optimizer: Adam, x 10-3 learning rate annealed to x 10-4 over 100 epochs
Delayed KL Weight: between epochs=(5,25), linearly annealed from 0.01 to 1
Batch-size: 25
Number of Epochs: 100
Training time: 15 hours
Number of matrices ox: 8
Code vector dim wy: 8
Feature dim 2 8
State dim s; 1024
RNN Hidden Size: 100
Inference Size: 32
Transition Size: 64
Encoder CNN Filters: [8, 32, 8)
Decoder CNN Filters: [8, 32, 8]
18

---

Published as a conference paper at ICLR 2023
DVBF Architecture Mixed-Physics
Input: observation and 12 prediction timesteps of 32 x 32 dimensions
Optimizer: Adam, x 10 learning rate annealed to x 10 over 100 epochs
Delayed KL Weight: between epochs=[5,25), linearly annealed from 0.01 to 1
Batch-size: 25
Number of Epochs: 100
Training time: 15 hours
Number of matrices ay:
Code vector dim wy:
Feature dim zz:
State dim 8; 1024
RNN Hidden Size: 100
Inference Size: 32
Transition Size: 64
Encoder CNN Filters: [8, 32, 8]
Decoder CNN Filters: [8, 32, 8]
H.0.5 ARCHITECTURE FOR KVAE
The public implementation of VVA FFaccaro et al. 2017) we used can be found here: https //github. com/ simonk amo onn xvae No sigmificant modifications were made the repo primarily just output formatting. When attempting to scale the hyperparameter K (number of mixture components for the LG-SSM matrices), we ran into numerical loss instability and lacking convergence in reconstruction. We detail the hyperparameter values chosen per experiment with respect to the hyperparameters found in this repository.
KVAE Architecture Gravity-16
Input: observation and 12 prediction timesteps of 32 x 32 dimensions
Optimizer: Adam, 7 x 10-3 learning rate exponentially decayed by 0.85 over 20 epochs
Batch-size: 32
Number of Epochs: 80
Training time: 2 hours
K: 3
Latent variable dim dy:
Latent state dim 2, 8
Control dim U, (unused):
RNN Hidden Size: 50
RNN Layers: 2
Encoder CNN Filters: 32, 32]
Decoder CNN Filters: [3, 32, 32, 32]
VAE Architecture on Mixed-Physies
Input: observation and 12 prediction timesteps of 32 x 32 dimensions
Optimizer: Adam, 7 x 10-3 learning rate exponentially decayed by 0.85 over 20 epochs
Batch-size: 32
Number of Epochs: 80
19

---

Published as a conference paper at ICLR 2023
Training time: 2 hours
K: 3
Latent variable dim dy:
Latent state dim žz: 8
Control dim Ue (unused):
RNN Hidden Size: 50
RNN Layers: 2
Encoder CNN Filters: [3, 32, 32, 32]
Decoder CNN Filters: [3, 32, 32, 32]
H.0.6 ARCHITECTURE FOR VRNN
The DVAE code implementation Girin et (2021) VRNN (Chung et al. 2013) we used can be found here: https://git thub. com/ / X 1iioyyBBEET 994 /D/AE/ b ob wwt./mmatterrs/mmmmmmmmmm /dvae/ modellvrrn. pyj The model was modified to take its own reconstructions as inputs to the feature extractor at every timestep after the true sequence observation period. We detail the hyperparameter values chosen per experiment with respect to the hyperparameters found this repository.
VRNN Architecture Gravity-16
Input: observation and 12 prediction timesteps of 32 x 32 dimensions
Optimizer: Adam, 1 x 10-3 learning rate
Batch-size: 50
Number of Epochs: 80
Training time: 8 hours
Latent state dim =e 32
Dropout probability: 0.2
Dense X Size: 128
Dense Z Size: 128
Dense H(X)-Z Size: 128
Dense H(Z)-X Size: 128
Dense H(Z) Size: 128
RNN Layers: 2
RNN Dim: 64
Beta coefficient: 1.0
Activation: Leak)ReLU(1.0)
VRNN Architecture on Mixed-Physics
Input: observation and 12 prediction timesteps of 32 x 32 dimensions
Optimizer: Adam, x 10-3 learning rate
Batch-size: 50
Number of Epochs: 80
Training time: 8 hours
Latent state dim द5: 32
Dropout probability: 0.2
Dense X Size: 128
Dense Z. Size: 128
20

---

Published as a conference paper at ICLR 2023
Dense H(X)-Z Size: 128
Dense H(Z)-X Size: 128
Dense H(Z) Size: 128
RNN Layers: 2
RNN Dim: 64
Beta coefficient: 1.0
Activation: LeakyReLU(1.0))
21


Reviews:
[{"review_id": "xLvP_Fmrpz", "reviewer": "Reviewer_wdac", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes a framework for sequential latent variable models for high-dimensional data few-shot forecasting problems. The model encodes the support set of few-short samples into a distribution of parameters of met -models. The model is optimized in a VAE framework and used for forecasting tasks. Experiment results on image sequence data simulated using diverse dynamics show strong performance of the model against baselines. ", "summary_of_the_review": "Few-shot forecasting is an interesting application for existing sequential latent variable models. However, the experiment settings are flawed and comparisons against baseline approaches are not fair. Important meta-learning baselines including MAML are missing from the experiment results. There are also minor concerns about the writing quality of the paper.", "strength_and_weaknesses": "Strengths:\n\nThe problem setting of few-short forecasting for time-series data is important. I also appreciate the motivation of using latent representation for modeling high-dimensional time-series data. The set of models being considered in the experiments is also extensive.\n\n\nWeakness:\n\n1. Despite considering an extensive set of models, the experiments are not thorough and comparisons are not fair. The training and evaluation settings of the proposed model are essentially meta-learning settings. Why popular meta-learning frameworks like MAML [1] and Probablistic MAML[2] are not considered? I do not think there’re big challenges to extending MAML frameworks to sequential LVMs. For the proposed framework, both the evaluation setting and training settings are episodic where the support set is fed to the model in each training episode. For the two baseline frameworks being compared, the support set is not present during training.\n2. Despite the images themselves being high-dimensional, the actual dynamics that generate the image sequences are all quite simple and low-dimensional. I’m not fully convinced the set of experiments shows the model’s capacity to handle real high-dimensional sequential data. As long as the model could uncover the simple low-dimensional ground-truth dynamics from the support set dynamics and generate images of high quality, it should perform well.\n3. The technical contribution of the work is incremental. It claims it proposes a framework for few-shot forecasting for sequential data but I do not see significant differences between the proposed framework and existing sequential latent variable models. More specifically, I think the difference between embedding c and latent variable z is only nominal with c being viewed as a conditional latent variable. Actually, the model’s code in the supplementary material also shows no significant difference. This is not necessarily bad. However, without extensive and convincing experiment results, I think the contribution of simply extending existing methods to a new problem setting is very limited.\n4. I also have some concerns about the quality writing and clarity of the paper. Please refer to Clarity, Quality, and Novelty.\n\n\n[1] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. \"Model-agnostic meta-learning for fast adaptation of deep networks.\" International conference on machine learning. PMLR, 2017.\n\n[2] Finn, Chelsea, Kelvin Xu, and Sergey Levine. \"Probabilistic model-agnostic meta-learning.\" Advances in neural information processing systems 31 (2018).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nOui7hcS-O6", "reviewer": "Reviewer_EkYh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors suggest few-shot learning of sequential latent variable models. The paper considers an amortized variational inference setup. This is obtained by an average-set-pooling/encoding of the support set to learn a context variable for each time series that drives the deterministic dynamics of the latent states. The method is illustrated for meta-learning image sequences and cardic electrical dynamics, where it performs favourable compared to sequential latent variable models that are not trained based on a meta-learning approach. \n\n---\nUpdate post-rebuttal:\nI increased my score to a weak accept after reading the authors' response and the other reviews. However, I still think this is a borderline case.\nThe additional experiments provide empirical evidence that a meta-learning setting of DKF following the Sequential Neural Process (SNP) approach yields inferior forecasting performance compared to the suggested method.\n\n---", "summary_of_the_review": "The paper addresses few-shot learning for sequential latent variable models. My main concern is that it is not clear to me how this improves on sequential neural processes. For the latter, the context observations can even by varying through time, and do not have to be the support set? There could be a misunderstanding on my side, and I will consider increasing my score if the authors address these concerns.", "strength_and_weaknesses": "Strengths:\n- The dynamic setting has received less attention for meta-learning compared to the iid case, so this work fills a gap in the current literature.\n- Empirical results suggest that the proposed method outperforms previous benchmark models significantly for a variety of applications/data sets.\n\nNegatives:\n- I feel that the approach is somewhat incremental from a methodology perspective in that it is an extension from previous work in few-shot applied to the iid setting (e.g., Garnelo et al., 2018), in combination with variational inference techniques for sequential latent variables.\n- It is not clear to me how is this different to the sequential neural process setting and why it cannot be included here as a comparison? The other benchmark models presented in this work are similarly not designed for a few-shot setup.\n\nComments/Actionable Feedback:\n- How exactly is the variational approximation of c given by the query and support sets constructed? Are the embedded features of the query set just incorporated into the averaging function? Is there some weighting of the query vs support set?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kdRCXl7zkB", "reviewer": "Reviewer_Chgq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Sequential Latent Variable Models for Few-Shot High-Dimensional Time-Series Forecasting introduces a sequential latent variable model for meta-learning various high-dimensional time-series. This model allows for learning diverse dynamics, and adapting in a few-shot setting during evaluation, evaluating on a variety of benchmark datasets against known models from prior literature.", "summary_of_the_review": "This paper introduces an interesting method, and both the background writing and method discussion make clear the relevant materials, and why this method is interesting. However the scope of the experiments, along with the presentation of the experiments, currently limit the potential interest of this work.", "strength_and_weaknesses": "Strengths:\n\nStrong presentation of the method, background, and related work. Logical extension to meta/few-shot setting, with direct references to how and why the architecture is logical. Results compare favorably with benchmarks, benchmarks are pretty thoroughly set up and tested on the chosen datasets. \n\nWeaknesses:\n\nFigure 4 color scheme is pretty difficult to discern, specifically two slightly different shades of green. Generally Figures 2, 3, and 4 (and to some extent 5) have many graphical results which should probably be condensed to a table, with a small example of frames from 1 or 2 chosen methods. More extensive videos/gifs can be linked in a project page, or in drive links like those provided in the appendix. Generally it would be nice to see improvements to the experimental results presentation in this section.\n\nThe comparisons are thorough but FiVO seems to be missing - I do not think a full comparison is totally necessary to show the value here, but some discussion or citation is probably in order given the close relation of FiVO to the related benchmark models tested https://arxiv.org/abs/1705.09279 .\n\nAddition of another experiment beyond the existing would broaden the pool of potential readers. Particularly there are a variety of benchmarks in the related work (midi music, speech, handwriting) that could make compelling comparison points. Despite the discussion of multi-dimensional data, it is worth considering or testing low-dimensional forecasting, as the formulation of few shot dynamics has direct relevance in that subfield, and there are many benchmarks in GluonTS which DeepAR used. Multi-speaker TTS (or audio modeling ala RAVE https://arxiv.org/abs/2111.05011 ) is another compelling area, but may be out of scope due to the limits of the reviewer/author feedback window. Online handwriting is at least multi-dimensional, and has many prior applications related to stylistic generation (see VRNN or https://arxiv.org/abs/2110.02891 ). The experiments in the paper show the value of the method, but the scope of experiments seems narrow compared to some of the prior work, making it difficult to asses the full impact in the design choices of this method.\n\nThe cardiac electrical dynamics seems to be an interesting task, but hard to understand or compare due to limited space. Consider redistributing importance (in terms of writing space allocated) between these experiments. However, a problem may arise given the comparison method - what do *failures* look like for this model in the cardiac electrical case? Are they disastrous, or tolerable for the setting? Given the strong baseline approach (PVH), the errors from this method are probably qualitatively different - discussion of what types of failures could occur seem important for practical applications.\n\nIt would be nice to mention use of exemplar based methods in static VAE settings, such as Exemplar VAE https://proceedings.neurips.cc/paper/2020/hash/63c17d596f401acb520efe4a2a7a01ee-Abstract.html , probably as related work.\n\nThe primary points I would like to see addressed are:\n1) modified presentation of results in figures 2/3/4/5, with a focus on moving numerical results into tables instead of bar charts for easier comparison - as it stands these results are difficult to parse.\n2) Further work on the experiments - either introducing an additional task with more common ground with the related work, or a deeper dive into the results on cardiac electrical dynamics, what the results mean, and what the potential impact of those results could be on the relevant application field.\nAddressing these points (either, or both - in whole or in part) would improve the quality of the presentation, and justify a higher score.\n\nAs mentioned in the paper, comparison to Lagrangian and Hamiltonian networks could be of interest if the focus remains on physics-based experimental benchmarks.\n\nSome minor typos to fix:\n\"abstracts a a latent\", paragraph 2\n\"Deepar\" -> \"DeepAR\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%