Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Sequential Latent Variable Models for Few-Shot High-Dimensional Time-Series Forecasting introduces a sequential latent variable model for meta-learning various high-dimensional time-series. This model allows for learning diverse dynamics, and adapting in a few-shot setting during evaluation, evaluating on a variety of benchmark datasets against known models from prior literature.
strength_and_weaknesses: Strengths:

Strong presentation of the method, background, and related work. Logical extension to meta/few-shot setting, with direct references to how and why the architecture is logical. Results compare favorably with benchmarks, benchmarks are pretty thoroughly set up and tested on the chosen datasets. 

Weaknesses:

Figure 4 color scheme is pretty difficult to discern, specifically two slightly different shades of green. Generally Figures 2, 3, and 4 (and to some extent 5) have many graphical results which should probably be condensed to a table, with a small example of frames from 1 or 2 chosen methods. More extensive videos/gifs can be linked in a project page, or in drive links like those provided in the appendix. Generally it would be nice to see improvements to the experimental results presentation in this section.

The comparisons are thorough but FiVO seems to be missing - I do not think a full comparison is totally necessary to show the value here, but some discussion or citation is probably in order given the close relation of FiVO to the related benchmark models tested https://arxiv.org/abs/1705.09279 .

Addition of another experiment beyond the existing would broaden the pool of potential readers. Particularly there are a variety of benchmarks in the related work (midi music, speech, handwriting) that could make compelling comparison points. Despite the discussion of multi-dimensional data, it is worth considering or testing low-dimensional forecasting, as the formulation of few shot dynamics has direct relevance in that subfield, and there are many benchmarks in GluonTS which DeepAR used. Multi-speaker TTS (or audio modeling ala RAVE https://arxiv.org/abs/2111.05011 ) is another compelling area, but may be out of scope due to the limits of the reviewer/author feedback window. Online handwriting is at least multi-dimensional, and has many prior applications related to stylistic generation (see VRNN or https://arxiv.org/abs/2110.02891 ). The experiments in the paper show the value of the method, but the scope of experiments seems narrow compared to some of the prior work, making it difficult to asses the full impact in the design choices of this method.

The cardiac electrical dynamics seems to be an interesting task, but hard to understand or compare due to limited space. Consider redistributing importance (in terms of writing space allocated) between these experiments. However, a problem may arise given the comparison method - what do *failures* look like for this model in the cardiac electrical case? Are they disastrous, or tolerable for the setting? Given the strong baseline approach (PVH), the errors from this method are probably qualitatively different - discussion of what types of failures could occur seem important for practical applications.

It would be nice to mention use of exemplar based methods in static VAE settings, such as Exemplar VAE https://proceedings.neurips.cc/paper/2020/hash/63c17d596f401acb520efe4a2a7a01ee-Abstract.html , probably as related work.

The primary points I would like to see addressed are:
1) modified presentation of results in figures 2/3/4/5, with a focus on moving numerical results into tables instead of bar charts for easier comparison - as it stands these results are difficult to parse.
2) Further work on the experiments - either introducing an additional task with more common ground with the related work, or a deeper dive into the results on cardiac electrical dynamics, what the results mean, and what the potential impact of those results could be on the relevant application field.
Addressing these points (either, or both - in whole or in part) would improve the quality of the presentation, and justify a higher score.

As mentioned in the paper, comparison to Lagrangian and Hamiltonian networks could be of interest if the focus remains on physics-based experimental benchmarks.

Some minor typos to fix:
"abstracts a a latent", paragraph 2
"Deepar" -> "DeepAR"
clarity,_quality,_novelty_and_reproducibility: This work seems original, though there are similar efforts in related areas, this method both unifies and justifies the approach with background/prior work. The addition of few shot setups in dynamics learning is of broad interest, and well supported by the proposed model.

"We then present the first framework of few-shot forecasting for high-dimensional time-series" in the abstract is a bit of an overclaim (given extensive prior work on few, one, and zero shot learning for video and audio generation, as well as the broader relationship of prompting and transformers to few shot settings), and in this case not necessary for the core method to be novel and of interest.

The appendix is extensive, and related links are helpful. Code is included, and the effort in terms of reproducibility of the approach is top notch. In addition the authors detail what packages were used for the benchmark comparisons, which is critical to any followup comparisons.
summary_of_the_review: This paper introduces an interesting method, and both the background writing and method discussion make clear the relevant materials, and why this method is interesting. However the scope of the experiments, along with the presentation of the experiments, currently limit the potential interest of this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors suggest few-shot learning of sequential latent variable models. The paper considers an amortized variational inference setup. This is obtained by an average-set-pooling/encoding of the support set to learn a context variable for each time series that drives the deterministic dynamics of the latent states. The method is illustrated for meta-learning image sequences and cardic electrical dynamics, where it performs favourable compared to sequential latent variable models that are not trained based on a meta-learning approach. 

---
Update post-rebuttal:
I increased my score to a weak accept after reading the authors' response and the other reviews. However, I still think this is a borderline case.
The additional experiments provide empirical evidence that a meta-learning setting of DKF following the Sequential Neural Process (SNP) approach yields inferior forecasting performance compared to the suggested method.

---
strength_and_weaknesses: Strengths:
- The dynamic setting has received less attention for meta-learning compared to the iid case, so this work fills a gap in the current literature.
- Empirical results suggest that the proposed method outperforms previous benchmark models significantly for a variety of applications/data sets.

Negatives:
- I feel that the approach is somewhat incremental from a methodology perspective in that it is an extension from previous work in few-shot applied to the iid setting (e.g., Garnelo et al., 2018), in combination with variational inference techniques for sequential latent variables.
- It is not clear to me how is this different to the sequential neural process setting and why it cannot be included here as a comparison? The other benchmark models presented in this work are similarly not designed for a few-shot setup.

Comments/Actionable Feedback:
- How exactly is the variational approximation of c given by the query and support sets constructed? Are the embedded features of the query set just incorporated into the averaging function? Is there some weighting of the query vs support set?
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. The presentation is generally clear, and references are adequately cited. 
summary_of_the_review: The paper addresses few-shot learning for sequential latent variable models. My main concern is that it is not clear to me how this improves on sequential neural processes. For the latter, the context observations can even by varying through time, and do not have to be the support set? There could be a misunderstanding on my side, and I will consider increasing my score if the authors address these concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work proposes a framework for sequential latent variable models for high-dimensional data few-shot forecasting problems. The model encodes the support set of few-short samples into a distribution of parameters of met -models. The model is optimized in a VAE framework and used for forecasting tasks. Experiment results on image sequence data simulated using diverse dynamics show strong performance of the model against baselines. 
strength_and_weaknesses: Strengths:

The problem setting of few-short forecasting for time-series data is important. I also appreciate the motivation of using latent representation for modeling high-dimensional time-series data. The set of models being considered in the experiments is also extensive.


Weakness:

1. Despite considering an extensive set of models, the experiments are not thorough and comparisons are not fair. The training and evaluation settings of the proposed model are essentially meta-learning settings. Why popular meta-learning frameworks like MAML [1] and Probablistic MAML[2] are not considered? I do not think there’re big challenges to extending MAML frameworks to sequential LVMs. For the proposed framework, both the evaluation setting and training settings are episodic where the support set is fed to the model in each training episode. For the two baseline frameworks being compared, the support set is not present during training.
2. Despite the images themselves being high-dimensional, the actual dynamics that generate the image sequences are all quite simple and low-dimensional. I’m not fully convinced the set of experiments shows the model’s capacity to handle real high-dimensional sequential data. As long as the model could uncover the simple low-dimensional ground-truth dynamics from the support set dynamics and generate images of high quality, it should perform well.
3. The technical contribution of the work is incremental. It claims it proposes a framework for few-shot forecasting for sequential data but I do not see significant differences between the proposed framework and existing sequential latent variable models. More specifically, I think the difference between embedding c and latent variable z is only nominal with c being viewed as a conditional latent variable. Actually, the model’s code in the supplementary material also shows no significant difference. This is not necessarily bad. However, without extensive and convincing experiment results, I think the contribution of simply extending existing methods to a new problem setting is very limited.
4. I also have some concerns about the quality writing and clarity of the paper. Please refer to Clarity, Quality, and Novelty.


[1] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." International conference on machine learning. PMLR, 2017.

[2] Finn, Chelsea, Kelvin Xu, and Sergey Levine. "Probabilistic model-agnostic meta-learning." Advances in neural information processing systems 31 (2018).
clarity,_quality,_novelty_and_reproducibility: Novelty: 

The problem setting and application of sequential latent variable models to few-shot forecasting is super interesting. The overall structure of the presentation and writing is easy to follow.


Quality and Clarity:

Many details of the writing are not satisfying, including the following concerns:
1. In Equation 4 and Equation 7, density functions of p(c) and p($\theta$) are missing in the integrand.
2. Experiments are conducted on forecasting. Nowhere in the paper shows how the forecasting results are obtained. Are the forecasting results obtained by sampling from the variational posterior distribution?
3. Section 3 claims two unifying frameworks for existing sequential latent variable models. For the sake of clarity and self-containedness, the author should consider including, at least in the supplementary material, examples of how existing sequential LVMs can be interpreted under the proposed unifying framework.


I have no concerns about the reproducibility of the experiment results.

summary_of_the_review: Few-shot forecasting is an interesting application for existing sequential latent variable models. However, the experiment settings are flawed and comparisons against baseline approaches are not fair. Important meta-learning baselines including MAML are missing from the experiment results. There are also minor concerns about the writing quality of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose Sequential Latent Variable Models (SLVM) for Few-Shot High-Dimensional Time-Series Forecasting, a method to learning diverse dynamics with meta-learning. The ideas will be of interest to the community, the presentation is strong, and the experiments thorough. The only minor issue is that the fundamental idea is only relatively novel, but on the balance this is a good paper.
justification_for_why_not_higher_score: The limited technical novelty likely precludes it from receiving an oral relative to other papers.
justification_for_why_not_lower_score: Particularly after the rebuttal, the experiments section is now quite strong, and is of interest to the community.
Decision: Accept: notable-top-25%