Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new model for unsupervised event detection, which has the following three technical contributions:
- It proposes individual criterion and mechanisms to detect expected and unexpected observations based on the comparison of prior and posterior latent representations. (Novel)
- It uses the above criterion to control the update rate of the latent states in a hierarchy of RNN layers. (Somewhat similar to VTA [Kim et al., 2019])
- The paper includes some interesting experiments to support the effectiveness of the proposed method. (But not convincing enough)


main_review: Strengths: In my knowledge, the proposed method of learning hierarchical temporal structures by measuring prior and posterior belief states is new in the area of unsupervised representation learning for video data.

Weaknesses: My major concern is about the insufficient experimental results.
- The model is compared with VTA only on the Moving Ball dataset, which in my view, is not enough to validate the superiority of the proposed model. The authors may consider to compare with VTA on a more challenging dataset such as Bouncing Balls. I believe this dataset can better support the claim in this paper, because it has more obvious event changes represented by the interactions of multiple balls and is also used in the work of VTA.
- The model is not compared with any existing approaches on the other three datasets. 
- How does the rollout result in Fig 5, especially the color of the object, support the claim in the caption that "the produced rollouts illustrate model’s ability to...produce accurate...jumpy rollouts"?

Other concerns:
- How to determine the number of layers if we do not know how many latent factors (e.g., color, shape, floor...) there are in the environment?
- Although the authors claim in the appendix and the hyperparameters in Criterion U are robust across datasets, it is still not clear to me how to tune these hyperparameters. It would be good if the authors can give a sensitivity analysis on them. 
summary_of_the_review: This paper explores an interesting problem of unsupervised representation learning in videos. Although the hierarchical architecture based on latent variables is very similar to the temporal abstraction structure in VTA [Kim et al., 2019], I think the proposed criteria for event detection are novel and reasonable. However, my major concern is about the insufficient experimental results (see my comments above), and so I give this paper a Weak Reject at this moment.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Variational Predictive Routing (VPR) models continuous data as a hierarchical renewal process. More specficially, it is a hierarchical generative model that organizes the temporal hierarchy based on the rates of change of its latent representations. The core contribution hereby is that event detection is formulated as a comparison between semantically consistent KL divergences (or, in the case of CU, a moving average of KL divergences). VPR is time agnostic and can model a wide variety of temporal structures. Compared to a baseline method (VTA), VPR detects changes in synthetic datasets with high accuracy.
main_review: I like the general idea of modeling change detection as the difference in divergences, I think this indirection is novel.
I miss some more details on the following things:
* What is the generative model for the change detection? To my understanding, change detection is modeled as an interplay between the inference  model and the generative model, but shouldn't there be some part in the generative model alone that is handling 'changes'?
* Can you expand on the "non markovian property' of the model (paragraph before eq 5)? I assume this makes s_t independent from all x_t', t' > t?
* A pseudo algorithm would be very helpful to see exactly where/how the decision to stop propagating information upwards happens.
* It would be great (but admittingly hard) to have more expressive datasets.
* Related to datasets: In a real world setting, I believe that one can have maybe a bit of time to look at a prefix part of a sequence to be handled, and needs to 'extract' the core properties of that sequence with respect to event onsets. How could that be handled? I assume this becomes some sort of meta learning?
* I think you should cite "Neural Sequnce Chunker" by J. Schmidhuber and Unsupervised Real-Time Control through Variational Empowerment  by M. Karl et al.
summary_of_the_review: I believe the paper has a very nice core idea, and a well thought out 'harness' around that idea. I miss more conceptualized explanations, and would have loved more experiments (and also more comparisons to comparable models).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Variational Predictive Routing (VPR), a spatiotemporal deep generative model for hierarchical video representation learning. It is motivated by a desire to implement predictive routing---a model of cognition---within a deep generative modeling framework. VPR automatically detects expected and unexpected changes at each level of the latent hierarchy, which it uses to “route” bottom-up and top-down signals. Interactions between bottom-up (from observations) and top-down signals (from the prior) are crucial for implementing this hierarchy and are a core aspect of the predictive routing framework. Each level of the hierarchy learns a time-agnostic transition model for a distinct dynamic latent factor. By design, “slower” dynamic features are modeled at higher levels of the hierarchy and “faster” dynamic features are modeled at lower levels. Experiments on synthetic video datasets validate the success of VPR at implementing predictive routing within a variational representation learning framework.
main_review: ########################################################################## 

Pros:

-I think that exchanging ideas between neuroscience and machine learning is a great way to develop novel generative models with new capabilities such as VPR. I felt as if this work was also indirectly motivated by a desire to improve our knowledge about how to design better world models, which is a really important topic for model-based reinforcement learning.  This paper has the potential to be of significant interest to that community as well. 

-I was surprised at how cleanly VPR is able to learn the dynamics of distinct latent factors in each level, as evidenced clearly by the qualitative results. Despite the fact that the datasets are by and large toy-like and synthetic, this was still great to see.

-The analytical experiments are well-thought-out and validate the role that the various components of the VPR play (e.g., in Figures 3,4, and 7). 

-The presentation is solid---I provide some suggestions to improve it below. The visualized qualitative results are really helpful for quickly grasping how VPR works and what it is capable of learning. 

##########################################################################

Cons:  

-The introduction motivates this work as one which can be “beneficial for predicting future sequences”, and the beginning of the experiments section even lists “future event prediction” as a task to be studied. However, I was disappointed to find this is missing from the model design (as explained in Section 5: Event duration estimation) and is missing from the experiments.

-I found it strange that TD-VAE  by Gregor et al. (which is cited in the introduction) is not discussed again in more detail under “Hierarchical generative models” in Section 2. This model learns a jumpy state-space model, supports a hierarchy of latent variables with transition functions operating at different time scales, and can handle future prediction. On the surface level, it does not have the explicit event detection mechanism of VPR, but it still seems capable of learning a transition function over “important states”. 

-I would have liked to see at least one comparison between TD-VAE and CW-VAE and VPR (if/when able to support future prediction) to judge VPR’s ability to be useful as a world model and perhaps to highlight where it excels whereas these previous methods do not. The experiment comparing fixed-interval versions of VPR in terms of feature disentanglement (Figure 7) is illuminating but is not really satisfying for this since it is not evaluating on long-term prediction. Without such an experiment and a comparison against these rigorous baselines, the maturity of the work still feels to be at somewhat of an exploratory stage.

-I would have liked to see more analysis of the Miniworld Maze results as it is the most visually challenging environment and the dynamics of the changing latent factors are less hand-engineered. Comparing VPR against VTA here would be helpful.

-Section 3 needs some work to make the presentation more clear:

--I couldn’t understand Figure 2. Figure 8 in the appendix was much more clear. 

--I didn’t see where $p(o_t | s_t^{1:N})$ is defined.

--The usage of $t$ and $\tau$ got very confusing.  It would be good to stick with one throughout the paper, if possible.

--I believe the posterior models should be conditioned on $s_{\tau -1}^{>n}$, not $s_{\tau}^{>n}$. 

-A discussion on how $\gamma$ and $\tau_w$ are selected, and how sensitive the model is to these variables, is missing. These values should be affected by the size of the latent dimension, which in turn affects the value of the KL used for event detection.

##########################################################################

Some suggestions for improvement:

-In Figure 3, use a), b), c), and d) to describe sub-figures

-In figure captions where the figures have error bars or confidence intervals, tell how many seeds/runs are used in the caption
summary_of_the_review: I currently believe this paper is borderline and I am leaning to recommend to reject. 
This is because I think the way the paper frames VPR as being comparable with prior methods like CW-VAE and as a potentially-useful framework for long-term prediction is promising, yet more work is needed (described in my review) to take this work from an exploratory stage to ready for publication. I look forward to hearing from the authors during the rebuttal period on this matter.
==========================================================================
Update after author response: The authors have provided satisfactory responses to my concerns and revised the paper with extra results and discussions accordingly. I have increased my score to a 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces VPR, an unsupervised hierarchical latent variable model. The latent representation is organized into a temporal hierarchy, with deeper layers representing features with slower rates of change. This hierarchy is enabled by an event detection mechanism that finds expected events in the input using VPR's internal transition model, and finds unexpected events through changes in the latent distributions at each level of the hierarchy. The paper demonstrates VPR's performance on unsupervised event boundary discovery, and how it is able to learn disentangled representations, on a variety of synthetic video datasets.
main_review: strengths
---------
VPR represents a significant advance over previous models like CW-VAE (which requires a fixed "event detection" interval) and VTA (where event detection must be learned separately).

For the most part the method is clearly explained; Figs 1 and 2 are very helpful.

The 1D synthetic dataset is useful for understanding the method in a simple setting. Do similar results to Fig. 4 emerge from the other datasets as well?

The layerwise rollouts are an interesting way to probe the representations at each layer.


concerns
--------
I still don't fully understand the event prediction (specifically CE). Though the paper does say in words what D_{st} and D_{ch} are (which is much appreciated), the fact that their comparison is a reasonable way to define a predictable event is not clear to me. Does D_{ch} represent a difference we would expect from the structure of the model with/without seeing new data? Why does it make to compare D_{st} to this? Spelling this out a bit more clearly would help explain this crucial piece of the method.

How is the F1 score computed for change detection? Do you arbitrarily define successive events (in both the data and the model) as 0->1->0->1->... and then compute F1 on these binary sequences? Would be useful to spell this out more clearly.

The current figure layout is convenient space-wise, but makes following the results harder than it needs to be.

Fig 4 legend is hard to parse. 
* "CE" appears to be "CE with CU enabled"; what then is CE+CU?
* "CU disabled" appears to be "CE with CU disabled"; what then is CU? CU only, but in the context of "CE with CU enabled"?
* why is there no CU curve in Fig 4A?
* why do 4A and 4C show different models? would make more sense to show the same models under the noise and no-noise condition

Fig 5A - do you have an intuition for why the color changes in L2 on every frame? Given the frequency of these events (on wall bounce or otherwise with probability 0.1) I would have expected the model to learn slower transitions between colors.

Experiments are run with other baselines, but inconsistently. Running VTA on the 1D synthetic data could be instructive, and would especially be interesting to see how it performs under different noise levels. It should also be easy enough to add a CW-VAE curve to Fig. 3B. This will perform terribly, but that also opens up an opportunity to revisit the fundamental differences between VPR and CW-VAE and why the new model is an advance. I think both rollouts and random samples should be added in appendix figures for the CW-VAE and VTA. In general, the results section could start off with an explanation of the datasets (as it does already) AND an explanation of the baseline models, and why those models were chosen. That gives a bit more structure to (and motivation for) the comparisons made later on.

How will this model behave when there are slow, continuous changes in high-level features, rather than discrete changes? For example, what if colors slowly cycled through hue space in the Moving Ball dataset (much more slowly than position)? Would kind of features would VPR learn at L2? Not suggesting this as an experiment to perform, but might be an interesting direction to address in the discussion.

summary_of_the_review: This paper presents a novel, well-motivated method for inferring temporal hierarchy in latent representations of sequence data. I lean towards accepting this paper, and would be willing to further increase my score if some of the concerns above (especially regarding the experiments) are addressed in revision.
review_title: null

Meta-review: Thanks for your submission to ICLR.

This paper considers a variational inference hierarchical model called Variational Predictive Routing.

Prior to discussion, several reviewers were on the fence about the paper, most notably having concerns about some of the experimental results as well as various clarity issues throughout the paper.  However, the authors did a really nice job addressing many of these concerns.  Ultimately, several of the reviewers updated their scores, leading to a clear consensus view that this paper is ready for publication.  We really appreciate your effort in providing additional details and results.

Please do keep in mind the concerns of the reviewers when preparing a final version of the manuscript.
Decision: Accept (Poster)