Variational Predictive Routing with Nested Subjective Timescales
Authors: Alexey Zakharov
Qinghai Guo
Zafeirios Fountas
Venue: ICLR 2022 Poster (2022)
Keywords: Hierarchical temporal abstraction
event discovery
hierarchical generative models
variational inference

Abstract: Discovery and learning of an underlying spatiotemporal hierarchy in sequential data is an important topic for machine learning. Despite this, little work has been done to explore hierarchical generative models that can flexibly adapt their layerwise representations in response to datasets with different temporal dynamics. Here, we present Variational Predictive Routing (VPR) – a neural probabilistic inference system that organizes latent representations of video features in a temporal hierarchy, based on their rates of change, thus modeling continuous data as a hierarchical renewal process. By employing an event detection mechanism that relies solely on the system’s latent representations (without the need of a separate model), VPR is able to dynamically adjust its internal state following changes in the observed features, promoting an optimal organisation of representations across the levels of the model’s latent hierarchy.  Using several video datasets, we show that VPR is able to detect event boundaries, disentangle spatiotemporal features across its hierarchy, adapt to the dynamics of the data, and produce accurate time-agnostic rollouts of the future. Our approach integrates insights from neuroscience and introduces a framework with high potential for applications in model-based reinforcement learning, where flexible and informative state-space rollouts are of particular interest.

Published as a conference paper at ICLR 2022
VARIATIONAL PREDICTIVE ROUTING WITH NESTED SUBJECTIVE TIMESCALES
Alexey Zakharov
Qinghai Guo
Zafeirios Fountas
Huawei Technologies London, UK
Huawei Technologies
Huawei Technologies London, UK
Shenzhen, China
ABSTRACT
Discovery and learning of an underlying spatiotemporal hierarchy in sequential data is important topic for machine learning. Despite this, little work has been done to explore hierarchical generative models that can flexibly adapt their layerwise representations in response to datasets with different temporal dynamics. Here, we present Variational Predictive Routing (VPR) neural probabilistic inference system that organizes latent representations of video features in temporal hierarchy, based on their rates of change, thus modeling continuous data as a hierarchical renewal process. By employing an event detection mechanism that relies solely on the system's 's latent representations (without the need of separate model), VPR is able dynamically adjust its internal state following changes in the observed features, promoting an optimal organisation of representations across the levels of the model's latent hierarchy. Using several video datasets, we show that VPR is able to detect event boundaries, disentangle spatiotemporal features across its hierarchy, adapt to the dynamics the data, and produce accurate time- agnostic rollouts of the future. Our approach integrates insights from neuroscience and introduces a framework with high potential for applications in model-based reinforcement learning, where flexible and informative state-space rollouts of particular interest.
INTRODUCTION
key theoretical benefit of hierarchically-ssrrettui latent models is the ability to learn spatial and temporal features that are characterised by an increasing level of abstraction in deeper layers. In most current implementations
such models, state transitions are realized using recurrent neural networks that operate at fixed rates determined by the timescale of the recorded data (Wichers et al. 2018) Castrejon et al. 2019 Kumar et al. 2019: Wu et al., (022]. Though quite successful, this approach neglects the role of the transition rates in the process of learning spatiotemporal representations across the model's latent hierarchy. Furthermore, is computationally inefficient as more abstract features tend to remain static for longer periods than features represented in lower levels. This redundancy in state transitions introduces noise during both inference and future predictions, which is particularly detrimental to the learning of long-term dependencies (Bengio al. 1994).
Having the ability to update the representation of these high-level features only when change occurs in the environment can save significant computational resources and minimize the effect of accumulated noise in state transitions. Such 'event- based' hierarchical model can also be beneficial for predicting future sequences (e.g. planning in model-hased reinforcement learning) as the same future point time can be reached with shorter and more accurate state rollouts (Pertsch et al.) 2020 Zakharov et al.) 2021). These benefits have recently prompted a number of proposed models that either perform transitions with temporal jumps of arbitrary length (Koutnik al. 2014 Buesing et al.) 2018 Gregor et al.) 2018; Saxena et al. 2021), or aim to identify significant events (or key-frames) in
sequential data and model the transitions between these events (Schmidhuberi 1991 Chung et
2017, Neitz et al. 2018 J Jayaraman et al., 20188 Shang et al. 2019 Kipf et al. 2019 Kim et al.
2019
Pertsch et al.. 2020 Zakharov et al. 2021).
The large variety of different event criteria defined in these studies demonstrates the lack of widely established definition of this concept in this literature. For instance, important events are either
"Corresoonding author; fei ri fountas@huawei.com

---

Published as a conference paper at ICLR 2022
selected as the points in time that contain maximum information about full video sequence (Pertsch iet al. 2020), about the agent's s actions (Shang et al. 2019), as the most predictable (Neitz et al. 2018 Jayaraman et al., 2018) or the most surprising (Zakharov et al., 202I) points in time. A clearer picture can be drawn when viewing events from the perspective of cognitive psychology. where events defined as segments of time "conceived by an observer to have a beginning and an end" (Zacks & Tversky (2001). Neuroimaging evidence suggests that the human brain segments continuous experience hierarchically in order to infer the structure of events, with timescales that increase from milliseconds to minutes across nested hierarchy in the cortex (Baldassano et al. 2017) while their boundaries tend to track changes in scene features and mental representations (Hard et al. 2006 Kurby& & Zacks 2008).
The processing of changes in the brain's hierarchical representations can be mathematically described using the framework of predictive coding (PC). According to PC, cortical representations of the world's state are constantly compared against new sensory information in multiple local inference loops that communicate via prediction error signals (Mumford 1992. Rao & Ballard 1999 Friston 2010). In contemporary view of this framework, called predictive routing, inference is realized via bottom-up propagation of sensory signals (rather than errors), which can be blocked by top- down predictions if considered uninformative (Bastos et al. 2020). This simple connection between salient representation changes and PC has led studies to propose plausible mechanisms for subjective experience of time (Roseboom et al. 2019) and the formation of episodic memories (Fountas et al. 2021) based on the detection of such changes.
Inspired by these insights, we propose a variational inference hierarchical model, termed variational predictive routing (VPR), that relies on change detection mechanism to impose nested temporal hierarchy on its latent representations. We implement this model the domain of video using recurrent neural networks and amortised inference and show that our model is able to detect changes in dataset features even in early learning stages, regardless of whether they are expected or unexpected changes, with significant benefits for the learnt latent representations.
In summary, we make the following contributions. (a) We propose mechanism that can detect the onset of both expected and unexpected events individually. (b) We show that event boundary discovery can rely on layerwise disentangled representations without the need of dedicated learning mechanism. (c) We formulate a generative model (VPR) that integrates various insights from neuroscience and uses the detection mechanism to learn hierarchical spatiotemporal representations. (d) We propose a dataset based on 3D Shapes (Burgess & Kim, 2018) that allows the analysis of hierarchical latent spaces with nested temporal structure. (e) In different experiments and datasets, we demonstrate the ability of VPR to discover changes, be cost-effective, adjust representations to the dataset's temporal factors of variation and produce farsighted and diverse rollouts in the environment, all of which are key advantages for agent learning and planning.
2 RELATED WORK
Hierarchical generative models Our proposed model relates to the extensive body of literature
on hierarchical generative models. Incorporation of the hierarchy into variational latent models has
been used for improving the expressiveness of the variational distributions (Ranganath et al. 2016)).
modeling image (Rasmus et al. 20155 Bachman 2016 Vahdat & Kautz 2020) and speech data (Hsu
et al. (219)). Notably, NVAE (Vahdat& & Kautz, 2020) implement stable colloyyonvllutional and hierarchical variational autoencoder (VAE) with the use of separate deterministic bottom-up and top-down information channels. NVAE incorporates similar architectural components to VPR but designed to model stationary, as opposed to temporal, data. more closely related hierarchical generative model is CW-VAE (Saxena et al. 2021). CW-VAE is designed to model temporal data using a hierarchy of latent variables that update over fixed time intervals. Unlike CW-VAE our model employs adaptive strategy in which update intervals rely on event detection.
Furthermore, hierarchical generative models are also subject to research in the area of PC. Deep neural PC was originally proposed in PredNet by Lotter et al. (2016), who implemented local inference loops that propagate bottom-up prediction errors using stacked layers of LSTMs. Although PredNet is able to predict complex naturalistic videos and to respond to visual illusions similar to the human visual system (Watanabe et al. 2018), it does not perform well in learning high-level latent representations (Rane et al., 2020). This led to its alternative implementations such as CortexNet (Canziani & Culurciello 2017). More recently, series of studies including (Song al. 2020); Millidge et al.
2

---

Published as a conference paper at ICLR 2022
(2020) showed that deep PC can be implemented in single computational graph and can, under some conditions, replace back- propaaroongttion with local learning updates. Although local inference loops in PC are suitable for identifying event boundaries, none of the above models involve the learning of temporal abstractions. An exception the hybrid episodic memory model in Fountas et al. (2021), where temporal abstraction
introduced as a means to model human reports of subjective time durations. Finally, to our knowledge, there is currently no computational model of predictive routing (Bastos et al. 2020) the contemporary view of PC that inspired our work.
Hierarchical temporal structure A number of hierarchical approaches that do not rely on fixed intervals have also been put forward. [Cung al. (017)) proposed Hierarchical Multiscale RNNs and showed that a modified discrete-space LSTT can discover multiscale structure in data in an unsupervised manner. This method relies on parameterized boundary detector and introduces additional operations to RNNs for controlling this detection, as well as copying and updating the current hidden state of the network. HM-RNNs have been combined with more advanced state transition functions (Mujika et al. 2017) and have been extended to stochastic version using variational inference (Kim et al.) 2019). A crucial difference between VPR and HM-RNNs that, in our model, event boundaries are determined via mechanism that does not involve learning. Instead, VPR detects changes in latent representations of each hierarchical layer independently, and enforces a nested temporal structure by blocking bottom-up propagation between the detected boundaries. As result, event detection performance is high even in the very first stages of training while the emerging temporal structures change organically along with the latent space throughout training.
3 VARIATIONAL PREDICTIVE ROUTING
We introduce variational predictive routing (VPR) with nested subjective timescales a hierarchical event-based generative model capable of learning disentangled spatiotemporal representations using video datasets. The representational power of VPR lends itself to several key components: (1) distinct pathways of information flow between VPR blocks, (2) selective bottom-up communication that is blocked if no change in the layerwise features is detected, and (3) the resultant subjective-timescale transition models that learn predict feature changes optimally.
Block architecture VPR consists of computational blocks that are stacked together in hierarchy. Fig. [ shows the insides of a single block with key vari- ables and channels of information flow. Each block in layer n consists of three deterministic variables (x.) ct, dz)) that represent the three channels of com- munication between the blocks: bottom-up up (encod- ing), top-down (decoding), and temporal (transi- tioning), respectively. These variables are used to parametrise random variable 8 that contains the leamned representations in a given hierarchical level.
top-down
botlom-up
temporal
dF
change?
Communication between the blocks is a crucial com- ponent of the system. Top-down decoding from level +1 to is realised by passing the latest context c++ and sample from n*+1 through a neural net- work to retrieve c" -dee(ca*11 (n,)). Tempo- ral transitioning is implemented with the use of a recurrent GRU model (Cho et al.] 2014), such that d = ftron (sy, dz). Finally, bottom-up encoding of new observations iteratively computes layerwise observation variables, n++ Senc (x ). Fig. 2 shows a three-level VPR model unrolled over five timesteps, demonstrating how these communication channels interact over time.
Figure 1: VPR block architecture. Distinct information channels are mediated using dif- ferent deterministic variables: di temporal, c top-down, x, bottom-up. In turn, they are used to parametrise random variable s..
Subjective timescales As will be described next, each level n updates its state only when an event boundary has been detected. This results in updates that can occur arbitrarily far apart in time driven only by the changes inferred from the observations and dependent on the model's internal representations. For this reason, we use the term subjective timescales to refer to level-specific
3

---

Published as a conference paper at ICLR 2022
Stacked model
Unrolled over time
Level 3
Level 2
Level
Figure Example of three-level VPR model unrolled over five timesteps. are block variables as demonstrated in Fig. 1 > indicates the latest top-down context from a level above. indicates that bottom-up up encoding channel is open, while O indicates that it blocked.
succession of time and denote it by Tn E {l,..., J. where Ta is the total number of states produced in level n. The subjective time variable Tn should not be confused with variable E {1, T) that denotes the objective timescale (i.e. normal succession of time). Lastly, we set t = Tn for n = 1.
Formal description We consider sequences of observations {oi,.... or), modelled using a col- J:N lection of latent variables 8i.T...' The generative model of VPR can be written factorised distribution,
T. II r==1
Ta II T===1
I&N P S1.T.N)
p(o, Is I:N
II
where 1:N denotes all latest hierarchical states at timestep Tu. p(s s <TT. 722 represents a prior distribution of the state s", conditioned past states in level n and all above states in the hierarchy n, and p(x(") (), subscript from Tn and I, in the following sections and use and T instead. Since the true posterior p(s (o:T) is intractable, we resort to approximate using parametrised approximate posterior distribution { (o1:r) = I Ih (s [x." 8" s?"), where x. is the layerwise encoding of o, and @ are the parameters of the posterior model.
diagonal Gaussian prior. To simplify the notation, we omit the
Notably, the proposed generative model (ser and s>") by virtue of the deterministic variables d and that mediate block communica- tions during the generative process. All of the model components are summarised follows,
Markovian in both temporal and top-down directions
Posterior model,
(o(s") 1 s2r, s77)
(2) Encoder,
n+1 (xn
(5)
Prior model, Transition model,
Po (3 8² d = fir (s dh)
(3) Decoder,
a- = (0"
(6)
(4)
Reconstruction,
O, = (re)(co)
(7)
Finally, to train VPR, we define a variational lower bound over Inp(ouT)), LELBO,
T
N Th
LELBO = Σ
B (lag,,(lnpp p(o, 1:NN)
o( .z
[D 0s (8 (5
(8)
7=0
x=1 ==11
and (o() = po (s s (s s?²). where (s7)) = (0 s2,)
Event detection At the heart of VPR is mechanism that is used for detecting predictable and unpredictable changes in the observable features over time, and thus determine the boundaries of events. Occurring at every level of the model, the event detection is used for controlling the structure
4

---

Published as a conference paper at ICLR 2022
of the unrolled model over time by allowing (Fig. 2 or disallowing (Fig. 2 )) propagation of bottom-up information to the higher levels of the hierarchy.
Specifically, for some level n, the model retrieves the latest posterior belief state, pst
p(s"_1 1x,,, 822 8²") go ( Jx,,, 8 822,,8 87"), which represents model's belief over subset of observable features (represented in level n) at the latest observed timestep. т. Given a new ob- servation, 2-+1- VPR then computes an updated posterior belief under the static assumption, qet = (<(**+1<0" - 57" =²n
s²-). Notice that qet is conditioned on the same set of latent states, which implies that Pat qst iff x7 ==+1. This means that the model's posterior belief over the inferred features will remain constant if the features have not changed between T and T + 1. The model's 's update in the posterior belief is then measured using KL-divergence, D.c = Dx z. (g-<|[p=<) This key quantity can be seen as a measure of how much the relevant layerwise features have changed since the last encoded timestep T.
Next, VPR has two ways of identifying event boundary. Criterion E (CE): predictable events detected with the use of the model's transition model that computes the next subjective timestep's s belief state, Pch. po po(*"=11s"; 8 = 82²)) This prediction can be seen as the model's prior belief over the features it expects to observe next (the change assumption). Upon receiving the new s²"). Similarly to the computations for the static assumption, the KL divergence between these two predictable event is considered detected D (gae llp.c) > D (.Ptt) Satisfying this criterion indicates that the model's prediction produced a belief state more consistent with the new observation + suggesting that it contains a predictable change in the features that are represented in the corresponding level of the hierarchy. Criterion U (CU): events that are not or cannot be captured well by the transition models (i.e. unexpected changes) require moving average statistics of the latest Ta timesteps over the observed values of Dst. and identifies an event when inequality Dst,7+1 > >1 -i Dst.k/ Tur (where Y is threshold weight) is satisfied. //T.
information 11 the model then calculates the posterior belief. 4ch 1 s²
distributions is calculated, Det = D (9ch (Pc.). (qchilPet)
additional method for their detection. To this end, VPR keeps
As will be shown, this criterion proved to be crucial in the beginning of the training when the model's representations and layerwise transition models are not yet trained.
Bottom-up communication Event detection serves two primary functions in our model. First, detecting layerwise events in sequence of observations is used for triggering update block's state (Fig. 2 ) by inferring its new posterior state. Matching block updates with detectable changes in layerwise features (event boundaries) prompts VPR to represent spatiotemporal features in levels that most closely mimic their rate of change over time. Similarly, learning to transition between states only when they signify a change in the features of the data allows VPR to make time-agnostic (or jumpy) transitions from one event boundary to another. Second, the detection mechanism is used for blocking bottom-up communication (Fig. () and thus stopping the propagation of new information to the deeper levels of the hierarchy. This encourages the model to better organise its spatiotemporal representations by enforcing a temporal hierarchy onto its generative process.
Practically, when bottom-up information channel blocked at level n, the variables of the model in levels 2 n will remain unchanged. By virtue of the hierarchy, this similarly means that the top-down
information provided to level n the next timestep will stay constant (Fig. 2 Ph). As result, the blockage mechanism encourages the model to represent slower changing features in the higher levels of the hierarchy.
Importantly, there is no constraint on when and how often a state update can occur over time. In other words, VPR is time-agnostic and is designed to model the changing parts of sequences with temporal hierarchy, irrespective of how slow or fast their rates are. This leads to the model's ability to dynamically adjust to the datasets with different temporal structures and to learn more optimal, disentangled, and hierarchically-strrctuue the rate at which blockage occurs to datasets with slower and faster temporal dynamics.
representations. Fig. 3 shows the ability VPR to adjust
4 EXPERIMENTS
Using a variety of datasets and different instances of VPR, we evaluate its performance in the following areas: (a) unsupervised event boundary discovery, (b) generation of hierarchical representations, (c) adaptation to the temporal dynamics of incoming data, and (d) future event prediction. We employ the following datasets and their corresponding VPR model instances:
5

---

Published as a conference paper at ICLR 2022
b
4
Aver. intervals in L2
0.8
0.8
CE+CU
6-
F1 score
0.6
CE
5-
0.6
Fast
CU
Feature 2 Feature
4-
Slow
VTA
33
0
2 4
2
6 8
2
CW-
Training iterations (x10K) Tr. Iterations (x10K) Tr. iterations (x10K) Tr. iterations (x10k)
VPR
Figure 3: Performance of VPR's event detection system using Moving Ball and 3DSD. (a.) VPR adapts its rate of state updates to the temporal dynamics of the Moving Ball dataset, (a:) while maintaining the same accuracy of event detection. (b) Fl score of event detection using different sets of criteria in Moving Ball, with a comparison against VTA (Kim et al. 2019). (c) Fl score of levels 2 and in detecting changes in their corresponding features using 3DSD. (d) Comparison against baseline methods using 3DSD. Shaded regions indicate one standard deviation over 5 seeds.
Synthetic dataset is implemented to validate our approach for event boundary discovery and analyse the roles of the chosen event criteria. The dataset consists of 1D sequences generated using semi- Markov process with random value jumps every 10 steps. The VPR instance employed a minimal version, where N 1, the transition model Pos ($ s,) is memoryless and the posterior state calculation is replaced by a direct map (st = Jon = O, + c, where e is Gaussian noise.
Moving Ball consists of a coloured ball that travels in straight lines and bounces off walls within the boundaries of an image. Importantly, the ball's colour changes either upon wall bounce or at random time with probability p = 0.1. As there are two dynamic features position and colour of a ball - we employ VPR with N 2 (two levels) and measure the accuracy of the model to detect and represent changes the ball's colour in level 2 of the hierarchy.
3D Shapes Dynamic (3DSD) dynamic extension to the 3D Shapes dataset (Burgess & Kim 2018) large image collection of shapes generated using 6 underlying factors of variation. By manipulating a subset these factors, we produce sequences of images where different factors vary over time with different periodicity. Specifically, we impose a temporal hierarchy onto three factors colours of objects, walls, and floor (from slow to fast, respectively). We test the ability of VPR with N to detect and represent the changes in these features in levels 2 and 3 of the hierarchy.
Miniworld Maze. To evaluate the behaviour of VPR in more perceptually challenging setting, we use 3D environment Syym-Miniworld (Chevalier-Boise Boisvert, 2018). For this, we designed a maze environment that consists of sequence of connected hallways each with different colour and of random length - and collect sequecnes agent observations when traversing these hallways.
To validate event boundary discovery in VPR, we compare against the temporal abstraction model that employs a parametrised boundary detection system, VTA (Kim et al., 2019). Further, to emphasise the benefit of flexible timescales we demonstrate the event detection accuracy of a CW-VPR version of VPR where latent transitions occur at fixed rates, analogous to CW-VAE (Saxena et al., 2021). Finally, we compare VPR's future event prediction accuracy against a model for long-term video prediction, CW- VAE (Saxena et al. 2021
4.1 UNSUPERVISED EVENT DISCOVERY
VPR's 's event detection mechanism proves be effective across three different datasets. Specifically, it quickly achieves high F1 score in the Synthetic (Fig.A. Moving ball (bi F1 = 0.97 + 0.03), and 3DSD (Fig. 3e:
0.97 0.02) datasets. Furthermore, we observe a significantly better performance compared to the baseline models VTA (see Fig.3b: Moving ball, Fig. 3 3DSD, Fig Td; Synthetic) and CW-VPR (see Fig. 31: 3DSD).
Detection criteria analysis As mentioned, the detection mechanism employs two criteria for identifying expected (CE) and unexpected (CU) events. Using the Synthetic and Moving Ball datasets, we show that the optimal performance is only achieved when both of these criteria are used in the decision-makee making. First, observe that the role of CU is particularly crucial in the early stages of training, in order to provide initial supervision signal for the untrained transition model. Fig. 4h shows the F1 score of the CE criterion detecting event boundaries with (blue curve) and

---

Published as a conference paper at ICLR 2022
1.0
Uneea events (5)
detected by:
scate
60
30
VPR
CE or CU CE
F1 score
0.8
CU
20
9.4
VPR with only CE
10
50 160
150 200 250
50
160
150
100 150 200 250
0.0 0.2
0.4
0.8
1.0
Training sleps
Training steps
Training steps
Noise level
Figure 4: Event discovery performance in synthetic data. Posterior noise set to 0.0 in (a-b) and 0.4 (c). Shaded regions indicate one standard deviation over seeds.
without (pink curve) the CU criterion enabled at the same time. It is evident that the transition model cannot improve performance with a disabled CU. contrast, expected event recognition with both CE and CU quickly converges to optimality after some training. Notably, the better CE becomes in detecting events, the less number of events are classified as unexpected (Fig. 4b). Furthermore, in the later stages of learning, the roles of the two criteria switch, as CE becomes the driving force for improving the model's s performance. Since detection with CU relies on a con-parametric threshold, its performance has theoretical limit and reduces substantially for high levels of posterior noise (Fig. 4)) In contrast, CE continues to improve beyond CU's supervision signal, being significantly more robust noise. In addition, training VPR the Moving Ball dataset using the different detection criteria demonstrates similar behaviour. Fig. Bp shows that employing both CE and CU results in the significantly better and faster convergence of the detection Fl score.
of
4.2 HIERARCHICAL TEMPORAL DISENTANGI EGEENTANG.EMENT
VPR employs its hierarchical structure to disentangle the temporal features extracted from the dataset. By matching the rates of event boundaries with the rates level updates, the model learns to represent particular temporal attributes of the dataset in the appropriate layers of the hierarchy. We analyse the property hierarchical disentanglement by (1) producing rollouts using separate levels of the model, (2) generating random samples from different levels, and (3) analysing factor variability of the samples as measure of disentanglement
Each layer of VPR is equipped with an event detection mechanism that guides the model distribute representations of temporal features across the hicrarchy and learn jumpy predictions between event boundaries. Fig. shows layerwise rollouts of VPR in the Moving Ball and 3DSD datasets. For Moving Ball in Fig. 5a, VPR learns represent the ball''s position and colour in the separate levels (L1 and L2, respectively). LI rollout accurately predicts positional changes of the ball, while
Rollout steps
Rollout steps
(a) Moving Ball
(b) 3D Shapes Dynamic
Layerwise rollouts using VPR. GT denotes ground-truth sequence, LI rollouts made using level 1, and To produce layerwise rollouts, the model predicts the next state 8h41 in the relevant level n and decodes under fixed states all other levels. The produced rollouts illustrate model's ability learn disentangled representations and produce accurate and feature- speciifc jumpy rollouts.
Figure 5:
7

---

Published as a conference paper at ICLR 2022
Samples from Level
Samples from Level
Samples from Level
Miniw. Maze 3D Shapes
Figure 6: Random samples taken from the different levels of VPR. The model generates diverse images with respect to spatiotemporal features represented in the sampled level, while keeping all other features fixed.
keeping the colour constant. At the same time, 1.2 rollout produces jumpy predictions of the ball colour under constant position. Similar behaviour can be observed in 3DSD, using three layers.
VPR's representational power be further dissected by taking random samples from the different levels of the hierarchy. Fig. shows 3DSD and Miniworld Maze reconstructions VPR conditioned sampling only of the levels and fixing all others. In line with the produced rollouts, 3D Shapes samples indicate that LI contains representations of the floor colour and object shape, L2 of wall colour and angle, and L3 of object colour. For the Miniworld Maze samples, we similarly observe that LI represents the short-term feature of the agent's position in room, L2 encodes the colour of the walls, while L3 provides the wider context of the agent's ' location
maze.
To quantify temporal feature disentanglement,
measure the average entropy in the distribution of each of the features associated with the reconstructed samples, H, - M/ EM Σ' p(v;) log p(u.). where I = 32 is the number of samples trial, . 100 is the number of sampling trials, and is factor of variation extracted from each reconstructed image using pre-trained extractor model. Factor will be associated with higher average entropy if the layerwise random samples produce more uniform distributions of this factor across the reconstructed images. Fig, 7 shows that, for each level, the average entropy is high only for temporal feature that ranks the same in the order of the dataset's 's temporal hierarchy (factor 1: fastest, factor 3: slowest). This implies that VPR distributes representations of features in way that preserves the underlying temporal hierarchy of the dataset.
We contrast this result with instances of VPR that perform block updates over fixed intervals (anal- ogous to KW-VAE (Saxena et al.) 202T)) in Fig. 7 can be seen that different intervals result in the different representational properties of the model, suggesting the difficulty of selecting the appropriate interval values. Furthermore, fixed intervals unnecessarily bound the unrolled structure of arbitrary time intervals are likely to not be abstracted
the model, meaning that features that to the higher levels. For instance, as shown in the Appendix Fig. 12, a fixed-interval VPR model cannot represent the ball colour in level 2 entirely, in contrast VPR with subjective timescales.
Samples from level
Samples from level 2 Samples from level 3
Average entropy (nats)
Update intervals:
subjective
4
6
3
Factor index (fast to slow)
Figure 7: Feature disentanglement in 3DSD. VPR with subjective timescales finds the most appropri- ate distribution of representations, in contrast to fixed-interval models.
4.3 ADAPTATION TO TEMPORAL DYNAMICS
Since VPR is an event-based based system, it relies solely on the underlying change dynamics a dataset layerwise computations only initiated when event boundary has been detected and can thus
8

---

Published as a conference paper at ICLR 2022
adapt datasets with different temporal dynamics. To test this property, instantiate two versions of the Moving Ball dataset with fast and slow ball movements. Fig. Bl compares the average time intervals between block updates (in level 2) of VPR trained using these two datasets. It can be that VPR performs about 30% fewer block updates under slow ball movement compared to VPR under fast ball movement, while maintaining high F1 score of event detection in both cases. We also validate that both models produce the same hierarchical organisation of features reported in Section This event-driven property implies significant computational savings, promoting a more optimal usage of computational resources with respect to the temporal dynamics of different datasets which will be even more significant in VPR instances with larger number of hierarchical levels.
4.2
4.4 FUTURE EVENT PREDICTION
The ability of VPR to learn jumpy and feature-specifie transitions in its hierarchical levels has beneficial implications on the accuracy with which long-term event sequences can be predicted. To test this, we perform jumpy layerwise rollouts of length 200 for the Moving Ball and 3DSD datasets and evaluate them against the ground-truth event sequences - colour changes in the Moving Ball (VPR level 2), and wall and object colour changes in the 3DSD (VPR levels and 3). Here we compare VPR against a powerful video prediction model, CW- VAE (Saxena et al.) (021), which produces rollouts over the physical timescale and thus requires longer rollouts to reach an equivalent of VPR's jumpy predictions. For instance, predicting 200 events of object colour changes in the
3DSD corresponds to a physical rollout of 1600 steps. For the 3DSD, we find that VPR make more accurate event predictions of both wall (acc. 0.99 + 0.017) and object (acc. 0.90 + 0.04) colour changes than CW-VAE (acc. 0.92 + 0.05 and 0.71 + 0.02, respectively). For the Moving Ball, we find that VPR significantly outperforms CW-VAE predicting changes in the ball colour, reaching perfect accuracy of 1.0 + 0.0 against only 0.47 = 0.09 by CW-VAE.
5 DISCUSSION
Event duration estimation The presented model in its current form is not able to generate far- sighted rollouts over the physical timescale of the data. This process requires an additional method to estimate the number of transitions in each layer n, before layer n + performs a transition. This can be implemented with neural network that estimates the probability p(changelss, S2, 8n+1), trained in self-supervised manner, using the criteria CE and CU. Analogous estimation has been employed in previous studies parametrised event boundary detector (Chung al., 2017 Mujika al. 2017] Kim et af.) 2019). However, unlike these proposals, this estimation is not necessary for any of the results presented here, nor for potential applications such agent planning or solving downstream tasks. To highlight this important difference, we chose to not explore event duration estimation here.
Expected and unexpected events The criterion for event boundary detection used to identify expected changes (CE) can be viewed as a model selection method based on the shortest belief update, given the prior assumption that a change has either occurred or not. If geh = qu. then the incquality CE is equivalent to maximizing the likelihood of the latent state given new observations. Indeed, can see in Appendix Fig. 16 that the absolute difference between the entropies Hs*(s") and Ha (s") is negligible compared to the difference between the corresponding relative entropies, indicating that the CE criterion is predominantly governed by the latter difference. Furthermore, although detecting the onset of anticipated events is crucial for constructing hierarchical temporal latent spaces, these events do not denote salience, unless CU is also satisfied. Visual salience (in the form of attention in humans) has been often directly linked the value of DK (q [p). i.c. Bayesian surprise, with empirical evidence provided by Ittl & Baldi (2006). Thresholding this quantity has been proposed as a model of attention to perceptual changes. which determines which changes can be deemed salient (Roseboom et al., 2019 Fountas et al. (021), while the accumulation of such changes is shown to correlate well with cortical activity when humans perform duration judgments (Sherman et al. 2020). Finally, the distinction between expected and salient events, captured by CE and CU here, has been extensively discussed by Yu & Dayan (005). who proposed a connection with two neuromodulators and inextricably intertwined role in attention. Modeling hierarchical attention in VPR using CE and CU constitutes an interesting avenue for future work and yet another potential example of how cognitive neuroscience and machine learning can facilitate each other's progress.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like to thank Yansong Chua for his valuable contributions and comments on the early versions of the model presented in the current manuseript.
REFERENCES
Philip Bachman. An architecture for deep, hierarchical generative models. In NIPS, 2016.
Christopher Baldassano, Janice Chen, Asieh Zadbood, Jonathan W Pillow, Uri Hasson, and Kenneth A Norman. Discovering event structure in continuous narrative perception and memory. Neuron, 95 10,109-721, 2017.
André M. Bastos, M. Lundqvist, Ayan S. Waite, N. Kopell, and E. Miller. Layer and rhythm specificity for predictive routing. bioRxiv, 2020.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Lars Buesing, Theophane Weber, Sébastien Racaniere, SM Eslami, Danilo Rezende, David P Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, et al. Learning and querying fast generative models for reinforcement learning. arXiv preprini arX1v//802.030000 2018.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gihhbbooommcccppiiillssss 2018.
Alfredo Canziani and Eugenio Culurciello. Cortexnet: a generic network family for robust visual temporal representations. arXiv preprint 1rXiv:/706.02777 2017.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrmns for video prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7608-7617, 2019.
Maxime Chevalier-Boisvert. gym-miniworld environment for openai gym. https://githuu com/maa wtcm//axieebb///cccccoccmoo wec.imecb/bcee wwtmmmmmiiiwoooo.oo 2018.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations, ICLR, 2017.
Zafeirios Fountas, Anastasia Sylaidi, Kyriacos Nikiforou, Anil K. Seth, Murray Shanahan, and War- rick Roseboom. A predictive processing model of episodic memory and time perception. bioRxiv, 2021. doi: 10. 00.1001/022002.17..553
URL http::ttt////tctt //ttps::/www... biorx www.ooowwv..oooooo.ooooooooooo www.booor..w...rrr/ccttttttttt content.///////////o//m
eazly/2021/07/0//20000 02. 953133
Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 1 11(2):
127-138, 2010.
Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. Temporal difference variational auto-encoder. arXiv preprint /rXiv:/806.. 0018800003107,. 2018.
David R Ha and Jürgen Schmidhuher. World models. ArXiv, abs/1 803. 10122, 2018.
Bridgette M Hard, Barbara Tversky, and David S Lang. Making sense of abstract events: Building event schemas. Memory & cognition, 34(6): 1211-1335 2006.
Wei-Ning Hsu, Y. Zhang, Ron J. Weiss, H. Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Z. Chen, Jonathan Shen. P. Nguyen, and Ruoming Pang. Hierarchical generative modeling for controllable speech synthesis. ArXiv, abs/1810.07211 2019.
10

---

Published as a conference paper at ICLR 2022
Laurent Itti and Pierre F Baldi. Bayesian surprise attracts human attention. In Advances in neural information processing systems, pp. 547-554. Citeseer, 2006.
Dinesh Jayaraman, Frederik Ebert, Alexei A Efros, and Sergey Levine. Time-agnostic prediction: Predicting predictable video frames. arXiv preprini arXiv: arXiv:1808.0771 1808.07784, 2018.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Neural Information Processing Systems, 12:11570-11579, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015.
Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzales Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. International Conference on Machine Learning, pp. 2418-3428, PMLR, 2019.
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork mn. In International Conference on Machine Learning, pp. 1863-1871. PMLR, 2014.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A conditional flow-based model for stochastic video generation. arXiv preprint arXiv: 1903.01444 2019.
Christopher A Kurby and Jeffrey M Zacks. Segmentation in the perception and memory of events. Trends in cognitive sciences, 12(2):72-79, 2008
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.00 000004 2016.
Andrew L. Maas. Rectifier nonlincarities improve neural network acoustic models. 2013.
Beren Millidge, Alexander Tschantz, and Christopher Buckley. Predictive coding approximates backprop along arbitrary computation graphs. arXiv preprint wwxii:2006.04182, 2020.
Asier Mujika, Florian Meier, and Angelika Steger. Fast-slow recurrent neural networks. arXiv preprint 3005v:/770..86660 2017.
David Mumford. On the computational architecture of the neocortex. Biological cybernetics, 66(3): 241-251, 1992.
Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf. Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. arXiv preprint arXiv: arXiv:1808.04411 047688 2018.
Karl Pertsch, Oleh Rybkin, Jingyun Yang, Shenghao Zhou, Konstantinos Derpanis, Kostas Daniilidis, Joseph Lim, and Andrew Jaegle. Keyframing the future: Keyframe discovery for visual prediction and planning. In Learning for Dynamics and Control, pp. 969-979. PMLR, 2020.
Roshan Prakash Rane, Edit Szügyi, Vageesh Saxena, André Ofner, and Sebastian Stober. Prednet and predictive coding: critical review. In Proceedings of the 2020 International Conference
on Multimedia Retrieval, ICMR '20, pp. 233-241, New York, NY, USA, 2020. Association doi: 10.1145337228..390694, URL
for Computing Machinery. ISBN 9781450370875.
htttps://doi.ore wwwg/oorg10000 0terg//1..14/337222220 1wl.111/33228.33333333 3390694
R. Ranganath, Dustin Tran, and David M. Blei. abs/1511.02386. 2016.
Hierarchical variational models. ArXiv,
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-fis -feeld effects. Nature neuroscience, 2(1):79-87, 1999.
Antti Rasmus, Mathias Berglund, M. Honkala, H. Valpola, and T. Raiko. Semi-supervised learning with ladder networks. In NIPS, 2015.
11

---

Published as a conference paper at ICLR 2022
Warrick Roseboom, Zafeirios Fountas, Kyriacos Nikiforou, David Bhowmik, Murray Shanahan, and Anil Seth. Activity in perceptual classification networks as a basis for human subjective time perception. Nature communications, 10(1):1-9, 2019.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders, In Advances in Neural Information Processing Systems, volume 34, 2021.
Jürgen Schmidhuber. Neural sequence chunkers. 1991.
Wenling Shang. Alex Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Learning world graphs to accelerate hierarchical reinforcement learning. arXiv preprint 1XXiv:907.00060 2019.
Maxine T. Sherman, Zafeirios Fountas, Anil K. Seth, and Warrick Roseboom. Accumulation of salient perceptual events predicts human subjective time. bioRxiv, 2020. doi: 10. 101/ 2020.01.09.900423. URL htttps://www... http://wwww......ooo////t/aaa/// 26/2020. 01. 09. 900423
Yuhang Song, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Can the brain do backpropa- gation?-exace implementation of backpropagation in predictive coding networks. Advances in neural information processing systems, 33:22566, 2020.
Arash Vahdat and J. Kautz. Nvae: A deep hierarchical variational autoencoder. ArXiv, abs/2007.03898, 2020.
Biji Watanabe, Akiyoshi Kitaoka, Kiwako Sakamoto, Masaki Yasugi, and Kenta Tanaka. Illusory motion reproduced by deep neural networks trained for prediction. Frontiers in psychology, 9:345, 2018.
Nevan Wichers, Ruben Villegas, Dumitru Erhan, Honglak Lee, et al. Hierarchical long-term video prediction without supervision. In International Conference on Machine Learning, pp. 6038-6046. PMLR, 2018.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn, Greedy hierarchical variational autoencoders for large-scale video prediction. CVPR, 2021.
Angela J Yu and Peter Dayan. Uncertainty, neuromodulation, and attention. Neuron, 46(4):681-692, 2005.
Jeffrey M Zacks and Barbara Tversky, Event structure in perception and conception. Psychological bulletin, 127(1):3, 2001.
Alexey Zakharov, Matthew Crosby, and Zafeirios Fountas. Episodic memory for subjective-timesett models. In ICML 2021 Workshop Unsupervised Reinforcement Learning, 2021.
12

---

Published as a conference paper at ICLR 2022
A EVENT DETECTION MECHANISM
A.1 ARCHITECTURE AND PARAMETERS
Fig. @ shows the insides of the event detection mechanism implemented each level the model's hierarchy. Given the current timestep + 1, the mechanism at level n is triggered if the bottom-up communication has not been blocked by the level below, n 1. Upon receiving new bottom-up infor- mation +++ the model proceeds in evaluating four key variables. First, the latest posterior is assigned to be the new prior under the static assumption, Pst = p(="*(((*"))0 [ di, () &- go [ clarity,
|r"" d.,, df,c))) For the deterministic variables di and c represent 8²r and s?", respectively. Sec- ond, the new posterior under the static assumption is computed using the deterministic variables of the latest block, qst (o(56+1 |1 dl, e.) As explained in the main body, we then compute the Kl-divergence between the two states, Dat DKt((sst [Px)). Third, we trigger the transition model to predict the next temporal context, +1 in order produce the prior under the change assumption of the model, Pet. pe |dd - ).). Lastly, the posterior under the change assump- tion can also be computed using the new bottom-up encoding, qch = qo (s" + d=+1, c²). As with the static assumption, we calculate the KL-ddiveggence between the prior and posterior states,
Doh D k (gos P.a.)
Practically, prior state Poh. is computed only once, after which it is stored for subsequent comparisons until the event criteria are satisfied and the block is updated.
Additionally, criterion CU (Dst,r+1 > ? Σi=r- Dtt,k/Tw)) involves two hyperparameters: Tu is the length of a sliding window used for calculating the moving average, and y is the threshold factor that multiplies the value of the moving average. In running the experiments, we found that the optimal values are 7 1.1 and Tu 100. These values also proved be robust for use across different datasets, as we kept their values constant for all of the reported experiments.
Latest block state
Decision mechanism
d
d++1
Pst
Use
qah
Prior prediction
Pab
New observation
Dxr(an lip..)
DEI lipa)
Figure 8: Event detection mechanism relics on the computation of key variables shown in the figure. Given the latest updated state of VPR block at level and timestep denoted as Pst (and the corresponding deterministic states d' and c,), the model receives a new observation x; at some timestep + 1. ① This observation is used to compute the model's posterior belief, Ist, using the latest deterministic variables of the block. 2 Concurrently, VPR makes prediction using its generative model, producing Pch (and the corresponding di d representing the model's prior belief about the features at the next subjective timestep. 3 Lastly, VPR produces a posterior belief state, Jeh, under the updated temporal context variable d
A.2 DECISION METRICS
We can visualise the values calculated as part of the decision-makingg process in the event detection mechanism in Figure 9 As described in Section 1 the CU criterion acts as the initial supervision signal, which subsequently results in the rapidly improving transition model and thus the CE-based detection. Figure shows two examples of the computed values over the length of an observation sequence the early (left) and later (right) stages of training. It can be observed that at only 500 training iterations the decision- making is primarily driven by the CU criterion. At 18500 iterations, their roles tend switch, the CE criterion becomes significantly more accurate at detecting events.
13

---

Published as a conference paper at ICLR 2022
Tr. iteration 500
Tr. iteration 18500
10-1
10a
Dal Ppelee
10-3
10
CU threshold
Object changes colour
10
10
Sequence timesteps
Sequence tim timeeseps
Figure 9: Key KI KL-divergence values computed using the level 2 detection mechanism at different stages of the training process using the Moving Ball dataset. Left-hand side graph shows the early stages of the training (training iteration 500), while right-hand side the later stages (training iteration 18500). line with the described detection criteria in Section [ one of the blue lines falls below the orange line, event is considered to be detected. As such, it be seen that the decision- making is dominated by the CU criterion (light blue) in the early stages (left). On the other hand, representations mature and the transition model learns, the CE criterion (dark blue) begins to dominate the detection process (right).
B MODEL ARCHITECTURE AND TRAINING
The model consists several components implemented using neural networks:
Encoder is combination of (a) convolutional neural network that embeds nigh-dimensional image data into lower-dimensional representation, (b) stacked fully- connected networks with residual connections fehe- and (c) fully-connected networks in each layer that compress layerwise observation embeddings, x", prior to being passed into a posterior model.
Bottom-up.
Top-down. Decoder is a combination of (a) transpose convolutional neural network, Srees for reconstructing images using top-down information c and (b) stacked fully-connected networks with residual connections f
Temporal. Layerwise transition models are reccurrent GRU models (Cho et al. 2014 with hidden states of size |dd] 200 and an additional fully-connected network with neurons.
Prior and posterior models. These models are implemented using four fully-connected networks and parametrise diagonal Gaussian, thus outputting vector of size | 2.
The convolutional components of the encoder and decoder are analogous those used in Ha Schmidhuber (2018). Fully-connected top-down and bottom-up components are all made out of the same building block, which consists of four fully-connected layers with a residual connection the output (e.g. x =++1 (frn((")) + 0.1 .x")) and Leaky ReLU activations (Maas 2013). The number of neurons is kept the same throughout and is equal to the dimensionality of the block's input (e.g. |x:] l). Component (c) of the bottom-up model, as well the posterior and prior models, also consist of four fully-connected layers but with no residual connections.
We use the same VPR architecture for the Moving Ball and 3DSD datasets. Specifically, the latent |s = 20, while the temporal, top-down, and bottom-up deterministic variables are | = |d 200. Moving Ball and 3DSD, observations O, R61x61x3 so we
states are of size
set to be |x-||| set the model's input layer have the same shape. For the Bouncing Balls dataset (see section C.3 we increase the capacity of the model, such that |>"|| 1024 and |s 60.
For training, we use Adam optimizer (Kingma Ba 2015) with learning rate of 0.0005 and cosine decay to 0.00005 over a period of 15,000 iterations. We employ linear annealing of the KL. coefficient from to 1 over the first 3000 iterations. Although we also used KL balancing (Vahdat & Kautz 2020) in the models presented in this paper, we find that it does not have significant effect on their resultant properties. Further, we use binary cross entropy reconstruction loss and sequences of length 15 for the Moving Ball and Bouncing Balls datasets, and mean squared error loss and sequences of length 50 for 3DSD and Miniworld Maze datasets. Batch size 32 is used for all datasets.
14

---

Published as a conference paper at ICLR 2022
B.1 PSEUDOCODE OF EVENT DETECTION
Algorithm 1: Event detection and inference in VPR
for video to do for r 1 to max length do
Retrieve a new observation, 0,
Initialise an empty decision mask, list M
// Event detection (bottom-up) for n = 1 to N do
Compute layerwise encoding via fenc #7
// Level decision is always True n then M.append(True) continue
// If propagazion is blocked
M[n - 1] is False then M.append(False) continue
Compute variables Jchs 9sts Pehs Pst of level
/1 CE or CU criterion if (xz(g=<lp=s) > Drz(gehllpaN) or D > y Ek=t D then
M.append(rrue)
else
M.append(aalse)
// Inference (top-down) for n = N to do
if M[n] is False then continue
if n 0 then
Compute of using decoder, of fa f ct) Compute reconstruction image, o, (reo(ct) break
Compute cf using decoder, ct t- fieec ( .c7t1) Infer the new posterior state s using the posterior model, (5(5 [27 sn s s2n)
C ADDITIONAL RESULTS
C. EXPLORING INFORMATION PATHWAYS
VPR equipped with two distinct generative information pathways temporal and top-down - that give the model flexibility in generating diverse sequences. We investigate this property further by performing layerwise rollouts using level n = n, the model and setting d: = 0. We call the rolled out level targer level. Setting the temporal context variable to for all the levels below the target level effectively means that the model would have no temporal information about previously inferred states in those levels, <n.
827
15

---

Published as a conference paper at ICLR 2022
show layerwise rollouts under empty temporal priors (in all levels below) using the Moving Ball and 3DSD datasets. Additionally, we differentiate between rollouts with and without sampling (a) rollouts with sampling are decoded by taking random samples from all states below the (b) rollouts without sampling are decoded using the means of the states' Gaussian distributions 8 ~,"r, instead.
Figures 10
target level, 8 =
Moving Ball Temporal information carried in level I relates the ball's position and direction of travel. After depriving the model this information and performing a rollout using level 2, we observe a distinct behaviour, in which VPR correctly predicts the changes the ball's colour while randomly assigning it position within an image (Fig. TOa). When no sampling is done, the position remains constant in the middle of the box (Fig. 10b).
in the main body of the paper, VPR learns to represent different features of the 3D Shapes dataset in the different levels its hierarchy. As result, performing layerwise rollouts under empty temporal priors in the levels below rolled out level produces interesting properties. Rolling out level 3 while sampling during decoding (Fig. [1 creates a sequence in which the changes in the object colour (a feature that represented in level 3) are correctly predicted over the steps of the rollout; however, all other features including object shape, angle, wall and floor colours are decoded at random. Given that these features are represented in levels 1-2 and that no temporal information was passed to these levels, the model predicts high uncertainty for latent variables 85"r and thus samples represented factors at random. Level rollouts show the same behaviour while the wall colour and object shape correctly predicted over the timesteps of the rollout, the floor colour (represented level 1) is sampled at random. These rollouts once again demonstrate VPR's disentanglement properties, well as its versatility in generating sequences of quality reconstructions even when information from one of the channels is blocked.
3DSD
As
Rollout steps
(a) with sampling
Rollout steps
(b) without sampling
Layerwise rollouts under empty temporal priors in the Moving Ball dataset. GT denotes the ground-truth sequence, LI level rollout, and (a) Decoding performed while sampling at all levels below the target level; (b) decoding is done using the means of the Gaussians for s
Figure 10:
<n.
C.2 FIXED-INTERVAL MODELS COMPARISON
One of the advantages of VPR with subjective timescales (VPR-ST) over equivalent fixed-interval models is the ability abstract temporal features to the deeper levels of its hierarchy even when changes in these features occur over arbitrary number timesteps. The Moving Ball dataset allows us to analyse this property of VPR, since the ball's colour does not change over fixed time intervals.
To test this, we train four different fixed-interval VPR models that update their level 2 state every 2, 4, 6, and timesteps. Figure 12 shows layerwise rollouts of these models, which are analogous to Figure 5 where we analysed subjective-timescale VPR. As can be seen, the model struggles to abstract the colour of the object entirely to level 2, in contrast to VPR-ST. Because the colour of the object does not change with fixed periodicity, fixed-interval models do not seem to be suitable for learning temporally disentangled features in such datasets.
16

---

Published as a conference paper at ICLR 2022
Rollout steps
with sampling
Rollout steps
(b) without sampling
Figure 11: Layerwise rollouts under empty temporal priors in the 3DSD dataset. GT denotes the
ground-truth sequence, L/ level rollout, and so on. (a) Decoding performed while sampling at all levels below the target level; (b) decoding is done using the means of the Gaussians for <n,
17

---

Published as a conference paper at ICLR 2022
Rollout steps
(a) Trained with level intervals of: timesteps.
steps
(b) Trained with level intervals of: timesteps.
Rollout steps
(c) Trained with level intervals of: timesteps.
Rollout steps
(d) Trained with level intervals of: timesteps.
Figure 12: Layerwise rollouts using fixed-interval VPR models with manually assigned intervals of
4, 6, and 8. can be observed that the ball colour does not get entirely represented in level (L2) of the model, in contrast the VPR with subjective timescales in Figure
18

---

Published as a conference paper at ICLR 2022
C.3 VPR ON THE BOUNCING BALLS DATASET
To further illustrate the effectiveness of the VPR's event detection system, additionally train it using a more dynamically complex dataset, analogous to the one used Kim et al. (2019). The results demonstrate that VPR outperforms the VTA model (Kim et al. 2019) in determining event boundaries, as seen from Figure 13 The resultant representational properties VPR with respect to the two temporal factors of variation (position and colours) proved to exhibit the same hierarchical disentanglement features discussed in Section 4.2 and demonstrated using samples taken from the different latent levels of the model in Figure
1.0
1.0
True- Dositiviiie rate
False- positive rate
score
0.8
0.8
0.6
VTA
0.6
0.2
0.2
0.0
10
20
30
40
50
60
20
30
40
50
10
20
30
50
60
Training iterations (x1000)
Training iterations (x1000)
Training iterations (x1000)
Figure 13: Comparison of VPR against VTA on the task of event detection using the Bouncing Balls
dataset. (a) F1 score. (b) True-positive rate. (c) malse-positive rate. Shaded region indicates one standard deviation using different seeds.
(a) Samples from Level
(b) Samples from Level 2.
Figure 14: Random samples taken from the different levels of VPR. Similar to the analysis in Section 4.2 it is evident that VPR learns to represent the position of the balls in Level I and their individual colours in Level 2 of the hierarchy.
19

---

Published as a conference paper at ICLR 2022
C.4 CU SENSITIVITY ANALYSIS
As shown, the CU threshold is important component of the event detection mechanism that incorporates two main hyperparameters window size of the moving average Tu and threshold weight 7. Using the Moving Ball dataset, we evaluate the sensitivity of the VPR's s performance the different values of these two parameters in Figure [5] The graphs indicate an average achieved F1 score by VPR using the specified parameter value. VPR's performance for each of the parameter values is averaged across all VPR instances trained using all of the values of the other parameter. In total, 100 different models were trained (5 sliding window sizes threshold weights 5 runs each).
We find that changing the size of the moving average window does not have a significant effect on the model's performance. This is not the case for the threshold weight parameter that resulted in the deterioration of VPR's performance when was increased to 1.2. This likely due to the fact that significant increase in the value of y results in more sparse event detection that poorly matches the rate of feature changes, Overall, however, we find that the two hyperparameters are quite robust, as the same set of values (? 1.1 and Tw 100) was used across all of the datasets presented in the paper.
1.0
1.0
0.9
0.9
score
0.7
0.7
0.6
0.4
0.4
5
10
50
100
500
1.01
1.05
1.10 Threshold weight (y)
1.20
Sliding window size (T.)
Figure 15: Average achieved F1 score against the two hyperparameters of the CU threshold: window size Tw and threshold weight y. Shaded region indicates standard deviation.
20

---

Published as a conference paper at ICLR 2022
C.5 DECOMPOSING THE KL TERMS OF THE CE CRITERION
The KL terms of the CE criterion can be conveniently decomposed into the entropy and the cross entropy terms,
(et (Peh) = (geh)) +
Byan (log Pch. )
(9)
earropy
cruss entropy
DKI(gellpas) (1st))
E.g (log Pst):
(10)
entropy
cross entropy
where CE inequality can be re-written as,
(gst) + Eag. (log > - ([eh)) + Rag. Peh).
(11)
By recording the differences between the entropy and cross entropy components (see Fig. T6p. we find that decision-making is largely dominated by the cross entropy terms. This implies that the "decision-making" inequality above can be approximated as,
Eg, (log Par) Eq (logPen),
(12)
since (qx:) H (qch) = 0. This interprets the model's CE event detection as selection more well-predicted state based on the static change assumptions.
200
150
Nats
100
Entropy
Cross entropy
20
40
50
Training iterations (x1000)
Figure Comparison of the difference in entropies (qcx) ((gh)) and cross entropies EEg, (logPeh) - Egg (log Pa)l of the two competing hypotheses denoting whether or not state change has occurred at timestep and layer n. Data for this graph was acquired by training multiple instances of VPR with the Moving Ball dataset and for level = 2.
C.6 EVENT DETECTION IN THE MINIWORLD MAZE.
21

---

Published as a conference paper at ICLR 2022
VPR
VPR
VPR
L3
VTA
12
VTA
GT
L2
VTA
Figure 17: Comparison between VPR and VTA event detection using random sequences from the MiniWorld Maze dataset. GT denotes the ground-truth sequence. L2 and L3 denote events that have been captured by the second and third level of the VPR model respectively. 1.2 is able to detect events that correspond to colour changes, 1.3 detects the maze, while the VTA model detects turns between corridors. Best seen in the electronic version of the manuscript.
sparse events relating the agent's location in
22


Reviews:
[{"review_id": "xGDApLVjmW2", "reviewer": "Reviewer_RjVA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the general idea of modeling change detection as the difference in divergences, I think this indirection is novel.\nI miss some more details on the following things:\n* What is the generative model for the change detection? To my understanding, change detection is modeled as an interplay between the inference  model and the generative model, but shouldn't there be some part in the generative model alone that is handling 'changes'?\n* Can you expand on the \"non markovian property' of the model (paragraph before eq 5)? I assume this makes s_t independent from all x_t', t' > t?\n* A pseudo algorithm would be very helpful to see exactly where/how the decision to stop propagating information upwards happens.\n* It would be great (but admittingly hard) to have more expressive datasets.\n* Related to datasets: In a real world setting, I believe that one can have maybe a bit of time to look at a prefix part of a sequence to be handled, and needs to 'extract' the core properties of that sequence with respect to event onsets. How could that be handled? I assume this becomes some sort of meta learning?\n* I think you should cite \"Neural Sequnce Chunker\" by J. Schmidhuber and Unsupervised Real-Time Control through Variational Empowerment  by M. Karl et al.", "summary_of_the_paper": "Variational Predictive Routing (VPR) models continuous data as a hierarchical renewal process. More specficially, it is a hierarchical generative model that organizes the temporal hierarchy based on the rates of change of its latent representations. The core contribution hereby is that event detection is formulated as a comparison between semantically consistent KL divergences (or, in the case of CU, a moving average of KL divergences). VPR is time agnostic and can model a wide variety of temporal structures. Compared to a baseline method (VTA), VPR detects changes in synthetic datasets with high accuracy.", "summary_of_the_review": "I believe the paper has a very nice core idea, and a well thought out 'harness' around that idea. I miss more conceptualized explanations, and would have loved more experiments (and also more comparisons to comparable models).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oJEzODq17JG", "reviewer": "Reviewer_xcsC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: In my knowledge, the proposed method of learning hierarchical temporal structures by measuring prior and posterior belief states is new in the area of unsupervised representation learning for video data.\n\nWeaknesses: My major concern is about the insufficient experimental results.\n- The model is compared with VTA only on the Moving Ball dataset, which in my view, is not enough to validate the superiority of the proposed model. The authors may consider to compare with VTA on a more challenging dataset such as Bouncing Balls. I believe this dataset can better support the claim in this paper, because it has more obvious event changes represented by the interactions of multiple balls and is also used in the work of VTA.\n- The model is not compared with any existing approaches on the other three datasets. \n- How does the rollout result in Fig 5, especially the color of the object, support the claim in the caption that \"the produced rollouts illustrate model’s ability to...produce accurate...jumpy rollouts\"?\n\nOther concerns:\n- How to determine the number of layers if we do not know how many latent factors (e.g., color, shape, floor...) there are in the environment?\n- Although the authors claim in the appendix and the hyperparameters in Criterion U are robust across datasets, it is still not clear to me how to tune these hyperparameters. It would be good if the authors can give a sensitivity analysis on them. ", "summary_of_the_paper": "This paper presents a new model for unsupervised event detection, which has the following three technical contributions:\n- It proposes individual criterion and mechanisms to detect expected and unexpected observations based on the comparison of prior and posterior latent representations. (Novel)\n- It uses the above criterion to control the update rate of the latent states in a hierarchy of RNN layers. (Somewhat similar to VTA [Kim et al., 2019])\n- The paper includes some interesting experiments to support the effectiveness of the proposed method. (But not convincing enough)\n\n", "summary_of_the_review": "This paper explores an interesting problem of unsupervised representation learning in videos. Although the hierarchical architecture based on latent variables is very similar to the temporal abstraction structure in VTA [Kim et al., 2019], I think the proposed criteria for event detection are novel and reasonable. However, my major concern is about the insufficient experimental results (see my comments above), and so I give this paper a Weak Reject at this moment. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ALCBecHZz5u", "reviewer": "Reviewer_A5x3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n---------\nVPR represents a significant advance over previous models like CW-VAE (which requires a fixed \"event detection\" interval) and VTA (where event detection must be learned separately).\n\nFor the most part the method is clearly explained; Figs 1 and 2 are very helpful.\n\nThe 1D synthetic dataset is useful for understanding the method in a simple setting. Do similar results to Fig. 4 emerge from the other datasets as well?\n\nThe layerwise rollouts are an interesting way to probe the representations at each layer.\n\n\nconcerns\n--------\nI still don't fully understand the event prediction (specifically CE). Though the paper does say in words what D_{st} and D_{ch} are (which is much appreciated), the fact that their comparison is a reasonable way to define a predictable event is not clear to me. Does D_{ch} represent a difference we would expect from the structure of the model with/without seeing new data? Why does it make to compare D_{st} to this? Spelling this out a bit more clearly would help explain this crucial piece of the method.\n\nHow is the F1 score computed for change detection? Do you arbitrarily define successive events (in both the data and the model) as 0->1->0->1->... and then compute F1 on these binary sequences? Would be useful to spell this out more clearly.\n\nThe current figure layout is convenient space-wise, but makes following the results harder than it needs to be.\n\nFig 4 legend is hard to parse. \n* \"CE\" appears to be \"CE with CU enabled\"; what then is CE+CU?\n* \"CU disabled\" appears to be \"CE with CU disabled\"; what then is CU? CU only, but in the context of \"CE with CU enabled\"?\n* why is there no CU curve in Fig 4A?\n* why do 4A and 4C show different models? would make more sense to show the same models under the noise and no-noise condition\n\nFig 5A - do you have an intuition for why the color changes in L2 on every frame? Given the frequency of these events (on wall bounce or otherwise with probability 0.1) I would have expected the model to learn slower transitions between colors.\n\nExperiments are run with other baselines, but inconsistently. Running VTA on the 1D synthetic data could be instructive, and would especially be interesting to see how it performs under different noise levels. It should also be easy enough to add a CW-VAE curve to Fig. 3B. This will perform terribly, but that also opens up an opportunity to revisit the fundamental differences between VPR and CW-VAE and why the new model is an advance. I think both rollouts and random samples should be added in appendix figures for the CW-VAE and VTA. In general, the results section could start off with an explanation of the datasets (as it does already) AND an explanation of the baseline models, and why those models were chosen. That gives a bit more structure to (and motivation for) the comparisons made later on.\n\nHow will this model behave when there are slow, continuous changes in high-level features, rather than discrete changes? For example, what if colors slowly cycled through hue space in the Moving Ball dataset (much more slowly than position)? Would kind of features would VPR learn at L2? Not suggesting this as an experiment to perform, but might be an interesting direction to address in the discussion.\n", "summary_of_the_paper": "The paper introduces VPR, an unsupervised hierarchical latent variable model. The latent representation is organized into a temporal hierarchy, with deeper layers representing features with slower rates of change. This hierarchy is enabled by an event detection mechanism that finds expected events in the input using VPR's internal transition model, and finds unexpected events through changes in the latent distributions at each level of the hierarchy. The paper demonstrates VPR's performance on unsupervised event boundary discovery, and how it is able to learn disentangled representations, on a variety of synthetic video datasets.", "summary_of_the_review": "This paper presents a novel, well-motivated method for inferring temporal hierarchy in latent representations of sequence data. I lean towards accepting this paper, and would be willing to further increase my score if some of the concerns above (especially regarding the experiments) are addressed in revision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0x6hPe33RGy", "reviewer": "Reviewer_ovNG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "########################################################################## \n\nPros:\n\n-I think that exchanging ideas between neuroscience and machine learning is a great way to develop novel generative models with new capabilities such as VPR. I felt as if this work was also indirectly motivated by a desire to improve our knowledge about how to design better world models, which is a really important topic for model-based reinforcement learning.  This paper has the potential to be of significant interest to that community as well. \n\n-I was surprised at how cleanly VPR is able to learn the dynamics of distinct latent factors in each level, as evidenced clearly by the qualitative results. Despite the fact that the datasets are by and large toy-like and synthetic, this was still great to see.\n\n-The analytical experiments are well-thought-out and validate the role that the various components of the VPR play (e.g., in Figures 3,4, and 7). \n\n-The presentation is solid---I provide some suggestions to improve it below. The visualized qualitative results are really helpful for quickly grasping how VPR works and what it is capable of learning. \n\n##########################################################################\n\nCons:  \n\n-The introduction motivates this work as one which can be “beneficial for predicting future sequences”, and the beginning of the experiments section even lists “future event prediction” as a task to be studied. However, I was disappointed to find this is missing from the model design (as explained in Section 5: Event duration estimation) and is missing from the experiments.\n\n-I found it strange that TD-VAE  by Gregor et al. (which is cited in the introduction) is not discussed again in more detail under “Hierarchical generative models” in Section 2. This model learns a jumpy state-space model, supports a hierarchy of latent variables with transition functions operating at different time scales, and can handle future prediction. On the surface level, it does not have the explicit event detection mechanism of VPR, but it still seems capable of learning a transition function over “important states”. \n\n-I would have liked to see at least one comparison between TD-VAE and CW-VAE and VPR (if/when able to support future prediction) to judge VPR’s ability to be useful as a world model and perhaps to highlight where it excels whereas these previous methods do not. The experiment comparing fixed-interval versions of VPR in terms of feature disentanglement (Figure 7) is illuminating but is not really satisfying for this since it is not evaluating on long-term prediction. Without such an experiment and a comparison against these rigorous baselines, the maturity of the work still feels to be at somewhat of an exploratory stage.\n\n-I would have liked to see more analysis of the Miniworld Maze results as it is the most visually challenging environment and the dynamics of the changing latent factors are less hand-engineered. Comparing VPR against VTA here would be helpful.\n\n-Section 3 needs some work to make the presentation more clear:\n\n--I couldn’t understand Figure 2. Figure 8 in the appendix was much more clear. \n\n--I didn’t see where $p(o_t | s_t^{1:N})$ is defined.\n\n--The usage of $t$ and $\\tau$ got very confusing.  It would be good to stick with one throughout the paper, if possible.\n\n--I believe the posterior models should be conditioned on $s_{\\tau -1}^{>n}$, not $s_{\\tau}^{>n}$. \n\n-A discussion on how $\\gamma$ and $\\tau_w$ are selected, and how sensitive the model is to these variables, is missing. These values should be affected by the size of the latent dimension, which in turn affects the value of the KL used for event detection.\n\n##########################################################################\n\nSome suggestions for improvement:\n\n-In Figure 3, use a), b), c), and d) to describe sub-figures\n\n-In figure captions where the figures have error bars or confidence intervals, tell how many seeds/runs are used in the caption", "summary_of_the_paper": "This paper introduces Variational Predictive Routing (VPR), a spatiotemporal deep generative model for hierarchical video representation learning. It is motivated by a desire to implement predictive routing---a model of cognition---within a deep generative modeling framework. VPR automatically detects expected and unexpected changes at each level of the latent hierarchy, which it uses to “route” bottom-up and top-down signals. Interactions between bottom-up (from observations) and top-down signals (from the prior) are crucial for implementing this hierarchy and are a core aspect of the predictive routing framework. Each level of the hierarchy learns a time-agnostic transition model for a distinct dynamic latent factor. By design, “slower” dynamic features are modeled at higher levels of the hierarchy and “faster” dynamic features are modeled at lower levels. Experiments on synthetic video datasets validate the success of VPR at implementing predictive routing within a variational representation learning framework.", "summary_of_the_review": "I currently believe this paper is borderline and I am leaning to recommend to reject. \nThis is because I think the way the paper frames VPR as being comparable with prior methods like CW-VAE and as a potentially-useful framework for long-term prediction is promising, yet more work is needed (described in my review) to take this work from an exploratory stage to ready for publication. I look forward to hearing from the authors during the rebuttal period on this matter.\n==========================================================================\nUpdate after author response: The authors have provided satisfactory responses to my concerns and revised the paper with extra results and discussions accordingly. I have increased my score to a 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your submission to ICLR.

This paper considers a variational inference hierarchical model called Variational Predictive Routing.

Prior to discussion, several reviewers were on the fence about the paper, most notably having concerns about some of the experimental results as well as various clarity issues throughout the paper.  However, the authors did a really nice job addressing many of these concerns.  Ultimately, several of the reviewers updated their scores, leading to a clear consensus view that this paper is ready for publication.  We really appreciate your effort in providing additional details and results.

Please do keep in mind the concerns of the reviewers when preparing a final version of the manuscript.
Decision: Accept (Poster)