ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning
Authors: Zhiyu Yao
Yunbo Wang
Haixu Wu
Jianmin Wang
Mingsheng Long
Venue: ICLR 2022 Submitted (2022)
Keywords: Predictive Learning
Video Prediction

Abstract: Learning predictive models for unlabeled spatiotemporal data is challenging in part because visual dynamics can be highly entangled in real scenes, making existing approaches prone to overfit partial modes of physical processes while neglecting to reason about others. We name this phenomenon \textit{spatiotemporal mode collapse} and explore it for the first time in predictive learning. The key is to provide the model with a strong inductive bias to discover the compositional structures of latent modes. To this end, we propose ModeRNN, which introduces a novel method to learn structured hidden representations between recurrent states. The core idea of this framework is to first extract various components of visual dynamics using a set of \textit{spatiotemporal slots} with independent parameters. Considering that multiple space-time patterns may co-exist in a sequence, we leverage learnable importance weights to adaptively aggregate slot features into a unified hidden representation, which is then used to update the recurrent states. Across the entire dataset, different modes result in different responses on the mixtures of slots, which enhances the ability of ModeRNN to build structured representations and thus prevents the so-called mode collapse. Unlike existing models, ModeRNN is shown to prevent spatiotemporal mode collapse and further benefit from learning mixed visual dynamics. 

Under review as a conference paper at ICLR 2022
MODERNN: HARNESSING SPATIOTEMPORAL MODE COLLAPSE IN UNSUPERVISED PREDICTIVE LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning predictive models for unlabeled spatiotemporal data is challenging in part because visual dynamics can be highly entangled in real scenes. Due the interference and competition between the learning of various dynamics modes, we find that most existing approaches often degenerate to learning ambiguous motion patterns and thus producing blurry prediction results. We name this phenomenon spatiotemporal mode collapse (STMC) and explore for the first time in the context of unsupervised predictive learning. The key idea is to provide the model with a strong inductive bias to discover the modular, compositional structures latent modes. To this end, we propose ModeRNN, which introduces decoupling- aggregation framework to learn structured hidden representations between recurrent states. It first introduces set of mode slots with independent parameters extract individual components of visual dynamics. Considering that multiple spatiotemporal modes may co-exist in a sequence, we then use learnable importance weights to adaptively aggregate the slot features into a unified hidden representation for recurrent updates.
series of experiments on large-scale, real-world datasets, ModeRNN is shown to better mitigate the so-called mode collapse and thus further benefit from the learning process on diverse visual dynamics.
INTRODUCTION
Predictive learning is unsupervised learning paradigm that has shown the ability discover the spatiolemporal modex of visual dynamics (Xu et al. 2019 Goyal et al.) 2021). However, for large- scale and real-world datasets (see Figure []. the modes visual dynamics can be highly entangled
and difficult to learn due to the richness of data environments, the diversity of object interactions, and the complexity of motion patterns. For clarity, in the following discussion, spatiotemporal modes are considered to have the following properties:
1. A spatiotemporal mode refers to representation subspace that corresponds to a family of similar, but not predefined, visual dynamics.
2. Multiple spatiotemporal modes naturally exist in real-world data, even in single frame.
3. We assume the i..dd setup to allow all videos to share the same set of spatiotemporal modes in a dataset. Different data may have different compositional structures over the modes.
Under these assumptions, video prediction models are required to (i) decouple the potentially mixed spatiotemporal modes from raw video frames, (ii) understand the compositional structures on top of the learned modes, and (iii leam the state transitions based on the compositional structures.
Otherwise, since the learned dynamics with respect to different modes may interfere and compete during training, remains challenging for the prior art in video prediction to generate less blurry future frames based on an ambiguous understanding of mixed physical processes. We refer to this empirical phenomenon as spatiotemporal mode collapse (STMC), which is mainly caused by the collapse of learned representations into invalid subspaces when compromising to multiple spatiotemporal modes in the training set. Unlike the widely concerned mode collapse problem in generative adversarial networks. STMC has not drawn much attention because predictive learning is supposed to be well constrained by the image reconstruction loss. However, due to the limitation of model size, STMC occurs when the model cannot effectively decouple mixed spatiotemporal modes and infer their

---

Under review as a conference paper at ICLR 2022
Train: Future prediction
Test results
1.00
PhyDNelISP)
Env 1: Berkeley
Env 2: Google
Env 1
Env 2
0.95
PhyDNet
ModeRNNNI SP)
0.90
ModeRNN
0.85
Penn
Stanford
Env 3
Env 4
0.80
Predictive models collapse to blurry motion in the presance of complex dynamics modes
0.65
Berkeley
Google
Penn
Stanford
Figure 1: Qualitative and quantitative illustrations of spatiotemporal mode collapse on the large-
scale, real-world RoboNet dataset. (Left) RoboNet contains complicated dynamics modes in videos collected in various environments. The prediction results collapse to blurry motions due to the incompatibility of learning various dynamics modes. (Right) Unlike the prior art (Guen & Thome 2020) that performs better when separately trained in the subset of each environment (denoted as "SP"), the proposed ModeRNN manages to benefit from large-scale learning in all environments.
underlying structures. As result, its responses to different modes tend to lose diversity and may collapse to a meaningless average of multiple representation subspaces of valid modes.
In Figure T (left), we can observe the existence of STMC on a large-scale video dataset named RoboNet Dasari et al. 2019), in which potential spatiotemporal modes may come from seven different robot platforms (e.g., Baxter and WidowX), four data collection environments (e.g., Berkeley and Stanford), and a variety of unlabeled robot control tasks (e.g., pushing and grasping). An additional outcome of STMC is that we can achieve performance gain when training individual models in separate subsets with remarkably different visual dynamics, as shown in Figure [ (right). However, such dilemma prevents the model from growing into big ones that allow scalable training on large-scale, natively multimodal spatiotemporal sequences.
We explore STMC for the first time in unsupervised predictive learning. The core idea to provide a strong inductive bias for the predictive model to discover the compositional structures of latent modes. To this end, we propose ModeRNN, a new modular recurrent architecture that learns structured hidden representations through set of mode s where each of them responds to the representation subspace of a single spatiotemporal mode. ModeRNN also introduces decoupling aiiingggggggggtton framework to process the slot features in three stages, which is completely different from existing predictive models with modular architectures (Xu al. 2019 Goyal et al. 202T).
The first stage is recurrent state interaction and slot binding, in which we use the multi-head attention mechanism (Vaswani et al. (017) to enable the memory state to interact with the input state and previous hidden state of RNNs. We name the memory state "slot bus", because for each sequence, it is initialized from a using the slot features at each time step. By using the slot bus the queries, multi-head attention can naturally decouple modular components from hidden representations and bind them to particular mode slots. Features in each slot are then independently modeled using per-slot convolutional parameters. The second stage in each ModeRNN unit is slot fusion, motivated by the assumption that, there can be multiple spatiotemporal modes in a single video and similar videos can be represented by similar compositional structures over the mode slots. Therefore, we assign slot features with learnable importance weights and aggregate them into a unified hidden representation, which is then used in the third stage to update the slot bus and generate the output state of the ModeRNN unit.
Gaussian distribution with learnable parameters, and thereafter refined
We empirically show the existence of STMC on five datasets, and include the results on three real- world datasets in the manuscript, including the large-scale RoboNet dataset that has various data collection environments and multiple robot control tasks, the KTH dataset with six types of human actions that has been widely used by previous literature, and the radar echo dataset for precipitation forecasting that contains time-varying modes of seasonal climates. In addition, we include results on Mixed Moving MNIST dataset and the Human3.6N dataset in the appendix. In series of quantitative and visualization results, we demonstrate the effectiveness of ModeRNN in mitigating STMC and learning from highly entangled visual dynamics.
'The concept of "slor" was initially introduced by I Locatello et al. 2020) to denote the object-centric features in static scene understanding. We horrow this term here for the subspaces of spatiotemporal representations.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
RNN-based predictive models. Many deep learning models based on RNNs have been proposed
for spatiotemporal prediction (Ranzato al., 2014 Srivastava et al. 2015 Shi et al. 2015 Oh
et al. 2015 De Brabandere et 2016; Villegas et al. 2018). Shi et al. 2015) integrated 2D
convolutions into the recurrent state transitions of standard LSTM and proposed the convolutional
LSTM (ConvLSTM) network, which can model the spatial correlations and temporal dynamics in a
unified recurrent unit. More recent approaches have extended the prediction ability of ConvLSTM in
different aspects (Wang et al., 2017 Olu et al. 2018 Wang et al. 2019b a: Yao et al. 2020 Guen
2021 For example, as an important compared model of our approach, SA-ConvLSTM Lin et al. 2020 incorporates self-
& Thome 2020: Yu et al. 2019 Su al. 2020] Lin et al. 2020 Lee et al.
attention in the recurrent state transitions in ConvLSTM to obtain more global context information across time. However, unlike our approach, it does not learn decoupled representations to understand individual components in complex visual dynamics. Besides deterministic models, probabilistic models were proposed to explicitly consider the uncertainty in future prediction (Mathieu et al. 2016
Vondrick et al.] 2016; Tulyakov et al.) 2018 Xu et al. 2018;; Wang et al.] 2018 Denton & Fergus
2018|Castree et al.) 2019 Kwon & Park 2019: Bhagat et al., 2020). We use typical stochastic
video generation approach (Denton & Fergus 2018) based on conditional VAE as a compared model.
Unsupervised predictive learning for spatiotemporal disentanglement. Previous work has fo- cused on learning to disentangle the spatial and temporal features from visual dynamics (Denton) ctt al. 2017 Guen & Thome 2020 Hsiih et al., 2018: Wu et al. 202I). These methods factorize spatiotemporal data into feature subspaces with strong priors, e.g., assuming that the spatial infor- mation is temporally invariant. Another line of work is to learn predictive models for unsupervised scene decomposition such as (Xu et al. 2019: Hsieh et al., 2018). Unlike the above models, our approach uses a set of modular architectures in the recurrent unit to represent the mixed spatiotem- poral dynamics. The most relevant work to our method is the Recurrent Independent Mechanism (RIM) (Goyal et al. 202Tp, which consists of largely independent recurrent modules that are sparsely activated and interact via soft attention. ModeRNN different from RIM in three aspects. First, is specifically designed to tackle STMC in real-workee environments. Second, it learns modular features by incorporating multi-head attention in the recurrent unit, and performs state transitions on composi- tional features with learnable importance weights. Third, the modular structures in ModeRNN are frequently activated responding to the mixed visual dynamics. ModeRNN is compared with the state of the art in Section 4 including SA-ConvLSTM (Lin et al. 2020)) PhyDNet (Guen & Thome 2020
CrevNet (Yu ct al. 2019, RIM GGoa et al. 202 and LMC (Lee et al. 2021
MODERNN
We propose ModeRNN reduce spatiotemporal mode collapse (STMC) in unsupervised predictive learning. The key idea is to build cooouplinnggggregatin framework to model the recurrent state transitions of mixed spatiotemporal modes. In this section, we first discuss the basic network components in ModeRNN and then describe the details in the decoupling-aggregatioon recurrent unit.
3.1
MODE SLOTS & SLOT BUS
Mode slots. The decouplin--aggregation framework is built upon a set of hidden representations named mode slots. The term slot is in part borrowed from previous work for unsupervised scene decomposition (Locatello et al.) 2020). We use here to respond to a family of similar visual dynamics, that is, we aim to bind each mode slot to the representation subspace of each spatiotemporal mode one-to-one. Slot features can be viewed as latent factors that can explicitly improve the unsupervised decoupling of mixed dynamics across the dataset.
Slot bus. Assuming that multiple spatiotemporal modes naturally co-exist in real-world videos, all slots dynamically respond with different importance weights to form compositional representations, which are then used to update a long-term memory state, termed slot bus. The hierarchical structure of mode slots and the slot bus leads to a better understanding of the complex and highly mixed dynamic patterns without mode annotations. From similar data samples, the model is allowed to learn similar compositional structures over the slots. On the contrary, for distinct visual dynamics, it shows significant differences in the learned importance weights update the slot bus features. Therefore, it
3

---

Under review as a conference paper at ICLR 2022
provides solution to STMC. Specifically, the slot bus is initialized from learnable, varii-v Gaussian distribution, whose mean and variance encode the global priors for the entire dataset.
3.2
MODECELL
To learn and leverage the mode slots, we introduce a novel recurrent unit named ModeCell, which follows a decoupling-aggree aggregation framework with three modules, i.e. the state interaction and slot binding module, the adaptive slot fusion module, and the slot bus transition module.
3.2.1 STATE INTERACTION AND SLOT BINDING
This module decouples the mixed spatiotemporal modes from raw video frames to mode slots. To achieve this, as shown Figure it first uses multi-head attention to allow the slot bus to interact with the input and hidden states of the unit, and thereby divides them into separate subspaces. It then binds the features to each mode slot using neural networks with per-slot independent parameters.
Multi-head attention (Vaswani al. 2017) widely used in neural language and image processing, and in this work, it is incorporated in the state transitions of ModeRNN. This mechanism allows interactions between the previous slot bus B, the current input state Xx, and the previous hidden state H (see Figure 2 Formally, at each time step, we first apply 2D convolution projections to B.-1. We then flatten the result to ID and split into N mode slots along the channel dimension, such that fslot, slot, } Split (Reshape (We * B(-1)). Note that B;- E R. xd x (d,+ and sloth_ E (ddadu(d
where d, is the channel number of input state, d, that of hidden state, and d, du indicates the spatial resolution of the slot bus tensor. To improve efficiency, we two 3 3 depth-wise separable convolutions (Chollet 2017 for We- We use {slot, 1, ................................ slot, the queries (2),.... ON) } in multi- attention, and apply similar operations obtain keys [K)..... and values {V!,... Vi2 based on the concatenation of input state and hidden state, I, [X,, H.-1]. We then perform multi-head attention and reshape the N output slot features back to 3D tensors:
ORKTT vdn
slot," = Reshape
softmax
Vi
n (1,..,,,NN
(1)
where dg is the dimensionality of the key vectors used as a scaling factor.
Multi-head attention brings two benefits to the forward modeling of spatiotemporal data. First, since B.-1 can unrolled along the recurrent state transition path to be represented as the transformation of slot features at the previous time step, using B,-1 attention queries allows the model to extract features from X₂ and H4-1 by jointly attending to prior information at different slots. Second, the architecture with N attention heads naturally help factorize the hidden representation into N subspaces, corresponding to N spatiotemporal modes. The output at each attention head is then updated by per-slot feed-forward network (FFN) with independent parameters:
slot, FFN" (slot") = max (0, W * slot,"), ; n [1,
where denotes the convolution operator and Wen are x 3 convolution kernels. Through random parameter initialization and stochastic gradient descent, the independent networks (WFrx.... W WFN) would most likely be optimized into parameter subspaces far from each other, thus forcing the slots bind various modes in mixed visual dynamics.
3.2.2 ADAPTIVE SLOT FUSION
We explicitly consider the co-existence of various modes in video frame, and use the adaptive slot fusion module to aggregate the decoupled slot features through importance weights. The similarity of visual dynamics is reflected in the similar importance weights of mode slots, while different visual dynamics can be distinguished by different significance of each slot feature. This mechanism prevents ModeRNN from making ambiguous predictions in highly non-stationary data environments.
The implementation of this module is largely inspired by the mixture of experts (Shazeer et al. 2017). which introduces the gated networks control the information flow from base models an ensemble. We improve the gated architecture to dynamically aggregate decoupled slots [slot) ...., slor," } with
4

---

Under review as a conference paper at ICLR 2022
B
State interaction & slot binding
Adaptive slot fusion
Slot bus transition
B
Spatiotemporal slots
Compositional representation
Multi-head attention
slot
F
stot,
0
stot,
Pervlla
slot
tanh
tanh
Per-slot
Peneee
H4
PC
L
Ot
H
Concat
0
Generate
X
importance
Output state ModeCell
FC
weights
Figure 2: ModeCell tackles spatiotemporal mode collapse via decoupling-aggrenn aaggregation framework based on mode slots. Multiple ModeCells are stacked form the complete architecture of ModeRNN.
learnable importance weights {w! ...., w and finally have the compositional representation F based on the learned importance weights and corresponding slot features:
w/ FC FC
Σ> 1=1
I((i)))), (O,
(3)
dn di
N w/
Fi = w I
slot,.
(4)
n=1
In the first line, use the global average pooling encode the contextual information, which is the concatenation of current input state and previous hidden state, into dimensionality (d.+ For memory efficiency, we use a simple fully connected (FC) layer to reduce the dimensionality and get the compact feature in RR +d,)/2 Then we introduce (N + 1) slot-independent FC layers FC to generate the importance weights for the original input I, and each mode slot,, where a denotes the Sigmoid activation function. In the second line, we aggregate all mode slots as well as the input into a compositional representation Fe based on the learned importance weights.
3.2.3 SLOT BUS TRANSITION
The compositional state F; builds a hicrarchical representation on top of the slot features. We use four sets of F; and I, to form the input gate i. forget gate fú, output gate or. and modulated slot bus input 9t. We then update the slot bus state following an LSTM-style recurrent transition mechanism:
9t
tanh σ
W W
i
[r,
B. = 1. J....... + Og.
(5)
W
W.
Finally, we generate the output state ModeCell as H4 = or (B.h(B.). He taken as inputs by the next ModeCell the upper level when multiple ModeCells are being stacked in ModeRNN, In other words, ModeCell is to ModeRNN what LSTM is to the stacked LSTM network.
EXPERIMENTS
We quantitatively and qualitatively evaluate ModeRNN on three real-world datasets the main paper. We also conduct experiments the Human3.6N, dataset and the Mixed Moving MNIST dataset. Due to page limitation, we include these additional results Appendix and Appendix
RoboNet:
The RoboNet dataset (Dasari et al. 2019 includes more than 15 million video frames collected by different robot arms from 4 environments. thus contains a large diversity of spatiotemporal modes of rigid motions. We randomly select 4,000 videos for
5

---

Under review as a conference paper at ICLR 2022
Derkeley Google
Barkaley
Google
Slot
Importnnca wnight
Perin Stanfard
Pern Stanfordd
Slot 4
5.6
Berkuley
Google
Stanford
[6] tsme for PhyDNet
(6) tome for slot bus
(c) tsne for mode slots
(d) importance waights
Figure 3: (a) Demonstration of STMC on RoboNet using existing vidco prediction model for disentangling visual dynamics (Guen & Thome 2020). (b) The slot bus ModeRNN learns distinct representations for samples from different environments. (c) The four slots in ModeRNN learn decoupled features for various spatiotemporal modes. (d) The importance weights of mode slots respond differently to different data environments in RoboNet, i.e., families of similar video sequences.
testing and use the others for training. We follow the action-free and cetion-conditionel video prediction setups from the work Babaeizdeeh et al. (2018). In the fetien-free setup, models are trained to predict the next 10 frames from the previous 5 observations. In the action-conditioned setup, models are trained to predict 10 frames based on observations along with the robot action vectors at all time steps. All images are resized to 64 64.
The KTH dataset (Schuldt et al. 2004) contains 6 action categories and involves 25 subjects in 4 different scenarios. It thus naturally contains various modes responding to similar motion dynamics. We use person 1-16 for training and 17-25 for testing, resize each frame to the resolution of 128 x 128, and predict 20 frames from 10 observations.
KTH:
This dataset contains 30,000 sequences of radar echo maps for training, and 3, 3,769 for testing. It naturally contains various spatiotemporal modes of fluid dynamics due seasonal climate. Models are trained to predict the next 10 radar echoes based on the previous 10 observations. All frames are resized to the resolution of 384 x 384.
Radar Echo:
We train the models with the L2 reconstruction loss and use the ADAM optimizer (Kingma & Ba 2015) with starting learning rate of 0.0003. The batch size is set to 8, and the training process is stopped after 80,000 iterations. All experiments are implemented Py Torch (Paszke et al.) (019) and conducted on NVIDIA TITAN-RTX GPUs. We run all experiments three times and use the average results for quantitative evaluation. Typically, we use 4 x 64-channel stacked recurrent units in most RNN-based compared models, including ModeRNN, ConvLSTM (Shi al. 2015). PredRNN (Wang ct al. 2017), SA-ConvLSTM (Lin et al. 2020), and PhyDNet (Guen & |
ModeRNN is
also compared with state-of-the-art methods, including SVG (Denton & Fergus 2018
SAVP Lee
et al. 2018), CrevNet (Yu et al. 2019), RIM (Goyal et al. 2021), and LMC (Lee et al.
2021
4.1
DEMONSTRATION OF SPATIOTEMPORAL MODE COLLAPSE
STMC occurs large-scale, real-world datasets. RoboNet naturally has the label of the data
collection environments, including Berkeley, Google, Penn and Stanford. To demonstrate that STMC does exist in real-world datasets, and our approach can overcome STMC, assume that different enviromments correspond to different combinations of the spatiotemporal modes. As we have seen in Figure ! training the existing models using data samples from all environments leads to ambiguous predictions of object's 's future movements; While training separate models on the subset of each environment leads to better overall performance. From these results, we may conclude that previous methods degenerate drastically when using all training samples with mixed visual dynamics. These results perfectly match the t-SNE (Van der Maaten & Hinton 2008) visualization in Figure a),, where the cell output states of PhyDNet (Guen & Thome (2220] are severely entangled and collapss to less discriminative subspaces. In contrast, from Figure 3(b), the compositional slot bus features in ModeRNN show 4 clusters with clear boundaries, corresponding to four robot environments.
Does STMC still exist in supervised predictive learning? One may concern that why not use the environment labels as input to help learning the environment-specine representations. There are two reasons. First, in reality, most spatiotemporal modes are implicit and cannot be pre-defined annotated, even in RoboNet. Therefore, simply using the sparse labels for the environments

---

Under review as a conference paper at ICLR 2022
Table 1: Results the RoboNet dataset in the action-free setup.
MODEL
SSIM (1)
MSE ())
PARAM (MB)
MEM (GB)
CONVLSTM (SHI ET AL.
2015
0.725
133.4
8.2
2.6
PREDRNN (WANG ET AL.
2017
0.787
110.9
11.8
3.5
SVG (DENTON & FERGUS
2018
0.792
108.2
15.2
6.5
SA-CONVLSIM ET
2020
0.753
116.5
10.5
3.4
PHYDNET (GUEN & THOME
2020
0.742
122.5
14.4
4.5
PHYDNET W/ ENVIRONMENT LABEL
0.750
116.9
14.4
4.5
LMC LEE ET AL., 2021)
0.783
113.4
12.4
5.9
CREVNET (YUETA 2019 w/ ST-LSTM
0.794
109.4
7.0
3.3
MODERNN
0.831
91.9
6.4
3.2
Table 2: Results on the RoboNet dataset for action-conditioned video prediction.
MODEL
SSIM (t)
MSE (t)
TRAINING TIME (H)
PHyDNET (GUEN & THOME 2020)
0.813
106.2
20
SVG DENG ET AL. 2016)
0.835
99.1
23
SAVP PLEELEEE 2018)
0.842
96.5
25
MODERNN
0.874
83.5
robot types cannot completely address the STMC problem. Second, as shown in the 6-th line in Table 1 using the environment labels does not empirically improve the prediction results by large margin. We here follow the well-established practice Conditional-GAN (Mirza & Osindero 2014)
to encode one-hot environment label to PhyDNet (Guen & Thome 2020
4.2 VISUALIZATION OF REPRESENTATIONS LEARNED BY MODERNN
Besides the visualization slot bus in Figure b), we testify the mode decoupling ability of ModeRNN by visualizing the slot features Figure c).. We can see that features of the mode slots are clustered into 4 groups, indicating the ability to disentangle various spatiotemporal modes. In Figure d), we use the averaged importance weights {w(] Jn=1 on each slot to analyze how the adaptive slot fusion module works. We can see that different robot environments lead to different dependence distribution over the mode slots.
4.3 RESULTS ON THE ROBONET DATASET
Action-free video prediction. In Table we show the per-frame quantitative results and computa- tional efficiency for action-free video prediction. As we can see, ModeRNN achieves state-of-the-art overall performance with fewer parameters compared with existing approaches. It can consistently benefit from training with complex visual dynamics in the entire dataset (see the bar chart Figure D Furthermore, shown in the first example in Figure 4 ModeRNN is the only method that captures the exact movement of the robot arm, while other models make blurry predictions in the motion area.
Action-conditioned video prediction. We also conduct experiments under the action-conditioned setup. encoding the inputs of robot action signals using the action fusion module from PredRNN-V.
(Wang et al. 22221. We mainly compare the performance of ModeRNN with that of SVG (Denton) & Fergus 2018) and SV2P (Lee et al. 2018). which are strong baselines as their effectiveness on RoboNet has been well validated in the prior literature. For these models, we draw 100 prediction samples from the prior distribution given a testing sequence and report the results with the best
SSIM scores. From Table 2 ModeRNN achieves the best performance in the shortest training time. We show the qualitative results in the second case in Figure 4 With the help of the action inputs, ModeRNN has more accurate predictions about the moving trajectories the robot arm and the object, while the compared models still suffer from the blur effect.
7

---

Under review as a conference paper at ICLR 2022
Inputs
True future Predictions
ModeRNN
SA-ConVLSTM
PhyDNet
Inputs
True future & Predictions
ModeRNN
SAVP
SVG
Figure 4: Showcases of future prediction RoboNet in (Top) action-free and (Bottom) action- conditioned setups. These examples are randomly sampled from the Stanford environment. We provide examples for other data collection environments in Appendix
Table Results on the KTH dataset and the radar echo dataset. For SVG, we report the best results from 100 output samples per input sequence. indicates the result directly copied from the reference.
KTH
RADAR
MODEL
PSNR (t)
LPIPS (L)
CSI30 (t)
MSE (L)
CONVLSTM (SII ET AL
2015)
24.12
0.231
0.354
97.6
TRAIGRU SHLET AL.
2017)
*66.97
*0.219
0.357
89.2
PREDRNN WWNGG AL.
2017
$27.47
0.212
0.359
84.2
SVG (DENTON FERGUS
2018
27.73
0.196
CONY-TT-LSTM (SU ET AL.
2020
27.59
0.198
0.363
87.6
SA-CONVLSTM (LIN ET AL
2020
29..33
0.193
0.362
86.1
PHYDNET (GUEN & THOME
2020
28.69
0.188
0.358
92.1
CREVNET YY AL. 2019
28.82
0.183
0.381
81.5
LMC (LEE ET AL 2021
*28.61
0.195
0.361
93.5
MODERNN (KTH. N
6; RADAR: N =4
29.45
0.173
0.428
65.1
4.4 RESULTS ON THE KTH DATASET
this dataset, we use the frame-wise peak signal-to-noise ratio (PSNR) and learned perceptual image patch similarity (LPIPS) (Zhang et al. 2018) as evaluation metrics. We use 6 mode slots in each ModeCell. In the left column in Table 3 we show the quantitative results and find that
ModeRNN performs best among all compared methods, including the state of the art proposed in
provide the qualitative comparisons in Appendix where observe that ModeRNN 2021). We
recent two years (in et al. 2020 Guen & Thome 2020 Yu et al. 2019 Lee et al.
predict
the precise position of the moving person. Notably, we also evaluate ModeRNN on a larger human action dataset, Human3.61 3.6M, and provide the results in Appendix
8

---

Under review as a conference paper at ICLR 2022
Table 4: Ablation study (Left) the effectiveness of each model component and (Right) the number of mode slots in each ModeCell. Experiments are conducted on the oction-free RoboNet dataset.
MODEL
MSE
MODEL (# MODE SLOTS)
MSE
MODERNN
91.9
MODERNN (N 1)
118.1 I
MODERNN W/O SLOTT BINDING
132.5
MODERNN (N = 2)
103.4
MODERNN w/o PER-SLOT FFN
110.7
MODERNN (N = 3)
94.5
MODERNN w/o ADAPTIVE SLOT FUSION
121.2
MODERNN (N =
91.9
MODERNN w/O GATED SHORTCUT (wl Z.)
128.3
MODERNN (N =5)
93.3
4.5 RESULTS ON THE RADAR ECHO DATASET
Besides the frame-wise MSE, we use the Critical Success Index (CSI) metric, which is defined as Hits
CSI Hiss--Misses FalseAlarms' FaleeAll where hits correspond to the true positive, misses correspond to the false positive, and false alarms correspond to the false negative. A higher CSI indicates better forecasting performance, and it is particularly sensitive to high-intensity echoes. We set the alarm threshold to 30 dBZ for this radar benchmark. As shown in the right column in Table 3 ModeRNN achieves the state-of-the-ar overall performance and significantly outperforms the prior art for precipitation forecasting, i.e., TrajGRU (Shi et al. (017), with a CSI result of 0.428 vs. 0.357 and an MSE result of 65.1 vs. 89.2. We include prediction showeases of the compared models in Figure [4in Appendix E Both the quantitative and qualitative results show that ModeRNN effectively learns the non- non-stationary dynamics of rich spatiotemporal modes from complicated meteorological systems in the real world.
4.6 ABLATION STUDY
Effectiveness of each model component. In the left column of Table we analyze the efficacy of each component in ModeRNN on the action-free RoboNet dataset and have the following observations. First, removing the slot binding module increases the prediction error by 44.2%, showing the necessity of learning to decouple the dynamics using separate mode slots based on multi-head attention. Second, removing the adaptive slot fusion module increases the prediction error by 31.9%, which strongly demonstrates that is crucial to learn the state transitions upon the compositional representations based on the slot features. Third, the per-slot FFN in the slot binding module and the gated shortcut (wf Z,) in the slot fusion module also show significant impact on the final performance. These results verify that parameter isolation is effective in mode decoupling, and reveal the positive effect of an adaptive fusion rich appearance information and compact spatiotemporal dynamics. Finally, the framework that integrates the above techniques in unified modular model achieves the best performance.
entire
Number of mode slots. In the right column of Table 4 we adjust the number of mode slots RoboNet. We find that the performance first increases rapidly with the growth of the slot number and achieves the best performance at N 4. Notably, using a single slot in each ModeCell achieves similar performance to SA-ConyLST recurrent state transitions but does not have mode decoupling framework. We perform similar grid search on other datasets and finally set N = 6 on KTH and N = 4 on the radar echo dataset.
(Lin et al. 2020). which incorporates self-attention in the
CONCLUSION
In this paper, we demonstrated new phenomenon of spatiotemporal mode collapse (STMC) when training unsupervised predictive models on real-world datasets with highly mixed visual dynamics. Accordingly, we proposed ModeRNN that effectively learns modular features using a set of mode slots. To discover the compositional structures in spatiotemporal modes, ModeRNN adaptively aggregates the slot features with learnable importance weights. Compared with existing models, ModeRNN was shown to prevent the collapse of future predictions, improving qualitative and quantitative results on five datasets. A potential limitation of this work is that, although ModeRNN can be easily generalized as the world model for model- based robot control and has been evaluated on the RoboNet dataset for action-conditioned video prediction, its effectiveness on downstream tasks has not been explored by performing the entire pipeline of model predictive control. We would like study it in future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. ICLR, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, pp. 151-175, 2010.
Sarthak Bhagat, Shagun Uppal, Zhuyun Yin, and Nengli Lim. Disentangling nultiple features in video sequences using gaussian processes in variational autoencoders. In ECCV, pp. 102-117, 2020.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional VRNNs for video prediction. In CVPR, pp. 7608-7617, 2019.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pp. 1251-1258, 2017.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. RoboNet: Large-scale multi-robot learning. arXiv preprint arXiv: 1910. 1215, 2019.
Bert De Brabandere, Xu Jia, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In NeurlPS, pp. 667-675, 2016.
Dingxiong Deng, Cyrus Shahabi, Ugur Demiryurek, Linhong Zhu, Rose Yu, and Yan Liu. Latent space model for road networks to predict time-varying traffic. In SIGKDD, pp. 1525-1534, 2016.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In ICML, pp. 1182-1191, 2018.
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In NeurIPS, pp. 4417-4426, 2017.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In ICIR,, 2021.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In CVPR, pp. 11474-11484, 2020.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In NeurIPS, 2018.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.66 3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, pp. 1325-1339, 2013.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Yong-Hoon Kwon and Min-Gyu Park. Predicting future frames using retrospective cycle GAN, In CVPR, pp. 1811-1820, 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.00 01523, 2018.
Sangmin Lee, Hak Gu Kim, Dae Hwi Choi, Hyung-ll Kim, and Yong Man Ro. Video prediction recalling term motion context via memory alignment learning. In CVPR, 2021.
Zhihui Lin, Maomao Li, Zhuobin Zheng, Yangyang Cheng. and Chun Yuan. Self-attention convlstm for spatiotemporal prediction. In AAAI, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NewrlPS, 2020.
10

---

Under review as a conference paper at ICLR 2022
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv: 14f1:1411..784 2014.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard I. Lewis, and Satinder Singh. conditoaan video prediction using deep networks in atari games. In NeurlPS, pp. 2863-28711 2015.
Marc Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video
prediction. In ECCV, pp. 716-731, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, geghpperformnce deep 2019.
learning library. In NeurlPS, pp.
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv: /4/. 2014.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: a local SVM approach. In ICPR, pp. 32-36, 2004.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint 1xX.v:11011.66388. 2017.
Xingjian Shi, Zhourong Chen, Hao Wang, Din Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In NeurlPS, pp. 802-810, 2015.
Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit- Din Yeung, Wai-kin Wong, and Wang- chun Woo. Deep learning for precipitation nowcasting: A benchmark and a new model. In NeurIPS, pp. 5617-5627, 2017.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using LSTMs. In ICML, pp. 843-852, 2015.
Jiahao Su, Wonmin Byeon, Furong Huang, Jan Kautz, and Animashree Anandkumar, Convolutional tensor-train Istm for spatio-temporal learning. In NewrlPS, 2020.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In CVPR, pp. 1526-1535, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint arXiv:1812.017177 2018
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 1706.v:/000.00 03762, 2017.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hicrarchical prediction. In ICML, pp. 3560-3569, 2018.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurlPS, pp. 613-621, 2016.
Ting-Chun Wang. Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. video-to-video synthesis. arXiv preprint arXiv: 0868006601, 2018
11

---

Under review as a conference paper at ICLR 2022
Yunbo Wang, Mingsheng Long, Jianmin Wang. Zhifeng Gao, and S Yu Philip. PredRNN: Recurrent neural networks for predictive learning using spatiotemporal Istms. In NeurlPS, pp. 879-888, 2017.
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li. Fei-Fei. Eidetic 3D LSTM: A model for video prediction and beyond. In ICLR, 2019a.
Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S. Yu. Memory memory: predictive neural network for learning higher-order con-stationarity from spatiotemporal dynamics. In CVPR, pp. 9154-9162, 2019b.
Yunbo Wang, Haixu Wu, Jianjin Zhang. Zhifeng Gao, Jianmin Wang, Philip S Yu, and Mingsheng Long. Predrnn: recurrent neural network for spatiotemporal predictive learning. arXiv preprint 0rX1v:2103.09504 2021.
Haixu Wu, Zhiyu Yao, Jianmin Wang. and Mingsheng Long. MotionRNN: flexible model for video prediction with spacetime-varying motions. In CVPR, pp. 15435-15444, 2021.
Jingwei Xu, Binghing Ni, and Xiaokang Yang. Video prediction via selective sampling. In NeurlPS, pp. 1712-1722, 2018.
Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T Freeman, Joshua B Tenenbaum, and Jiajum Wu. Unsupervised discovery of parts, structure, and dynamics. In ICLR, 2019.
Zhiyu Yao, Yunbo Wang, Mingsheng Long, and Jianmin Wang. Unsupervised transfer learning for spatiotemporal predictive networks. In ICML, pp. 10778-10788, 2020.
Wei Yu, Yichao Lu, Steve Easterbrook, and Sanja Fidler. Efficient and information-preserving future frame prediction and beyond. In ICLR, 2019.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features perceptual metric. In CVPR, pp. 586-595, 2018.
12

---

Under review as a conference paper at ICLR 2022
A QUALITATIVE RESULTS ON ROBONET FOR DIFFERENT ENVIRONMENTS
We further provide the prediction samples from other environments of RoboNet under both the action-free and action- contitooon setups, including Google, Penn, and Stanford.
Inputs
True future & Predictions
ModeRNN
SA-ConvLSTM
PhyDNet
Inputs
True future & Predictions
ModeRNN
SAVP
SVG
Figure 5: Examples of (Top) action-free and (Bottom) oetion-conditionas video prediction from the Google environment.
Inputs
True future & Predictions
ModeRNN
SA-ConVLSTM
PhyDNet
Inputs
True future & Predictions
ModeRNN
SAVP
SVG
Figure 6: Examples of (Top) action-fred free and (Bottom) action- cetion-coniitooed video prediction from the Berkeley environment.
13

---

Under review as a conference paper at ICLR 2022
Inputs
True future Predictions
ModeRNN
SA-CorvLSTM
PhyDNet
Inputs
True future & Predictions
ModeRNN
SAVP
SVG
Figure 7: Examples of (Top) action free and (Bottom) action-conditions video prediction from the Penn environment.
B EXPERIMENTS ON THE HUMAN3, 6M DATASET
We further conduct experiments on complex human action dataset, Human3.6M (Ionescu al. 2013). The Human3.6M dataset contains 2,220 sequences for training, 300 for validation, and 1,056 Tor testing, involving 17 different scenarios. Compared with KTH, the Human3.66 M.M contains larger diversity and complexity in spatiotemporal modes, thereby more challenging. We follow the protocol from Wang al. 2019b) resize each RGB frame to the resolution of 128 128 x 3 and make the model predict 4 future frames based 4 previous ones. For evaluation metrics, we use the learned perceptual image patch similarity (LPIPS) (Zangg al.) 2018), the frame-wise peak signal-to-noiss models. Besides these advanced frame-wise metrics, we also the Fréchet Video Distance (FVD) (Unterthiner et al. 2018). which is video-wise metric for qualitative human judgment of generated frame sequences. The FVD measure both the temporal coherence of the video content and the quality of each frame.
ratio (PSNR) to evaluate
As shown in Table 5 ModeRNN significantly outperforms the previous state-of-the-arr method MotionRNN [Wu et al. 2021 (PSNR: 24.2 vs. 22.1, FYD: 16.4 vs. 18.3, LPSIS: 0.123 vs. 0.136). Note that approach prediction results are better in terms motion consistency. As for the qualitative results, ModeRNN predicts the sharpest sequence compared with other methods and enriches the details for each part of the body, especially for the arms. These results verify the capability of ModeRNN on dealing with diverse and complex spatiotemporal modes in fully unsupervised way.
also obtain great promotion on the FVD metric, which means the
Table 5: Quantitative results the Human3.66 dataset.
MODEI.
PSNR (t)
FVD (L)
LPSIS (L)
SA-CONVLSTM (LIN ET AL..
2020
21.3
19.2
0.153
PHYDNET (GUEN & THOME
2020)
22.0
18.3
0.145
MOTIONRNN (WO ET AL.
202T) BASED ON MIM
22.1
18.3
0.136
LMC (LEE ET AL. 2021
21.5
18.7
0.151
MODERNN
24.2
16.4
0.123
14

---

Under review as a conference paper at ICLR 2022
t=
8
ModeRNN
PhyDNet
MotionRNN
LMC
SA-ConVLSTM
aa)
t=3
4
5
6
ModeRNN
PhyDNet
MotionRNN
LMC
SA-ConVLSTM
(b)
Figure 8: Examples of predicted future frames the Human3.61 6M dataset.
15

---

Under review as a conference paper at ICLR 2022
C EXPERIMENTS ON THE MIXED MOVING MNIST DATASET
In the prior works, the number of the flying digits of the Moving MNIST dataset (Shi et al. 2015 is fixed, such as the classic methods ConvLSTM, and recent advanced SA-ConvLSTM, Phy DNet. To approximate the multi-mode phenomenon of the real world, for each sequence, we randomly set the number of the flying digits in the range of to 3. This setup is more challenging than previous convention, which requires the model to handle various spatiotemporal dynamics due to different frequencies of occlusions among sequences. We name this new dataset the Mixed Moving MNIST; which contains 20,000 training sequences, 6,000 validation sequences, and 9,000 testing sequences. Each sequence consists of 20 consecutive frames. The first 10 frames for the input, and the next 10 frames are for prediction. All the frames are in the resolution of 64 x 64.
Table 6: Quantitative results on the Mixed Moving MNIST dataset.
MODEL
SSIM (1)
MSE (4)
CONVLSTM (SmIl ET AL.
2015
0.836
78.7
PREDRNN (WAGG ET AL. 2017
0.851
67.3
MIM (WANG AL.
20T9E)
0.851
64.4
RIM (GOYAL ET AL.
2021)
0.874
57.5
SA-CONVLSTM (LIN ET AL.
2020
0.854
70.3
PHYDNET (GUEN & THOME
2020
0.871
60.6
LMC LEE ET AL. 202TT
0.856
72.5
CREVNET YYU BT AL. 2019
0.862
58.9
MODERNN
0.898
44.7
Inputs
True future & Predicitions
5
5
5
5
5
5
5
5
5
5
5
ModeRNN
5
5
5
5
5
5
5
5
ConvLSTM
5
5
5
6
6
6
6
6
6
6
RIM
5
5
5
5
5
5
5
5
5
(a) digit
Inputs
True future Predicitions
5
-5
5
4
9
$
5
5
5
5
1
107
17
ModeRNN
$
1
1
1
1
14
ConvLSTM
7
4
$
5/1
4
5
5
1
w
RIM
(b) digits
Figure 9: Examples of predicted future frames on the Mixed Moving MNIST dataset.
16

---

Under review as a conference paper at ICLR 2022
Slot 1
digit digits
Slot 2
digits
Slot 3
Slot
Cell state of CSMVLSTM
(b) Mode slots of ModeRNN
digit
2 digits
digits
1 digit
2 digits
3 digits
(c) Importance weights of ModRNN
(d) Slot bus of ModeRNN
Figure 10: i-SNE visualization on the Mixed Moving MNIST dataset. (a) Illustration of STMC on existing approach. (b) Different mode slots learn different components of visual dynamics. (c-d) The importance weights and slot bus show discriminative representations, which implies the ability to leam less blurry motions and thus alleviates STMC.
In Table 6 we show the overall quantitative results as well as computational efficiency of the see, ModeRNN achieves
compared models the Mixed Moving MNIST dataset. As state-of-theearr overall performance (SSIM: 0.897, MSE: 44.7) compared with existing approaches, including the state-of-the-ar approaches proposed in recent two years. Furthermore, as shown in Figure 9 ModeRNN is the only method that can capture the exact movement of each digit, while other models predict the blurry results and the digit "5" is even vanished. All in all, ModeRNN effectively overcomes STMC. It achieves the best performance on the synthetic data which is more difficult than the original Moving MNIST dataset due to a larger variety of visual dynamics.
In Figure T((a) we visualize the memory state C. of ConvLSTM using t-SNE (Van der Maaten & Hinton 2008)., and find that they are entangled under different digit modes the Mixed Moving MNIST dataset. It provides evidence that this widely used predictive model cannot learn mode structures effectively. Training the model on dataset with mixed dynamics leads to severe mode collapse in feature learning, resulting in the entanglement of hidden representations.
D ADDITIONAL RESULTS ON THE KTH DATASET
We show examples of predicted future frames on the KTH action dataset in Figure II
A-distance. A-distance [Ben-David et al. 2010) is defined as d = 2(1 - 2e) where e error rate of domain classifier trained to discriminate two visual domains. In Figure 12(a) use the A-distance to quantify the STMC in the real- world KTH action dataset. In this experiment, we divide the KTH dataset into two groups according to the visual similarities of human actions. According to the scale of the actions, we can simply group the existing six categories in the KTH dataset into two typical groups:
17

---

Under review as a conference paper at ICLR 2022
Input frames
Ground truth predicted future frames
9
12
15
18
21
24
27
30
ModeRNN
PredRNN
Conv-TT-LSTM
SVG
Figure 11: Examples of predicted future frames the KTT action dataset.
2.0
A-Dist Mode_1___ 1_D,, Mode_1_GT
2.0
A-Dist Mode. Mode_1_PD, Mode_
1.8
A-Dist MMdd__1_P, Mode_2_PD)
1.8
A-Dist Mode_ PD,, Mode_2_PP)
P.gistanss
A-distance
1.6
1.6
1.4
1.4
1.2
1.2
ConVLSTM
PredANN
ModeRNNN
ConvLSTM
PredRNN
ModeRNN
(a)
(b)
Demonstration of spatiotemporal mode collapse on the KTH dataset using A-distance. (a) A-distance based on the memory states, i.e., B, ModeRNN. (b) A-distance based on the output states, i.e., H, for ModeRNN).
Figure 12:
The first group corresponds to the global movement of the torso, including the categories of running, walking, and jogging.
The second group corresponds to the local movement of hands, including the categories of hand-clapping, hand waving, and boxing.
We here use the memory state C, in ConvLSTM and PredRNN, and the slot bus B, ModeRNN to calculate A-distance. As shown by the blue bars (higher is better), the lower A-distance between the two groups indicates that the learned representations from the two groups are highly entangled. The red bars (lower is better) show the domain distance between features taking as inputs the ground truth frames X; and those taking the predictions &. STMC happens when the A-distance between predictions of different groups (in blue) becomes much smaller than that between predictions and ground truth (in red).
As shown in Figure (3(a) a we visualize the memory state of ConvLSTM using t-SNE (Vn der Maaten & Hinton 2008)_ observed that the learned cell states by ConvLSTM are entangled among different action groups. The t-SNE visualization result matches the PhyDNet visualization on the RoboNet dataset shown Figure a Thus, these results verify that the STMC also exists under the real-world human motion dataset. While in ModeRNN, we further visualize the learned features the slot bus in Figure (3(b) which shows 2 clusters with clear boundaries, corresponding two action groups in the KTH dataset. According these t-SNE results, find that directly training the previous methods on the mixed dynamics will lead to severe STMC in representation learning, shown the entanglement of hidden representations. These entangled
1-SNE.
18

---

Under review as a conference paper at ICLR 2022
KTH Group (Body mation)
Group (Hand motion)
KTH Group (Body motion)
KTH Group 2 (Hand motion)
(a) Cell state of ConvLSTM on KTH
(b) Slot bus of ModeRNN on KTH
Radar Group (Winter)
Radar Group (Winter)
Radar Group
Radar Group (Summer)
(Summer)
(c) Cell state of ConvLSTM on Radar
(d) Slot ModeRNN Radar
Figure 13: (a, c) Illustaaiion of STMC on the existing ConvLSTM model on KTT and radar echo dataset of Guangzhou (GZ). (b, d) The slot bus ModeRNN shows discriminative representations on different groups of video dynamics. The two groups in KTH respectively correspond subtle hand motion (e.g., lavinw hand-clapping, and boxing) and more global body motion (e.g., running, walking, and jogging). The two groups in Radar are divided by different seasons.
representations will make the model provide poor ambiguous prediction. In contrast, ModeRNN can effectively overcome the STMC by learning an accurate decoupling for mixed dynamics.
E QUALITATIVE RESULTS ON THE RADAR ECHO DATASET
We show examples of predicted future frames the radar echo dataset in Figure 44
14
16
10
18
20
ModeRNN
ModeRNN
PredRNN
PredRNN
Figure 14: Examples of predicted future frames the radar echo dataset.
19

---

Under review as a conference paper at ICLR 2022
Considering the climate change among different seasons in Guangzhou, roughly consider the radar echo dataset into two typical meteorology groups:
The first group: It corresponds to the windier part of the year, from March to May, with average wind speeds of more than 7.5 miles per hour. There will be drizzles from time to time in these months. We use the radar maps from 2016/3 to 2016/5 and 2017/3 3 to 2017/4 for training, and use those in 2017/5 for testing.
The second group: It corresponds to the summer in Guangzhou, which experiences heavier cloud cover, with the percentage of time that the sky is overcast or mostly cloudy is around 80%. We use the radar maps from 2016/6 to 2016/8 and 2017/6 2017/7 for training. and use those in 2017/8 8 for testing.
As shown in Figure c3 we also visualize the cell state of ConvLSTM using t-SNEand find that the learned cell state are entangled under different climate groups. It shows that the STMC also exists under the real-world precipitation dataset. In Figure 3 we further visualize the features in the slot bus, which show 2 clusters with clear boundaries, corresponding to two climate groups.
F FURTHER COMPARISON WITH SA-CONVLSTM
To better position our ModeRNN, we provide further comparison with the competitive baseline SA- ConvI.STM (Lin et al. 2020) as follows, which combines the self-attention and ConvLSTM to capture the global context information. In the motivation aspect, SA-ConvLSTTM does not observe the STMC issue in the large-scale complex dataset, which is a key problem that blocks the predictive model capacity. We demonstrate STMC with extensive visualization and further propose the ModeRNN with full insights to tackle this problem. Technically, there two differences between ModeRNN and SA- LSSM:
SA-ConvLSTM uses self-attention only for the representation aggregation, leading to an inherent lack of the ability to decouple the mixed visual dynamics into several modes. On the contrary, ModeRNN separates the learned representations in several subspaces mode slots and adopts the slot bus to connect the decoupled slots along the temporal dimension. This design is hithite-otivated by the observation of STMC. As shown in Figure in the main text, the learned mode slots from ModeRNN are clustered into 4 groups. It proves that ModeRNN can ravel out various spatiotemporal modes and handle the mixed dynamics.
SA-ConyLSTM could not dynamically capture the mixed visual dynamics and adjust to different environments effectively. It only uses the self-attention between recurrent states to capture the global context regardless of various spatiotemporal mode information across different environments. In contrast, ModeRNN develops modular structure, which adaptively produces compositional features via the mode slots and adaptive slot fusion.
Berkeley
Google
Penn
Stanford
Figure 15: I-SNE visualization of the SA-ConvLSTM hidden states on the RoboNet dataset.
20

---

Under review as a conference paper at ICLR 2022
As shown in Figure 15
we further conduct the t-SNE visualization RoboNet, where the memory states of SA-ConvLSTM are entangled and collapse to the ambiguous representation subspaces, leading to the severe STMC. On the other hand, the quantitative results in RoboNet also show that SA-ConvI.STM, does not behave well compared with ModeRNN (SSIM: 0.753 vs. 0.831, MSE; 116.5 vs. 91.9). Thus, SA-ConvI.STY cannot capture mixed visual dynamics to overcome the STMC in larger complicated real-world dataset.
All in all, ModeRNN is different from SA-COnVLSTM in both motivation and technical design, which is compared distinctly from the combination of the multi-head attention and ConvLSTM. The carefully designed mode slots, slot bus and adaptive slot fusion form framework, directly aiming at the STMC, which is the key problem of unsupervised predictive learning. Benefiting from this compact connection between the STMC and model design, ModeRNN achieves the state-of-the-art performance and explainable slot features on extensive datasets with complex spatiotemporal modes.
21


Reviews:
[{"review_id": "c2jrokZm7db", "reviewer": "Reviewer_Wt6k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** \n* The paper presents a novel temporal model to capture spatiotemporal structures in the data and perform better video prediction. The quantitative experiments were performed on three datasets, mixed moving MNIST, KTH action, and Radar Echo datasets.\n\n**Weaknesses:**\n* The cell of the proposed ModeRNN is quite similar to an RNN cell with multi-head attention. One relevant work using a similar idea was not cited. Further, it performs better or on par with the proposed approach in the KTH action dataset.  \n   Lin et al., \"Self-Attention ConvLSTM for Spatiotemporal Prediction,\" AAAI 2020, https://doi.org/10.1609/aaai.v34i07.6819\n\n* In generative models, mode collapse means the generated samples being identical or very similar to each other. It may be caused by the imbalanced distribution of the training set or dependent structures and bias in the data. The motivation of spatiotemporal mode collapse is valid, however, the examples and used datasets are not enough to represent. For instance, in moving MNIST or other datasets do not contain any imbalanced or entangled factor of variation that will cause mode collapse.\n\n* Even though it was argued on the contrary in the related work, the proposed work is highly related to feature disentanglement. PhyDNet (Guen & Thome 2020) is cited but did not included in the moving MNIST or other dataset results.\n\n * How were the previous methods in Table 2 and 4 compared? Did you retrain/implement or they were taken from the reference papers? There are several confusing points: for instance, PredRNN's MSE on Radar dataset was reported 44.2 in the reference paper, but it is84.2 in Table 4. \n\n* What is \"mixed moving MNIST\"? In reference works, mostly moving MNIST (either freshly rendered or the version in Shrivastava et al. 2015) is being used.\n\n* Several typo issues in the entire text (for instance, all titles \"Resuls\"->\"Results\").\n ", "summary_of_the_paper": "This paper defines a phenomenon, spatiotemporal mode collapse in the training of unsupervised predictive models. They propose an RNN-based approach to learning structural hidden representations in temporal data. The proposed idea was experimented with and compared with respect to the convolutional LSTM baseline and several other temporal modeling methods (i.e., RIM or Conv-TT-LSTM).", "summary_of_the_review": "I think the proposed problem of spatiotemporal mode collapse was not covered and described clearly. This makes it difficult to understand where the contribution of the proposed method comes from and what it improves. Spatiotemporal slots idea reminds the attention modules on the temporal data and leads to a weighted fusion of temporal weights. It is very similar to self-attention ConvLSTMs. Mainly the introduced problem is not clear and also approach is limited in novelty. These are the points affected my decision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7obihHBY4hk", "reviewer": "Reviewer_W8mj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n- Overall well motivated novel architecture for addressing the issue of mode collapse in unsupervised predictive training. \n- The proposed decoupling-aggregation framework is demonstrated to learn a highly diverse set of mode-factors which are combined to cover a diverse set of output modes with high accuracy.\n- The analytical experiments show that the performance of ModeRNN consistently (across datasets) benefits from increased diversity of spatio-temporal modes, whereas the performance of other methods often diminishes when trained on more diverse data.\n- The description seems sufficiently detailed for reproducing the experiments.\n\nComments/Questions:\n1. Inconsistencies between text, equations and figure:\n- Eq. 3 is missing the shared FFN for dimensionality reduction\n- Text should explicitly state that the slot bus input $g_t$ is tanh activated unlike the gates. I’d also name it something like slot bus input instead of input modulation “gate”, since the input gate $i_t$ is usually considered to do the modulation of the cell input.\n2. Motivation for implementation details in adaptive slot fusion missing.\n3. For clarity: Is $\\sigma(I_t) \\cdot (W_\\text{fuse}^0 * I_t)$ what you call the residual connection? I’m not exactly sure about the terminology, but I believe residual connections are usually linear.\n4. I think instead of $slot_{t-1}^*$, you could just write $Q_{t}^*$, since these queries do not correspond to the spatio-temporal slots of the previous time-step. Using the same notation is confusing.\n5. Since the figure doesn’t have a descriptive caption, I’d consider placing it on the same page as the description of the architecture.\n6. A question about the ModeRNN (i.e. the stacked ModeCell): Do the higher layers receive only the cell output $\\mathcal{H}_t$ as input or do they also have access to the slot bus $\\mathcal{B}_t$?\n7. Typo: Figure 5: Ture should be True\n8. Did you also calculate the A-distance using the cell outputs instead of the memory states?\n9. The t-SNE visualization of Figures 3 and 5 for KTH and the Radar Echo dataset would be interesting. Are the modes as clearly separated?\n10. There’s a broken figure reference in Appendix B.3\n11. I might have overlooked it, but I didn’t see a comparison of ModeRNNs with different numbers of ModeCells. I can imagine you played a bit with this hyperparameter, I’d be interested to know your observations.\n12. The method for selecting hyperparameters is not described.\n13. How do training times compare to ConvLSTM, RIM, and other architectures?\n14. No discussion of limitations\n15. Do you have initial results on datasets with a larger diversity of spatio-temporal modes (e.g. Human3.6M or KITTI)?\n\nI have updated my score as my concerns are mostly addressed.", "summary_of_the_paper": "The manuscript proposes a novel architecture with a slot-based decoupling-aggregation framework for unsupervised sequence prediction. The model is motivated by preventing spatio-temporal mode collapse, which affects many existing methods. The experiments clearly show that the model addresses this issue and performance comparisons across three commonly used datasets are presented.", "summary_of_the_review": "Overall, interesting work. However, for stronger recommendation of acceptance some of the points above should be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "28OGcVH0r8F", "reviewer": "Reviewer_VMMf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The first property mentioned in Introduction section seems a bit simple. Why one would expect each sequence correspond to a single spatio-temporal mode? While this may be true for very simple scenarios, e.g., moving mnist or KTH, for more realistic scenarios, which may not hold.\n- Authors made a claim multiple times that they have demonstrated the effectiveness of their approach in dataset with highly mixed visual dynamics. However, I disagree that KTH or Moving MNIST represent such scenarios. Moreover, the training set of such dataset is very small given the complexity of the proposed method.\n- The baselines authors compared their approach to don't represent the SOTA or even rather old but strong approaches, such as (Kalchbrenner et al., 2017). Comparing against ConvLSTM or PredRNN is far from enough for ICLR 2022.\n- Regarding the method, I'd see it as a good engineering effort of combining existing methods rather than proposing a novel one, with new insights.\n", "summary_of_the_paper": "This paper proposes a mechanism to reduce spatiotemporal mode collapse in unsupervised predictive learning. To achieve this goal, the proposed method is built upon the idea that different latent modes in the same data domain should share a set of hidden representation subspaces, which can be represented with various compositional structures based on the features in each subspace. Experimental results show improvement over simpler baselines such as PredRNN or ConvLSTM.", "summary_of_the_review": "Although this approach seems to be a good engineering effort to come up with an effective model that works reasonably well on very small-scale datasets with relatively low complexity, I believe the novelty of this method is very limited. Moreover, comparison to SOTA and evaluation on larger, complicated datasets are missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel architecture, ModeRNN, for unsupervised video prediction by learning spatiotemporal attention in the latent subspace (slots).  ModeRNN effectively learns modular features using a set of mode slots and adaptively aggregates
the slot features with learnable importance weights. The paper has promising results on several benchmark video prediction datasets. 

During the post-rebuttal discussion, the reviewer Wt6k and VMMf responded to the authors' rebuttal, but there was no discussion among them. The consensus is that even though the paper is a very strong engineering effort, it was not clear how the proposed architecture addresses the spatiotemporal mode collapse problem.  T-SNE in Fig. 3/10/13 is insufficient to show disentangled feature space. In fact, PhyDnet was designed to disentangle different factors (physical vs unknown), hence not a good baseline.  [Hsieh et al 2018] is a better fit. In addition, synthetic data examples would be helpful to explain the underlying mechanism of the model and provide more insights for the video prediction community.

Based on this reason, I recommend rejecting this paper as it is now and encourage the authors to revise the draft and submit to future venues.

Hsieh, J. T., Liu, B., Huang, D. A., Li, F. F., & Niebles, J. C. (2018, January). Learning to Decompose and Disentangle Representations for Video Prediction. In NeurIPS.
Decision: Reject