Question: Why one would expect each sequence correspond to a single spatiotemporal mode?
Evidence: Predictive learning is an unsupervised learning paradigm that has shown the ability to discover the spatiotemporal modes of visual dynamics. However, for largescale and real-world datasets (see Figure), the modes in visual dynamics can be highly entangled and difficult to learn due to the richness of data environments, the diversity of object interactions, and the complexity of motion patterns. For clarity, in the following discussion, spatiotemporal modes are considered to have the following properties:
1. A spatiotemporal mode refers to a representation subspace that corresponds to a family of similar, but not predefined, visual dynamics.
2. Multiple spatiotemporal modes naturally exist in real-world data, even in a single frame.
3. We assume the i.i.d. setup to allow all videos to share the same set of spatiotemporal modes in a dataset. Different data may have different compositional structures over the modes.
Under these assumptions, video prediction models are required to (i) decouple the potentially mixed spatiotemporal modes from raw video frames, (ii) understand the compositional structures on top of the learned modes, and (iii) learn the state transitions based on the compositional structures.
Otherwise, since the learned dynamics with respect to different modes may interfere and compete during training, it remains challenging for the prior art in video prediction to generate less blurry future frames based on an ambiguous understanding of mixed physical processes. We refer to this empirical phenomenon as spatiotemporal mode collapse (STMC), which is mainly caused by the collapse of learned representations into invalid subspaces when compromising to multiple spatiotemporal modes in the training set. Unlike the widely concerned mode collapse problem in generative adversarial networks, STMC has not drawn much attention because predictive learning is supposed to be well constrained by the image reconstruction loss. However, due to the limitation of model size, STMC occurs when the model cannot effectively decouple mixed spatiotemporal modes and infer their
Predictive models collapse to blurry motion in the presence of complex dynamics modes underlying structures. As a result, its responses to different modes tend to lose diversity and may collapse to a meaningless average of multiple representation subspaces of valid modes.
In Figure (left), we can observe the existence of STMC on a large-scale video dataset named RoboNet, in which potential spatiotemporal modes may come from seven different robot platforms (e.g., Baxter and WidowX), four data collection environments (e.g., Berkeley and Stanford), and a variety of unlabeled robot control tasks (e.g., pushing and grasping). An additional outcome of STMC is that we can achieve a performance gain when training individual models in separate subsets with remarkably different visual dynamics, as shown in Figure (right). However, such a dilemma prevents the model from growing into big ones that allow scalable training on large-scale, natively multimodal spatiotemporal sequences.
We explore STMC for the first time in unsupervised predictive learning. The core idea is to provide a strong inductive bias for the predictive model to discover the compositional structures of latent modes. To this end, we propose ModeRNN, a new modular recurrent architecture that learns structured hidden representations through a set of mode slots 1 , where each of them responds to the representation subspace of a single spatiotemporal mode. ModeRNN also introduces a decoupling-aggregation framework to process the slot features in three stages, which is completely different from existing predictive models with modular architectures.
The first stage is recurrent state interaction and slot binding, in which we use the multi-head attention mechanism to enable the memory state to interact with the input state and previous hidden state of RNNs. We name the memory state "slot bus", because for each sequence, it is initialized from a multi-variate Gaussian distribution with learnable parameters, and thereafter refined using the slot features at each time step. By using the slot bus as the queries, multi-head attention can naturally decouple modular components from hidden representations and bind them to particular mode slots. Features in each slot are then independently modeled using per-slot convolutional parameters. The second stage in each ModeRNN unit is slot fusion, motivated by the assumption that, there can be multiple spatiotemporal modes in a single video and similar videos can be represented by similar compositional structures over the mode slots. Therefore, we assign slot features with learnable importance weights and aggregate them into a unified hidden representation, which is then used in the third stage to update the slot bus and generate the output state of the ModeRNN unit.
We empirically show the existence of STMC on five datasets, and include the results on three realworld datasets in the manuscript, including the large-scale RoboNet dataset that has various data collection environments and multiple robot control tasks, the KTH dataset with six types of human actions that has been widely used by previous literature, and the radar echo dataset for precipitation forecasting that contains time-varying modes of seasonal climates. In addition, we include results on a Mixed Moving MNIST dataset and the Human3.6M dataset in the appendix. In a series of quantitative and visualization results, we demonstrate the effectiveness of ModeRNN in mitigating STMC and learning from highly entangled visual dynamics.
For clarity, in the following discussion, spatiotemporal modes are considered to have the following properties: 1. A spatiotemporal mode refers to a representation subspace that corresponds to a family of similar, but not predefined, visual dynamics. 2. Multiple spatiotemporal modes naturally exist in real-world data, even in a single frame. 3. We assume the i.i.d. setup to allow all videos to share the same set of spatiotemporal modes in a dataset. Different data may have different compositional structures over the modes.
Answer: We have carefully rethought the notion of *spatiotemporal mode* and agreed with the reviewer that this statement is not rigorous enough. In the revised draft, we have corrected all relevant statements. In particular with the Introduction section, we rewrite the first paragraph as follows: The spatiotemporal modes in visual dynamics can be highly entangled and difficult to learn due to the richness of data environments, the diversity of object interactions, and the complexity of motion patterns. Spatiotemporal modes are considered to have the following properties:
- A spatiotemporal mode refers to a representation subspace that corresponds to a family of similar, but not predefined, visual dynamics.- Multiple spatiotemporal modes naturally exist in real-world data, even in a single frame.- We assume the i.i.d. setup to allow all videos to share the same set of spatiotemporal modes in a dataset. Different data may have different compositional structures over the modes.Through the new statements, one would **NOT** expect each sequence to correspond to a single spatiotemporal mode. More precisely, each sequence may be mixed up by a sequence-independent weighted combination from a set of spatiotemporal modes, and each mode may correspond to a compact dynamical component (modeled by the *mode slot*) underlying many sequences. Reasonably, the **decoupling-aggregation** framework proposed with this paper is well fit into the modeling of mixed spatiotemporal dynamics.