# Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations
## 3 Methods
### Model  
Overview. The proposed architecture makes use of the hierarchy of representations obtained from cascaded/multi-layer CNNs. Features output from different layers of cascaded CNNs capture visual information at different levels: say in object detection, high level features may represent object identities like cars, whereas mid level (resp. low level) features may represent shapes like circles (resp. edges). We posit that features at different levels tend to exhibit different temporal dynamics. For example, features representing a walking person as a whole will change slowly over time as compared to those of the motions of the person's limbs. Separating and capturing features at multiple levels of hierarchy is useful, e.g., capturing high-level features makes the prediction task simpler, whereas capturing low-level features helps generating realistic frames. We thus propose to use multiple ConvLSTMs, each of which is dedicated to model the temporal dynamics of each of multiple levels of CNN features. Our goal is to automatically capture sequential structures of low-, mid-, and high-level features with ConvLSTMs, and is to generate future frames by combining hierarchical features estimated by the ConvLSTMs. HRPAE consists of three modules: an Encoder CNN, ConvLSTMs and a Decoder CNN. These modules are used recurrently over the time horizon. An overview of 4-level HRPAE is shown in Figure 1.  
**Hierarchical Recurrent Predictive Auto-Encoder. ** The HRPAE architecture consists of three parts: a multi-layer Encoder CNN (EncCNN) with pooling layers, four ConvLSTMs and a multi-layer Decoder CNN (DecCNN) with up-sampling convolutional layers as shown in Figure 1. The EncCNN is used to extract hierarchical CNN features of different levels from a frame. Four parallel ConvLSTMs are used to model the dynamics of hierarchical (e.g., low-, mid-low-, mid-high- and high-level) features of the input sequence and DecCNN is used to generate predictions from the estimated hierarchical features. During training, our model computes context features by first obtaining different levels of features from EncCNN. Next, each of the ConvLSTMs receives the corresponding level of features to compute the hidden states given the previous hidden states. DecCNN combines the outputs from ConvLSTMs of corresponding level and the previous layer, and sequentially propagates the context information. The last layer of decoder network is CNN with \(1\times 1\) convolutional layer to produce output consistent with the frame size. The output from the last layer is fed back into the EncCNN to generate the next frame. The equations representing the model in Figure 1 are given as follows:  
\[(F_{t}^{1},F_{t}^{2},F_{t}^{3},F_{t}^{4})=\text{EncCNN}(\mathbf{x }_{t}),\] \[(H_{t}^{m},C_{t}^{m})=\text{ConvLSTM}_{m}(F_{t}^{m},C_{t-1}^{m}), \ m=1,2,3,4,\] \[\mathbf{\hat{x}}_{t+1}=\text{DecCNN}(H_{t}^{1},H_{t}^{2},H_{t}^{ 3},H_{t}^{4}),\]  
where \(F_{t}^{1}\) to \(F_{t}^{4}\) represent from the low- to high-level CNN features of the \(t\)-th frame \(\mathbf{x}_{t}\) extracted by EncCNN. \(H_{t}^{m}\) and \(C_{t}^{m}\) represent the hidden and the cell states of level-\(m\) ConvLSTMs at \(t\)-th frame. Our model can be expanded in terms of the number of levels, by adding the corresponding CNN layers and ConvLSTMs to the model. Following the VGG structure [9], each block of the EncCNN consists of two successive \(3\times 3\) convolutional layers with Batch Normalization [50] and leaky ReLU activation function. The max pooling layer is placed at the end of each block. The DecCNN can be viewed as mirrored version of EncCNN; the difference is that we use bilinear interpolation to upsample the CNN feature maps by a factor of 2. Finally, a \(1\times 1\) convolutional layer is used as the last block of the DecCNN. The ConvLSTMs are equivalent to standard convLSTM except that we add Layer Normalization [51] after each convolutional layer.