# Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations
## 1 Introduction  
The goal of video prediction is to predict future frames given past observations. Being able to model and predict the future is essential to many applications of machine learning and computer vision, such as human pose estimation and recognition [1], pedestrian detection and tracking [2], weather forecasting [3]. In recent years, a number of video prediction methods have been proposed [4; 5; 6; 7], driven by the success of convolutional neural networks (CNN) [8; 9; 10; 11; 12], recurrent neural networks (RNN) [13], and generative adversarial networks (GAN) [14]. However, video prediction remains a challenging task in computer vision, because of not only the uncertain nature of future events, but also the unpredictability of spatio-temporal dynamics as well as the high dimensionality. Recent approaches for video prediction focus on separating representations that exhibit different temporal dynamics. For example, DRNet [15] and MCnet [16] propose to decompose video frames into stationary and time-varying parts, and use RNNs to model the dynamics of time-varying components. Other recent works [17; 18; 19] consider extracting high-level features from input frames, then predict the temporal evolution of such high-level features, and use generative models to reconstruct future frames. However, such latent representations to be decomposed are often chosen manually. The aforementioned methods require the estimation of high-level structures which needs domain-specific knowledge or relies on heuristics, e.g., joint locations in human poses [17] or temporal stationarity of frames [15; 16]. Thus, it would be desirable if (i) hierarchical features can be _automaticallylearned from the input video; (ii) we can learn multiple, possibly more than two, levels of hierarchical features in a systematic way so as to fully exploit information in the video. In this paper, we propose an end-to-end trainable neural network for video prediction called Hierarchical Recurrent Predictive Auto-Encoder (HRPAE). We consider hierarchical cascaded/multi-layer CNNs encoder-decoder incepted by RNN network architecture. Features output from different CNN layers will exhibit different temporal dynamics. Thus, we propose to use multiple Convolutional LSTMs (ConvLSTMs) [3] to separately model sequential structures from multiple levels of hierarchy of features. The key idea is to automatically and separately capture temporal dynamics of hierarchical features without any prior information on the input video. To facilitate the generation of realistic frames, we propose a recurrent variation of the skip-connection architectures inspired by U-Net [20] or Hourglass networks [21]. Specifically, hierarchical features output at each encoder layer skip the \"bottleneck\" formed at encoder-decoder network, and are fed to ConvLSTMs which combine hidden variables from previous frames, and propagate the updated/predicted feature maps to the decoder layer, which we call skip-and-update. The proposed architecture enables the encoder, decoder CNNs and ConvLSTMs to share useful spatio-temporal information at multiple levels of hierarchy, which helps producing sharp output frames. Through experiments we show that HRPAE is able to obtain high quality results on the Bouncing Balls [22; 23; 24], the Moving MNIST (MMNIST) [4] and the KTH human action dataset [25]. With the Bouncing Balls dataset, experiments show that HRPAE is capable of modeling complex dynamics associated with the physical states of balls with a high accuracy. The normalized error in the ball positions are shown to be within 2% of the frame size with cosine similarity in the velocity of balls exceeding 0.97 over 10 frames of predictions. Experimental results on the MMNIST show that HRPAE is able to generate sharp predictions, and outperforms existing schemes in Mean Square Error (MSE), and achieves high Peak Signal-to-Noise Ratio (PSNR) [5] and Structural Similarity Index (SSIM) [26].