# Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations
## 4 Experiments
### Moving MNIST  
The MMNIST dataset consists of two randomly moving hand-written digits in a gray-scale frame. The MMNIST contains 10,000 sequences of 20 frames. Each sequence shows a pair of digits bouncing around in a \(64\times 64\) frame. The digits can bounce off from the boundary, and may overlap with each other. We randomly split the dataset into a fixed 9000 training and validation set, and 1000 testing set. For all methods, we use the unmodified/original version of MMNIST dataset from [4], publicly available at [55] (meanwhile, in DRnet [15] and PredRNN++ [41], the authors synthesized their own Moving MNIST dataset for training and testing). Figure 3: Experimental results on Bouncing Balls. This figure shows the ball center differences, velocity differences and cosine similarity. Figure 2: Qualitative results with Bouncing Balls dataset. **Implementation. ** The MMNIST image pixel values are re-scaled to the range \([0,1]\). All compared models are trained to make 10 future predictions given the first 10 input frames. Our model is trained under similar training settings as the PredRNN++. We use the Adam optimizer [56] with a constant learning rate of 0.001 with the batch size 8. Our model is trained for 80,000 iterations (about 72 training epochs). During the first 50,000 iterations, the probability of Scheduled Sampling is gradually increased to 1. **Ablation Study. ** HRPAE consists of multiple levels of CNN blocks and convLSTMs. Our experiment contains the ablation study by varying the number of levels, and compare the following three variants. * The default 4-level HRPAE as shown in Figure 1a with the output channel sizes given by 32, 64, 128 and 256. * The 3-level HRPAE which contains the first 3 levels of CNN blocks and convLSTMs from the network in Figure 1a (e.g., uses EncCNN 1-3, LSTM 1-3, and DecCNN 1-3) with the output channel sizes given by 32, 64 and 128. * The 2-level HRPAE which consists of the first 2 layers of CNN blocks and convLSTMs with output channel sizes given by 32 and 64. **Results. ** The experimental results are summarized in Table 2 which shows the comparison with the state-of-the-art models in terms of the MSE, PSNR and SSIM metrics. We observe that 4-level HRPAE outperforms DRNet, MCnet, PredRNN++ and gives the best performance. All three variants of our model show reasonable results on the testing MMNIST dataset. We find that variants with a higher number of hierarchical levels exhibit better performances. For instance, the averaged MSE loss decreases from 66 to 49 by increasing the number of levels from 2 to 4. The results demonstrate that it is important to exploit sequential structures of features at various levels of hierarchy. The frame-wise average losses over 10 prediction steps are shown in Figure 4. As shown in the figure in which we excluded the results for DRNet and MCnet results, 4-level HRPAE achieves the best results. Also 3-level HRPAE performs comparable to PredNet++. The boost in the performance is greater when the number of levels of HRPAE is increased from 2 to 3 than from 3 to 4. Thus, 3-level HRPAE achieves a good trade-off between the model size and performances; this aspect will be discussed in follow-up research. In Figure 5, we present examples of the predicted testing frames for the discussion of qualitative results. As shown in the figure, even though the digits get entangled with each other, our model is able to make accurate predictions. For example, in the left-side example of Figure 5, the two digits of \"2\" are overlapped with each other in the beginning of the target future sequence. Our model is able to maintain the shape of digits and make accurate predictions of the locations. This proves our models because even a small one, e.g., 2-level HRPAE, is able to preserve the distance information between digits. Similar observations can be made for the right-side example.