# Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations
## 4 Experiments
### Bouncing Balls  
The Bouncing Balls dataset simulates 4 balls bouncing within the frame. It is challenging to predict future frames of bouncing balls because of the underlying physical dynamics and interactions, e.g., the balls can bounce off walls, and may collide with each other and change directions. The purpose of this experiment is to evaluate the capacity of HRPAE of predicting object trajectories in an interactive environment.  
**Implementation.** Following the settings in [23; 40], we generated 10,000 training and 1000 testing sequences with \(128\times 128\) frame size where each sequence has 30 frames. The batch size is set to 8, and the default HRPAE (as shown in Figure 1) is trained for \(10^{5}\) iterations (about 80 epochs) by randomly sampling 20 frames. The rate of Scheduled Sampling is gradually increased to 1 during the first 50,000 iterations. We trained PredRNN++ under the same experimental settings, whereas the first layer of PredRNN++ is expanded to 64 channels; as compared to MMNIST, we doubled the patch size in order to maintain the width and height of input frames, because the frame height and width are 128 pixels.  
**Results.** We present qualitative results with the Bouncing Ball dataset in Figure 2. We observe that HRPAE can successfully predict how balls interact with their environment, e.g., colliding with other balls and bouncing off walls. As shown in Figure 2, HRPAE correctly predicts that the balls will collide, but will not be merged/entangled, unlike the digits in MMNIST dataset. Furthermore, HRPAE is able to predict complex interactions such as the balls bouncing off the wall and then colliding with another ball as shown in the 6 th row of Figure 2.  
Next we present quantitative results with the Bouncing Ball dataset. As shown in Table 1, HRPAE shows the superior performance in the overall MSE and PSNR per frame; this is achieved with the model size less than one-third of that of PredRNN++.  
Next, we evaluate the prediction accuracy, measured in terms of ball center positions. HRPAE only outputs predicted frames, but does not explicitly compute the ball positions. Thus, we perform image processing such as binary erosion [54] to extract ball center positions, denoted by \(c_{t}\) at time step \(t\), from the predicted frames. In Figure 3a we show the average error in the ball center positions in the \(L_{2}\) distance. The results show HRPAE is able to predict the ball positions with high accuracy. The positional error is observed to be the within 2% of the frame size on average across 10 frames (less than 2 pixels).  
Following the experimental settings in [23; 40], the normalized velocity, defined as \(v_{t}=c_{t+1}-c_{t-1}\), of the balls, and the differences in \(L_{2}\) norm between the ground-truth and the extracted ball velocities are shown in Figure 3b. The cosine similarity between the ground truth and predicted velocities is presented in Figure 3c. We observe that HRPAE outperforms PredRNN++ method with the smaller model size, e.g., the average cosine similarity of our method is 0.973, while that of PredRNN++ is 0.962. The results indicate that HRPAE excels at capturing high-level dynamics of physical states associated with the motion of the balls in a complex interactive environment.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Methods** & **MSE** & **PSNR** & **Parameters** \\ \hline PredRNN++ & 27.22 & 27.87 & 31,565 K \\ HRPAE & 20.25 & 29.36 & 9391 K \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results with the Bouncing Balls dataset.