Title: GENERALIZABLE FEATURES FROM UNSUPERVISED LEARNING

Abstract: Humans learn a predictive model of the world and use this model to reason about future events and the consequences of actions. In contrast to most machine predictors, we exhibit an impressive ability to generalize to unseen scenarios and reason intelligently in these settings. One important aspect of this ability is physical intuition (Lake et al., 2016) . In this work, we explore the potential of unsupervised learning to find features that promote better generalization to settings outside the supervised training distribution. Our task is predicting the stability of towers of square blocks. We demonstrate that an unsupervised model, trained to predict future frames of a video sequence of stable and unstable block configurations, can yield features that support extrapolating stability prediction to blocks configurations outside the training set distribution.

1. INTRODUCTION:

Humans learn a tremendous amount of knowledge about the world with almost no supervision and can construct a predictive model of the world. We use this model of the world to interact with our environment. As also argued by Lake et al. (2016) one of the core ingredients of human intelligence is intuitive physics. Children can learn and predict some of the common physical behaviors of our world just by observing and interacting without any direct supervision. And they form a sophisticated predictive model of the physical environment and expect the world to behave based on their mental model and have a reasonable expectation about unseen situations Téglás et al. (2011) .

Despite impressive progress in the last few years in the training of the supervised models, we have not yet quite been able to achieve similar results in unsupervised learning, and it remains one of the challenging research areas in the field. The full potential of the application of unsupervised learning is yet to be realized.

In this work, we leverage unsupervised learning to train a predictive model over sequences. We use the imagined and predicted future sequence data to help a physical environment prediction model generalize better to unseen settings.

More specifically we focus on the task of predicting if a tower of square bricks will fall or not, as introduced by Lerer et al. (2016) . They showed that a deep convolution neural network could predict the fall of the towers with super-human accuracy. But despite the strengths of convolution neural networks, Zhang et al. (2016) shows how deep neural networks have a hard time generalizing to novel situations in the same way as humans or simulation-based models can do. In this work, we show that deep neural networks are capable of generalizing to novel situations through a form of unsupervised learning. The core idea is to observe the world without any supervision and build a future predictive model of it, and in a later stage leverage and utilize the imagined future to train a better fall prediction model.

2. RELATED WORK:

In the beginning, unsupervised learning and generative models emerged as pre-training method Hinton & Salakhutdinov (2006) ; Hinton et al. (2006) ; Bengio et al. (2007) to help other tasks such as supervised learning. But since Krizhevsky et al. (2012) many other regularization Srivastava et al. (2014) , weight initialization Glorot & Bengio (2010) and normalization Ioffe & Szegedy (2015) techniques and architecture designs He et al. (2015) has been introduced that diminish the effect of pre-training. Although pre-training still could be useful in data scarce domains they are many other ways and applications that unsupervised learning are still very interesting models and it is a very active area of research. Just to name a few applications are semi-supervised learning Kingma et al. (2014) ; Salimans et al. (2016); Dumoulin et al. (2016) super resolution Sønderby et al. (2016) .

Video generation is one active area of research with many applications, and many of the recent works have been using some of the states of the art neural networks for video generation. Srivastava et al. (2015) uses LSTM recurrent neural networks to train an unsupervised future predictive model for video generation. And here we use a very similar architecture as described in Section 4.1. Mathieu et al. (2015) combines the common mean-squared-error objective function with an adversarial training cost in order to generate sharper samples. Lotter et al. (2016) introduce another form of unsupervised video prediction training scheme that manages to predict future events such as the direction of the turn of a car which could have potential use in training of the self-driving cars.

Model-based reinforcement learning (RL) is an active research area that holds the promise of making the RL agents less data hungry. Learning agents could explore, learn in an unsupervised way about their world, and learn even more by dreaming about future states. We believe that action-condition video prediction models are an important ingredient for this task. Fragkiadaki et al. (2015) learn the dynamics of billiards balls by supervised training of a neural net. Action-conditioned video prediction models have been applied to Atari playing agent Oh et al. (2015) as well as robotics (Finn et al., 2016; Finn & Levine, 2016) .

3. DATASET:

Recent datasets for predicting the stability of block configurations (Lerer et al., 2016; Zhang et al., 2016) only provide binary labels of stability, and exclude the video simulation of the block configuration. We, therefore, construct a new dataset, with a similar setup as Lerer et al. (2016) ; Zhang et al. (2016) , that includes this video sequence. We use a Javascript based physics engine1 to generate the data.

We construct towers made of 3 -5 square blocks. To sample a random tower configuration, we uniformly shift each block in its x and y position such that it touches the block below. Because taller towers are more unstable, this shift is smaller when we add more blocks. To simplify our learning setting, we balance the number of stable and unstable block configurations. For each tower height, we create 8000, 1000 and 3000 video clips for the training, validation, and test set, respectively. The video clips are sub-sampled in time to include more noticeable changes in the blocks configurations. We decided to keep 39 number of frames which with our sub-sampling rate was enough time for unstable towers to collapse. Each video frame is an RGB image of size 64x64. In addition to binary stability label, we include the number of blocks that fell down.

4. ARCHITECTURE:

The core idea of this paper is to use future state predictions of a generative video model to enhance the performance of a supervised prediction model. Our architecture consists of two separate modules:

Frame predictor A generative model to predict future frames of a video sequence. This model is trained to either generate the last frame or the complete sequence of frames.

Stability predictor In the original task, stability is predicted from a static image of a block configuration. We explore whether, in addition to initial configuration, the last frame prediction of our unsupervised model improves the performance of the stability prediction.

In the following sections, we explore several different architectures for both modules.

4.1. FUTURE FRAME PREDICTION:

We consider two different model architectures for this task. The first one, named ConvDeconv, only takes the first frame as input and predicts the last frame of the video sequence. The architecture consist of a block of convolution and max-pooling layers. To compensate for the dimensionality reduction of the max-pooling layers, we have a fully-connected layer following the last max-pooling layer. And finally a subsequent block of deconvolution layers with the output size same as the model input size. All activation functions are ReLU (Nair & Hinton, 2010) . See Table 1 for more details of the architecture. The objective function is the mean squared error between the generated last frame and the ground-truth frame; as a result, this training will not require any labels. We also experimented with an additional adversarial cost as in Mathieu et al. (2015) but did not observe any improvement for the stability prediction task. We hypothesize that although the adversarial objective function helps to have sharper images, such improved sample quality does not transfer to better stability prediction. Figure 1 shows a few examples of the generated data on the test set. Mean squared error is minimized using the AdaM Optimizer (Kingma & Ba, 2014) and we use earlystopping when the validation loss does not improve for 100 epochs.

We extend this ConvDeconv model in a second architecture, named ConvLSTMDeconv, to predict the next frame at each timestep. This model is composed of an LSTM architecture. The same convolutional and deconvolutional blocks as ConvDeconv is utilized to respectively input the current frame to the LSTM transition and output the next frame from the current LSTM state. The details of the ConvLSTMDeconv model architecture are shown in Table 2 and Figure 3 shows the diagram of the both architectures. During the training at each time step the ground-truth data feeds in to the model, but during the test time only the initial time step gets the first frame from the data and for subsequent time steps the generated frames from the previous time steps feed in to the model. The is similar setup to recurrent neural network language models Mikolov (2012) , and this is necessary as during the test time we only have access to the first frame. As before, the model is trained to predict the next frame at each time step by minimizing the predictive mean-squared-error using AdaM optimizer and early-stopping. For training, we further subsample in time dimension and reduce the sequence length to 5-time steps. Figure 2 shows some sample generated sequences from the test set. 

5. RESULTS:

Figure 4 shows the classification results for each of the 9 models described in Section 4.2 tested on 3, 4 and 5 blocks. Each test case is shown with a different color. And Table 3 shows all the 27 test case results' numerical values. In almost all cases the generated data improves the generalization performance to test cases with a different number of blocks than it was trained on. For comparison we have included results from Zhang et al. (2016) in Table 4 . Since Zhang et al. (2016) only reports the results when the models are trained on tower of 4 blocks, the corresponding results would be the second block row in Table 3 , models 4S, 4CD and 4CLD. Even though the datasets are not the same, but it can be observed that the range of performance of the baseline 4S model is consistent with the range of performance of AlexNet model on Table 4 . It can be seen that how the results of the 4CD model are significantly better than both IPE and human performance reported in Zhang et al. (2016) , while the baselines have similar performances.

One observation is the fact that the improvements are more significant when it's been tested on scenarios with more bricks than during training. It also improves the reverse case, i.e. fewer bricks than during training, but the improvement is not as significant. It is worth mentioning that testing on a lower number of bricks is a much harder problem as pointed out in Zhang et al. (2016) too. In their case, the prediction performance was almost random when going from 4 blocks to 3 blocks, which is not the case in our experiments2 . One possible explanation for performance loss is that a balanced tower with fewer blocks corresponds to an unstable configuration for a tower with more blocks e.g. a tower with 3 blocks is classified as unstable for a prediction model trained on towers of 5 blocks. One solution could be to train these models to predict how many blocks have fallen instead of a binary stability label. Because we have access to this data in our dataset, we explored the same experiments using these labels. Unfortunately, we did not observe any significant improvement. The main reason could be that the distribution of the number of fallen blocks is extremely unbalanced. It is hard to collect data with a balanced number of fallen blocks because some configurations are thus very unlikely e.g. a tower of 5 blocks with only two blocks falls (the majority of the time the whole tower collapses).

The another observation is the fact that models that use ConvDeconv generated data performed slightly better than those that use ConvLSTMDeconv. As seen in Figure 2 the samples in the Con-vLSTMDeconv case are more noisy and less sharper than those in Figure 1 . This could be caused since after the first time step the model outputs from the last time step is used as input for the next time step, the samples degenerates the longer the sequence is.

Data augmentation was crucial to increase the generalization performance of the stability prediction e.g. 5CD model tested on 4 bricks achieved only 50% without data augmentation while reaching 74.5% accuracy with data augmentation. This significant improvement from data augmentation could be partly because our dataset was relatively small. (2016) . We emphasize that these results are on a different dataset.

6. CONCLUSION:

In this paper, we showed that data generated from an unsupervised model could help a supervised learner to generalize to unseen scenarios. We argue that this ability of transfer learning and generalization by observing the world could be one of the ingredients to construct a model of the world that could have applications in many tasks, such as model-based RL. We aim to extend this work in future by looking at the videos of robots manipulating objects and being able to predict their failure beforehand, which could help an RL agent to explore more intelligently.

:

https://chandlerprall.github.io/Physijs/

We are not using the same dataset asZhang et al. (2016) and hence direct comparison is not possible.

Figures and Tables:

Figure 2: Samples from the ConvLSTMDeconv model. Each row is a different sample. The left sequence is the data and the right sequence is the generated data. Note that during generation model only see the first frame and for next time steps uses its own output from the last timestep.

Figure 4: Accuracy in percentage for each of the 9 models tested on test sets with a different number of blocks. Each color represents the number of blocks that the model was tested on. 50% is chance.

The results from our experiments

Table content: <table><tr><td>Model</td><td colspan="3">Train set Test set Accuracy</td></tr><tr><td>AlexNet</td><td>4</td><td>3</td><td>51 %</td></tr><tr><td>AlexNet</td><td>4</td><td>4</td><td>95 %</td></tr><tr><td>AlexNet</td><td>4</td><td>5</td><td>78.5 %</td></tr><tr><td>IPE</td><td>N/A</td><td>3</td><td>72 %</td></tr><tr><td>IPE</td><td>N/A</td><td>4</td><td>64 %</td></tr><tr><td>IPE</td><td>N/A</td><td>5</td><td>56 %</td></tr><tr><td>Human</td><td>N/A</td><td>3</td><td>76.5 %</td></tr><tr><td>Human</td><td>N/A</td><td>4</td><td>68.5 %</td></tr><tr><td>Human</td><td>N/A</td><td>5</td><td>59 %</td></tr></table>

The results reported on Zhang et al.

Table content: <table/>

Peer review (iclr2017):
Summary === This paper trains models to predict whether block towers will fall down or not. It shows that an additional model of how blocks fall down (predicting a sequence of frames via unsupervised learning) helps the original supervised task to generalize better. This work constructs a synthetic dataset of block towers containing 3 to 5 blocks places in more or less precarious positions. It includes both labels (the tower falls or not) and video frame sequences of the tower*s evolution according to a physics engine. Three kinds of models are trained. The first (S) simply takes an image of a tower*s starting state and predicts whether it will fall or not. The other two types (CD and CLD) take both the start state and the final state of the tower (after it has or has not fallen) and predict whether it has fallen or not, they only differ in how the final state is provided. One model (ConvDeconv, CD) predicts the final frame using only the start frame and the other (ConvLSTMDeconv) predicts a series of intermediate frames before coming to the final frame. Both CD and CLD are unsupervised. Each model is trained on towers of a particular heigh and tested on towers with an unseen height. When the height of the train towers is the same as the test tower height, all models perform roughly the same (with in a few percentage points). However, when the test height is greater than the train height it is extremely helpful to explicitly model the final state of the block tower before deciding whether it has fallen or not (via CD and CLD models). Pros === * There are very clear (large) gains in accuracy from adding an unsupervised final frame predictor. Because the generalization problem is also particularly clear (train and test with different numbers of blocks), this makes for a very nice toy example where unsupervised learning provides a clear benefit. * The writing is clear. Cons === My one major concern is a lack of more detailed analysis. The paper establishes a base result, but does not explore the idea to the extent to which I think an ICLR paper should. Two general directions for potential analysis follow: * Is this a limitation of the particular way the block towers are rendered? The LSTM model could be limited by the sub-sampling strategy. It looks like the sampling may be too coarse from the provided examples. For the two towers in figure 2 that fall, they have fallen after only 1 or 2 time steps. How quickly do most towers fall? What happens if the LSTM is trained at a higher frame rate? What is the frame-by-frame video prediction accuracy of the LSTM? (Is that quantity meaningful?) How much does performance improve if the LSTM is provided ground truth for only the first k frames? * Why is generalization to different block heights limited? Is it limited by model capacity or architecture design? What would happen if the S-type models were made wider/deeper with the CD/CLD fall predictor capacity fixed? Is it limited by the precise task specification? What would happen if networks were trained with towers of multiple heights (apparently this experiment is in the works)? I appreciate that one experiment in this direction was provided. Is it limited by training procedure? What if the CD/CLD models were trained in an end-to-end manner? What if the double frame fall predictor were trained with ground truth final frames instead of generated final frames? Minor concerns: * It may be asking too much to re-implement Zhang et. al. 2016 and PhysNet for the newly proposed dataset, but it would help the paper to have baselines which are directly comparable to the proposed results. I do not think this is a major concern because the point of the paper is about the role of unsupervised learning rather than creating the best fall prediction network. * The auxiliary experiment provided is motivated as follows: *One solution could be to train these models to predict how many blocks have fallen instead of a binary stability label.* Is there a clear intuition for why this might make the task easier? * Will the dataset, or code to generate it, be released? Overall Evaluation === The writing, presentation, and experiments are clear and of high enough quality for ICLR. However the experiments provide limited analysis past the main result (see comments above). The idea is a clear extension of ideas behind unsupervised learning (video prediction) and recent results in intuitive physics from Lerer et. al. 2016 and Zhang et. al. 2016, so there is only moderate novelty. However, these results would provide a valuable addition to the literation, especially if more analysis was provided.