Title: OBJECT-ORIENTED MODEL LEARNING THROUGH MULTI-LEVEL ABSTRACTION

Abstract: Object-based approaches for learning action-conditioned dynamics has demonstrated promise for generalization and interpretability. However, existing approaches suffer from structural limitations and optimization difficulties for common environments with multiple dynamic objects. In this paper, we present a novel self-supervised learning framework, called Multi-level Abstraction Objectoriented Predictor (MAOP), for learning object-based dynamics models from raw visual observations. MAOP employs a three-level learning architecture that enables efficient dynamics learning for complex environments with a dynamic background. We also design a spatial-temporal relational reasoning mechanism to support instance-level dynamics learning and handle partial observability. Empirical results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different static object layouts. In addition, MAOP learns semantically and visually interpretable disentangled representations.

1. INTRODUCTION:

Model-based deep reinforcement learning (DRL) has recently attracted much attention for improving sample efficiency of DRL, such as (Heess et al., 2015; Schmidhuber, 2015; Gu et al., 2016; Racanière et al., 2017; Finn & Levine, 2017) . One of the core problems for model-based reinforcement learning is to learn action-conditioned dynamics models through interacting with environments. Pixel-based approaches have been proposed for such dynamics learning from raw visual perception, achieving remarkable performance in training environments (Oh et al., 2015; Watter et al., 2015; Chiappa et al., 2017) .

To unlock sample efficiency of model-based DRL, learning action-conditioned dynamics models that generalize over unseen environments is critical yet challenging. Finn et al. (2016) proposed an action-conditioned video prediction method that explicitly models pixel motion and thus is partially invariant to object appearances. Zhu et al. (2018) developed an object-oriented dynamics predictor, taking a further step towards generalization over unseen environments with different object layouts. However, due to structural limitations and optimization difficulties, these methods do not learn and generalize well for common environments with a dynamic background, which contain multiple moving objects in addition to controllable objects.

To address these limitations, this paper presents a novel self-supervised, object-oriented dynamics learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP). This framework simultaneously learns disentangled object representations and predicts object motions conditioned on their historical states, their interactions to other objects, and an agent's actions. To reduce the complexity of such concurrent learning and improve sample efficiency, MAOP employs a threelevel learning architecture from the most abstract level of motion detection, to dynamic instance segmentation, and to dynamics learning and prediction. A more abstract learning level solves an easier problem and has lower learning complexity, and its output provides a coarse-grained guidance for the less abstract learning level, improving its speed and quality of learning convergence. This multi-level architecture is inspired by humans' multi-level motion perception from cognitive science studies (Johansson, 1975; Lu & Sperling, 1995; Smith et al., 1998) and multi-level abstraction search in constraint optimization (Zhang & Shah, 2016) . In addition, we design a novel CNN-based spatial-temporal relational reasoning mechanism, which includes a Relation Net to reason about spatial relations between objects and an Inertia Net to learn temporal effects. This mechanism offers a disentangled way to handle physical reasoning in the setting with partial observability.

Empirical results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different object layouts. It can learn from few examples and accurately predict the dynamics of objects as well as raw visual observations in previously unseen environments. In addition, MAOP learns disentangled representations and gains semantically and visually interpretable knowledge, including meaningful object masks, accurate object motions, disentangled reasoning process, and the discovery of the controllable agent.

2. RELATED WORK:

Object-oriented reinforcement learning has received much research attention, which exploits efficient representations based on objects and their interactions. This learning paradigm is close to that of human cognition in the physical world and the learned object-level knowledge can be robustly generalized across environments. Early work on object-oriented RL requires explicit encodings of object representations, such as relational MDPs (Guestrin et al., 2003) , OO-MDPs (Diuk et al., 2008) , object focused q-learning (Cobo et al., 2013) , and Schema Networks (Kansky et al., 2017) . In this paper, we present an end-to-end, self-supervised neural network framework that automatically learns object representations and dynamics conditioned on actions and object relations from raw visual observations. Action-conditioned dynamics learning aims to address one of the core problems for model-based DRL, that is, constructing an environment dynamics model. Several pixel-based approaches have been proposed for learning how an environment changes in response to actions through unsupervised video prediction and achieve remarkable performance in training environments (Oh et al., 2015; Watter et al., 2015; Chiappa et al., 2017) . Fragkiadaki et al. (2016) propose an object-centric prediction method to learn the dynamics model when given the object localization and tracking. Finn et al. (2016) develop a dynamics prediction method that explicitly models pixel motions and is partially invariant to object appearances, and its usage for model-based DRL is demonstrated with model predictive controller (Finn & Levine, 2017) . Recently, Zhu et al. (2018) propose an object-oriented dynamics learning paradigm that enables its learned model to generalize over unseen environments with different object layouts and be robust to changes of object appearances. However, this paradigm focuses environments with a single dynamic object. In this paper, we take a further step towards learning object-oriented dynamics model in more general environments with multiple controlled and uncontrollable dynamic objects. In addition, we design an instance-aware dynamics mechanism to support instance-level dynamics learning and handle partial observations. Relation-based deep learning approaches make significant progress in a wide range of domains such as physical reasoning (Chang et al., 2016; Battaglia et al., 2016) , computer vision (Watters et al., 2017; Wu et al., 2017) , natural language processing (Santoro et al., 2017) , and reinforcement learning (Zambaldi et al., 2018; Zhu et al., 2018) . Relation-based nets introduce relational inductive biases into neural networks, which facilitate generalization over entities and relations and enables relational reasoning (Battaglia et al., 2018) . OODP (Zhu et al., 2018 ) is similar to this paper, learning object-level dynamics conditioned on actions and object-to-object relations. In contrast to OODP, this paper proposes a novel spatial-temporal relational reasoning mechanism, which includes an Inertia Net for learning temporal effects in addition to a CNN-based Relation Net for reasoning about spatial relations. This mechanism offers a disentangled way to handle physical reasoning in the setting of partial observability.

Instance Segmentation has been one of the fundamental problems in computer vision. Instance segmentation can be regarded as the combination of semantic segmentation and object localization. Many approaches have been proposed for instance segmentation, including DeepMask (Pinheiro et al., 2015) , InstanceFCN (Dai et al., 2016) , FCIS (Li et al., 2017) , and Mask R-CNN (He et al., 2017) . Most models are supervised learning and require a large labeled training dataset. Liu et al. (2015) proposes a weakly-supervised approach to infer object instances in foreground by exploiting dynamic consistency in video. In this paper, we design a self-supervised, three-level approach for learning dynamic rigid object instances. At the most abstract level, the foreground detection module provides a coarse-grained guidance for producing region proposals at the instance segmentation level. The instance segmentation level then learns coarse instance segmentation. This coarse instance segmentation provides a guidance for learning accurate instances at the dynamics learning level, whose instance segmentation considers not only object appearances but also motion prediction conditioned on object-to-object relations and actions.

3. MULTI-LEVEL ABSTRACTION OBJECT-ORIENTED PREDICTOR (MAOP):

In this section, we will present a novel self-supervised deep learning framework, aiming to learn object-oriented dynamics models that are able to generalize over unseen environments with different object layouts and multiple dynamic objects. Such an object-oriented dynamics learning approach requires simultaneously learning object representations and motions conditioned on their historical states, their interactions to other objects, and an agent's actions. This concurrent learning is very challenging for an end-to-end approach in complex environments with a dynamic background. Evidences from cognitive science studies (Johansson, 1975; Lu & Sperling, 1995; Smith et al., 1998) show that human beings are born with prior motion perception ability (in Cortical area MT) of perceiving moving and motionlessness, which enables learning more complex knowledge, such as object-level dynamics prediction. Inspired by these studies, we design a multi-level learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP), which incorporates motion perception levels to assist in dynamics learning. Figure 1 illustrates three levels of MAOP framework: dynamics learning, dynamic instance segmentation, and motion detection. The dynamics learning level is an end-to-end, self-supervised neural network, aiming to learn object representations and instance-level dynamics, and predict the next visual observation conditioned on an agent's action. To guide the learning of the object representations and instance localization at the level of dynamics learning, the more abstracted level of dynamic instance segmentation learns a guiding network in a self-supervised manner, which can provide coarse dynamic instance mask proposals. It exploits the spatial-temporal information of locomotion property and appearance pattern to capture the region proposals of dynamic instances. To facilitate the learning of dynamic instance segmentation, MAOP employs the more coarse-grained level of motion detection, which detects changes in image sequences and provides guidance on proposing regions potentially containing dynamic instance. As the learning proceeds, the knowledge distilled from the more coarse-grained level are gradually refined at the more fine-grained level by considering additional information. When the training is finished, the coarse-grained levels of dynamic instance segmentation and motion detection will be removed at the testing stage. In the rest of this section, we will describe in detail the design of each level and their connections.

3.1. OBJECT-ORIENTED DYNAMICS LEARNING LEVEL:

The semantics of this level is formulated as learning an object-based dynamics model with the region proposals generated from the more abstracted level of dynamic instance segmentation. The whole architecture is shown in the top part of Figure 1 , which is an end-to-end neural network and can be trained in a self-supervised manner. It takes a sequence of video frames and an agent's actions as input, learns the disentangled representations (including objects, relations and effects) and the dynamics of controllable and uncontrollable dynamic object instances conditioned on the actions and object relations, and produce the predictions of raw visual observations. The whole architecture includes four major components: A) an Object Detector that decomposes the input image into objects; B) an Instance Localization module responsible for localizing dynamic instances; C) a Dynamics Net for learning the motion of each dynamic instance conditioned on the effects from actions and object-level spatial-temporal relations; and D) a Background Constructor that computes the background image from the learned static object masks. Algorithm 1 illustrates the interactions of these components and the learning paradigm of object based dynamics, which is a general framework and agnostic to the concrete form of each component. In the following paragraphs, we will describe the detailed implementation of every components.

Algorithm 1 Basic paradigm of object-oriented dynamics learning.

Input: A sequence of video frames I t-h:t with length h, input action a t at time t.

1: Object masks O t-h:t ← ObjectDetector(I t-h:t ), O include dynamic and static masks D, S 2: Instance masks X t-h:t ← InstanceLocalization(I t-h:t , D 

i ∈ [0, 1] H M ×W M (1 ≤ i ≤ n M )

, where H M and W M denote the height and width of the bounding box of this instance and n M denotes the maximum number of localized instances. As shown in Figure 1 , Instance Localization Module first samples a number of bounding boxes on the dynamic object masks and then select the regions, each of which contains only one dynamic instance. As we focus on the motion of rigid objects, the affine transformation is approximatively consistent for all pixels of each dynamic instance mask. Inspired by this, we define a discrepancy loss L instance for a sampled region that measures the motion consistence of its pixels and use it as a selection score for selecting instance masks. To compute this loss, we first compute an average rigid transformation of a sampled region between two time steps based on the instance masks and masked image in this region, then apply this transformation to this region at the previous time step, and finally compared this predicted region with the region at the current time. Obviously, when a sampled region contains exactly one dynamic instance, this loss will be extremely small, and even zero when the object masks are perfectly learned. More details of the region proposal sampling and instance mask selection can be found in Appendix A. 2018) to enable the inference of dynamics focusing on the relations with neighbour objects. This module crops a w-size "horizon" window from the concatenated masks of all objects O t centered on the expected location of X t :,:,i , where w denotes the maximum effective range of relations. Then, the cropped object masks are respectively concatenated with the constant x-coordinate and y-coordinate meshgrid map (to make networks more sensitive to the spatial information) and fed into the corresponding Relation Nets (RN) according to their classes. We use C t :,:,i,j to denote the cropped mask that crops the j-th object class O t :,:,j centered on the expected location of the i-th dynamic instance (the class it belongs to is denoted as c i , 1 ≤ c i ≤ n D ). The effect of object class j on class c i , E t (c i , j) ∈ R 2×na (n a denotes the number of actions) is calculated as, E t (c i , j) = RN ci,j concat C t :,:,i,j , Xmap, Ymap . Note that there are total n D × n O RNs for n D × n O pairs of object classes that share the same architecture but not their weights. To handle the partial observation problem, we add an Inertia Nets (IN) to learn the self-effects of an object class through historical states, E t self (c i ) = IN ci concat X t :,:,i , X t+1 :,:,i , . . . , X t+h :,:,i

, where h is the history length and there are total n D INs for n D dynamic object classes that share the same architecture but not their weights. To predict the motion vector m t i ∈ R 2 for the i-th dynamic instance, all these effects are summed up and then multiplied by the one-hot coding of action a t ∈ {0, 1} na , that is,

m t i = j E t (c i , j) + E t self (c i ) • a t .

Background Extractor. This module extracts the static background of input image based on the static object masks learned by Object Detector and then it is combined with the predicted dynamic instances to predict the next visual observation. As Object Detector can decompose its observation into objects in an unseen environment with a different object layout, Background Constructor is able to generate a corresponding static background and support the visual observation prediction in novel environments. Specifically, Background Constructor maintains an external background memory B ∈ R H×W ×3 which is continuously updated (via moving average) by the static object mask learned by Object Detector. Denoting α as the decay rate, the updating formula is given by, B t = αB t-1 + (1 -α)I t i S t :,:,i , B 0 = 0. Prediction and Training Loss. At the output end of our model, the prediction of the next frame is produced by merging the learned object motions and the background B t . The pixels of a dynamic instance can be calculated by masking the raw image with the corresponding instance mask and we can use Spatial Transformer Network (STN) (Jaderberg et al., 2015) to apply the learned instance motion vector m t i on these pixels. First, we transform all the dynamic instances according to the learned instance-level motions. Then, we merge all the transformed dynamic instances with the background image calculated from Background Extractor to generate the prediction of the next frame. In this paper, we use the pixel-wise l 2 loss to restrain image prediction error, denoted as L prediction . To get earlier feedback before reconstructing images and facilitate the training process, we add a highway loss,

L highway = i (ū i , vi ) t + m t i -(ū i , vi ) t+1 2 2

, where (ū i , vi ) t is the excepted location of i-th instance mask X t :,:,i . In addition, we add a proposal loss to utilize the dynamic instance proposals provided by the abstracted problem to guide our optimization, which is given by L proposal = i (D t :,:,i -P t :,:,i ) 2 2 , where P denotes the dynamic instance region proposals computed by the more abstract learning level (i.e., dynamic instance segmentation level). The total loss of the dynamics learning level is given by combining the previous losses with different weights,

L DL = L highway + λ 1 L prediction + λ 2 L proposal

3.2. DYNAMIC INSTANCE SEGMENTATION LEVEL:

This level aims to generate region proposals of dynamic instances to guide the learning of object masks and facilitate instance localization at the level of dynamics learning. The architecture is shown in Figure 1 . Instance Splitter aims to identify regions, each of which potentially contains one dynamic instance. To learn to divide different dynamic object instances onto different masks, we use the discrepancy loss L instance described in Section 3.1 to train Instance Splitter. Considering that one object instance may be split into smaller patches on different masks, we append a Merging Net (i.e., a two-layer CNN with 1 kernel size and 1 stride) to Instance Splitter to learn to merge redundant masks by a merging loss L merge based on the prior that the patches of the same instance are adjacent to each other and share the same motion. In addition, we add a foreground proposal loss L forground to encourage attentions on the dynamic regions. The total loss of this level is given by combining these losses with different weights, L DIS = L instance + λ 3 L merge + λ 4 L forground For more complex domains with arbitrary deformation and appearance change, MAOP is also readily to incorporate the vanilla Mask R-CNN (He et al., 2017) or other off-the-shelf supervised object detection methods (Liu et al., 2018) as a plug-and-play module into our framework to generate region proposals of dynamic instances. In addition, although the network structure of this level is similar to Object Detector in the level of dynamics learning, we do not integrated them together as a whole network because the concurrent learning of both object representations and dynamics model is not stable. Instead, we first learn the coarse object instances only based on the spatial-temporal consistency of locomotion and appearance pattern, and then use them as proposal regions to perform object-oriented dynamics learning at the more fine-grained level, which in turn fine-tunes the object representations. 

3.3. MOTION DETECTION LEVEL:

At this level, we employ foreground detection to detect changing regions from a sequence of image frames and provide coarse dynamic region proposals F p for assisting in dynamic instance segmentation. In our experiments, we use a simple unsupervised foreground detection approach proposed by Lo & Velastin (2001) . Our framework is also compatible with many advanced unsupervised foreground detection methods (Lee, 2005; Maddalena et al., 2008; Zhou et al., 2013; Guo et al., 2014) that are more efficient or more robust to moving camera. These complex unsupervised foreground detection methods have the potential to improve the performance but are not the focus of this work.

4. EXPERIMENTS:

We evaluate our model on two games, Monster Kong and Flappy Bird, from the Pygame Learning Environment (Tasfi, 2016) , which allows us to test generalization ability over various scenes with different layouts. Here, the Monster Kong is the advanced version of that used by Zhu et al. (2018) , which has a more general and complex setting. The monster wanders around and breathes out fires randomly, and the fires also move with some randomness. The agent explores with actions up, down, left, right, jump and noop. All these dynamic objects interact with the environments and objects according to their own physics engine. Moreover, gravity and jump model has a long-term dynamics effects, leading to a partial observation problem. To test whether our model can truly learn the underlying physical mechanism behind the visual observations and perform relational reasoning, we set the k-to-m zero-shot generalization experiment, where we use k different environments for training and m different unseen environments for testing. Flappy Bird is a side-scroller game, where a bird flies between columns of green pipes with action jump and noop. Since the unseen environments will be similar with the training ones without limitation of samples in this game, we limit the samples for training. Two experimental settings are shown in Figure 3 . We use random exploration on Monster Kong, and an expert guided random exploration on Flappy Bird because in this domain a totally random exploration will lead to an early death of the agent even at the very beginning.

We compare MAOP with state-of-the-art action-conditioned dynamics learning baselines, AC Model (Oh et al., 2015) , CDNA (Finn et al., 2016) , and OODP (Zhu et al., 2018) . AC Model adopts an encoder-LSTM-decoder structure, which performs transformations in hidden space and constructs pixel predictions. CDNA explicitly models pixel motions to achieve invariance to appearance. OODP and MAOP both aim at learning object-level dynamics through an object-oriented learning paradigm, which decomposes raw images into objects and perform prediction based on object-level relations. OODP is only designed for class-level dynamics, while MAOP is able to learn instancelevel dynamics. Implementation details of MAOP can be found in Appendix C.

4.1. ZERO-SHOT GENERALIZATION AND SAMPLE EFFICIENCY:

To make a sufficient comparison with the previous methods on the generalization ability and sample efficiency of object dynamics learning and image prediction, we conduct 1-5, 2-5 and 3-5 generalization experiments with a variety of evaluation indices on Monster Kong. We use n-error accuracy to measure the performance of object dynamics prediction, which is defined as the proportion that the difference between the predicted and ground-true agent locations is less than n pixel. We also add an extra pixel-based measurement (denoted by object RMSE), which compares the pixel difference near dynamic objects between the predicted and ground-truth images. To evaluate the image prediction, we adopt a typical image prediction loss (pixel-wise RMSE). In Figure 4 and B7 (Appendix B), we plot the learning curve for better visualization of the comparison in unseen environments. Further, we add a video (https://github.com/maop2018/maop-video/ blob/master/video.avi) for better perceptual understanding of the prediction performance in unseen environments.

Models:

Training environments Unseen environments MAOP and OODP are trained for 1-step prediction, while AC Model and CNDA are trained for 3-step prediction. They are all tested for 3-step and 8-step prediction.

1-5 † 1-5 2-5 3-5 1-5 † 1-5 2-5 3-

In addition, we evaluate our performance of long-range prediction, as shown in Table 2 . For actionconditioned dynamics prediction, our model only trained for 1-step prediction can achieve better performance than AC Model and CDNA trained for 3-step prediction. Specifically, our model has 88% probability that the predicted and ground-true agent locations are below 2 pixel when predicting 3 steps of the future, while the probability is 62% when predicting 8 steps of the future, which is also a satisfactory performance. Because the uncontrolled background objects (e.g., fires) tend to move according to a certain pattern and LSTM is good at remembering specific patterns of dynamics, CDNA has the highest prediction performance for background objects but fails to predict the complex dynamics that is conditioned on actions and object relations. Figure B8 illustrates a case to visualize the 8-step prediction of our model in unseen environments.

We also test our model on Flappy Bird, where we limit the training samples to 100 and 300 to form a sufficiently challenging generalization task. As shown in Visual interpretability. To demonstrate the visual interpretability of MAOP in unseen environments, we visualize the learned masks of dynamic and static objects. We highlight the attentions of the object masks by multiplying the raw images by the binarized masks. Note that MAOP does not require the actual number of objects but a maximum number and some learned object masks may be redundant. Thus, we only show the informative object masks. As shown in Figure 6 , our model captures all the key objects in the environments including the controllable agents (the cowboy and the bird), the uncontrollable dynamic objects (the monster, fires and pipes), and the static objects that have effects on the motions of dynamic objects (ladders, walls and the free space). We also observe that model can learn disentangled object representations and distinguish the objects by both appearance and dynamic property.

Monster Kong Flappy Bird Discovery of the controllable agent. With the learned knowledge in MAOP, we can easily uncover the action-controlled agent from all the dynamic objects, which is useful semantic information that can be used in heuristic algorithms. Specifically, the object that has the maximal variance of total effects over actions is the action-controlled agent. Denote the total effects as E i = ( j E(c i , j)) + E self (c i ), E i ∈ R 2×na , the label of the action-controlled agent is calculated as, arg max i V ar(E i ).

The histogram in Figure 5 plotting the ground-truth label distribution of our discovered actioncontrolled agents clearly demonstrates that our discovery of the controllable agent achieves perfect 100% accuracy.

Dynamical interpretability. To show the dynamical interpretability behind image prediction, we test our predicted motions by comparing RMSEs between the predicted and ground-truth motions in unseen environments (Table B4 in Appendix B). Intriguingly, most predicted motions are quite accurate, with the RMSEs less than 1 pixel. Such a visually indistinguishable error also verifies our dynamics learning.

5. CONCLUSIONS AND FUTURE WORK:

This paper presents a self-supervised multi-level learning framework for learning action-conditioned object-based dynamics. This framework is sample-efficient and generalizes object dynamics and prediction of raw visual observations to complex unseen environments with multiple dynamic objects. The learned dynamics model potentially enables an agent to directly plan or efficiently learn for unseen environments. Although a random policy or an expert's policy is used for exploration in our experiments, our framework can support smarter exploration strategies, e.g., curiosity-driven exploration. Our future work includes extending our model for deformation prediction (e.g., object appearing, disappearing and non-rigid deformation) and incorporating a camera motion prediction network module introduced by Vijayanarasimhan et al. (2017) for applications such as FPS games and autonomous driving.

:



Figures and Tables:

Figure 1: Multi-level abstraction framework from a top-down decomposition view. First, we perform motion detection to produce foreground masks. Then, we utilize the foreground masks as dynamic region proposals to guide the learning of dynamic instance segmentation. Finally, we use the learned dynamic instance segmentation networks (including Instance Splitter and Merging Net) as a guiding network to generate region proposals of dynamic instances and guide the learning of Object Detector in the level of dynamics learning.

Figure 2: Architecture of Dynamics Net (left) and its component of Effect Net (right). Different classes are distinguished by different letters (e.g., A, B, ... , F).

Figure 3: Examples of 1-to-3 generalization experiments.

Figure 4: Learning curves for the dynamics prediction of the agent in unseen environments on Monster Kong.

Figure 5: The ground-truth label distribution of our discovered controllable agents in unseen environments on Monster Kong.

Figure 6: Visualization of the masked images in unseen environments. Left is the raw image.

Object Detector and Instance Localization Module. Object Detector is a CNN module aiming to learn object masks from input image. An object mask describes the spatial distribution of a class of objects, which forms the fundamental building block of our object-oriented framework. Considering that instances of the same class are likely to have different motions, we append an Instance Localization Module to Object Detector to localize each dynamic instance to support instance-level dynamics learning. The class-specific object masks in conjunction with instance localization build the bridge to connect visual perception (Object Detector) with dynamics learning (Dynamics Net), which allows learning objects based on both appearances and dynamics.

Table content: <table><tr><td colspan="5">3: Predicted instance masks</td><td>Xt+1</td><td>← ∅</td></tr><tr><td>9:</td><td>Xt+1</td><td>←</td><td>Xt+1</td><td>xt+1</td></tr><tr><td colspan="6">10: end for 11: Background image B t+1 ← BackgroundConstructor(I t , S t )</td></tr><tr><td colspan="4">12: Predicted next frame</td><td>Ît+1</td><td>← Merge(</td><td>Xt+1</td><td>, B t+1 )</td></tr></table>

Prediction performance on Monster Kong. k-m means the k-to-m generalization problem. † indicates training with only 1000 samples. ALL represents all dynamic objects.As shown in Table1, MAOP significantly outperforms other methods in all experiment settings in terms of generalization ability and sample efficiency of both object dynamics learning and image prediction. It can achieve 0.84 0-error accuracy, even with a single training environment, which suggests MAOP is good at relational reasoning. Although AC Model achieves high accuracy in training environments, its performance in unseen scenes is much worse, which is probably because its pure pixel-level inference easily leads to overfitting. CDNA performs better than AC Model in those uncontrolled objects, but still cannot deal with complicated interactions in lack of knowledge on object-to-object relations. By the structural limitation of OODP, it has innate difficulty on frames with multiple dynamic objects.

Table content: <table><tr><td>5</td></tr></table>

Performance of long-range prediction on Monster Kong. n-acc means n-error accuracy.

Table content: <table/>

Table 3, our performance is similar with that on Monster Kong. Our generalization ability and sample efficiency significantly outperform other baselines. Surprisingly, only 100 samples are enough to reach almost perfect 1-error accuracy.

Table content: <table><tr><td/><td/><td colspan="3">Training environments</td><td/><td colspan="3">Unseen environments</td><td/></tr><tr><td/><td>Models</td><td>1-5  †</td><td/><td>1-5  ‡</td><td/><td>1-5  †</td><td/><td>1-5  ‡</td><td/></tr><tr><td/><td/><td>Agent</td><td>All</td><td>Agent</td><td>All</td><td>Agent</td><td>All</td><td>Agent</td><td>All</td></tr><tr><td/><td>MAOP</td><td>0.84</td><td>0.90</td><td>0.87</td><td>0.93</td><td>0.83</td><td>0.89</td><td>0.83</td><td>0.92</td></tr><tr><td>0-error</td><td>OODP</td><td>0.01</td><td>0.29</td><td>0.01</td><td>0.32</td><td>0.01</td><td>0.18</td><td>0.02</td><td>0.15</td></tr><tr><td>accuracy</td><td>AC Model</td><td>0.39</td><td>0.64</td><td>0.48</td><td>0.75</td><td>0.03</td><td>0.18</td><td>0.04</td><td>0.23</td></tr><tr><td/><td>CDNA</td><td>0.13</td><td>0.78</td><td>0.41</td><td>0.84</td><td>0.10</td><td>0.77</td><td>0.16</td><td>0.79</td></tr><tr><td/><td>MAOP</td><td>0.99</td><td>1.00</td><td>0.97</td><td>0.97</td><td>0.99</td><td>0.99</td><td>0.98</td><td>0.97</td></tr><tr><td>1-error</td><td>OODP</td><td>0.05</td><td>0.52</td><td>0.04</td><td>0.56</td><td>0.06</td><td>0.39</td><td>0.07</td><td>0.39</td></tr><tr><td>accuracy</td><td>AC Model</td><td>0.48</td><td>0.80</td><td>0.57</td><td>0.87</td><td>0.07</td><td>0.37</td><td>0.14</td><td>0.45</td></tr><tr><td/><td>CDNA</td><td>0.26</td><td>0.82</td><td>0.57</td><td>0.89</td><td>0.22</td><td>0.81</td><td>0.36</td><td>0.84</td></tr><tr><td/><td>MAOP</td><td>1.00</td><td>1.00</td><td>0.99</td><td>0.99</td><td>1.00</td><td>1.00</td><td>0.99</td><td>0.98</td></tr><tr><td>2-error</td><td>OODP</td><td>0.14</td><td>0.66</td><td>0.12</td><td>0.67</td><td>0.16</td><td>0.59</td><td>0.16</td><td>0.56</td></tr><tr><td>accuracy</td><td>AC Model</td><td>0.53</td><td>0.85</td><td>0.63</td><td>0.90</td><td>0.12</td><td>0.53</td><td>0.24</td><td>0.64</td></tr><tr><td/><td>CDNA</td><td>0.37</td><td>0.84</td><td>0.66</td><td>0.92</td><td>0.36</td><td>0.84</td><td>0.49</td><td>0.87</td></tr></table>



Table content: <table/>

Peer review (iclr2019):
 In this paper, the novel MAOP model is described for self-supervised learning on past video game frames to predict future frames. The presented results indicate that the method is capable of discovering semantically important visual components, and their relation and dynamics, in frames from arcade-style video games. Key to the approach is a multi-level self-learning approach: more abstract stages focus on simpler problems that are easier to learn, which in turn guide the learning process at the more complex stages. A downside is that it the method is complex, consisting of many specific sub-components and algorithms, which in turn have again other sub-components. This makes the paper a long read with a lot of repetition, and various times the paper refers to the names of sub-components that are only explained later. Other methodological details that are relevant to understand how the method operates are described in the Appendices. I expect that if the paper would be better structured, it would be easier to understanding how all the parts fit together. Another downside of this complexity is that the method seems designed for particular types of video game frames, with static backgrounds, a fixed set of objects or agents. It is unclear how the method would perform on other types of games, or on real-world videos. While the method therefore avoids the need for manual annotation, it instead encodes a lot of domain knowledge in its design and components. I also didn*t fully understand how the self-supervised model is used for Reinforcement Learning in the experiments. Is the MAOP first trained, and the fixed to perform RL with the learned agent models, or is the MOAP learned end-to-end during RL? Pros: + MAOP seems successful on the tested games in the experiments + Demonstrates that, with a sufficiently engineered method, self-supervised learning can be used to discover different types of objects, and their dynamics. Cons: - writing could be improved, as the methodology currently reads as a summation of facts, and some parts are written out of order, resulting in various forward references to components that only become clear later. Several times, the paper states that some novel algorithm is used, but then provides no further explanation in the text as all description of this novelty is deferred to an appendix. - method does not seem generic, hence it is unclear how relevant this architecture it is to other use cases - many hyperparameters for the individual components, algorithms. Unclear how these parameter setting affect the results Below are more detailed comments and questions: General comments: * The proposed MOAP method consists of many subalgorithms, resulting in various (hyper)parameters which may impact the results (e.g. see Appendix A, B). Appendix D lists several used hyperparameter settings, though various parameters for the algorithms are still missing (e.g. thresholds alpha, beta in Algo.2). Were the used parameters optimized? How are these hyperparameters set in practice? How does changing them impact your results? * Methods seems particularly designed for *video games*, where the object and background structures have well defined sizes, appearance, etc. How will the MOAP fair in more realistic situations with noisy observations, occluded objects, changing appearances and lighting conditions, etc.? * How about changing appearance of an agent during an action, e.g. a *walking animation* ? Can your method learn the sequence of sprites to accurately predict the next image? Is that even part of the objective? * Appendix D has important implementation details, but is never mentioned in the text I believe! Didn*t realize it existed on first read through. * Introduction: * What prediction horizon are you targeting? 1 step, T steps into the future, 1 to T steps in the future simultaneously? What are you trying to predict? Object motion? Future observations? * "... which includes a CNN-based Relation Net to ... ", the names Relation Net, Inertia Net, etc.. are used as if the reader is expected to know what these are already. If these networks were introduced in related work already, please add citations. Otherwise please rephrase to clarify that these are networks themselves are part of your novel design. * Section 3.1 * "It takes multiple-frame video images ... and produce the predictions of raw visual observations.". As I understand from this, the self-supervised approach basically performs supervised learning to predict a future frame (target output) given past frames (input). I do not understand how this relates to Reinforcement Learning (RL) as mentioned in the introduction and Related Work. Is there still some reward function in play when learning the MAOP parameters? Or is the idea to first self-supervised learn the MAOP, and afterwards fix its parameters and use it in separate a RL framework? I believe RL is not mentioned anymore until Section 4.2. This connection between self-supervised and reinforcement learning should be clarified, or otherwise the related work should be adjusted to include other (self-supervised) work on predicting future image frames. * "An object mask describes the spatial distribution of an object ..." Does the distribution capture uncertainty on the object*s location, or does it capture the spread of the object*s extent (*mass distribution*) ? * "Note that Object Detector uses the same CNN architecture with OODP". What does OODP stand for? Add citation here. (first mention of OODP is in Experiments section) * "(similar with Section 3.2)" → "similar to". Also, I find it a confusing to say something is similar to what will be done in a future section, which has not yet been introduced. Can you not explain the procedure here, and in Section 3.2 say that the procedure is "similar to Section 3.1" instead? * "to show the detailed structure of the Effect Net module." First time I see the name *Effect Net*, what is it? This whole paragraph different nets are named, with a rough indication of their relation, such as "Dynamic Net", "Relation Net" and "Inertia Net". Is "Effect Net" a different name for any of the three previous nets? The paper requires the reader to puzzle from Fig.2 that Relation Net and Inertia Net are parts of Effect Net, which in turn is part of Dynamics Net. This wasn*t clear from the text at all. * Section 3.2: * p7.: "Since DISN leans" → "Since DISN learns" ? * There are many losses throughout the paper, but I only see at the end of Section 3.1 some mentioning that multiple losses are combined. How is this done for the other components, .e.g is the total loss for DISN a weighted sum of L_foreground and L_instance ? Are the losses for all three three MAOP levels weighted for full end-to-end learning? * This section states various times "we propose a novel [method]", for which then no explanation is given, and all details are explained in the Appendix. While the Appendix can hold important implementation details, I would still expect that novelties of the paper are clearly explained in the paper itself. As it stands, the appendix is used as an extension of the methodological section of an already lengthy paper. * "Conversely, the inverse function is ... " M has a mask for each of the n_o "object classes", hence the "Instance Localization Module" earlier to split out instances from the class masks. So how can there be a single motion vector STN^-1(M,M*) if there are multiple instances for an object mask? How will STN^-1 deal with different amount of instances in M and M* ? * Section 3.3: * What is the output of this level? I expect some mathematical formulation as in the previous sections, resulting in some symbol, that is then used in Section 3.2. E.g. is the output "foreground masks F" (found in Appendix A) ? This paper is a bit of a puzzle through the pages for the reader. * Section 4: * "We compare MAOP with state-of-the-art action-conditioned dynamics learning baselines, ..." Please re-iterate how these methods differ in assumptions, what they model, with respect to your novel method? For instance, is the main difference your "novel region proposal method" and such? Is the overall architecture different? E.g. explain here already the AC Model uses "pixel-level inference", and that OODP has "lacks knowledge on object-to-object relations" to underline their difference to your approach, and provide context for your conclusions in Section 4.1. * Appendix A: * Algorithm 1, line 7: "sample a pixel coordinate" → is this non-deterministically sampling?