Title: OBJECT-ORIENTED MODEL LEARNING THROUGH MULTI-LEVEL ABSTRACTION

Abstract: Object-based approaches for learning action-conditioned dynamics has demonstrated promise for generalization and interpretability. However, existing approaches suffer from structural limitations and optimization difficulties for common environments with multiple dynamic objects. In this paper, we present a novel self-supervised learning framework, called Multi-level Abstraction Objectoriented Predictor (MAOP), for learning object-based dynamics models from raw visual observations. MAOP employs a three-level learning architecture that enables efficient dynamics learning for complex environments with a dynamic background. We also design a spatial-temporal relational reasoning mechanism to support instance-level dynamics learning and handle partial observability. Empirical results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different static object layouts. In addition, MAOP learns semantically and visually interpretable disentangled representations.

1. INTRODUCTION:

Model-based deep reinforcement learning (DRL) has recently attracted much attention for improving sample efficiency of DRL, such as (Heess et al., 2015; Schmidhuber, 2015; Gu et al., 2016; Racanière et al., 2017; Finn & Levine, 2017) . One of the core problems for model-based reinforcement learning is to learn action-conditioned dynamics models through interacting with environments. Pixel-based approaches have been proposed for such dynamics learning from raw visual perception, achieving remarkable performance in training environments (Oh et al., 2015; Watter et al., 2015; Chiappa et al., 2017) .

To unlock sample efficiency of model-based DRL, learning action-conditioned dynamics models that generalize over unseen environments is critical yet challenging. Finn et al. (2016) proposed an action-conditioned video prediction method that explicitly models pixel motion and thus is partially invariant to object appearances. Zhu et al. (2018) developed an object-oriented dynamics predictor, taking a further step towards generalization over unseen environments with different object layouts. However, due to structural limitations and optimization difficulties, these methods do not learn and generalize well for common environments with a dynamic background, which contain multiple moving objects in addition to controllable objects.

To address these limitations, this paper presents a novel self-supervised, object-oriented dynamics learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP). This framework simultaneously learns disentangled object representations and predicts object motions conditioned on their historical states, their interactions to other objects, and an agent's actions. To reduce the complexity of such concurrent learning and improve sample efficiency, MAOP employs a threelevel learning architecture from the most abstract level of motion detection, to dynamic instance segmentation, and to dynamics learning and prediction. A more abstract learning level solves an easier problem and has lower learning complexity, and its output provides a coarse-grained guidance for the less abstract learning level, improving its speed and quality of learning convergence. This multi-level architecture is inspired by humans' multi-level motion perception from cognitive science studies (Johansson, 1975; Lu & Sperling, 1995; Smith et al., 1998) and multi-level abstraction search in constraint optimization (Zhang & Shah, 2016) . In addition, we design a novel CNN-based spatial-temporal relational reasoning mechanism, which includes a Relation Net to reason about spatial relations between objects and an Inertia Net to learn temporal effects. This mechanism offers a disentangled way to handle physical reasoning in the setting with partial observability.

Empirical results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different object layouts. It can learn from few examples and accurately predict the dynamics of objects as well as raw visual observations in previously unseen environments. In addition, MAOP learns disentangled representations and gains semantically and visually interpretable knowledge, including meaningful object masks, accurate object motions, disentangled reasoning process, and the discovery of the controllable agent.

2. RELATED WORK:

Object-oriented reinforcement learning has received much research attention, which exploits efficient representations based on objects and their interactions. This learning paradigm is close to that of human cognition in the physical world and the learned object-level knowledge can be robustly generalized across environments. Early work on object-oriented RL requires explicit encodings of object representations, such as relational MDPs (Guestrin et al., 2003) , OO-MDPs (Diuk et al., 2008) , object focused q-learning (Cobo et al., 2013) , and Schema Networks (Kansky et al., 2017) . In this paper, we present an end-to-end, self-supervised neural network framework that automatically learns object representations and dynamics conditioned on actions and object relations from raw visual observations. Action-conditioned dynamics learning aims to address one of the core problems for model-based DRL, that is, constructing an environment dynamics model. Several pixel-based approaches have been proposed for learning how an environment changes in response to actions through unsupervised video prediction and achieve remarkable performance in training environments (Oh et al., 2015; Watter et al., 2015; Chiappa et al., 2017) . Fragkiadaki et al. (2016) propose an object-centric prediction method to learn the dynamics model when given the object localization and tracking. Finn et al. (2016) develop a dynamics prediction method that explicitly models pixel motions and is partially invariant to object appearances, and its usage for model-based DRL is demonstrated with model predictive controller (Finn & Levine, 2017) . Recently, Zhu et al. (2018) propose an object-oriented dynamics learning paradigm that enables its learned model to generalize over unseen environments with different object layouts and be robust to changes of object appearances. However, this paradigm focuses environments with a single dynamic object. In this paper, we take a further step towards learning object-oriented dynamics model in more general environments with multiple controlled and uncontrollable dynamic objects. In addition, we design an instance-aware dynamics mechanism to support instance-level dynamics learning and handle partial observations. Relation-based deep learning approaches make significant progress in a wide range of domains such as physical reasoning (Chang et al., 2016; Battaglia et al., 2016) , computer vision (Watters et al., 2017; Wu et al., 2017) , natural language processing (Santoro et al., 2017) , and reinforcement learning (Zambaldi et al., 2018; Zhu et al., 2018) . Relation-based nets introduce relational inductive biases into neural networks, which facilitate generalization over entities and relations and enables relational reasoning (Battaglia et al., 2018) . OODP (Zhu et al., 2018 ) is similar to this paper, learning object-level dynamics conditioned on actions and object-to-object relations. In contrast to OODP, this paper proposes a novel spatial-temporal relational reasoning mechanism, which includes an Inertia Net for learning temporal effects in addition to a CNN-based Relation Net for reasoning about spatial relations. This mechanism offers a disentangled way to handle physical reasoning in the setting of partial observability.

Instance Segmentation has been one of the fundamental problems in computer vision. Instance segmentation can be regarded as the combination of semantic segmentation and object localization. Many approaches have been proposed for instance segmentation, including DeepMask (Pinheiro et al., 2015) , InstanceFCN (Dai et al., 2016) , FCIS (Li et al., 2017) , and Mask R-CNN (He et al., 2017) . Most models are supervised learning and require a large labeled training dataset. Liu et al. (2015) proposes a weakly-supervised approach to infer object instances in foreground by exploiting dynamic consistency in video. In this paper, we design a self-supervised, three-level approach for learning dynamic rigid object instances. At the most abstract level, the foreground detection module provides a coarse-grained guidance for producing region proposals at the instance segmentation level. The instance segmentation level then learns coarse instance segmentation. This coarse instance segmentation provides a guidance for learning accurate instances at the dynamics learning level, whose instance segmentation considers not only object appearances but also motion prediction conditioned on object-to-object relations and actions.

3. MULTI-LEVEL ABSTRACTION OBJECT-ORIENTED PREDICTOR (MAOP):

In this section, we will present a novel self-supervised deep learning framework, aiming to learn object-oriented dynamics models that are able to generalize over unseen environments with different object layouts and multiple dynamic objects. Such an object-oriented dynamics learning approach requires simultaneously learning object representations and motions conditioned on their historical states, their interactions to other objects, and an agent's actions. This concurrent learning is very challenging for an end-to-end approach in complex environments with a dynamic background. Evidences from cognitive science studies (Johansson, 1975; Lu & Sperling, 1995; Smith et al., 1998) show that human beings are born with prior motion perception ability (in Cortical area MT) of perceiving moving and motionlessness, which enables learning more complex knowledge, such as object-level dynamics prediction. Inspired by these studies, we design a multi-level learning framework, called Multi-level Abstraction Object-oriented Predictor (MAOP), which incorporates motion perception levels to assist in dynamics learning. Figure 1 illustrates three levels of MAOP framework: dynamics learning, dynamic instance segmentation, and motion detection. The dynamics learning level is an end-to-end, self-supervised neural network, aiming to learn object representations and instance-level dynamics, and predict the next visual observation conditioned on an agent's action. To guide the learning of the object representations and instance localization at the level of dynamics learning, the more abstracted level of dynamic instance segmentation learns a guiding network in a self-supervised manner, which can provide coarse dynamic instance mask proposals. It exploits the spatial-temporal information of locomotion property and appearance pattern to capture the region proposals of dynamic instances. To facilitate the learning of dynamic instance segmentation, MAOP employs the more coarse-grained level of motion detection, which detects changes in image sequences and provides guidance on proposing regions potentially containing dynamic instance. As the learning proceeds, the knowledge distilled from the more coarse-grained level are gradually refined at the more fine-grained level by considering additional information. When the training is finished, the coarse-grained levels of dynamic instance segmentation and motion detection will be removed at the testing stage. In the rest of this section, we will describe in detail the design of each level and their connections.

3.1. OBJECT-ORIENTED DYNAMICS LEARNING LEVEL:

The semantics of this level is formulated as learning an object-based dynamics model with the region proposals generated from the more abstracted level of dynamic instance segmentation. The whole architecture is shown in the top part of Figure 1 , which is an end-to-end neural network and can be trained in a self-supervised manner. It takes a sequence of video frames and an agent's actions as input, learns the disentangled representations (including objects, relations and effects) and the dynamics of controllable and uncontrollable dynamic object instances conditioned on the actions and object relations, and produce the predictions of raw visual observations. The whole architecture includes four major components: A) an Object Detector that decomposes the input image into objects; B) an Instance Localization module responsible for localizing dynamic instances; C) a Dynamics Net for learning the motion of each dynamic instance conditioned on the effects from actions and object-level spatial-temporal relations; and D) a Background Constructor that computes the background image from the learned static object masks. Algorithm 1 illustrates the interactions of these components and the learning paradigm of object based dynamics, which is a general framework and agnostic to the concrete form of each component. In the following paragraphs, we will describe the detailed implementation of every components.

Algorithm 1 Basic paradigm of object-oriented dynamics learning.

Input: A sequence of video frames I t-h:t with length h, input action a t at time t.

1: Object masks O t-h:t ← ObjectDetector(I t-h:t ), O include dynamic and static masks D, S 2: Instance masks X t-h:t ← InstanceLocalization(I t-h:t , D 

i ∈ [0, 1] H M ×W M (1 ≤ i ≤ n M )

, where H M and W M denote the height and width of the bounding box of this instance and n M denotes the maximum number of localized instances. As shown in Figure 1 , Instance Localization Module first samples a number of bounding boxes on the dynamic object masks and then select the regions, each of which contains only one dynamic instance. As we focus on the motion of rigid objects, the affine transformation is approximatively consistent for all pixels of each dynamic instance mask. Inspired by this, we define a discrepancy loss L instance for a sampled region that measures the motion consistence of its pixels and use it as a selection score for selecting instance masks. To compute this loss, we first compute an average rigid transformation of a sampled region between two time steps based on the instance masks and masked image in this region, then apply this transformation to this region at the previous time step, and finally compared this predicted region with the region at the current time. Obviously, when a sampled region contains exactly one dynamic instance, this loss will be extremely small, and even zero when the object masks are perfectly learned. More details of the region proposal sampling and instance mask selection can be found in Appendix A. 2018) to enable the inference of dynamics focusing on the relations with neighbour objects. This module crops a w-size "horizon" window from the concatenated masks of all objects O t centered on the expected location of X t :,:,i , where w denotes the maximum effective range of relations. Then, the cropped object masks are respectively concatenated with the constant x-coordinate and y-coordinate meshgrid map (to make networks more sensitive to the spatial information) and fed into the corresponding Relation Nets (RN) according to their classes. We use C t :,:,i,j to denote the cropped mask that crops the j-th object class O t :,:,j centered on the expected location of the i-th dynamic instance (the class it belongs to is denoted as c i , 1 ≤ c i ≤ n D ). The effect of object class j on class c i , E t (c i , j) ∈ R 2×na (n a denotes the number of actions) is calculated as, E t (c i , j) = RN ci,j concat C t :,:,i,j , Xmap, Ymap . Note that there are total n D × n O RNs for n D × n O pairs of object classes that share the same architecture but not their weights. To handle the partial observation problem, we add an Inertia Nets (IN) to learn the self-effects of an object class through historical states, E t self (c i ) = IN ci concat X t :,:,i , X t+1 :,:,i , . . . , X t+h :,:,i

, where h is the history length and there are total n D INs for n D dynamic object classes that share the same architecture but not their weights. To predict the motion vector m t i ∈ R 2 for the i-th dynamic instance, all these effects are summed up and then multiplied by the one-hot coding of action a t ∈ {0, 1} na , that is,

m t i = j E t (c i , j) + E t self (c i ) • a t .

Background Extractor. This module extracts the static background of input image based on the static object masks learned by Object Detector and then it is combined with the predicted dynamic instances to predict the next visual observation. As Object Detector can decompose its observation into objects in an unseen environment with a different object layout, Background Constructor is able to generate a corresponding static background and support the visual observation prediction in novel environments. Specifically, Background Constructor maintains an external background memory B ∈ R H×W ×3 which is continuously updated (via moving average) by the static object mask learned by Object Detector. Denoting α as the decay rate, the updating formula is given by, B t = αB t-1 + (1 -α)I t i S t :,:,i , B 0 = 0. Prediction and Training Loss. At the output end of our model, the prediction of the next frame is produced by merging the learned object motions and the background B t . The pixels of a dynamic instance can be calculated by masking the raw image with the corresponding instance mask and we can use Spatial Transformer Network (STN) (Jaderberg et al., 2015) to apply the learned instance motion vector m t i on these pixels. First, we transform all the dynamic instances according to the learned instance-level motions. Then, we merge all the transformed dynamic instances with the background image calculated from Background Extractor to generate the prediction of the next frame. In this paper, we use the pixel-wise l 2 loss to restrain image prediction error, denoted as L prediction . To get earlier feedback before reconstructing images and facilitate the training process, we add a highway loss,

L highway = i (ū i , vi ) t + m t i -(ū i , vi ) t+1 2 2

, where (ū i , vi ) t is the excepted location of i-th instance mask X t :,:,i . In addition, we add a proposal loss to utilize the dynamic instance proposals provided by the abstracted problem to guide our optimization, which is given by L proposal = i (D t :,:,i -P t :,:,i ) 2 2 , where P denotes the dynamic instance region proposals computed by the more abstract learning level (i.e., dynamic instance segmentation level). The total loss of the dynamics learning level is given by combining the previous losses with different weights,

L DL = L highway + λ 1 L prediction + λ 2 L proposal

3.2. DYNAMIC INSTANCE SEGMENTATION LEVEL:

This level aims to generate region proposals of dynamic instances to guide the learning of object masks and facilitate instance localization at the level of dynamics learning. The architecture is shown in Figure 1 . Instance Splitter aims to identify regions, each of which potentially contains one dynamic instance. To learn to divide different dynamic object instances onto different masks, we use the discrepancy loss L instance described in Section 3.1 to train Instance Splitter. Considering that one object instance may be split into smaller patches on different masks, we append a Merging Net (i.e., a two-layer CNN with 1 kernel size and 1 stride) to Instance Splitter to learn to merge redundant masks by a merging loss L merge based on the prior that the patches of the same instance are adjacent to each other and share the same motion. In addition, we add a foreground proposal loss L forground to encourage attentions on the dynamic regions. The total loss of this level is given by combining these losses with different weights, L DIS = L instance + λ 3 L merge + λ 4 L forground For more complex domains with arbitrary deformation and appearance change, MAOP is also readily to incorporate the vanilla Mask R-CNN (He et al., 2017) or other off-the-shelf supervised object detection methods (Liu et al., 2018) as a plug-and-play module into our framework to generate region proposals of dynamic instances. In addition, although the network structure of this level is similar to Object Detector in the level of dynamics learning, we do not integrated them together as a whole network because the concurrent learning of both object representations and dynamics model is not stable. Instead, we first learn the coarse object instances only based on the spatial-temporal consistency of locomotion and appearance pattern, and then use them as proposal regions to perform object-oriented dynamics learning at the more fine-grained level, which in turn fine-tunes the object representations. 

3.3. MOTION DETECTION LEVEL:

At this level, we employ foreground detection to detect changing regions from a sequence of image frames and provide coarse dynamic region proposals F p for assisting in dynamic instance segmentation. In our experiments, we use a simple unsupervised foreground detection approach proposed by Lo & Velastin (2001) . Our framework is also compatible with many advanced unsupervised foreground detection methods (Lee, 2005; Maddalena et al., 2008; Zhou et al., 2013; Guo et al., 2014) that are more efficient or more robust to moving camera. These complex unsupervised foreground detection methods have the potential to improve the performance but are not the focus of this work.

4. EXPERIMENTS:

We evaluate our model on two games, Monster Kong and Flappy Bird, from the Pygame Learning Environment (Tasfi, 2016) , which allows us to test generalization ability over various scenes with different layouts. Here, the Monster Kong is the advanced version of that used by Zhu et al. (2018) , which has a more general and complex setting. The monster wanders around and breathes out fires randomly, and the fires also move with some randomness. The agent explores with actions up, down, left, right, jump and noop. All these dynamic objects interact with the environments and objects according to their own physics engine. Moreover, gravity and jump model has a long-term dynamics effects, leading to a partial observation problem. To test whether our model can truly learn the underlying physical mechanism behind the visual observations and perform relational reasoning, we set the k-to-m zero-shot generalization experiment, where we use k different environments for training and m different unseen environments for testing. Flappy Bird is a side-scroller game, where a bird flies between columns of green pipes with action jump and noop. Since the unseen environments will be similar with the training ones without limitation of samples in this game, we limit the samples for training. Two experimental settings are shown in Figure 3 . We use random exploration on Monster Kong, and an expert guided random exploration on Flappy Bird because in this domain a totally random exploration will lead to an early death of the agent even at the very beginning.

We compare MAOP with state-of-the-art action-conditioned dynamics learning baselines, AC Model (Oh et al., 2015) , CDNA (Finn et al., 2016) , and OODP (Zhu et al., 2018) . AC Model adopts an encoder-LSTM-decoder structure, which performs transformations in hidden space and constructs pixel predictions. CDNA explicitly models pixel motions to achieve invariance to appearance. OODP and MAOP both aim at learning object-level dynamics through an object-oriented learning paradigm, which decomposes raw images into objects and perform prediction based on object-level relations. OODP is only designed for class-level dynamics, while MAOP is able to learn instancelevel dynamics. Implementation details of MAOP can be found in Appendix C.

4.1. ZERO-SHOT GENERALIZATION AND SAMPLE EFFICIENCY:

To make a sufficient comparison with the previous methods on the generalization ability and sample efficiency of object dynamics learning and image prediction, we conduct 1-5, 2-5 and 3-5 generalization experiments with a variety of evaluation indices on Monster Kong. We use n-error accuracy to measure the performance of object dynamics prediction, which is defined as the proportion that the difference between the predicted and ground-true agent locations is less than n pixel. We also add an extra pixel-based measurement (denoted by object RMSE), which compares the pixel difference near dynamic objects between the predicted and ground-truth images. To evaluate the image prediction, we adopt a typical image prediction loss (pixel-wise RMSE). In Figure 4 and B7 (Appendix B), we plot the learning curve for better visualization of the comparison in unseen environments. Further, we add a video (https://github.com/maop2018/maop-video/ blob/master/video.avi) for better perceptual understanding of the prediction performance in unseen environments.

Models:

Training environments Unseen environments MAOP and OODP are trained for 1-step prediction, while AC Model and CNDA are trained for 3-step prediction. They are all tested for 3-step and 8-step prediction.

1-5 † 1-5 2-5 3-5 1-5 † 1-5 2-5 3-

In addition, we evaluate our performance of long-range prediction, as shown in Table 2 . For actionconditioned dynamics prediction, our model only trained for 1-step prediction can achieve better performance than AC Model and CDNA trained for 3-step prediction. Specifically, our model has 88% probability that the predicted and ground-true agent locations are below 2 pixel when predicting 3 steps of the future, while the probability is 62% when predicting 8 steps of the future, which is also a satisfactory performance. Because the uncontrolled background objects (e.g., fires) tend to move according to a certain pattern and LSTM is good at remembering specific patterns of dynamics, CDNA has the highest prediction performance for background objects but fails to predict the complex dynamics that is conditioned on actions and object relations. Figure B8 illustrates a case to visualize the 8-step prediction of our model in unseen environments.

We also test our model on Flappy Bird, where we limit the training samples to 100 and 300 to form a sufficiently challenging generalization task. As shown in Visual interpretability. To demonstrate the visual interpretability of MAOP in unseen environments, we visualize the learned masks of dynamic and static objects. We highlight the attentions of the object masks by multiplying the raw images by the binarized masks. Note that MAOP does not require the actual number of objects but a maximum number and some learned object masks may be redundant. Thus, we only show the informative object masks. As shown in Figure 6 , our model captures all the key objects in the environments including the controllable agents (the cowboy and the bird), the uncontrollable dynamic objects (the monster, fires and pipes), and the static objects that have effects on the motions of dynamic objects (ladders, walls and the free space). We also observe that model can learn disentangled object representations and distinguish the objects by both appearance and dynamic property.

Monster Kong Flappy Bird Discovery of the controllable agent. With the learned knowledge in MAOP, we can easily uncover the action-controlled agent from all the dynamic objects, which is useful semantic information that can be used in heuristic algorithms. Specifically, the object that has the maximal variance of total effects over actions is the action-controlled agent. Denote the total effects as E i = ( j E(c i , j)) + E self (c i ), E i ∈ R 2×na , the label of the action-controlled agent is calculated as, arg max i V ar(E i ).

The histogram in Figure 5 plotting the ground-truth label distribution of our discovered actioncontrolled agents clearly demonstrates that our discovery of the controllable agent achieves perfect 100% accuracy.

Dynamical interpretability. To show the dynamical interpretability behind image prediction, we test our predicted motions by comparing RMSEs between the predicted and ground-truth motions in unseen environments (Table B4 in Appendix B). Intriguingly, most predicted motions are quite accurate, with the RMSEs less than 1 pixel. Such a visually indistinguishable error also verifies our dynamics learning.

5. CONCLUSIONS AND FUTURE WORK:

This paper presents a self-supervised multi-level learning framework for learning action-conditioned object-based dynamics. This framework is sample-efficient and generalizes object dynamics and prediction of raw visual observations to complex unseen environments with multiple dynamic objects. The learned dynamics model potentially enables an agent to directly plan or efficiently learn for unseen environments. Although a random policy or an expert's policy is used for exploration in our experiments, our framework can support smarter exploration strategies, e.g., curiosity-driven exploration. Our future work includes extending our model for deformation prediction (e.g., object appearing, disappearing and non-rigid deformation) and incorporating a camera motion prediction network module introduced by Vijayanarasimhan et al. (2017) for applications such as FPS games and autonomous driving.

:



Figures and Tables:

Figure 1: Multi-level abstraction framework from a top-down decomposition view. First, we perform motion detection to produce foreground masks. Then, we utilize the foreground masks as dynamic region proposals to guide the learning of dynamic instance segmentation. Finally, we use the learned dynamic instance segmentation networks (including Instance Splitter and Merging Net) as a guiding network to generate region proposals of dynamic instances and guide the learning of Object Detector in the level of dynamics learning.

Figure 2: Architecture of Dynamics Net (left) and its component of Effect Net (right). Different classes are distinguished by different letters (e.g., A, B, ... , F).

Figure 3: Examples of 1-to-3 generalization experiments.

Figure 4: Learning curves for the dynamics prediction of the agent in unseen environments on Monster Kong.

Figure 5: The ground-truth label distribution of our discovered controllable agents in unseen environments on Monster Kong.

Figure 6: Visualization of the masked images in unseen environments. Left is the raw image.

Object Detector and Instance Localization Module. Object Detector is a CNN module aiming to learn object masks from input image. An object mask describes the spatial distribution of a class of objects, which forms the fundamental building block of our object-oriented framework. Considering that instances of the same class are likely to have different motions, we append an Instance Localization Module to Object Detector to localize each dynamic instance to support instance-level dynamics learning. The class-specific object masks in conjunction with instance localization build the bridge to connect visual perception (Object Detector) with dynamics learning (Dynamics Net), which allows learning objects based on both appearances and dynamics.

Table content: <table><tr><td colspan="5">3: Predicted instance masks</td><td>Xt+1</td><td>← ∅</td></tr><tr><td>9:</td><td>Xt+1</td><td>←</td><td>Xt+1</td><td>xt+1</td></tr><tr><td colspan="6">10: end for 11: Background image B t+1 ← BackgroundConstructor(I t , S t )</td></tr><tr><td colspan="4">12: Predicted next frame</td><td>Ît+1</td><td>← Merge(</td><td>Xt+1</td><td>, B t+1 )</td></tr></table>

Prediction performance on Monster Kong. k-m means the k-to-m generalization problem. † indicates training with only 1000 samples. ALL represents all dynamic objects.As shown in Table1, MAOP significantly outperforms other methods in all experiment settings in terms of generalization ability and sample efficiency of both object dynamics learning and image prediction. It can achieve 0.84 0-error accuracy, even with a single training environment, which suggests MAOP is good at relational reasoning. Although AC Model achieves high accuracy in training environments, its performance in unseen scenes is much worse, which is probably because its pure pixel-level inference easily leads to overfitting. CDNA performs better than AC Model in those uncontrolled objects, but still cannot deal with complicated interactions in lack of knowledge on object-to-object relations. By the structural limitation of OODP, it has innate difficulty on frames with multiple dynamic objects.

Table content: <table><tr><td>5</td></tr></table>

Performance of long-range prediction on Monster Kong. n-acc means n-error accuracy.

Table content: <table/>

Table 3, our performance is similar with that on Monster Kong. Our generalization ability and sample efficiency significantly outperform other baselines. Surprisingly, only 100 samples are enough to reach almost perfect 1-error accuracy.

Table content: <table><tr><td/><td/><td colspan="3">Training environments</td><td/><td colspan="3">Unseen environments</td><td/></tr><tr><td/><td>Models</td><td>1-5  †</td><td/><td>1-5  ‡</td><td/><td>1-5  †</td><td/><td>1-5  ‡</td><td/></tr><tr><td/><td/><td>Agent</td><td>All</td><td>Agent</td><td>All</td><td>Agent</td><td>All</td><td>Agent</td><td>All</td></tr><tr><td/><td>MAOP</td><td>0.84</td><td>0.90</td><td>0.87</td><td>0.93</td><td>0.83</td><td>0.89</td><td>0.83</td><td>0.92</td></tr><tr><td>0-error</td><td>OODP</td><td>0.01</td><td>0.29</td><td>0.01</td><td>0.32</td><td>0.01</td><td>0.18</td><td>0.02</td><td>0.15</td></tr><tr><td>accuracy</td><td>AC Model</td><td>0.39</td><td>0.64</td><td>0.48</td><td>0.75</td><td>0.03</td><td>0.18</td><td>0.04</td><td>0.23</td></tr><tr><td/><td>CDNA</td><td>0.13</td><td>0.78</td><td>0.41</td><td>0.84</td><td>0.10</td><td>0.77</td><td>0.16</td><td>0.79</td></tr><tr><td/><td>MAOP</td><td>0.99</td><td>1.00</td><td>0.97</td><td>0.97</td><td>0.99</td><td>0.99</td><td>0.98</td><td>0.97</td></tr><tr><td>1-error</td><td>OODP</td><td>0.05</td><td>0.52</td><td>0.04</td><td>0.56</td><td>0.06</td><td>0.39</td><td>0.07</td><td>0.39</td></tr><tr><td>accuracy</td><td>AC Model</td><td>0.48</td><td>0.80</td><td>0.57</td><td>0.87</td><td>0.07</td><td>0.37</td><td>0.14</td><td>0.45</td></tr><tr><td/><td>CDNA</td><td>0.26</td><td>0.82</td><td>0.57</td><td>0.89</td><td>0.22</td><td>0.81</td><td>0.36</td><td>0.84</td></tr><tr><td/><td>MAOP</td><td>1.00</td><td>1.00</td><td>0.99</td><td>0.99</td><td>1.00</td><td>1.00</td><td>0.99</td><td>0.98</td></tr><tr><td>2-error</td><td>OODP</td><td>0.14</td><td>0.66</td><td>0.12</td><td>0.67</td><td>0.16</td><td>0.59</td><td>0.16</td><td>0.56</td></tr><tr><td>accuracy</td><td>AC Model</td><td>0.53</td><td>0.85</td><td>0.63</td><td>0.90</td><td>0.12</td><td>0.53</td><td>0.24</td><td>0.64</td></tr><tr><td/><td>CDNA</td><td>0.37</td><td>0.84</td><td>0.66</td><td>0.92</td><td>0.36</td><td>0.84</td><td>0.49</td><td>0.87</td></tr></table>



Table content: <table/>

Peer review (iclr2019):
 This paper proposes a new architecture for learning dynamics models in 2D Atari-like game words. The architecture includes multiple layers of abstraction: a “motion detection” level, which looks at which pixels change over time in order to guess at which parts of the image are in the foreground or not; a “instance segmentation” level, which segments the foreground into regions and instances; and a “dynamics learning” level, which learns the dynamics of object instances using a interaction network-style approach. Pros: - Impressive-looking dynamics predictions in Atari-like games. - An object-based prediction model, which could enable predictions about specific entities in the scene rather than holistic frame predictions. Cons: - Very complicated and difficult-to-understand architecture. - No ablation studies to validate different components of the architecture. - No validation in a model-based RL or control setting. - Experiments are only done on one-step predictions, rather than long-term rollouts. Quality --------- The quality of the predictions seems quite high (based on Figure 6 and the results tables), though there are a number of opportunities to further strengthen the evaluation and analysis: - I wish that there were more than a single figure of qualitative results to go on. I highly recommend that a revision include a link to a video showing more predictions over time for each environment, ideally with comparisons to the other baselines as well. - The introduction of the paper motivates the learning of the model in terms of model-based RL, however, the model is not actually used in a model-based RL setting. It would be nice to see at least a simple validation that the model can be used with an off-the-shelf planner to solve one of the games which are evaluated in the paper. If it cannot, then that limits the significance of the model. - As far as I can tell, all the results reported in the tables are based on one-step predictions only. While it is great to show that even in this regime the other models struggle, it would be even better if results could be reported for longer rollouts (i.e., taking the model outputs and feeding it back in as input, and repeating this procedure say 50 steps into the future). Models are not particularly useful in a MBRL setting if they can only be used to predict a single timestep, so it is important to validate that longer-term predictions can be made as well. Overall the literature review is reasonably solid, but I am not sure the citations in the opening sentence are quite appropriate as model-based DRL has been around for longer than 2017 (see for example [1-3]). Moreover, Chiappa et al (2017) only learns a model and does not use it for planning, so I am not sure it is quite appropriate as a citation for MBRL. Clarity -------- Unfortunately, I had a very hard time understanding how exactly the architecture works and I felt like there were a lot of details missing. I am not confident that I would be able to reproduce the architecture from reading the paper alone. Below, I will list some of the specific points where I was confused, but I think overall the paper needs to be substantially reorganized in order to be clearer as to how the architecture actually works. More broadly, I think some of my confusion stems from the fact that there are very similar computations occurring across the three levels of abstraction but the paper does not really make it clear how these computations relate to one another or how they are similar/different. For example, in the “dynamics learning” level there are modules for performing object detection and instance localization. But then in the “instance segmentation” level, there are similarly modules for detecting and masking out instances. It is not clear to me why this needs to be done twice? In general, I would *strongly* recommend including at least in the appendix an algorithm box that sketches out the computational graph for the whole architecture (not in as much detail as the existing algorithm boxes, but in more detail than what is given in Figure 1). Specific places where I was confused: - Where do the region proposals (P) come from? - If I’m understanding correctly, the variable M is used multiple times in multiple different ways. It seems to be produced from the “instance localization” module in the “dynamics learning” level, but also from the “dynamic instance segmentation network” in the “instance segmentation” level. Are these M different or the same? - Where does F_foreground^(t) come from? Originality ------------- Overall, the idea of learning object-based transition models is not really new (and there are a few citations missing regarding prior work in this regard, e.g. [4-6]). However, there is yet to be an accepted solution for actually learning object-based models robustly and the present work seems to result in the cleanest separation between dynamic objects and background that I have seen so far, and is therefore quite original in that regard. This paper appears to be quite similar to Zhu & Zhang (2018), with the main difference being additional functionality to handle multiple dynamic objects in a scene rather than just a single dynamic object. This is a fairly significant difference and the improvement over Zhu & Zhang (2018) seems quite large, so even though the papers seem quite similar on the surface I think the difference is actually quite substantial. Significance ---------------- If it were clearer how to reproduce this paper, and if it could be shown to apply to a wider range of environments (e.g. the Atari suite, or even better the Sonic domains from the OpenAI Retro contest), then I believe this paper could be quite significant as it would open up new avenues for model-based learning in these domains. Unfortunately, however, it is not clear to me as the paper is currently written how well it would do on other 2D environments, thus limiting the significance. If the model only works on Monster Kong and Flappy Bird---neither of which are commonly used in the RL literature---then it has limited applicability to the rest of the model-based RL community. Similarly, as stated above, it is not clear how well the model will work with longer rollouts or in actual in MBRL settings, thus limiting its significance. References --------------- [1] Heess, Wayne, Silver, Lillicrap, Tassa, & Erez (2015). Learning Continuous Control Policies by Stochastic Value Gradients. NIPS 2015. [2] Gu, Lillicrap, Sutskever, & Levine (2016). Continuous Deep Q-Learning with Model-based Acceleration. ICML 2016. [3] Schmidhuber (2015). On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models. arXiv 2015. [4] Wu, Yildirim, Lim, Freeman, & Tenenbaum (2015). Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning. NIPS 2015. [5] Fragkiadaki, Agrawal, Levine, & Malik (2016). Learning visual predictive models of physics for playing billiards. ICLR 2016. [6] Kansky, Silver, Mely, Eldawy, Lazaro-Gredilla, Lou, Dorfman, Sido, Phoenix, & George (2017). Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics. ICML 2017.