Object-Oriented Model Learning through Multi-Level Abstraction

1 INTRODUCTION . Model-based deep reinforcement learning ( DRL ) has recently attracted much attention for improving sample efficiency of DRL , such as ( Heess et al. , 2015 ; Schmidhuber , 2015 ; Gu et al. , 2016 ; Racanière et al. , 2017 ; Finn & Levine , 2017 ) . One of the core problems for model-based reinforcement learning is to learn action-conditioned dynamics models through interacting with environments . Pixel-based approaches have been proposed for such dynamics learning from raw visual perception , achieving remarkable performance in training environments ( Oh et al. , 2015 ; Watter et al. , 2015 ; Chiappa et al. , 2017 ) . To unlock sample efficiency of model-based DRL , learning action-conditioned dynamics models that generalize over unseen environments is critical yet challenging . Finn et al . ( 2016 ) proposed an action-conditioned video prediction method that explicitly models pixel motion and thus is partially invariant to object appearances . Zhu et al . ( 2018 ) developed an object-oriented dynamics predictor , taking a further step towards generalization over unseen environments with different object layouts . However , due to structural limitations and optimization difficulties , these methods do not learn and generalize well for common environments with a dynamic background , which contain multiple moving objects in addition to controllable objects . To address these limitations , this paper presents a novel self-supervised , object-oriented dynamics learning framework , called Multi-level Abstraction Object-oriented Predictor ( MAOP ) . This framework simultaneously learns disentangled object representations and predicts object motions conditioned on their historical states , their interactions to other objects , and an agent ’ s actions . To reduce the complexity of such concurrent learning and improve sample efficiency , MAOP employs a threelevel learning architecture from the most abstract level of motion detection , to dynamic instance segmentation , and to dynamics learning and prediction . A more abstract learning level solves an easier problem and has lower learning complexity , and its output provides a coarse-grained guidance for the less abstract learning level , improving its speed and quality of learning convergence . This multi-level architecture is inspired by humans ’ multi-level motion perception from cognitive science studies ( Johansson , 1975 ; Lu & Sperling , 1995 ; Smith et al. , 1998 ) and multi-level abstraction search in constraint optimization ( Zhang & Shah , 2016 ) . In addition , we design a novel CNN-based spatial-temporal relational reasoning mechanism , which includes a Relation Net to reason about spatial relations between objects and an Inertia Net to learn temporal effects . This mechanism offers a disentangled way to handle physical reasoning in the setting with partial observability . Empirical results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different object layouts . It can learn from few examples and accurately predict the dynamics of objects as well as raw visual observations in previously unseen environments . In addition , MAOP learns disentangled representations and gains semantically and visually interpretable knowledge , including meaningful object masks , accurate object motions , disentangled reasoning process , and the discovery of the controllable agent . 2 RELATED WORK . Object-oriented reinforcement learning has received much research attention , which exploits efficient representations based on objects and their interactions . This learning paradigm is close to that of human cognition in the physical world and the learned object-level knowledge can be robustly generalized across environments . Early work on object-oriented RL requires explicit encodings of object representations , such as relational MDPs ( Guestrin et al. , 2003 ) , OO-MDPs ( Diuk et al. , 2008 ) , object focused q-learning ( Cobo et al. , 2013 ) , and Schema Networks ( Kansky et al. , 2017 ) . In this paper , we present an end-to-end , self-supervised neural network framework that automatically learns object representations and dynamics conditioned on actions and object relations from raw visual observations . Action-conditioned dynamics learning aims to address one of the core problems for model-based DRL , that is , constructing an environment dynamics model . Several pixel-based approaches have been proposed for learning how an environment changes in response to actions through unsupervised video prediction and achieve remarkable performance in training environments ( Oh et al. , 2015 ; Watter et al. , 2015 ; Chiappa et al. , 2017 ) . Fragkiadaki et al . ( 2016 ) propose an object-centric prediction method to learn the dynamics model when given the object localization and tracking . Finn et al . ( 2016 ) develop a dynamics prediction method that explicitly models pixel motions and is partially invariant to object appearances , and its usage for model-based DRL is demonstrated with model predictive controller ( Finn & Levine , 2017 ) . Recently , Zhu et al . ( 2018 ) propose an object-oriented dynamics learning paradigm that enables its learned model to generalize over unseen environments with different object layouts and be robust to changes of object appearances . However , this paradigm focuses environments with a single dynamic object . In this paper , we take a further step towards learning object-oriented dynamics model in more general environments with multiple controlled and uncontrollable dynamic objects . In addition , we design an instance-aware dynamics mechanism to support instance-level dynamics learning and handle partial observations . Relation-based deep learning approaches make significant progress in a wide range of domains such as physical reasoning ( Chang et al. , 2016 ; Battaglia et al. , 2016 ) , computer vision ( Watters et al. , 2017 ; Wu et al. , 2017 ) , natural language processing ( Santoro et al. , 2017 ) , and reinforcement learning ( Zambaldi et al. , 2018 ; Zhu et al. , 2018 ) . Relation-based nets introduce relational inductive biases into neural networks , which facilitate generalization over entities and relations and enables relational reasoning ( Battaglia et al. , 2018 ) . OODP ( Zhu et al. , 2018 ) is similar to this paper , learning object-level dynamics conditioned on actions and object-to-object relations . In contrast to OODP , this paper proposes a novel spatial-temporal relational reasoning mechanism , which includes an Inertia Net for learning temporal effects in addition to a CNN-based Relation Net for reasoning about spatial relations . This mechanism offers a disentangled way to handle physical reasoning in the setting of partial observability . Instance Segmentation has been one of the fundamental problems in computer vision . Instance segmentation can be regarded as the combination of semantic segmentation and object localization . Many approaches have been proposed for instance segmentation , including DeepMask ( Pinheiro et al. , 2015 ) , InstanceFCN ( Dai et al. , 2016 ) , FCIS ( Li et al. , 2017 ) , and Mask R-CNN ( He et al. , 2017 ) . Most models are supervised learning and require a large labeled training dataset . Liu et al . ( 2015 ) proposes a weakly-supervised approach to infer object instances in foreground by exploiting dynamic consistency in video . In this paper , we design a self-supervised , three-level approach for learning dynamic rigid object instances . At the most abstract level , the foreground detection module provides a coarse-grained guidance for producing region proposals at the instance segmentation level . The instance segmentation level then learns coarse instance segmentation . This coarse instance segmentation provides a guidance for learning accurate instances at the dynamics learning level , whose instance segmentation considers not only object appearances but also motion prediction conditioned on object-to-object relations and actions . 3 MULTI-LEVEL ABSTRACTION OBJECT-ORIENTED PREDICTOR ( MAOP ) . In this section , we will present a novel self-supervised deep learning framework , aiming to learn object-oriented dynamics models that are able to generalize over unseen environments with different object layouts and multiple dynamic objects . Such an object-oriented dynamics learning approach requires simultaneously learning object representations and motions conditioned on their historical states , their interactions to other objects , and an agent ’ s actions . This concurrent learning is very challenging for an end-to-end approach in complex environments with a dynamic background . Evidences from cognitive science studies ( Johansson , 1975 ; Lu & Sperling , 1995 ; Smith et al. , 1998 ) show that human beings are born with prior motion perception ability ( in Cortical area MT ) of perceiving moving and motionlessness , which enables learning more complex knowledge , such as object-level dynamics prediction . Inspired by these studies , we design a multi-level learning framework , called Multi-level Abstraction Object-oriented Predictor ( MAOP ) , which incorporates motion perception levels to assist in dynamics learning . Figure 1 illustrates three levels of MAOP framework : dynamics learning , dynamic instance segmentation , and motion detection . The dynamics learning level is an end-to-end , self-supervised neural network , aiming to learn object representations and instance-level dynamics , and predict the next visual observation conditioned on an agent ’ s action . To guide the learning of the object representations and instance localization at the level of dynamics learning , the more abstracted level of dynamic instance segmentation learns a guiding network in a self-supervised manner , which can provide coarse dynamic instance mask proposals . It exploits the spatial-temporal information of locomotion property and appearance pattern to capture the region proposals of dynamic instances . To facilitate the learning of dynamic instance segmentation , MAOP employs the more coarse-grained level of motion detection , which detects changes in image sequences and provides guidance on proposing regions potentially containing dynamic instance . As the learning proceeds , the knowledge distilled from the more coarse-grained level are gradually refined at the more fine-grained level by considering additional information . When the training is finished , the coarse-grained levels of dynamic instance segmentation and motion detection will be removed at the testing stage . In the rest of this section , we will describe in detail the design of each level and their connections . 3.1 OBJECT-ORIENTED DYNAMICS LEARNING LEVEL . The semantics of this level is formulated as learning an object-based dynamics model with the region proposals generated from the more abstracted level of dynamic instance segmentation . The whole architecture is shown in the top part of Figure 1 , which is an end-to-end neural network and can be trained in a self-supervised manner . It takes a sequence of video frames and an agent ’ s actions as input , learns the disentangled representations ( including objects , relations and effects ) and the dynamics of controllable and uncontrollable dynamic object instances conditioned on the actions and object relations , and produce the predictions of raw visual observations . The whole architecture includes four major components : A ) an Object Detector that decomposes the input image into objects ; B ) an Instance Localization module responsible for localizing dynamic instances ; C ) a Dynamics Net for learning the motion of each dynamic instance conditioned on the effects from actions and object-level spatial-temporal relations ; and D ) a Background Constructor that computes the background image from the learned static object masks . Algorithm 1 illustrates the interactions of these components and the learning paradigm of object based dynamics , which is a general framework and agnostic to the concrete form of each component . In the following paragraphs , we will describe the detailed implementation of every components . Algorithm 1 Basic paradigm of object-oriented dynamics learning . Input : A sequence of video frames It−h : t with length h , input action at at time t. 1 : Object masks Ot−h : t ← ObjectDetector ( It−h : t ) , O include dynamic and static masks D , S 2 : Instance masks Xt−h : t ← InstanceLocalization ( It−h : t , Dt−h : t ) 3 : Predicted instance masks X̂ t+1 ← ∅ 4 : for each instance mask x in X do 5 : Effects from spatial relations mt1 ← RelationNet ( xt , O t , at ) 6 : Effects from temporal relations mt2 ← InertiaNet ( xt−h : t , at ) 7 : Total effects mt ←mt1 + mt2 8 : Predicted instance mask x̂t+1 ← Transformation ( xt , mt ) 9 : X̂ t+1 ← X̂ t+1⋃ x̂t+1 10 : end for 11 : Background image Bt+1 ← BackgroundConstructor ( It , St ) 12 : Predicted next frame Î t+1 ← Merge ( X̂ t+1 , Bt+1 ) Object Detector and Instance Localization Module . Object Detector is a CNN module aiming to learn object masks from input image . An object mask describes the spatial distribution of a class of objects , which forms the fundamental building block of our object-oriented framework . Considering that instances of the same class are likely to have different motions , we append an Instance Localization Module to Object Detector to localize each dynamic instance to support instance-level dynamics learning . The class-specific object masks in conjunction with instance localization build the bridge to connect visual perception ( Object Detector ) with dynamics learning ( Dynamics Net ) , which allows learning objects based on both appearances and dynamics . Specifically , Object Detector receives image It ∈ RH×W×3 at timestep t and then outputs object masks Ot ∈ [ 0 , 1 ] H×W×nO , including dynamic object masks Dt ∈ [ 0 , 1 ] H×W×nD and static object masks St ∈ [ 0 , 1 ] H×W×nS , where H and W denote the height and width of the input image , nD and nS denotes the maximum class number of dynamic and static objects respectively , and nO = nD + nS . The entry Ou , v , i indicates the probability that the pixel Iu , v , : belongs to the ith object class . Then , Instance Localization Module uses the dynamic object masks to compute each single instance mask Xt : , : ,i ∈ [ 0 , 1 ] HM×WM ( 1 ≤ i ≤ nM ) , where HM and WM denote the height and width of the bounding box of this instance and nM denotes the maximum number of localized instances . As shown in Figure 1 , Instance Localization Module first samples a number of bounding boxes on the dynamic object masks and then select the regions , each of which contains only one dynamic instance . As we focus on the motion of rigid objects , the affine transformation is approximatively consistent for all pixels of each dynamic instance mask . Inspired by this , we define a discrepancy loss Linstance for a sampled region that measures the motion consistence of its pixels and use it as a selection score for selecting instance masks . To compute this loss , we first compute an average rigid transformation of a sampled region between two time steps based on the instance masks and masked image in this region , then apply this transformation to this region at the previous time step , and finally compared this predicted region with the region at the current time . Obviously , when a sampled region contains exactly one dynamic instance , this loss will be extremely small , and even zero when the object masks are perfectly learned . More details of the region proposal sampling and instance mask selection can be found in Appendix A. Dynamics Net . Dynamics Net is designed to learn instance-based motion effects of actions , objectto-object spatial relations ( Relation Net ) and temporal relations of spatial states ( Inertia Net ) , and then reason about the motion of each dynamic instance based on these effects . Its architecture is illustrated in Figure 2 , which has an Effect Net for each class of objects . An Effect Net consists of one Inertia Net and several Relation Nets . As shown in the left subfigure , instance-level dynamics learning is performed , which means the motion of each dynamic instance is individually computed . We take as an example the computation of the motion of the i-th instance Xt : , : ,i to show the detailed structure of the Effect Net . As shown in the right subfigure of Figure 2 , we first use a sub-differentiable tailor module introduced by Zhu et al . ( 2018 ) to enable the inference of dynamics focusing on the relations with neighbour objects . This module crops a w-size “ horizon ” window from the concatenated masks of all objects Ot centered on the expected location of Xt : , : ,i , wherew denotes the maximum effective range of relations . Then , the cropped object masks are respectively concatenated with the constant x-coordinate and y-coordinate meshgrid map ( to make networks more sensitive to the spatial information ) and fed into the corresponding Relation Nets ( RN ) according to their classes . We use Ct : , : ,i , j to denote the cropped mask that crops the j-th object class Ot : , : ,j centered on the expected location of the i-th dynamic instance ( the class it belongs to is denoted as ci , 1 ≤ ci ≤ nD ) . The effect of object class j on class ci , Et ( ci , j ) ∈ R2×na ( na denotes the number of actions ) is calculated as , Et ( ci , j ) = RNci , j ( concat ( Ct : , : ,i , j , Xmap , Ymap ) ) . Note that there are total nD × nO RNs for nD×nO pairs of object classes that share the same architecture but not their weights . To handle the partial observation problem , we add an Inertia Nets ( IN ) to learn the self-effects of an object class through historical states , Etself ( ci ) = INci ( concat ( Xt : , : ,i , X t+1 : , : ,i , . . . , X t+h : , : ,i ) ) , where h is the history length and there are total nD INs for nD dynamic object classes that share the same architecture but not their weights . To predict the motion vector mti ∈ R2 for the i-th dynamic instance , all these effects are summed up and then multiplied by the one-hot coding of action at ∈ { 0 , 1 } na , that is , mti = ( ( ∑ j E t ( ci , j ) ) + Etself ( ci ) ) · at . Background Extractor . This module extracts the static background of input image based on the static object masks learned by Object Detector and then it is combined with the predicted dynamic instances to predict the next visual observation . As Object Detector can decompose its observation into objects in an unseen environment with a different object layout , Background Constructor is able to generate a corresponding static background and support the visual observation prediction in novel environments . Specifically , Background Constructor maintains an external background memory B ∈ RH×W×3 which is continuously updated ( via moving average ) by the static object mask learned by Object Detector . Denoting α as the decay rate , the updating formula is given by , Bt = αBt−1 + ( 1− α ) It ∑ i S t : , : ,i , B 0 = 0 . Prediction and Training Loss . At the output end of our model , the prediction of the next frame is produced by merging the learned object motions and the background Bt . The pixels of a dynamic instance can be calculated by masking the raw image with the corresponding instance mask and we can use Spatial Transformer Network ( STN ) ( Jaderberg et al. , 2015 ) to apply the learned instance motion vector mti on these pixels . First , we transform all the dynamic instances according to the learned instance-level motions . Then , we merge all the transformed dynamic instances with the background image calculated from Background Extractor to generate the prediction of the next frame . In this paper , we use the pixel-wise l2 loss to restrain image prediction error , denoted as Lprediction . To get earlier feedback before reconstructing images and facilitate the training process , we add a highway loss , Lhighway = ∑ i ww ( ūi , v̄i ) t + mti − ( ūi , v̄i ) t+1ww22 , where ( ūi , v̄i ) t is the excepted location of i-th instance mask Xt : , : ,i . In addition , we add a proposal loss to utilize the dynamic instance proposals provided by the abstracted problem to guide our optimization , which is given by Lproposal = ww∑ i ( D t : , : ,i−P t : , : ,i ) ww2 2 , where P denotes the dynamic instance region proposals computed by the more abstract learning level ( i.e. , dynamic instance segmentation level ) . The total loss of the dynamics learning level is given by combining the previous losses with different weights , LDL = Lhighway + λ1Lprediction + λ2Lproposal

Summary: In this paper, the novel MAOP model is described for self-supervised learning on past video game frames to predict future frames. The presented results indicate that the method is capable of discovering semantically important visual components, and their relation and dynamics, in frames from arcade-style video games. Key to the approach is a multi-level self-learning approach:  more abstract stages focus on simpler problems that are easier to learn, which in turn guide the learning process at the more complex stages.