Occlusion  resistant  learning  of  intuitive physics from videos

1 Introduction . Learning intuitive physics has recently raised significant interest in the machine learning literature . To reach human performance on complex visual tasks , artificial systems need to understand the world in terms of macroscopic objects , movements , interactions , etc . Infant development experiments show that young infants quickly acquire an intuitive grasp of how objects interact in the world , and that they use these intuitions for prediction and action planning ( Carey , 2009 ; Baillargeon & Carey , 2012 ) . This includes the notions of gravity ( Carey , 2009 ) , continuity of trajectories ( Spelke et al. , 1995 ) , collisions ( Saxe & Carey , 2006 ) , etc . Object permanence , the fact that an object continues to exist when it is occluded , ( Kellman & Spelke , 1983 ) , is one of the first concepts developed by infants . From a modeling point of view , the key scientific question is how to develop general-purpose methods that can make physical predictions in noisy environments , where many variables of the system are unknown . A model that could mimic even some of infant ’ s ability to predict the dynamics of objects and their interactions would be a significant advancement in model-based action planning for robotics ( Agrawal et al. , 2016 ; Finn & Levine , 2017 ) . Importantly , to be applied to real-world problems , such a model needs to predict object motion in 3D and handle frequent inter-object occlusions . Yet , to our knowledge , most current works on learning intuitive physics get around this challenge by either i ) working in 2D spaces with no occlusions ( Battaglia et al. , 2016 ; Chang et al. , 2016 ; Fragkiadaki et al. , 2015 ) or ii ) learning end-to-end models without decomposing the scene into objects ( Agrawal et al. , 2016 ; Lerer et al. , 2016 ; Finn et al. , 2016 ) . The former methods have demonstrated that learning models of intuitive physics is possible but assume ground truth positions of objects are available at both training and test time . The latter methods can operate directly on pixel inputs without knowing ground truth positions of objects but are typically limited to a small number of objects and generalize poorly to new setups ( e.g . a new number of objects in the scene , see ( Lerer et al. , 2016 ) ) . A third class of methods has recently emerged ( Janner et al. , 2018 ) that first decomposes the input image of the 3D scene into layers corresponding to masks of individual objects and learns scene dynamics given such object-centric decomposition . Note that here the object dynamics is learnt from pixel masks of individual objects , rather than their ground truth positions . This is difficult for 3D scenes due to frequent inter-object occlusions that present two major challenges . First , estimating accurate position and velocity of objects is challenging due to partial occlusions by other objects . Second , objects can be fully occluded by other objects for a significant number of frames . This work falls into the third class of compositional methods , but develops an occlusion resistant model for learning intuitive physics that addresses both of these challenges due to inter-object occlusions . In detail we propose a compositional model that from object instance masks and depth fields in two consecutive frames , ( Mt , t+1 , Dt , t+1 ) , estimates the center , velocity and size of objects . This predicted state ŝt is then used as input of a Recurrent Interaction Network , which predicts a sequence of futures states ŝt+2 , ... , ŝt+L . This sequence of states is given to the Compositional Rendering Network which produces segmentation masks M̂t+2 , ... , M̂t+L and depth estimates D̂t+2 , ... , D̂t+L in future frames . The key innovation of the proposed model is dealing with partial and complete occlusions in the scene . To deal with partial occlusions , the obtained sequence of masks+depths is compared to the ground truth , and gradients are backpropagated through the pre-trained Compositional Rendering Network to refine state predictions . This allows us to refine positions of partially occluded objects where simply taking the centroid of the observed portion of the mask results in an incorrect estimate of the object position . With this refinement object positions are corrected taking into account the unobserved ( occluded ) portion of the object . The refined state estimates s̄t+1 , ... , s̄t+L are used at training time for learning parameters of the Recurrent Interaction Network and at test time to improve accuracy of object position prediction when following partially occluded objects . To deal with full occlusions , when the object is not visible in multiple frames , we use the learnt model of object dynamics ( Recurrent Interaction Network ) to predict the position of the object multiple frames ahead and thus recovering the object position after the occlusion . Using the proposed approach , we show that it is possible to learn object dynamics in 3D environments with severe inter-object occlusions and predict segmentation masks up to 30 frames in the future despite occlusion other objects thus mimicking object permanence . 2 Related work . Forward modelling in videos . Forward modelling in video has been studied for action planning ( Ebert et al. , 2018 ; Finn et al. , 2016 ) and as a scheme for unsupervised learning of visual features ( Lan et al. , 2014 ; Mathieu et al. , 2015 ) . In that setup , a model is given a sequence of frames and has to generate frames in future time steps . To succeed in this task , such models need to predict object movements , suggesting that they need to learn physical regularities from video . However , models for end-to-end future frame prediction tend to perform poorly on long-term prediction tasks ( say more 5-8 frames ( Lan et al. , 2014 ; Mathieu et al. , 2015 ; Finn et al. , 2016 ) ) , failing to preserve object properties and generating blurry outputs . This suggests that models for intuitive physics may require a more structured representation of objects and their interactions . Learning dynamics of objects . Longer term predictions can be more successful when done on the level of trajectories of individual objects . For example , in ( Wu et al. , 2017b ) , the authors propose `` scene de-rendering '' , a system that builds an object-based , structured representation from a static ( synthetic ) image . The recovered state can be further used for physical reasoning and future prediction using a physics engine on both synthetic and real data ( Battaglia et al. , 2013 ; Wu et al. , 2017a ) . Future prediction from static image is often multi-modal ( e.g . car can move forward or backward ) and hence models able to predict multiple possible future predictions , e.g . based on variational auto-encoders ( Xue et al. , 2016 ) , are needed . Others have developed structured models that factor object motion and object rendering into two learnable modules . Examples include ( Watters et al. , 2017 ; Marco Fraccaro , 2017 ; Ehrhardt et al. , 2017b ; a ) that combine object-centric dynamic models and visual encoders . Such models parse each frame into a set of object state representations , which are used as input of a `` dynamic '' model , predicting object motion . However , ( Marco Fraccaro , 2017 ) restrict drastically the complexity of the visual input by working on binary 32x32 frames , and ( Ehrhardt et al. , 2017b ; a ; Watters et al. , 2017 ) still need ground truth position of objects to train their models . None of these work explicitly models inter-object occlusions , which is the focus of our method . In our work , we build on learnable models of object dynamics ( Battaglia et al. , 2016 ) and ( Chang et al. , 2016 ) , which have the key property that they are compositional and hence can model a variable number of objects , but extend them to learn from visual input rather than ground truth object state vectors . Our work is related to ( Janner et al. , 2018 ) , done independently and concurrently with our work , who develop an object-oriented model of dynamics coupled with a differentiable object renderer to predict a single image with segmentation masks of objects in a future time , given a single still image as input . In contrast , our model predicts frame-by-frame object motion in scenes with partial and full object occlusion . This is possible because ( i ) our model of dynamics is recursive , predicting a whole sequence of object movements ( instead of one single image in future ( Janner et al. , 2018 ) ) that allows the model to be applied recursively to follow an object through complete occlusion by other objects ; ( ii ) we design a refinement procedure that allows to refine the estimated positions of objects in case of partial occlusions . In addition , in contrast to ( Janner et al. , 2018 ) our model predicts velocity of objects and depth of the scene ( also taking as input a pair of frames and the depth field ) . Others have proposed unsupervised methods to discover objects and their interactions in 2d videos ( van Steenkiste et al. , 2018 ) . It is also possible to construct Hierarchical Relation Networks ( Mrowca et al. , 2018 ) , representing objects as graphs and predicting interactions between pairs of objects . However , this task is still challenging and requires full supervision in the form of ground truth position and velocity of objects . Learning physical properties from visual inputs . Related are also methods for learning physical properties of objects . Learning of physical properties , such as mass , volume or coefficients of friction and restitution , has been considered in ( Wu et al. , 2016 ) . Others have looked at predicting the stability and/or the dynamics of towers of blocks ( Lerer et al. , 2016 ; Zhang et al. , 2016 ; Li et al. , 2016a ; b ; Mirza et al. , 2017 ; Groth et al. , 2018 ) . Our work is complementary . We don ’ t consider prediction of physical properties but focus on learning models of object dynamics handling inter-object occlusions at both training and test time . ( Greff et al. , 2019 ) Contributions . We describe a model that learns complex dynamics of objects in a 3D environment , where inter-object occlusions occur frequently . Our model combines an abstract representation of the scene ( position , velocity and depth of objects ) , with a compositional neural renderer predicting the resulting object masks with depth and explicitly modelling occlusions between objects . This procedure allows us to train the model even when some objects are partially or totally occluded . Unlike ( Watters et al. , 2017 ) , our model is fully compositional and handles variable number of objects in the scene . Moreover , it does not require as input annotated inter-frame correspondences during training . 3 Occlusion resistant modeling for intuitive physics . This section describes our model for occlusion resistant learning of intuitive physics . We first describe the learning set-up considered in this work . We then describe in detail the two main components of our model . In section 3.2 we outline the compositional renderer with occlusion reasoning that predicts object masks given a scene state representation , and in section 3.3 we detail the recurrent interaction network that predicts the scene state evolution over time . Finally , in section 3.4 we outline the training procedure .

Summary: This paper proposes a method that combines a recurrent neural network that predicts values that are used as inputs to a rendered which interprets them and generates an object shape map and a depth map for every step of the dynamics predicted by the recurrent neural network. The proposed method is able to handle object occlusions and interactions. In experiments, the authors show improved performance against baselines for future prediction, object tracking, and object permanence.