Learning to perceive objects by prediction

1 INTRODUCTION . Visual scenes are composed of various objects in front of backgrounds . Discovering objects from 2D images and inferring their 3D locations is crucial for planning actions in robotics ( Devin et al. , 2018 ; Wang et al. , 2019 ) and this can potentially provide better abstraction of the environment for reinforcement learning ( RL ) , e.g . Veerapaneni et al . ( 2020 ) . The appearance and spatial arrangement of objects , together with the lighting and the viewing angle , determine the 2D images formed on the retina or a camera . Therefore , objects are latent causes of 2D images , and discovering object is a process of inferring latent causes ( Kersten et al. , 2004 ) . The predominant approach in computer vision for identifying and localizing objects rely on supervised learning to infer bounding boxes ( Ren et al. , 2015 ; Redmon et al. , 2016 ) or pixel-level segmentation of objects ( Chen et al. , 2017 ) . However , the supervised approach requires expensive human labeling . It is also difficult to label every possible category of objects . Therefore , an increasing interest has developed recently to build unsupervised or self-supervised models to infer objects from images , such as MONet ( Burgess et al. , 2019 ) , IODINE ( Greff et al. , 2019 ) slot-attention ( Locatello et al. , 2020 ) , GENESIS ( Engelcke et al. , 2019 ; 2021 ) , C-SWM ( Kipf et al. , 2019 ) and mulMON ( Nanbo et al. , 2020 ) . Our work also focuses on the same unsupervised object-centric representation learning ( OCRL ) problem , but offers a new learning objective and architecture to overcome the limitation of existing works in segmenting more complex scenes and explicitly represents objects ’ 3D locations . The majority of the existing OCRL works are demonstrated on relatively simple scenes with objects of pure colors and background lacking complex textures . As recently pointed out , the success of several recent models based on a variational auto-encoder ( VAE ) architecture ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) depends on a reconstruction bottleneck that needs to be intricately balanced ( Engelcke et al. , 2020 ) . To evaluate how such models perform on scenes with more complex surface textures , we created a new dataset of indoor scenes with diverse texture patterns on the objects and background . We found that several existing unsupervised OCRL models overly rely on clustering pixels based on their colors . A challenge in our dataset and in real-world perception is that sharp boundaries between different colors exist both at contours of objects and within the surface of the same object . A model essentially has to implicitly learn a prior knowledge of which types of boundaries are more likely to be real object contours . The reconstruction loss in existing works appears to be insufficient for learning this prior . To tackle this challenge , we draw our inspiration from development psychology and neuroscience . Infants understand the concept of object as early as 8 months , before they can associate objects with names ( Piaget & Cook , 1952 ; Flavell , 1963 ) . The fact that infants are surprised when object are hidden indicates that they have already learned to segment discrete objects from the scene and that their brains constantly make prediction and check for deviation from expected outcome . As the brain lacks direct external supervision for object segmentation , the most likely learning signal is from the error of this prediction . In the brain , a copy of the motor command ( efference copy ) is sent from the motor cortex simultaneously to the sensory cortex , which is hypothesized to facilitate the prediction of changes in sensory input due to self-generated motion ( Feinberg , 1978 ) . What remains to be predicted are changes in visual input due to the motion of external objects . Therefore , we believe that the functional purpose of grouping pixels into object is to allow the prediction of the motion of the constituting pixels in a coherent way by tracking very few parameters ( e.g. , the location , pose , and speed of an object ) . Driven by this hypothesis , our contribution in this paper is : ( 1 ) we combine predictive learning and explicitly 3D motion prediction to learn 3D aware objectcentric representation , which we call Object Perception by Predictive LEarning ( OPPLE ) ; ( 2 ) we provide a new dataset1 with complex surface texture and motion by both the camera and objects to evaluate object-centric representation models ; ( 3 ) we found several previous models overly rely on clustering colors to segment objects ; ( 4 ) although our model leverages image prediction as learning objective , the architecture generalize the ability of object segmentation and spatial localization to single-frame images . 2 METHOD . 2.1 PROBLEM FORMULATION . We denote a scene as a set of distinct objects and a background S = { O1 , O2 , . . . , OK , B } , whereK is the number of objects in scene . At any moment t , we denote two state variables , the location and pose of each object relative to the perspective of an observer ( camera ) , as x ( t ) 1 : K andφ ( t ) 1 : K , where x ( t ) k is the 3-d coordinate of the k-th object and φ ( t ) k is its yaw angle from a canonical pose , as viewed from the reference frame of the camera ( for simplicity , we do not consider pitch and roll here and leave it for future work to extend to 3D pose ) . At time t , given the location of the camera o ( t ) ∈ R3 and its facing direction α ( t ) , S renders a 2D image on the camera as I ( t ) ∈ Rw×h×3 , where w × h is the size of the image . Our goal is to train a neural network that infers properties of objects given only a single image I ( t ) as the sole input without external supervision and with only the information of the intrinsics and egomotion of the camera without explicit knowledge of its extrinsic matrix : { z ( t ) 1 : K , π ( t ) 1 : K+1 , x̂ ( t ) 1 : K , p ( t ) φ1 : K } = fobj ( I ( t ) ) ( 1 ) Here , z ( t ) 1 : K is a set of view-invariant vectors representing the identity of each object k. “ Viewinvariant ” is loosely defined as |z ( t ) k − z ( t+∆t ) k | < |z ( t ) k − z ( t ) l | for k 6= l and ∆t > 0 in most cases , i.e. , the vector codes are more similar for the same object across views than they are different across objects . π ( t ) 1 : K+1 ∈ R ( K+1 ) ×w×h are the probabilities that each pixel belongs to any of the objects or the background ( ∑ k πkij = 1 for any pixel at i , j ) , which achieve object segmentation . To localize objects , x̂ ( t ) 1 : K are the estimated locations of each object relative to the observer and p ( t ) φ1 : K are estimated probability distribution of the poses of each object . Each p ( t ) φk ∈ R b is a probability distribution over b equally-spaced bins of yaw angles in ( 0 , 2π ) . 2.2 OVERALL PRINCIPLE : LEARNING OBJECT REPRESENTATION BY PREDICTING THE FUTURE . Our hypothesis is that the notion of object emerges to meet the need of efficiently predicting the future fates of all parts of an object . With the ( to be learned ) ability to infer an object ’ s pose and location from each frame , the object ’ s speed of translation and rotation can be estimated from consecutive frames . If depth is further inferred for each pixel belonging to an object , then the optical 1We will release upon publication of the paper flow of each pixel can be predicted based on the object ’ s speed and the position of each pixel relative to the object ’ s center . The pixel-segmentation of an object essentially prescribes which pixels should move together with the object . With the predicted optical flow , one can further predict part of the next image by warping the current image . The parts of the next image unpredictable by warping include surfaces of objects or the background that are currently occluded but will become visible , and the region of a scene newly entering the view due to self- or object-motion . These portions can only be predicted based on the learned statistics of the appearance of objects and background , which we call ” imagination ” . In this work , we will show that with the information of self-motion , knowledge of geometry ( rule of rigid-body movement ) and the assumption of smooth object movement , the object representations captured by function fobj and depth perception can be learned without supervision . 2.3 NETWORK ARCHITECTURE . To demonstrate the hypothesized principle above , we build our OPPLE networks as illustrated in Figure 1 , which process two consecutive images individually and make prediction for the next image with the information extracted from them . Object extraction network . We build an object extraction network fθobj by modificaiton of a U-Net ( Ronneberger et al. , 2015 ) to extract representation for each object , the pixels it occupies and its spatial location and pose . A basic U-Net is composed of a convolutional encoder and a transposed convolutional decoder , while each encoder layer sends a skip connection to the corresponding decoder layer , so that the decoder can combine both global and local information . Inside our fθobj , an image I ( t+1 ) first passes through the encoder . Additional Atrous spatial pyramid pooling layer ( Chen et al. , 2017 ) is inserted between the middle two convolutional layers of the encoder to expand the receptive field . The top layer of the encoder outputs a feature vector et capturing the global information of the scene . A Long-Short Term Memory ( LSTM ) network further repeatedly reads in e ( t ) and sequentially outputs one code for an object at a time . Each object code is then mapped through a one-layer fully connected network to predict object code z ( t ) k , object location x̂ ( t ) k and object pose probability p φ ( t ) k , k = 1 , 2 , · · · , K. The inferred location is restricted to the visible range of the camera with an upper limit of distance . The pose prediction is represented as log p φ ( t ) k for numerical stability . Each object code vector z ( t ) k is then independently fed through the decoder with shared skip connection from the encoder . The decoder outputs one channel for each pixel , representing an un-normalized log likelihood that the pixel belongs to the object k. The unnormalized logit maps for all objects are concatenated with a map of all zero for the background , and compete through a softmax function to output the probabilistic segmentation map π ( t ) k . Depth perception network . We use a standard U-Net for depth perception function hθdepth that processes images I ( t ) and output a single-channel depth mapD ( t ) . Object-based imagination network . We build the imagination network gθimag also with a modified U-Net . The input is concatenated image I ( t ) and log of depth log ( D ( t ) ) inferred by the depth perception network , both multiplied element-wise by one probabilistic mask π ( t ) k . The output of the encoder network is concatenated with a vector composed of the observer ’ s moving velocity v ( t ) obs and rotational speed ω ( t ) obs , and the estimated object location x̂ ( t ) k , velocity v̂ ( t ) k and rotational speed ω̂ ( t ) k before entering the decoder . The decoder outputs five channels for each pixel : three for predicting RGB colors , one for depth and one for the probability of the pixel belonging to any object k or background and is used to weight the predicted color and depth for the final ” imagination ” .

Summary: This paper presents an unsupervised object-centric scene representation technique that can decompose a scene into multiple objects (and segment the scene) and infer their 3D locations and pose. The overall setup is very similar to earlier models like MONet but this model works on sequences of images, more precisely on 3 consecutive images. It uses the first two images to infer the 3D position and pose of objects and combining this with known camera motion tries to predict the last (third) image. The main contribution here is an optical flow based method to warp the image at time t using the predicted object location/pose/depth to predict (some of) the pixels in image at time t+1.  In more detail, the object extraction network outputs the location and pose of each object. And a separate depth perception network outputs the depth for each pixel in the image. The location and pose of objects are used to estimate the velocity of each object (e.g., by subtracting the position at t-1 from position at t. note this requires matching each object at time t-1 to object in time t, which they do using a soft-matching approach). These along with the depth information are then used to warp the image at t to predict pixels in image at time t+1. This is possible only for a subset of the pixels so for the rest, they use a separate "imagination" network that takes in object information and predicts the color/depth and object masks at t+1. The predictions from warping and imagination network are then combined to form the final predicted color and depth images.  To train the model, they require images and camera motion, and use a combination of losses: reconstruction loss on predicted and ground truth image, self-supervised losses on object location, pose, and depth.