Learning to perceive objects by prediction
Authors: Tushar Arora
Li Erran Li
Ming Bo Cai
Venue: ICLR 2022 Submitted (2022)
Keywords: predictive learning
object-centric representation
3D perception
sensory grounding

Abstract: The representation of objects is the building block of higher-level concepts. Infants develop the notion of objects without supervision. The prediction error of future sensory input is likely the major teaching signal for infants. Inspired by this, we propose a new framework to extract object-centric representation from single 2D images by learning to predict future scenes in the presence of moving objects. We treat objects as latent causes whose function to an agent is to facilitate efficient prediction of the coherent motion of their parts in visual input. Distinct from previous object-centric models, our model learn to explicitly infer objects' location in 3D environment in addition to segmenting objects. Further, the network learns a latent code space where objects with the same geometric shape and texture/color frequently group together. The model requires no supervision or pre-training of any part of the network. We provide a new synthetic dataset with more complex textures on objects and background and found several previous models not based on predictive learning overly rely on clustering colors and lose specificity in object segmentation.  Our work demonstrates a new approach for learning symbolic representation grounded in sensation and action.

Under review as a conference paper at ICLR 2022
LEARNING TO PERCEIVE OBJECTS BY PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
The representation of objects is the building block of higher-level concepts. In- fants develop the notion of objects without supervision. The prediction error of future sensory input is likely the major teaching signal for infants. Inspired by this, we propose new framework extract object-centric representation from single 2D images by learning to predict future scenes in the presence of moving objects. We treat objects as latent causes whose function to an agent facilitate efficient prediction of the coherent motion of their parts in visual input. Distinct from pre- vious object-centric models, our model learns to explicitly infer objects' locations in 3D environment in addition to segmenting objects. Further, the network learns a latent code space where objects with the same geometric shape and texture/color frequently group together. The model requires no supervision pre-training of any part of the network. We provide a new synthetic dataset with more complex textures objects and background and found several previous models not based on predictive learning overly rely on clustering colors and lose specificity in ob- ject segmentation. Our work demonstrates new approach for leamning symbolic representation grounded in sensation and action.
INTRODUCTION
Visual scenes are composed of various objects in front of backgrounds. Discovering objects from 2D images and inferring their 3D locations is crucial for planning actions in robotics (Devin et al.]
2018 Wang et al. 2019) and this can potentially provide better abstraction of the environment for reinforcement learning (RL). e.g. Veerapaneni et al. (2020). The appearance and spatial arrangement of objects, together with the lighting and the viewing angle. determine the 2D images formed on the retina or a camera. Therefore, objects are latent causes of 2D images, and discovering object is a process of inferring latent causes (Kersten al. 2004). The predominant approach in computer vision for identifying and localizing objects rely on supervised learning to infer bounding boxes (Ren et al.) 2015] [Redmon et al.] (2016) or pixel-level segmentation of objects (Chen et al.. (017). However, the supervised approach requires expensive human labeling. It is also difficult to label every possible category of objects. Therefore, an increasing interest has developed recently to build unsupervised self-supervisee models to infer objects from images, such as MONet (Burgess al. 2019), IODINE (Greff et al. 2019) slot-attention (Locatello et al.] 22020), GENESIS (Engelcke et al, 2019 2021), C-SWM (Kipf al. 2019) and mulMON (Nanbo al. 2020)) Our work also focuses on the same unsupervised object-centric representation learning (OCRL) problem, but offers a new learning objective and architecture to overcome the limitation of existing works in segmenting more complex scenes and explicitly represents objects' 3D locations.
The majority of the existing OCRL works are demonstrated on relatively simple scenes with objects of pure colors and background lacking complex textures. As recently pointed out, the success of several recent models based on variational auto-encoder (VAE) architecture (Kingma & Welling 2013 Rezende et al. (2014) depends on a reconstruction bottleneck that needs to be intricately bal- anced (Engelcke et al. 2020). To evaluate how such models perform on scenes with more complex surface textures, we created a new dataset of indoor scenes with diverse texture patterns on the ob- jects and background. We found that several existing unsupervised OCRL models overly rely on clustering pixels based on their colors. A challenge in our dataset and in real-world perception is that sharp boundaries between different colors exist both at contours of objects and within the sur- face of the same object. A model essentially has to implicitly lear prior knowledge of which types of boundaries are more likely to be real object contours. The reconstruction loss in existing works appears to be insufficient for learning this prior.

---

Under review as a conference paper at ICLR 2022
To tackle this challenge, we draw our inspiration from development psychology and neuroscience. Infants understand the concept of object as early as 8 months, before they can associate objects with names (Piaget & Cook 1952 Flavell 1963). The fact that infants are surprised when object are hidden indicates that they have already learned to segment discrete objects from the scene and that their brains constantly make prediction and check for deviation from expected outcome. As the brain lacks direct external supervision for object segmentation, the most likely learning signal from the error of this prediction. In the brain, copy of the motor command (efference copy) is sent from the motor cortex simultaneously to the sensory cortex, which is hypothesized to facilitate the prediction of changes in sensory input due self- self-generated motion (Feinberg, 1978). What remains to be predicted are changes in visual input due to the motion of external objects. Therefore, we believe that the functional purpose of grouping pixels into object is to allow the prediction of the motion of the constituting pixels in coherent way by tracking very few parameters (e.g., the location, pose, and speed of an object). Driven by this hypothesis, our contribution in this paper is: (1) we combine predictive learning and explicitly 3D motion prediction to learn 3D aware object- centric representation, which we call Object Perception by Predictive LEarning (OPPLE); (2) we provide new datase [ with complex surface texture and motion by both the camera and objects to evaluate object-centric representation models; (3) we found several previous models overly rely on clustering colors to segment objects; (4) although our model leverages image prediction as learning objective, the architecture generalize the ability of object segmentation and spatial localization to single-frame frame images.
2 METHOD
2.1 PROBLEM FORMULATION
We denote scene as a set of distinct objects and background S 10,,, Oz,.... Ok, B), where K is the number of objects in scene. At any moment t, we denote two state variables, the location and pose of each object relative to the perspective of observer (camera), ii:k and Φix- where ai (t) is the 3-d coordinate of the k-th object and (/ is its yaw angle from a canonical pose, as viewed from the reference frame of the camera (for simplicity, we do not consider pitch and roll here and leave it for future work to extend to 3D pose). At time f, given the location of the camera (!)) R and its facing direction (*)), S renders 2D image the camera as I(t) R=xhx3, where is the size the image. Our goal is to train a neural network that infers properties of objects given only a single image (9) as the sole input without external supervision and with only the information of the intrinsies and egomotion of the camera without explicit knowledge of its extrinsic matrix:
(f)
h
(1) {z l:K: 1.K+I L:K P PoLK) } = fang (Ia)(((1))
(1)
Here, =1:K is set of view-invariant vectors representing the identity of each object k. "View- invariant" is loosely defined as (t) -
((-bi)) | - =100 for k o I and At > 0 in most cases, i.e., the vector codes are more similar for the same object across views than they are different (t) across objects. क -- E [(K+I) are the probabilities that each pixel belongs to any of the objects or the background (Ex Thij = 1 for any pixel at i, which achieve object segmentation. To localize objects, # are the estimated locations of each object relative to the observer and p (!)
are estimated probability distribution of the poses of each object. Each (+) E _g is a probability distribution over b equally-spaced bins of yaw angles in (0, 2m).
2.2 OVERALL PRINCIPLE: LEARNING OBJECT REPRESENTATION BY PREDICTING THE FUTURE
Our hypothesis is that the notion of object emerges to meet the need of efficiently predicting the future fates of all parts of an object. With the (to be learned) ability to infer an object's 's pose and location from each frame, the object's speed of translation and rotation can be estimated from con- secutive frames. If depth is further inferred for each pixel belonging to an object, then the optical
We will release upon publication of the paper
2

---

Under review as a conference paper at ICLR 2022
flow of each pixel can be predicted based on the object's speed and the position of each pixel relative to the object's ''' center. The sigmeettgtntntiionn of an object essentially prescribes which pixels should move together with the object. With the predicted optical flow, can further predict part of the next image by warping the current image. The parts of the next image unpredictable by warping include surfaces of objects the background that currently occluded but will become visible, and the region of a scene newly entering the view due to self- or object-motion. These portions can only be predicted based on the learned statistics of the appearance of objects and background, which we call "imagination".. In this work, we will show that with the information of self-motion, knowl- edge of geometry (rule of rigid-body movement) and the assumption of smooth object movement, the object representations captured by function Sobj and depth perception can be learned without supervision.
2.3 NETWORK ARCHITECTURE
To demonstrate the hypothesized principle above, build our OPPLE networks illustrated in Figure[] which process two consecutive images individually and make prediction for the next image with the information extracted from them.
networks
Object
Depth perceptin
Objechbased
prediction
Warping
Object extraction
Final prediction
Segmented
objects:
lserrity matching motig estiation
Combinstr weight
self-motion
obectbased imagination
Imagination
t+1
Ground truth
Learning from prediction error
Figure 1: Architecture for the Object Perception by Predictive LEarning (OPPLE) network Images at each time point are processed by the Object Extraction and Depth Perception Networks independently. All networks use U-Net structure. The dotted boxes indicate the entire OPPLE network acting at each time step (details omitted for - 1). Motion information of each object estimated from the spatial information extracted for each object between (-1 and (. Objects between frames are soft-matched by score depending on the distance between their latent codes. Self- and object motion information are used together with object segmentation and depth map predict the next image by warping the current image. The segmented object images and depth, together with their motion information and the observer's motion, used by the imagination network to imagine the next scene and fill the gap not predictable by warping. The error between the final combined prediction and the ground truth of the next image provides teaching signals for all three networks.
Object extraction network. We build an object extraction network fe,, by modificaiton of a U-Net |Rnnebergger al. 2015) extract representation for each object, the pixels occupies and its spatial location and pose. A basic U-Net is composed of convolutional encoder and a transposed convolutional decoder, while each encoder layer sends skip connection the corresponding de- coder layer, so that the decoder can combine both global and local information. Inside our fam.
an image I (1++1) first passes through the encoder. Additional Atrous spatial pyramid pooling layer (Chen et al. 2017) is inserted between the middle two convolutional layers of the encoder to expand the receptive field. The top layer of the encoder outputs a feature vector e' capturing the global infor- mation of the scene. A Long-Short Term Memory (LSTTM) network further repeatedly reads in e())
3

---

Under review as a conference paper at ICLR 2022
and sequentially outputs one code for an object a time. Each object code is then mapped through a one-layer fully connected network to predict object code = object location ( and object pose probability Po Po(t) k 2, 2....... K. The inferred location is restricted to the visible range of the camera with an upper limit of distance. The pose prediction is represented as log p for numerical
stability. Each object code vector 2 is then independently fed through the decoder with shared skip connection from the encoder. The decoder outputs one channel for each pixel, representing an un-normalized log likelihood that the pixel belongs to the object k. The unnormalized logit maps for all objects are concatenated with map of all zero for the background, and compete through
softmax function to output the probabilistic segmentation map n
Depth perception network. We use a standard U-Net for depth perception function heaph that processes images ((t) and output single-channel depth map (t),
Object-based imagination network. We build the imagination network 9ime also with a modified U-Net. The input is concatenated image [(t) and log of depth log (D()) inferred by the depth perception network, both multiplied element-wise by one probabilistic mask T (t) The output of the encoder network is concatenated with vector composed of the observer's ' moving velocity (t) and rotational speed whee and the estimated object location & velocity (()) and rotational speed ( before entering the decoder. The decoder outputs five channels for each pixel: three for predicting RGB colors, one for depth and one for the probability of the pixel belonging to any object it or background and is used to weight the predicted color and depth for the final "imagination".
2.4 LEARNING OBJECT REPRESENTATION BY PREDICTION
Below, explain details of the approach. The pseudocode of the algorithm is provided in the appendix.
2.4.1
PREDICTION BY WARPING
We first describe the prediction of part of the next image by warping the current image. Here we consider only rigid objects and the fates of all visible pixels belonging to object. With depth D E R he (1) of all pixels in a view inferred by the Depth Perception network based (e)
on visual features in the image [(t) the 3D location of a pixel m (i.j) at any coordinate (i. in the image, can be determined given the focal length of the camera. We use the inferred current and previous locations, 2
(!-1) to estimate the instantaneous velocity of the 1tth object (( = - & (t-1) for all pixels (i.j) attributed to that th object. Similarly, with the inferred the (t) current and previous pose probabilities of the object, p and p/r ((-1), we can obtain the likelihood
(1) and a
of its angular veloeity p((lt) ok (t-1) w = w). We obtain the posterior distribution of the object's next pose (1+1) p(o(+t)) ol og () with Von Mises prior. More details about warping and pose estimation are added in the Appendix.
Assuming pixel (i,j) belongs to object k. we use the estimated motion information a( and p(u (6) oo o (o((("1)) of the object, together with the current location and pose of the object and the current 3D location m. (1) of the pixel, to predict the 3D location mz (t+11) of the pixel at the next
(i.5)
k,(i.j)
(t+1) moment as m k.(i.)) = M -aa. [M (9) Se (i.)) ) a!a + 66 v (t)) where M wine and ((()) in are rotational matrices due to the rotation of the observer and the object, respectively, and db is the velocity of the observer (relative to its own reference frame at l). In this way, assuming objects move smoothly most of the time, if the self motion information is known, the 3D location of each visible pixel can be predicted. a pixel belongs to the background, WK+1 = and UK+1 = (K + 1 is the background's index). Given the predicted 3D location, the target coordinate (i',""((e (t+1) of the pixel on the image and its new depth D ((+)))++) can be calculated. This prediction pixel movement allows predicting the image I((+++) and depth 1'(++1 by weighting the colors and depth of pixels predicted to land near each pixel at the discrete grid of the next frame.
(m))) (t)
(2)
4

---

Under review as a conference paper at ICLR 2022
Considering that object attribution of each pixel has be inferred, we can write object attribution as a probability of belonging to each object, across all pixels, nx ,..k . 1,2, K The predicted motion of each pixel should be described as probability distribution over K + 1 discrete target locations p( p(x(()++))
E =f 5(
(+++1) ((a(t++))) i.e., pixel (i.j) has a probability of m to move to k.(i.)))
(K.)
((+1)) location x' k.(i,))* Using probabilistie prediction of pixel movements, we partially predict the color of next image at the pixel grids where some original pixels from the current will land by weighting 1'
their contribution I' Warp (p, q). This contribution term used to calculate a weight for effect of pixel (i.j) on position (p.q) on the grid, and written wg(i. J: 9). More details in the appendix.
2.4.2 IMAGINATION
For the regions not fully predictable by warping current image with appendix equation 77. i.e., for (p.4) where En,ss (1((,. < 1, we learn function g that "imagines" the appearance p(t+1)
kimag E Rwxax3 and the pixel-wise depth D( gg of the object or background k in the next frame, and the predicted probabilities that each pixel in the next frame belongs to each object or the background ((++1) E Ru The function takes as input portion of the current kimma
F and the inferred depth D O " both extracted image corresponding to each object (t) O
by element- -wise multiplying with the probabilistic segmentation mask π (t) the information of the camera's ' self motion, and the location and motion of that object:
[I' (t+1) (r(t+)) ((ttt)) klmug
MImag' klmug: π
= g (1( C
D
obs
(2)
The "imagination" specific for each object and the background can then be merged using the weights
prescribed by πt" 1:K1mag"
((++1)
I ((t+1)
Σn I (t+11
> ((t-1)
and Dr ((++1)
D
(t+1) π (t-1)
Imag
klmag
klimag'
Imag
klmag
klmag
2.4.3 COMBINING WARPING AND IMAGINATION
The final predicted image or depth map are weighted average of the prediction made by warping the current image predicted depth map and the corresponding predictions by imagination:
I ((++) I' (t+1)
Warp +
(++1)
(1
W Wwarp)
Here, Wwarp E R with each element Wwarp (P. g) = max Σ k. w(i.j. q), 1}. The intuition is that imagination is only needed when there is not sufficient contribution for predicting pixel by warping. The same weighting applies for generating the final predicted depth D (e++1)
In addition to the image, the states of the objects can also be predicted. The location of object k at
t ++ can be predicted as x (t+1)) (1
(1)
(B)
-lhh) Its new pose probability can be p( = w)p(o! = =1) for 72 equal to
k
predicted by p (t++1) (Óx + Wabs 72)
Σ
21 11.4472
each fixed yaw angle bin. To obtain p (t+1) (ox) instead of p (++1) (Øx + Wabs) at the same set of bins,
the vector p (1+1) (dhx + wwms) can be shifted by multiplication with a pre-computed matrix composed of bank of shifted Von Mises distributions to "move" the probability mass on the angle bins.
There is one important issue of object-centric representation when making prediction for future images: order to predict the spatial state of each object at + based on the views at t and 1- the network needs to match the representation of an object at ł from the representation of the same object at 1. As the dimensions of features (e.g.. shape, surface texture, size, etc.) grows, the number of possible objects grows exponentially. Therefore, representations based on the order by which an LSTM extracts objects, as this requires learning a consistent order over enormous amount of objects. Instead, take soft-matching approach:
cannot simply match object
take a subset of the features in ((()) extracted by f as an identity code for each object. For object l time we calculate the distance between its identity code and those of all objects at t 1, and pass the distances through radial basis function to serve as a matching score Tw indicating how closely the object l matches each of the previous objects. The scores are used weight all the
5

---

Under review as a conference paper at ICLR 2022
estimated translational and rotational speeds for object i each assuming different object / were the true object k at 1. We additionally introduce a fixed identity code 2K+1 = for the background and set the predicted motion of background to zero.
2.4.4 LEARNING OBJECTIVE
Above, we have explained how the next image input r((++)) the depth map (x and the spatial
states of each object, æ (t+1) and p (f+1) can be predicted based on object-centric representation
extracted by function f from the current and previous images I and r the depth D extracted by a function h. combined with the prediction from object-based imagination function that are all be learned. Among the three prediction targets, only the ground truth of visual input
I (t++1 is available, while the other can only be inferred by f and h from I [(++1), Therefore, for the prediction targets other than [(++), use the self-consistent loss between the predicted value based on / and ( - and the inferred value based on L + as additional regularization terms to learn the functions f and g.
To learn the functions f. g and h, we approximate them with deep neural networks with parameters 0 and optimize 0 to minimize the following loss function:
L Limage + Adopth + Aspatial + Amap-mmpp
Here, Trimage MSE(log(D logg is the error between the predicted and inferred depth. These pro-
MSE(T((+1), r((+1) is the image prediction error.
Istepth
vide major teaching signals. Lxpatial = =Σx1 |K*+ riare E Ex=1 |= Eig mijj Tkezl Σ (p (1+)))
Ex min [ ([+()))
4k =( 1,0) self-consistent loss on spatial information prediction. The first term is the error between inferred and predicted location of each object, while the calculation of the predicted location incorporates soft matching between objects in consecutive frames. The second term is the negative term of contrastive loss, which we found empirically prevents the network from reaching a local minimum where all objects are inferred at the same location relative the camera (and covering minimal regions of the picture). Trand is the inferred object location from a random sample within the same batch. The third term penalizes the discrepancy between the inferred object location and the average location of pixels in its segmentation mask. The last term is the KL-divergence between the predicted and inferred pose for each object at ++ Lmpp ReLu(10- T TTy Ti for le & I avoids loss of gradient due to zero probability of object belonging and discourages overlap between maps different objects.
(++1)
02 Ei=1 TRIP" ((tt))))))))) is the
K+
2.5 DATASET
We procedurally generated dataset composed of 306445 triplets of images captured by virtual camera with field of view of 90 degrees in a square room. The camera translates horizontally and pans with random small steps between consecutive frames to facilitate the learning of depth percep- tion. 3 objects with random shape, size, surface color or textures are spawned at random locations in the room and each move with randomly selected constant velocity and panning speed. The trans- lation and panning of the the camera is known to the networks. No other ground truth information provided. The first two frames serve as data and the last frame serve as the prediction target at t + 1. An important difference between this dataset and other commonly used synthetic datasets for OCRL is that more complex and diverse textures are used on both the objects and the background. We further evaluated on richer version of the Traffic dataset (Henderson & Lampert 2020) in A.6
2.6 COMPARISON WITH OTHER WORKS
To compare our work with the states-of-the art models of unsupervised object-centric representa- tion learning, we trained MONet (Burgess et al. 2019), slot-attention (Locatello et al. 2020) and GENESIS (Engelcke et al. 2021) on the same dataset. Although these models are trained on single images, all images of each triplets are used for training.
We failed to obtain reasonable result by training GENESIS V2 our dataset, thus adopted GENESTS network pre-trained GQN dataset and retrained on our dataset with K=7.

---

Under review as a conference paper at ICLR 2022
To address the concern that the original configura- tions of the models are not optimized for more diffi- cult dataset, trained variants of some of the models with large network size. For MONet, we tested chan- nel numbers of [32, 64, 128, 128] (MONet-128) and for the hidden layers of encoder of the component VAE in- stead of [32, 32, 64, 64] and adjusted decoder layers sizes accordingly, and increased the base channel from 64 to 128 for the attention network. For slot attention, we tested a variant which increased the number of fea- tures in the attention component from 64 to 128 (slot-
Model
ARI-fg
loU
MONet
0.31
0.08
MONet-128
0.33
0.22
MONet--28-bigge
0.33
0.15
slot-attention
0.41
0.31
[32, 64, 128, 256, 256] EMONet-128-bigee
slot-attention- 128
0.39
0.54
GENESIS
0.17
0.03
our model (OPPLE)
0.46
0.35
Table 1: Performance of different models on object segmentation.
attention- 128). Slot numbers were chosen as 4 except for GENESIS.
RESULTS
After training the networks, evaluate them 4000 test images unused during training but gen- erated randomly with the same procedure, thus coming from the same distribution. We compare the performance of different models mainly their segmentation performance. Additionally, demonstrate the ability unique to our model: inferring locations of objects in 3D space and the depth of the scene. The performance of depth perception is illustrated in the appendix.
3.1 OBJECT SEGMENTATION
Following prior works (Greff et al. 2019 Engelcke et al. 2019 2021), we evaluated segmentation with the Adjusted Rand Index of foreground objects (ARI). addition, for each image, we matched ground-true objects and background with each of the segmented class by ranking their Intersection over Union (IoU) and quantified the average IoU over all foreground objects] The performance is summarized in table 2.6)
Our model outperforms all com- pared models ARI and is
second to slot-attention-12 in IoU. As shown in Figure - MONet and GENESIS appear to heavily rely color to group pixels into the same masks. Even though some of these mod- els almost fully designate pix- els of an object to a mask, the masks lacks specificity in that they often include pixels with similar colors from other objects or background. Patterns on the backgrounds often treated objects as well. The reason of such drawbacks awaits fur- ther investigation but we postu-
Original images
Ground truth segmentation
MONet-128
Slot-attention-
128
GENESIS
model (OPPLE)
Figure 2: Example of object segmentation by different models
late there may be fundamental limitation in the approach that learns purely from static discrete im- ages. Patches in the background with coherent color offer room to compress information similarly as objects with coherent colors do, and their shapes re-occur across images just as other objects. Our model is able to learn object-specific masks because these masks are used to predict optical flow specific to each object. A wrong segmentation would generate large prediction error even the motion of an object is estimated correctly. Such prediction error forces the masks to be con- centrated on object surface. They emerge first at object boundaries where the prediction error the largest and gradually grow inwards during training. Figure 3 further compares the distribution Due the artifact the border of slot-attention model, IoU calculated excluding those border pixels

---

Under review as a conference paper at ICLR 2022
model (OPPLE)
B3300
MONey
C2400
sitt-atteiioo-12e
Pepso
GENESIS
3002
2300
209)
2000
1893
2000
0.0
0.6
00
10.
0.4
10.6
0.5
1.0
0.0
0,2
0.6
0.0
0-
1.0
rnterseds Graan 1868
H
distance
9 coo.ooopppee.... by dhjem
30000
distination
Interred object
2000
33-
d/n Ri1
210 811 rén
-10 -33 tro sdject poler anglo
5
5.3 oiject distance
nearest neight
now neighoout matrher matrher
Figure 3: A-D: distribution of IoU. All models have IoU < 0.01 for about 1/4 of objects. Only OPPLE shows a bi-modal distribution while other models' IoU are more skewed towards 0. E-F: object localization accuracy of OPPLE for object's polar angle and distance relative to the camera. Each dot is a valid object with color representing its segmentation IoU. Angle estimation is highly accurate for well segmented objects (red dots). Distance is under-estimated for farther objects. G: objects with failed segmentation (blue dots) are mostly far away and occupying few pixels. H The numbers of objects sharing the same shape texture with their nearest neighbour objects in latent space are significantly above chance.
of IoU across models. Most models have IoU < 0.01 for about 1/4 of objects. Only OPPLE and slot-attention-128 show bi-modal distributions while other models' IoU are more skewed towards O. Figure G plots each object's distance and size on the picture with colors corresponding to their ToUs in our model. Objects with poor segmentation (blue dots) mostly far away from the camera and occupy few pixels. This is reasonably because motion of farther objects causes less shift on the images and thus provide weaker teaching signal for the network to assign their pixels separate from the background. For other models, blue dots are more spread even for near objects (not shown).
3.2 OBJECT LOCALIZATION
The Object Extraction Network infers object location relative to the camera. We convert the inferred locations to angles and distance in polar coordinate relative to the camera. Figure 3-F plot the true and inferred angles and distance, color coded by objects' IoUs. For objects well segmented (red dots), their angles estimated high accurately (concentrated on the diagonal in E). Distance estimation is negatively biased for farther objects, potentially because the regularization term on the distance between the predicted and inferred object location at frame + 1 favors shorter distance when estimation is noisy. Note that the ability to explicitly infer object's 's location is not available in other models compared.
3.3 MEANINGFUL LATENT CODE
Because a subset of the latent code (10 dimensions) was used to calculate object matching scores between frames order to soft-match objects, this should force the object embedding 2 to be similar for the same objects. We explored the geometry of the latent code by examining whether the nearest neighbours of each of the object in the test data with IoU > 0.5 are more likely to have the same property as themselves. 772 out of 3244 objects' nearest neighbour had the same shape (out of 11 shapes) and 660 objects' nearest neighbour had the same color or texture (out of 15). These numbers are 28 to 29 times the standard deviation away from the means of the distribution expected if the nearest neighbour were random (Figure H) This suggests the latent code reflects meaningful features of objects. However, texture and shape are not the only factors determining latent code, as we found the variance of code of all objects with the same shape and texture to still be big.
4 RELATED WORK
Our work is on the same tracks as two recent trends in machine learning: object-centric represen- tation (Locatello al. 2020) and self-supervised learning (Chen et al. 2020). We take the same
8

---

Under review as a conference paper at ICLR 2022
logic as self- self-supervised learning that learning to predict part of the data based on another part forces a neural network to learn important structures in the data. However, most of the existing works in self-supervised learning do not focus object-based representation, but instead encode the entire scene as one vector. Other works on object-centric representations overcome this by assigning one representation to each object, as do. Although works such as MONet (Burgess et al.] 2019), I0- DINE (Greff' et al. 2211)), slot-attention (Locatello et al.) 2020), GENESIS (Engelcke et al. 2019) and PSGNet (Bear et al. 2020) can also segment objects and some of them can "imagine" com- plete objects based on codes extracted from occluded objects or draw objects in the correct order consistent with occlusion, few works explicitly infer an object's location in 3D space together with segmentation purely by self-supervised learning, with the exception of closely related work O3V (Henderson & Lampert 2020). Both our works learn from videos. One major distinction is that O3V interleaves spatial and temporal convolution, thus it still require video as input at test time. In contrast, our three major networks process each image independently. Therefore, once trained, our network can generalize to single images. Another distinction from Henderson & Lampert (2020) and many other works is that our model learns from prediction instead of reconstruction. Contrastive- learning of structured world model (Kipf et al., 2019) also learns object masks and predict their future states by linking each object mask with a node in a Graphic Neural Network (GNN). The order of mapping object slot nodes of GNN is fixed through time, and the actions to objects are coded with specific associated nodes. This arrangement may become infeasible with combinatorial number of different possible objects as the order of assigning different objects to a limited number of nodes may not be consistent across scenes. We solve this by a soft matching of object repre- sentation between different time points, which does not require the RNN in the Object Extraction Network to learn fixed order of extracting different types of objects. On the neuroscience side, our work is highly motivated by recent works on predictive learning (O'Reilly et al.) 202I) which also yields view-invariance representation while self-motion signal is available. O'Reilly et al. (2021) used biologically plausible but less efficient learning and applied their model to an easier dataset with objects without background, and did not learn object localization. We should note that explicit spatial localization and depth perception were not pursued in previous works on self-supervised object-centric learning. and the images in our dataset have significantly richer texture information than those demonstrated in previous works (Burgess et al. 2019] Kipf et al., 2019), making the task more challenging. Although view synthesis is not our central goal, the principle illustrated here can be combined with recent advancement in 3D-aware image synthesis (Wiles et al. 2020).
5 DISCUSSION
We provide a new approach to learn object-centric representation that includes explicit spatial lo- calization of objects, object segmentation from image, automatic matching the same objects across scene based on a learned latent code and depth perception as a by-product. All of the information extracted by networks are learned without supervision and no pre-training volved. The only additional information required is that of observer's s self- motton,,oo, which is available in the brain efference copy. This demonstrate the possibility of learning rich embodied informa- tion of object, one step toward linking neural networks with symbolic representation in general. We expect future works to develop self-supervised learning model for natural categories beyond simple object identity, building on our work.
other tasks is in-
The work demonstrates that the notion of object can emerge as a necessary common latent cause of the pixels belonging to the object for the purpose of efficiently explaining away the pixels' coherent movement across frames. In our experiment, object spatial location is inferred more easily than object pose (which we have not fully investigated), thus the predicted warping relies more on object translation than rotation. As a limitation, almost all existing object-centric representation works, including ours, focus on rigid bodies and simple environment. Future works need to explore how to learn object representation for deformable objects, objects with more complex shapes and lighting conditions, and more cluttered environment, towards more realistic application. There is important implication for learning 3D-aware object- object-based representation, because for such representation to be useful eventually for robotics and RL, agents need to understand an object's ''s spatial relation to itself based on vision. Learning object-centric representation with explicit 3D information is an important step toward embodied representation of the environment.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Daniel M Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter. Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Joshua Tenenbaum, et Learning physical graph repre- sentations from visual scenes. arXiv preprint arXiv:2006. /2373, 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint 1901.1190.111000 2019.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020.
Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representa- tions for generalizable robot learning. In 2018 IEEE International Conference Roboties and Automation (ICRA). pp. 7111-7118. IEEE, 2018.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13052, 2019.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Reconstruction bottlenecks in object- centric generative models. arXiv preprint arXiv:2007.06245, 2020.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object
representations without iterative refinement. arXiv preprint arXiv:2104.0991 099558 2021.
Irwin Feinberg. Efference copy and corollary discharge: implications for thinking and its disorders.
Schizophrenia bulletin, 4(4):636, 1978.
John H Flavell. The developmental psychology of jean piaget. 1963.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 2424 2433. PMLR, 09-15 Jun 2019.
Paul Henderson and Christoph H Lampert. Unsupervised object-centric video generation and de- composition 3d. arXiv preprint wxxiv:2007.06705 2020.
Daniel Kersten, Pascal Mamassian, and Alan Yuille. Object perception as bayesian inference. Annu. Rev. Psychol. 55:271-304, 2004.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 6980, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1312.61 14, 2013.
Thomas Kipf, Elise der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprint arXiv:1911.122. 2019.
Francesco Locatello, Dirk Weissenborn. Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arXiv preprint arXiv:2006. /5055, 2020.
10

---

Under review as a conference paper at ICLR 2022
Li Nanbo, Cian Eastwood, and Robert B Fisher. Learning object-centric representations of multi- object scenes from multiple views. 34th Conference on Neural Information Processing Sys- tems, 2020.
Randall C O'Reilly, Jacob L Russin, Maryam Zolfaghar, and John Rohrlich. Deep predictive learn- ing in neocortex and pulvinar. Journal of Cognilive Neuroscience, (3(6):1158-111 2021.
Jean Piaget and Margaret Trans Cook. The origins of intelligence in children. 1952.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference machine learning, pp. 1278-1286. PMIR, 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Dequan Wang, Coline Devin, Qi-Zhi Cai, Fisher Yu, and Trevor Darrell. Deep object-centric policies for autonomous driving. 2019 International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2019.
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7467-7477, 2020.
11

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 PSEUDO CODE OF THE OPPLE FRAMEWORK
Algorithm Developing object-centric representation by predicting future scene
InitInitialize: Network parameters 0
Input: images ((t-1), ((t) IRxxxx3, self-motion (t-1) (t-1)
abs oba wab
Output: prediction I((++),, segmentation (t-1) (t)
"I:4+1 iiK+1 objects' codes (t--1) (t)
=1:K objects'
locations and poses (t-11) (t-1) (1)
br.K
11K PoLK
for 7 = (t- do scene code e(r) t- U-NetEncoder,, fo (I(7)
object code ii.k' location a l:k' pose p di.
LSTMM)(e(7))
background code 2K+1 0
depth ( ho (T))
segmentation mask π r1K+1 + Softmax (U-NetDecoder, fo (I(T), (r) (2.k) 11
(r)
end for object matching scores for 1 to do
RBF(2 (1) k +1
object motion 01:K W1:K + Tk, (t) L Por Po:
(t-1) (1) (t-1) = 1 K +1
onject-specific optical flow; + U1 WI:K Vobs. Wobs D(1)
end for ((++1)
warp + Warp (1), optical flow 1l0w1:K+1)
(0) final image prediction: (1t+1] t ((t+t) (((+1)
((+1)
imagine + go ( @ T1:K41, Log( D(t)) @ π 11:++ bas Wobs: 01:K+1, 11:k)
(1)
warp imagine warping weights
update parameters: 0 + 0 -Vol[ 1(+ + regularization loss]
A.2 PERFORMANCE ON DEPTII PERCEPTION
We demonstrate few example images and the inferred depth. Our network can capture the global 3D structure the scene, although details on object surfaces are still missing. Because background occurs in every training sample, the network appears to bias the depth estimation on objects towards the depth the walls behind, as is also shown in the scatter plot.
Images
Inferred depth
Inferred depth
Ground truth depth
Ground truth depth
Figure 4: Comparison between ground truth depth and inferred depth
12

---

Under review as a conference paper at ICLR 2022
A.3 NETWORK TRAINING AND DATASET
We trained the three networks jointly using ADAM optimization Kingma Ba (2014) with learn- ing rate of 3e - 4, = le 6 and other default setting in PyTorrc,, with batch size of 24. 40 epochs were trained on the dataset. We set Aspatial = 1.0, Adepth 0.11 and Lmpp = 0.005 Images in datasets are rendered in Unity environment and downsampled to 128 x 128 resolution for training. Images are rendered in sequence of 7 steps each time a room with newly selected texture and object. Camera moves with random steps and random rotation between consecutive frames. All possible triplets equally spaced by 1, 2, and 3 frames form training samples.
The model implemented in PyTorch and trained on NVidia RTX 6000. We will release the code and dataset upon publication of the manuscript.
A.4 METHOD
A.4.1 PREDICTION BY WARPING
We first describe the prediction of part of the next image by warping the current image. Here we consider only rigid objects and the fates of all visible pixels belonging to an object. With depth D E Rwxx ho(I(t) (()) of all pixels in view inferred by the Depth Perception network based on visual features in the image [(t)_ the 3D location of pixel any coordinate
(i, in the 2 can be determined given the focal length d of the camera as V 22 (GJ) 2 . b-1 [... Here, we take the coordinate of the center of an image as (0.0). On
image, where < w-1
(t)
m (E.)
the other hand, with the inferred (1) and T% (t-11) the current and previous locations of the object k that the pixel (i belongs to, from (t)) and (t-11) respectively, we can estimate the instantaneous velocity of the object o( = at (t-1) 1) Similarly, with the inferred the current and previous pose probabilities of the object, Pou and Poz we can obtain the likelihood of its angular velocity
(1)
(t-1)
plok 1/
((-1)
w/ w) x
Σ
p( -7) 10(0p
(1-1) 72)
(5)
7:222-1 E (2-27
By additionally imposing a prior distribution (we use Von Mises distribution) over w that favors slow rotation, we can obtain the posterior distribution of the object's angular velocity plw (t) (t) of ((t-l)), and eventually the posterior distribution of the object's next pose p( (t+1)
Assuming a pixel (i.j) belongs object k using the estimated motion information of and p(w
1h (t- (t-1)) of the object, together with the current location and pose of the object and the current (t)
3D location mi())
of the pixel, we can predict the 3D location m m((1+) of the pixel at the next k.lk.))
(i.f)
moment as
= (t) (!) (m m k.()))) -igs Wane
(1+1)
(r) (4.)) 20(0 + (000 + (())
(t)
(6)
where (() -Wah and M. (1)
Wt are rotational matrices due to the rotation of the observer and the object,
respectively, and with is the velocity of the observer (relative to its own reference frame at f). In this way, assuming objects move smoothly most of the time, if the self motion information known, the 3D location of each visible pixel can be predicted. If a pixel belongs to the background, WK+1 0 and UK+1 = (K + the background's index). Given the predicted 3D location, the target coor- dinate (('.j") (1+1) of the pixel on the image and its new depth D (1. (i. (t-1) can be calculated. This prediction of pixel movement allows predicting the image [((++)) 1) and depth D by weighting the colors and depth of pixels predicted to land near each pixel at the discrete grid of the next frame, explained in Sec 2.4.2.
13

---

Under review as a conference paper at ICLR 2022
A.4.2 WARPING CONTRIBUTION WEIGHT
As the object attribution of each pixel is not known but is inferred by fatj (1 represented for every pixel as probability of belonging to each object and the background Th
(8) k = 1, ..... K Therefore, the predicted motion of each pixel should be described as a probability distribution "ki 8(x' 8(x=(t+1)
over K + 1 discrete target locations p("" (((x((+))
Ekel K+I (t)
k.(i.) ). i... pixel (i.j) has a
(8.5)
probability of Kki to move location a: (l+1) the next time point, for k 1, K+1. With
k.(i..)
such probabilistic prediction of pixel movement for all visible pixel (i, can partially predict the colors of the next image at the pixel grids where some original pixels from the current view will land nearby by weighting their contribution:
Ensy (xlldna)) (i.d)
if Ens mx(i...P.
I' (1+1) (p.9)
otherwise
We define the weight of the contribution from any source pixel (i,j) target pixel (p,
wa pi, "ka -- ---DD max{1 - - pl. 0} max 11 - 11'((t))) g 0} (8) The first term incorporates the uncertainty of which object a pixel belongs to. The second term --B-D"0 resolves the issue of occlusion when multiple pixels are predicted to move close to the same pixel grid by down-weighting the pixels predicted to land farther from the camera. These last two terms mean that only the source pixels predicted to land within a square of 2 x pixels centered at any target location (p, q) will contribute to the color 'Warp (p, q)- The depth map (2++) Warp can be predicted by the same weighting scheme after replacing 7 (f,j) with each predicted depth (x(++1)) (i.j) assuming the pixel belongs to object k.
A.5 DEPENDENCY OF SEGMENTATION PERFORMANCE ON OBJECT SIZE AND DISTANCE ACROSS MODELS
Our model (OPPLE)
MONet-128
pixels occupied by objects
pixels occupied by abjects
10000
10000
1000
100
100
10
10
1.
3.0
0.3
100
3.0
object distance
object distance
Slot-attention-128
Genesis
pinels occupied objects
10000
10000
1000
1000
0
pixels occupied
IoU
100
100
0.3
1.0
3.0
0.3
10
3.0
object distance
object distance
A.6 ILLUSTRATION OF PREDICTION QUALITY
In the figure below, we display the first, second image, one of the masked objects, the predicted third image, and the ground truth of the third image, both for our dataset, and for richer version of the Traffic dataset used by (Henderson Lampert 2020). We provide reference lines and some circles to aid the comparison between images and evaluate the warping quality. Imagination quality can be insprected usually at one side of the predicted images (the camera motion is typically larger in our dataset).
14

---

Under review as a conference paper at ICLR 2022
15


Reviews:
[{"review_id": "csQKLOEZWJ1", "reviewer": "Reviewer_8qC3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the paper quite interesting. I think the optical flow based warping to predict some subset of the pixels in the next timestep is in itself an important contribution and this paper would be a good addition to the emerging literature on unsupervised object-centric models.\n\nI think the main concern with the paper is limited experimental evaluation. The model is evaluated only on a single, rather simple dataset (that was generated by the authors). I know these models cannot usually handle complicated datasets so it's fine to have a simple dataset but I'd have liked to see the model evaluated on some of the datasets that other competing models were evaluated on. Some of those datasets might not have camera information etc. but I'm sure there are other datasets that have the necessary information (e.g., see [1], [2] for some potential datasets). \n\nAlso, the authors mention that their technique is the first to infer 3D position of objects while segmenting images. If I'm not mistaken, [1] also does both and would be a great model to compare against.\n\nOther notes\n\n- Std deviations in table 1 are too large. Is this a typo? If not, it looks like all models are doing equally well.\n- I don't have any specific recommendations here but I found the model description a bit hard to follow. It might be a good idea to do another pass and see if it can be organized better. For example, the fact that warping and imagination are combined to get the final image can be mentioned earlier so the reader knows where/how the imagination network is used.\n- There are many inline equations; these make it difficult to parse the text visually. It would be nice to take these out of the text and split the long section 2.3.1 to subsections and mark these.\n- Figure 1 is great but again hard to parse. Perhaps adding variable names (i.e., x, p, z etc.) to the figure might make it easier to understand what goes in and out of each network.\n\n[1] Henderson, Paul, and Christoph H. Lampert. 2020. “Unsupervised Object-Centric Video Generation and Decomposition in 3D.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/2007.06705.\n[2] Kabra et al., 2021. \"SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition\", https://arxiv.org/abs/2106.03849", "summary_of_the_paper": "This paper presents an unsupervised object-centric scene representation technique that can decompose a scene into multiple objects (and segment the scene) and infer their 3D locations and pose. The overall setup is very similar to earlier models like MONet but this model works on sequences of images, more precisely on 3 consecutive images. It uses the first two images to infer the 3D position and pose of objects and combining this with known camera motion tries to predict the last (third) image. The main contribution here is an optical flow based method to warp the image at time t using the predicted object location/pose/depth to predict (some of) the pixels in image at time t+1.\n\nIn more detail, the object extraction network outputs the location and pose of each object. And a separate depth perception network outputs the depth for each pixel in the image. The location and pose of objects are used to estimate the velocity of each object (e.g., by subtracting the position at t-1 from position at t. note this requires matching each object at time t-1 to object in time t, which they do using a soft-matching approach). These along with the depth information are then used to warp the image at t to predict pixels in image at time t+1. This is possible only for a subset of the pixels so for the rest, they use a separate \"imagination\" network that takes in object information and predicts the color/depth and object masks at t+1. The predictions from warping and imagination network are then combined to form the final predicted color and depth images.\n\nTo train the model, they require images and camera motion, and use a combination of losses: reconstruction loss on predicted and ground truth image, self-supervised losses on object location, pose, and depth.\n\n", "summary_of_the_review": "Overall, I think the paper is quite interesting and would be of interest to the community. However, the empirical evaluation is very limited and this makes it difficult to evaluate the full merit of the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WrDeHjS3hiC", "reviewer": "Reviewer_uocA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nI like the general thrust of this paper. While the basic idea has been around in cognitive psychology, which inspired the authors, I am not aware of any significant implementation of it. This is a nice first step. \n\nTo make this work, the authors develop some algorithmic bits that might be useful for followup work. \n\nThe results compare well to others in this space, showing that the training strategy has real promise. In some sense, the methods are not really comparable as the other methods do not have a viable way to make use of the additional training data. (Also see comment about number of objects below). \n\nWeaknesses\n\nThe paper is harder to read than needed. I appreciate there is a lot of stuff going, and I believe I was able to get most of it after a few iterations, so the lack of clarity is not extreme. \n\nIt is not clear how differing numbers of objects are handled. It seems that the number of objects might simply be provided (K=3 in the dataset)? But if the number of objects is known, then the comparison to other work that infers it might not be fair. K is in the pseudo code, but does not seam inferred. Should K be an input. \n\nA few more details on the LSTM would help.\n\nThe paper could use some polishing. Figure 1, which is informative and key, could be tidied up. Also, I am guessing that the \"Objects\" box is the LSTM. The English could be improved in places, and there are a number of grammar errors (e.g., the first two uses of \"pixel\" in 2.3.1 should be \"pixels\", and 'This last two terms mean' on page 5).\n\nThe authors do not say whether they will release their code. \n\nComments\n\nA clear limitation of this work is that the data is synthetic and very simple (although perhaps more complex than other work in this space). The authors acknowledge this in their discussion. While this might be the standard for this sub-area, real data from a robot or car should be relatively easy to get. Easy block-world-like real data might be better for pushing the work than adding more texture and diverse lighting in synthetic data. \n\nUsing just two time points is both a strength and a weakness. With just two, there is likely to be a lot of ambiguity between translation and rotation, especially if you generalize to more than one pose parameter. Looking at the effect of larger training sequences would be interesting. \n\nWhile the part of the system that is deployed at testing is a simple network, there are a number of hand constructed components (e.g., the warping function) that make use of what we know about cameras, it would be more interesting to see those learned.  \n", "summary_of_the_paper": "Inspired by ideas about how humans learn about objects, the authors detail a system to train a neural network to perceive generic objects using image triplets where objects move and the viewer also can move.  The viewer's motion is provided as an input. Object perception by parts of the network operating on the first two time points is rewarded by predicting what is seen at the third time point (the training signal). Having been trained, the object perception part of the system, which is a relatively basic neural network, can segment them from a single image.  This new setup requires different data that what has been used in this space, and the authors contribute a synthetic dataset as well.\n", "summary_of_the_review": "This is a good first step in this direction, and should inspire follow up work. The technical innovation is sensible. The results are good. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K8Z332m28Qn", "reviewer": "Reviewer_iu2Y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper is messily written, and proposes something that only works marginally better than prior methods, on a synthetic toy dataset. The performance in Table 1 illustrates this: the standard deviation of the segmentation IOU (0.34) is about even with the average IOU (0.35)! Looking at the qualitative results in Figure 2, it seems the model often misses objects completely, and produces segmentations that are fractional and have holes. I appreciate that the baselines are doing badly here also, but slot attention had very similar ARI-fg (0.42+-0.35 vs 0.46+-0.39). Statistically maybe these are equivalent. Is it possible at least to show that this method does better on the datasets where those previous papers managed to work (like CLEVR and those deepmind shapes datasets)? In general this area of work seems stuck in a setup where all methods work well on different toy data, but no methods work on real images or videos. \n\nIt seems like the pose estimation is not working at all, judging by Figure 3-E. The discussion mentions this too: \"object spatial location is inferred more easily than object pose (which we have not fully investigated), thus the predicted warping relies more on object translation than rotation\". Maybe the object pose estimation can be removed from the paper entirely, to make things simpler. \n\n\"our framework can easily generalize to 3D pose\" I am sure the formulation can easily be extended to capture this, but I would prefer that the wording here be a bit more careful, to not suggest that the model is expected to work when pitch and roll are unknowns as well. (If you do expect it to work, please try it out and add the results to the paper.)\n\n\"At time t, given the location of the camera o(t) ∈ R3 and its facing direction α(t)\" Does this mean the camera intrinsics and extrinsics are assumed known? It would be great to say this directly. The discussion section supports this interpretation, saying \"additional information required is that of observer’s self-motion, which is available both in the brain as efference copy and easily accessible in vehicles\". I do not really buy the argument about the brain, or easy accessibility in vehicles. Self-driving vehicles usually register themselves to a known map; inferring pose from odometry alone causes drift. \n\nThe LSTM reading objects from the full-frame encoding sounds like a very weak part of the model. Why not, for example, use a standard object detector, like MaskRCNN? I know you want to be self-supervised, but then why not self-supervise a well-known architecture that is proven to work, instead of inventing a new one?\n\n\"The location prediction is restricted to the range of possible value within the virtual environment by logistic function and linear scaling\" I don't know what this means. \n\nThe imagination network is clumsily introduced. The first instance of it is already \"the imagination network\", and the motivation for it is only written in the caption of Figure1. I found a helpful description later on in page5. These things should be re-arranged.\n\nThe description of the unprojection of a 2D coordinate into 3D space looks odd to me. Where is this coming from? Given that i,j are coordinates, what do |i| and |j| represent? Normally we start from x=fX/Z, and then invert this to X=Zx/f, where x is 2D and X is 3D, and f is the focal length. I am also not sure how the first term and second term here are able to multiply, since the first term is a 2-tuple (i,j) and the third term is a 3-tuple (i,d,j). \n\nI also got lost in the angular velocity equation. It seems the sum is over all \\gamma1, all \\gamma2, and all differences of the two within 2\\pi of \\omega. This is too many things to sum over! It seems like you won't end up with a valid probability distribution. I am probably misreading the notation here. In any case, why is this probability distribution useful? The object-based imagination network seems to require a scalar here for the rotational speed. So why not take the expectation of the first pose distribution, expectation of the second, and then take a difference?\n\nIt is interesting that you do not use depth supervision, but for toy settings like this I think it is OK to assume depth is known, and focus on other hard parts, like segmentation and tracking.\n\nI need some help understanding L_{spatial}. The second term is described as a contrastive loss, but it's a difference between known positions and random positions. Why is this a good idea, and why is it contrastive? Normally a contrastive loss compares two estimates and pushes them apart (rather than pulling every estimate to random).\n\nI was surprised that the evaluation talked about a model called \"OPPLE\" (\"Only OPPLE shows a bi-modal distribution.\"). Apparently this is the name of the proposed model, and the place to learn this is the caption of Figure 1! Please do not put critical information exclusively in figure captions. \n\n\nTypos: \n- probability mess -> probability mass \n- generated dataset -> generated a dataset\n- states-of-art -> state-of-the-art\n- network appear to -> network appears to\n- intersectin (in fig3) -> intersection\n- Figure 1 is never referred to in the text", "summary_of_the_paper": "This paper presents a method to learn how to parse 3-frame videos into object-centric representations, which include segmentation masks and 3D positions and yaws for those objects frame-by-frame, as well as an image representation of the background, and an overall depth map. This is accomplished with a depth network, an object network with an LSTM at the bottleneck to iteratively pick out objects and their positions and yaws, and a decoder to provide segmentations, a warping/re-compositing operation that pastes the inferred objects at their estimated positions for the NEXT frame (i.e., with a constant-velocity assumption), and finally an \"imagination\" network which refines the estimated image. The model learns with a combination of 4 losses, which include image reprojection/prediction, depth consistency, a spatial term that includes consistency and randomness (though I have complaints about this), and finally a penalty term that discourages object probabilities from being zero. The paper also introduces a new synthetic dataset where prior methods do badly, and the proposed method does slightly better. The learned depth maps look good, but this is perhaps expected because camera poses are known.\n", "summary_of_the_review": "I think the paper is not quite ready for publication. The method does not work particularly well, a part of the method (object pose estimation) seems to be not working at all, and the evaluation is only on a new toy dataset and does not include evaluation on established datasets. Also the text contains too much notation, and has some parts mixed up (with terms being used before they are introduced), but I think this can be fixed easily.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FLQMvcWmzpu", "reviewer": "Reviewer_Z7dM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strength\n\n1. The paper tackles the difficult problem of learning to segment objects from an image using no supervision during training.\n2. The problem setting and motivation for this task are explained clearly. A detailed description of the method, along with a pseudocode of the learning algorithm is provided in the paper.\n3. The paper introduces a new synthetic dataset of images taken from scenes with multiple objects with varying shapes and textures (11 and 15, respectively).\n4. Figures 3A-D are very helpful explaining the quantitative performance of the method in relation to the baselines. Figures 3E-G are also helpful showing the failure modes of the proposed method.\n \n# Weakness\n1. I am not fully convinced that the comparison to the baselines is entirely fair. If I understand correctly, the rest of the methods were trained on single images without having access to previous and next frames. While I appreciate the method’s usage of consecutive frames as part of the supervision, I think this should be stated clearly when comparing with the baselines -- to avoid any overclaims.\n\n    SynSin [1], for example, also predicts the next frame’s RGB and depth images without using additional supervision. Similar to the proposed method in this paper, SynSin synthesizes future images given their camera poses by warping the current frame using differentiable rendering. Combining an object-centric approach (like slot-attention) with such a method that performs future image prediction would make a fairer comparison, in my opinion.\n\n    [1] Wiles, O., Gkioxari, G., Szeliski, R. and Johnson, J., 2020. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7467-7477).\n\n2. I think there are some key references missing in the paper too. For instance, a paper from last year also learns an object-centric representation in an unsupervised fashion to decompose the objects in a scene while estimating their poses in 3D [2]. How does [2] compare to the method presented in this paper? What are the main differences between them?\n\n    [2] Henderson, P. and Lampert, C.H., 2020. Unsupervised object-centric video generation and decomposition in 3D. Advances in Neural Information Processing Systems, 33.\n\n3. The results are reported using only the dataset introduced in the paper. I suggest including results from datasets like the Objects Room or CLEVR, where the existing methods (MONet, Slot-attention, Genesis, [2]) have already reported results on. \n\n4. There are no ablation studies reported in the paper. Some parts of the loss function seem redundant judging from their current descriptions. I think it would be very helpful either presenting additional results by ablating the loss function / part of the model, or detail in the text why the method needs each of those components.\n\n# Recommended Decision\n\nI think in its current form the paper is not ready to be published. I strongly encourage the authors to clarify the positioning of this paper in relation to the state-of-the-art (see my comment on weaknesses). I vote for [reject, not good enough] for now, but I would be happy to increase my rating once the authors clear up my concerns.\n\n# Additional Feedback\n\n1. It should be \\hat(m) instead of \\hat(t) in the L_spatial loss’ third term.\n\n2. What is x_rand? If they are randomly sampled point positions then what is their underlying distribution?\n\n3. Have you tested the method on scenes with more than 3 objects? Slot attention, for instance, is able to segment up to 9 objects. Since the paper introduces a new dataset, I would’ve hoped to see a more challenging benchmark with a wider variety of objects and number of instances.\n\n4. Also, the dataset assumes that the camera moves with constant pose change. Have you tested the motion prediction (position, orientation and their time-derivatives) in the presence of different camera velocities? If not, are there any limitations of the method that prevents it?\n\n5. I suggest adding the equivalents of Figure 3G for each baseline in the appendix.\n\n6. What is the number of bins for the yaw angle prediction, b? Have you tried using a continuous representation for the rotation?\n\n7. What does the predicted image (I’) for the next time-step look like? I suggest including more qualitative results to the paper for evaluating the warping function.\n\n8. I think it is a good idea to introduce a more diverse benchmarking dataset for learning object-centric representations. However, I think the dataset proposed in the paper should be further expanded by including more daily life objects, instead of just geometric primitives like spheres, prisms and cones. I suggest taking a look at datasets like YCB or ShapeNet to include more realistic objects to your dataset.\n", "summary_of_the_paper": "This paper studies the problem of predicting the segmentations and poses (position + yaw orientation) of multiple objects, given the image of a scene. The paper introduces a method that is trained without supervision for the segmentations, similar to several other recent object-centric models. In contrast to these existing models, the method proposed in this paper additionally estimates the 3D location of each object by predicting a depth map and classifies the yaw angle by representing the pose domain as equally-spaced bins. To do so, during training the method operates on a short clip of the scene recorded by a moving camera, and uses self-supervision by predicting the scene’s image at the next time step. At test time, the model is able to infer a representation of each object in the scene and segment them given a single image of the scene.", "summary_of_the_review": "All in all, I think the paper attempts to tackle a very challenging problem. The method looks sound and the results might be interesting to the community in learning object-centric representations. However, there are some major concerns I have, mainly about the position and novelty of this paper with respect to a paper from last year, and the lack of results from datasets the state-of-the-art methods report on. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the difficult problem of learning to segment objects from an image using no supervision during training. The paper is clearly written and a new synthetic dataset is made available. Unfortunately, the reviewers raised a number of issues with the submission (missing citations and comparison to relevant related work / additional baselines  + ablation studies / missing empirical evaluation of the proposed method on standard dataset beyond the toy dataset proposed by the authors). The paper received 1 reject, 2 marginal rejects and 1 accept but even the positive reviewer agreed that these were limitations. The authors also conceded to these limitations and initiated experiments that are starting to address the reviewers' comments. At this time, the results of these experiments remain incomplete and hence most reviewers agree that the paper should go through another round of reviews before it is publishable. I thus recommend this paper be rejected in the hope that a subsequent revision will make it a much stronger contribution.
Decision: Reject