Learning to perceive objects by prediction
Authors: Tushar Arora
JOHN DAY
Li Erran Li
Ming Bo Cai
Venue: Submitted to ICLR 2023 (2023)
Keywords: self supervised learning
predictive learning
object-centric representation
3D perception
sensory grounding

Abstract: The representation of objects is the building block of higher-level concepts. Infants develop the notion of objects without supervision, for which the prediction error of future sensory input is likely a major teaching signal. We assume that the goal of representing objects distinctly is to allow the prediction of the coherent motion of all parts of an object independently from the background while keeping track of relatively fewer parameters of the object's motion. To realize this, we propose a framework to extract object-centric representations from single 2D images by learning to predict future scenes containing moving objects. The model learns to explicitly infer objects' locations in a 3D environment, generate 2D segmentation masks of objects, and perceive depth. Importantly, the model requires no supervision or pre-training but assumes rigid-body motion and only needs the observer's self-motion at training time. Further, by evaluating on a new synthetic dataset with more complex textures of objects and background, we found our model overcomes the reliance on clustering colors for segmenting objects, which is a limitation for previous models not using motion information. Our work demonstrates a new approach to learning symbolic representation grounded in sensation and action.

Under review as a conference paper at ICLR 2023
LEARNING TO PERCEIVE OBJECTS BY PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
The representation of objects is the building block of higher-level concepts. In- fants develop the notion of objects without supervision, for which the prediction error of future sensory input is likely a major teaching signal. We assume that the goal of representing objects distinctly is to allow the prediction of the coherent motion of all parts of an object independently from the background while keeping track of relatively fewer parameters of the object's 's motion. To realize this, we pro- pose framework to extract object-centric representations from single 2D images by learning to predict future scenes containing moving objects. The model learns to explicitly infer objects' locations in a 3D environment, generate 2D segmenta- tion masks of objects, and perceive depth. Importantly, the model requires no su- pervision or pre-training but assumes rigid-body motion and only needs observer's s self-motion at training time. Further, by evaluating on a new synthetic dataset with more complex textures of objects and background, the reliance on clustering colors for segmenting objects, which is a limitation for previous models not using motion information. Our work demonstrates new approach to learning symbolic representation grounded in sensation and action.
found
model overcomes
INTRODUCTION
Visual scenes are composed of various objects in front of backgrounds. Discovering objects from 2D images and inferring their 3D locations is crucial for planning actions in robotics (Devin al., 2018 Wang et al. 2019) and this can potentially provide better abstraction of the environment for reinforcement learning (RL), e.g. /Veeapanenn al. (2020). The appearance and spatial arrangement of objects, together with the lighting and the viewing angle, determine the 2D images formed on the retina or camera. Therefore, objects are latent causes of 2D images, and discovering objects is process of inferring latent causes (Kersten et al.) 2004]. The predominant approach in computer vision for identifying and localizing objects rely on supervised learning to infer bounding boxes Ren et al. 2015 Redmon et al.) 2016) or pixel-level segmentation of objects (Chen et al., (017). However, the supervised approach requires expensive human labeling. It is also difficult to label every possible category of objects. Therefore, an increasing interest has developed recently in the domain of object-centric representation learning (OCRL) to build unsupervised or self-supervised models to infer objects from images, such MONet (Burgess et al. 2019). IODINE (Greff al., 2019) slot-attention (Locatello et al.] 2020), GENESIS (Engelcke et al.
2019 202T). C-SWM (Kipf
al. 2019), mulMON (Nanbo al. 2020 and S al. 2022).
The majority of the early OCRL works are demonstrated on relatively simple scenes with objects of pure colors and background lacking complex textures. As recently pointed out, the success of several recent models based on a variational auto-encoder (VAE) architecture (Kingma & Welling 2013 Rezende et al. 2014) depends on a capacity bottleneck that needs to be intricately balanced against a reconstruction loss (Engelcke et al., 2020). Potentially due to the lack of sufficient inductive bias existing real-world environments, such methods often fail in scenes with complex textures on objects and background (Greff et al.] (2019). To overcome this limitation, recent works utilized opti- cal flow (either ground truth or estimated) as prediction target (Kipf et al. (202I) because optical flow is often coherent within objects and distinct from the background. Similarly, depth often exhibits sharp changes across object boundaries. When used as an additional prediction target, it further improves segmentation performance of the slot-attention model (Elsayed et al., 2022). Although these new prediction targets allow models to perform better in unsupervised object segmentation in realistic environments with complex textures, these models still pose sharp contrast to the learning

---

Under review as a conference paper at ICLR 2023
ability of the brain: neither depth nor optical flow available as external input to the brain, yet infants can learn to understand the concept of object by months old (Piaget & Cook 1952: Flavell
1963), with other evidence suggesting this may be achieved as early as 3.5-4.5 months (Bail (Bailargeon aageon
1987 The fact that this ability develops before they can name objects (around 12 months old) without supervision confirms the importance of learning object-centric representation for develop- ing higher-level concepts and the gap of current models from the brain. To narrow this gap, this paper starts with considering the constraints faced by the brain and proposes a new architecture and learning objective using signals similar to what the brain has access to.
As the brain lacks direct external supervision for object segmentation, the most likely learning signal is from the eπror of predicting the future. In the brain, copy of the motor command (efference copy) sent from the motor cortex simultaneously to the sensory cortex, which is hypothesized to facilitate the prediction of changes in sensory input due to self-generated motion (Feinberg, 1978). What remains to be predicted are changes in visual input due to the motion of external objects. Therefore, assume that the functional purpose of grouping pixels into objects is to allow the prediction of the motion the constituting pixels in an object (e.g., the location, pose, and speed of an object). Driven by this hypothesis, our contribution in this paper is: (I) we combine predictive learning and explicit 3D motion prediction to learn 3D- aware object-centric representation from RGB image input without any supervision or pre-training, which we call Object Perception by Predictive L.arning (OPPLE); (2) we provide a new d with complex surface texture and motion by both the camera and objects to evaluate object-centric representation models; we confirm that several previous models overly rely on clustering colors to segment objects on this dataset; (3) although our model leverages image prediction as a learning objective, the architecture generalizes the ability of object segmentation and spatial localization to single-fran frame images.
coherent way by tracking very few parameters
METHOD
Here, outline our problem statement then explain details of our model parts, the prediction ap- proach, and the learning objective. Pseudocode for our algorithm and the details of implementation are provided in the appendix A. A.2
2.1 PROBLEM FORMULATION
We denote a scene as a set of distinct objects and a background S = (0,,0 B), where K the number of objects in scene. At any moment I, we denote two state variables, the location and pose of each object from the perspective of an observer (camera), as 21.K (t) and d where æi is the 3-d coordinate of the k-th object and &f is its yaw angle from a canonical pose, as viewed from the reference frame of the camera (for simplicity, we do not consider pitch and roll here and leave it for future work to extend to 3D pose). At time t, given the location of the camera !!!) E R and its facing direction (+), S renders a 2D image on the camera as (+) E R= where w x h is the size of the image. Our goal is to develop a neural network that infers properties of objects given only single image ((t) as the sole input without external supervision and with only the information of the intrinsics and ego-motion of the camera:
(1) {21:K: : IIK+ 1.K K. P Ponix } Sang (T))
=1.K is set of view-invariant vectors representing the identity of each object k. "View- invariant" is loosely defined as |z z ((+A*) |= 2(() for + l and At > 0 in most cases, i.e., the vector codes are more similar for the same object across views than they are for different objects. (1)
Here, 200
(K+K))xxxl are the probabilities that each pixel belongs any of the objects or the background (Ex Thy 1 for any pixel at i, which achieves object segmentation. To localize objects, (t) are the estimated 3D locations of each object relative to the observer and P are the estimated probability distributions of the poses of each object. Each (() E TR& is probability distribution over b equally-spaced bins of yaw angles in (0, 2Tr.
We will release upon publication of the paper
2

---

Under review as a conference paper at ICLR 2023
2.2 PRINCIPLE BEHIND LEARNING OBJECT REPRESENTATION FROM PREDICTION
Our hypothesis is that the notion of object emerges to meet the need of efficiently predicting the future fates of all parts of an object. With the (to be learned) ability infer an object's pose and location from each frame, the object's s speed of translation and rotation can be estimated from con- secutive frames, assuming the camera's 's self-motion known. If depth is further inferred for each pixel belonging to object, then the optical flow of each pixel can be predicted based on the ob- ject's speed and the position of each pixel relative to the object's center. The pixel-segmennation of an object essentially prescribes which pixels should move together with the object. With the pre- dicted optical flow, one can further predict part the next image by warping the current image. The parts of the next image unpredictable by warping include surfaces of objects the background that are currently occluded but will become visible, and the region of a scene newly entering the view due to self- or abject-motion. These portions can only be predicted based the learned statisties of the appearance of objects and background, which we call "imagination". In this work, we will show that with the information of self-motion, knowledge of geometry (rule of rigid-body movement) and the assumption of smooth object movement, the object representations captured by function fatj and depth perception can be learned by minimizing the prediction error of the next scene in environments with motion of both the camera and objects, without supervision.
2.3 NETWORK ARCHITECTURE
To demonstrate the hypothesized principle above, we build our OPPLE networks as illustrated in Figure which process two consecutive images separately and make prediction for the next image with the information extracted from them.
Concateratt
Heddemerd
Adsittonn
t-
Object extration
U-Not.ads
Flow of the mechod: do0ed
procust
connectios
00
Depth perception
Inlaned dephi
t
Osjectoased
predictor by korping
Object extraction
Segiranted cljecta
Final prediction
warsing
self-motion
Mentty matrntng
Combriatio weight
g
Imagination
Imagginiton
t+1
Learning prediction error
Ground
Figure 1: Architecture for the Object Perception by Predictive LEarning (OPPLE) network: Our method uses three ordered frames. (a) At time 1-1, we only extract the object information which gives us the object location and pose. At time frame i, we extract depth using UNet and extract object information using a modified UNet. (b) Motion information of each object is estimated from the spatial information extracted for each object at t - 1 and t. Objects between frames are soft- matched by a score depending on the distance between their latent codes. Self- and object motion information are used together with object segmentation and depth map predict the next image by (c) The segmented object images and depth, together with their motion information and the observer's motion, are used by the imagination network to imagine the next time frame and fill the gap not predictable by warping. (d) The error between the final combined prediction and the ground truth of the time frame provides teaching signals for all three networks.
warping the current image.
Depth perception network. We use a standard U-Net for depth perception function ha that processes images (!) and output single-channel depth map (t),
3

---

Under review as a conference paper at ICLR 2023
Object extraction network. We build an object extraction network fea. by modificaiton of a U-Net [Ronneberger et al., 2015) to extract representation for each object, the pixels it occupies and its spatial location and pose. A basic U-Net is composed of convolutional encoder and a transposed convolutional decoder, while each encoder layer sends a skip connection to the corresponding de- coder layer, so that the decoder can combine both global and local information. Inside our f0ap an
image I (t) first passes through the encoder. Additional Atrous spatial pyramid pooling layer (Chen et al. (017) is inserted between the middle two convolutional layers of the encoder to expand the receptive field. The top layer of the encoder outputs a feature vector e' capturing the global informa- tion of the scene. A Long-Short Term Memory (LSTM) network further repeatedly reads in ((t) and sequentially outputs one vector for object time. Each object vector is then mapped through one-layer fully connected network to predict object code = object location # and object pose probability P P050. k- = 1,2, ... K. The inferred location is restricted within viewing angle of 1.2 times the field of view and a bounded distance from the camera. The pose prediction is represented (t) is then independently fed through the decoder with shared skip connection from the encoder. The decoder outputs one channel for each pixel, representing an un-normalized log probability that the pixel belongs to the object & (a logit map). The unnormalized logit maps for all objects are concatenated with a map of all zero (for the background), and they compete through softmax function to output the probabilistic segmentation map #
as Pato for numerical stability. Each object code 2k
(t) for the probability that each pixel belongs to any of the objects or the background.
Object-based imagination network. We build imagination network Pau also with modified U-Net. For each object and the background, the input concatenated image I and log of depth (D))) inferred by the depth perception network, both multiplied element wise by one proba- (t) The output of the encoder part of the network is concatenated with a vector (t) composed of the observer's moving veloeity vobs and rotational speed was and the estimated ob- (+) ject location # velocity o and rotational speed (t()) before entering the decoder. The decoder outputs five channels for each pixel: three for predicting RGB colors, one for depth and one for the probability of the pixel belonging to any object & background. Each predicted RGB and depth image are weighted by the probability image and summed to form the final "imagination". The imagined RGB and depth image for the next frame is further combined with the prediction of the next frame to form the final prediction, detailed in the next section.
bilistic mask from x
2.4 PREDICTING OBJECTS' SPATIAL STATES
We start with predicting the spatial states, namely 3D location and pose, of each object for & + 1. If the tthh object at t 1 is the same the ttth object at twe can the inferred current and previous locations ( and # (t-1) for that object to estimate its instantaneous velocity (treating
the interval between consecutive frames the time unit) e()) = =a(() st (t-1) Similarly, with
the inferred current and previous pose probabilities of the object, Pos and Pox we can calculate (t)
(e)
(t-1)
the likelihood function of its angular velocity w( as p(of off--)) (e) = w). Combining with a Von Mises prior distribution, we can calculate the posterior distribution of the angular velocity
p(w( k
(t) I 0% off ((1-1)). The location of object k at t+ 1 can be predicted as x (++1) -M()) -uin
(t) +
o uobs). ). where v v is the velocity of the observer relative to its own reference frame and (())
is the rotational matrices due the observer's self rotation. Its new probability for pose being equal
to each possible discrete yaw angle bin 72 can be predicted through p'(ol (++1) (( + Wobs = =72 =
Σ
plutk (0) w) (t) 7)), where Wahs is the angular velocity of the observer. Since
11,0 TE (w 2r 20, 2m.)
(t) as probability distributions on discrete bins of angle while Wahs can take we model o and W
continuous values, we convert pl (0( (t+1) + wobs) to p' (o ((-1)) on the same set of bins as of by interpolation.
However, there is one important issue: in order to predict the spatial state of each object at + based the views at & and t 1, the network needs match the representation of object
4

---

Under review as a conference paper at ICLR 2023
t from the representation of the same object at t 1. As the dimensions of features (e.g., shape, surface texture, size, etc.) grows, the number of possible objects grows exponentially. Therefore, we cannot simply assume that the k4 objects extracted by LSTM from consecutive scenes correspond to the same object, as this requires learning a consistent order over enormous amount of objects. Instead, we take soft-matching approach: we take subset (10 dimensions) of the object code in 2/² extracted by f as an identity code for each object. For object at time t. we calculate the distance between its identity code and those of all objects at t 1, and pass the distances through radial basis function to serve as a matching score Fil indicating how closely the object at matches each of the objects / at t. - 1. The scores are used weight all the translational and angular velocity for object k estimated with the equations above, each assuming a different object were the true object &
to yield the final estimated translational and angular velocity for object &t at We additionally introduce fixed identity code =k+1 0 for the background and set the predicted motion of background to zero.
2.4.1 PREDICTING IMAGE BY WARPING
The next image 1(++1) can be partially predicted by warping the current image ((t) based on the predicted optical flow of each pixel in ((5, To predict the optical flow, we need combine inferred depth of each pixel, inferred probabilities that it belongs to each object and the background, the estimated motion of each object and the knowledge of the camera's self-motion. The model assumes all objects are rigid.
We now consider the fates of all visible pixels belonging to an object. With depth D E R ho ( of all pixels in a view inferred by the Depth Perception network based on visual features in the image ((t), the 3D location relative to the camera of any pixel (a,g) at coordinate (i.j)
(+)
in the image can be calculated given the focal length d of the camera. Assuming a pixel (i.j)
belongs to object k; the estimated motion information 0% (t) and p(w] (1) ox 16 (f--)) of the object,
(2)
together with the current location and pose of the object and the current 3D location (i.)) of the pixel, the 3D location m (t++1)
(t)
k.(k.)) of the pixel at the next moment can be predicted m k,(k.))
(1+1)
M -Mie
M [
(4) We (m (E.) 2 + 26 + + - where M and M! are rotational matrices
(1)
due to the rotation of the observer and the object, respectively, and w(t) ab. is the velocity of the observer (relative to its own reference frame at t). In this way, assuming objects move smoothly most of the time, if the observer's self motion information is known, the 3D location of each visible pixel can be predicted. If a pixel belongs to the background, wK+1
(K + 1 is the background's index). Given the predicted 3D location, the target coordinate (i',")) (1+2) of the pixel
and vK+1
on the image and its new depth (i. (++1)) can be calculated. This prediction of pixel movement allows predicting the image I((++)) and depth D'(++1) by weighting the colors and depth of all pixels predicted to land near each discrete pixel grid of frame i + 1.
We use the weights of bilinear interpolation to decide how much each source pixel at contributes to drawing the four pixel grids surrounding the predicted landing location of that pixel in frame ( + 1, assuming the pixel fully belongs to one object or the background. Because each pixel may belong to any of the K object and the background, it has K + 1 potential landing point at L +- 1 each with probability prescribed by the segmentation output for that pixel. Therefore, the contribution weight is further multiplied by the segmentation probability that attributes the source pixel to the object or the background which would bring the source pixel to the target location at + 1 (more details in appendix). The sum of all contribution weights target pixel receives indicates how much that pixel can be predicted by warping. When the sum is smaller than 1, the final prediction for the pixel will be a weighted average of the prediction by warping and by imagination.
2.4.2 IMAGINATION
For the pixels not fully predictable by warping current image, we mix the prediction based on warp- ing with a prediction leannd from the statistical regularity of scenes which we call "imagination".
To do so, We learn function g that "imagines" the appearance (((+1) E Rwxxxx and the pixel-
klmag
5

---

Under review as a conference paper at ICLR 2023
wise depth D ((lttt) R of each object k or background in the next frame, and weight them with predicted probabilities that each pixel in the next frame belongs each object or the background ((++1)
' Almag E Ruxh The function takes as input portion of the current image corresponding to each (t)
object 70)) @ k
(t) and logarithm of the inferred depth log( D(O) @ T both extracted by element- (t)
wise multiplying with the probabilistic segmentation mask T the information of the camera's 's self motion, and the inferred location and motion of that object:
(I-(t-1) Dr(t+1) (((+)) klmag"
kimag
πI
klmag = ( O To (o)
② Th k
obs
(2)
The "imagination" specific for each object and the background can then be merged using the weights prescribed by N/ ((++1) 1:Kimug"' p Σn & kimug @ T ( and D D'(+tt)
Σ
D kimug
((++1)
1++
klmag
This is similar to the way several other OCRL methods (Burgess al. 2019 Locatello et al. 2020)
reconstruct the current image without using geometric knowledge, except that our "imagination"
predicts the next image by additionally conditioning on the motion information of the observer and objects.
2.4.3 COMBINING WARPING AND IMAGINATION
The final predicted image or depth map are weighted average of the prediction made by warping the current image inferred depth map and the corresponding predictions by Imagination network:
I (1++1
(t++1) Warp
Wwarp + I
(1+1) Imug
C (1 - Warp)
(3)
Here, each element of the weight map for the prediction based on warping W Warp E R is W Waap(P, q) ( w(i, J. g), 1} and w...j. p, 4) is the contribution weight of pixel (i.j) at for drawing pixel (p. 9) at + 1 (more details in appendix). The intuition is that imagination is only needed when not enough pixels from I will land near (p, 4) at l + based on the predicted optical flow. The same weighting applies for generating the final predicted depth (((++))
2.4.4 LEARNING OBJECTIVE
Above, we have explained how to predict the spatial states of each object, x' a/( (t+1) and p P' (t+1) the
next image p((++) and depth map (++++) based on object-centric representation extracted by function f from the current and previous images r!()) and r!(-1), the depth ( extracted by a function h, combined with the prediction from object-based imagination function that are all to be leamned. Among the three prediction targets, only the ground truth of visual input I (1+1) is available, while the other can only be inferred by f and h from ((++)) Therefore, for the prediction targets other than ((++1), we use the self-consistent loss between the predicted value based on and t and the inferred value based on t
additional regularization terms to learn f and g.
To learn the functions f. g and h, we approximate them with deep neural networks with parameters 0 and optimize 0 to minimize the following loss function:
L Limage + Adepth Eeepth + Asputtial Isppaial + Amap Immp
(4)
Here, Limage = MSE (SSEllo(Dr(tttggg
((++2) ( the error between the predicted and inferred depth. These pro-
is the image prediction error.
Ldepth
log(D(++)) Log
vide major teaching signals. Lspenial >x EEK+ Tkix" kt
((++1) - - (t++1))2 -EK, min(
a (²('),, 6) + Σk (a En, ! Tkey + Ex= DrI (1+1) |> Tup'' 0r consistent loss on spatial information prediction. The first term is the error between inferred and predicted location of each object, while the calculation of the predicted location incorporates soft matching between objects consecutive frames (rи is the matching probability between any ob- jects in two frames). The second term is the negative term of contrastive loss, which we found empirically helpful to prevent the network from reaching a local minimum where all objects are inferred at the same location relative to the camera (and covering minimal regions of the picture). aad is the inferred object location from a random sample within the same batch. The third term penalizes the discrepancy between the inferred object location and the average location of pixels in
((+T)) is the self-

---

Under review as a conference paper at ICLR 2023
its segmentation mask. The last term is the KL-divereence between the predicted and inferred pose for each object f. + 1. map ReI.u(10-1 x F: + 2k=1 ReL.u( 0.1) + क n,, for k & avoids loss of gradient due to zero probability of object belonging and discourages overlap between maps of different objects.
2.5 DATASET
We procedurally generated a dataset composed 306K triplets of images captured by a virtual camera with field of view of 90 degrees in square room. The camera translates horizontally and pans with random small steps between consecutive frames to facilitate the learning of depth perception. objects with random shape, size, surface color textures spawned at random locations in the room and each moves with a randomly selected constant velocity and panning speed. The translation and panning of the the camera relative its reference frame and its intrinsic known to the networks. No other ground truth information is provided. The first two frames serve as data and the last frame serves the prediction target at 4 An important difference between this dataset and other commonly used synthetic datasets for OCRL, such as CLEVRset (Johnson et al., 2017) and GQN data (Eslami al. 2018) is that more complex and diverse textures are used on both the objects and the background. We further evaluated on richer version of the Traffic dataset
(Henderson Lampert 2020) A.4
2.6 COMPARISON WITH OTHER WORKS
To compare our work with the state-of the art models of unsupervised object-centric representation learning, trained MONet (Burgess et al.) 2019), slot-attention (Locatello et al. 2020) and GEN- E Engelcke et al. 2021) on the same dataset. Although these models are trained on single images, all images of each triplets are used for training.
To address the concern that the original configura- tions of the models are not optimized for more difficult dataset, we additionally trained variants of some of the models with large network size. For MONet, we tested channel numbers of [32, 64. 128. 128] (MONet-122 128) and [32. 64, 128. 256, 256] (MONet- 128-bigger) for the hidden layers of encoder of the component VAE (while the original paper used [32, 32, 64, 64)) and ad- justed decoder layers' sizes accordingly, and increased the base channel from 64 to 128 for its attention net- work. For slot attention, creased the number features in the attention compo- nent from 64 to 128 (slot-attention-125, Slot numbers were chosen as except for GENESIS.
MODEL
ARI-FG
IoU
MONET
0.31
0.08
MONET- 128
0.33
0.22
MONET- 128-T-118-BR 128-BIGGER
0.36
0.20
SLOT-ATTENTION
0.40
0.34
SLOT-ATTENTION-
0.34
0.38
OUR MODEL. (OPPL.E)
0.46
0.35
tested variant which in-
Table 1: Performance of different models object segmentation.
3 RESULTS
After training the networks, evaluate them 4000 test images unused during training but gen- erated randomly with the same procedure, thus coming from the same distribution. We compare the performance of different models mainly on their segmentation performance. Additionally, we demonstrate the ability unique to our model: inferring locations of objects in 3D space and the depth of the scene. The performance of depth perception illustrated in the appendix.
3.1 OBJECT SEGMENTATION
Following prior works Greff et al. 2019 Engelekee et al. 2019 202T), we evaluated segmentation with the Adjusted Rand Index of foreground objects (ARI). In addition, for each image, we matched
We failed to obtain reasonable result by training GENESIS V2 our dataset, thus adopted a GENESIS network pre-trained on GQN dataset and retrained our dataset with K=7. The result is included in the appendix as are confident that we have exhausted hyper-parameter tuning for this model

---

Under review as a conference paper at ICLR 2023
ground-true objects and background with each of the segmented class by ranking their Intersection over Union (IoU) and quantified the average IoU over all foreground objects. The performance is summarized in table 2.6
Our model outperforms all com- pared models ARI and is second to a slot-attention-I1 128 in IoU, As shown in Figure [ MONet appear to heavily rely on color to group pixels into the same masks. Even though some of these models almost fully designate pixels of an ob- ject to a mask, the masks lack specificity in that they often in- clude pixels with similar col- from other objects back- ground. Patterns on the back- grounds are often treated ob-
Original images
Ground truth segmentation
MONet-128
Slot-attention-
model (OPPLE)
Figure 2: Example of object segmentation by different models jects as well. We postulate there may be fundamental limitation in the approach that learns purely from static discrete images. Patches in the background with coherent color offer room to compress information similarly as objects with coherent colors do, and their shapes re-occur across images just as other objects. Our model able learn object-specific masks because these masks are used to predict optical flow specific to each object. A wrong segmentation would generate large predic- tion error even if the motion of an object is estimated correctly. Such prediction error forces the masks concentrate on object surface. They emerge first at object boundaries where the prediction error is the largest and gradually grow inwards during training. Figure 33A-C further compares distribution of IoU across models. Only OPPLE and slot-attention-122 show bi-modal distributions while MONet skewed towards O. Figure 3 plots each object's distance and size on the picture with colors corresponding their IoUs in model. Objects with poor segmentation (blue dots) are mostly far away from the camera and occupy few pixels. This is because motion of farther ob- jects causes less shift on the images and thus provides weaker teaching signal. For other models, blue dots are more spread even for near objects (See appendix).
3.2 OBJECT LOCALIZATION
The Object Extraction Network infers object location relative to the camera. We convert the inferred locations to angles and distance in polar coordinate relative the camera. Figure33-FF plot the true and inferred angles and distance, color coded by objects' ToUs. For objects well segmented (red dots), their angles are estimated with high accuracy (concentrated on the diagonal in E). Distance estimation is negatively biased for farther objects, potentially because the regularization term on the distance between the predicted and inferred object location at frame + 1 favors shorter distance when estimation is noisy. Note that the ability to explicitly infer object's 's location is not available in other models compared.
3.3 MEANINGFUL LATENT CODE
Because a subset of the latent code (10 dimensions) was used to calculate object matching scores between frames in order to soft-match objects, this should force the object embedding = to be similar for the same objects. We explored the geometry of the latent code by examining whether the nearest neighbours of each of the object in the test data with loU > 0.5 are more likely to have the same property as themselves. 772 out of 3244 objects' nearest neighbour had the same shape (out of 1l shapes) and 660 objects' nearest neighbour had the same color or texture (out of 15). These numbers are 28 to 29 times the standard deviation away from the means of the distribution expected if the nearest neighbour were random (Figure 3H). This suggests the latent code reflects meaningful features of objects. However, texture and shape are not the only factors determining latent code, as found the variance of code of all objects with the same shape and texture to still be big.
8

---

Under review as a conference paper at ICLR 2023
A 2230
Our model (OPPLE)
MONett128
3900
S lot Attentoon-128
nut distibution
chservafion
2230
1200
1000
2000
00
02
100
0.0
0.0
1a0
0.6
0.0
0.0
0.2
044
0.6
0.5
nesrest neigh bour matchhes shape
nearest neighbour smalich besttrr
pok antt
Inforned Dolen distar
IoU
10 33
paler angle
Due object distance
coject
Figure 3: A-C: distribution of IoU. All models have IoU < 0.01 for about 1/4 of objects. Only OPPLE shows a bi-modal distribution while other models' loU are more skewed towards D The numbers of objects sharing the same shape texture with their nearest neighbour objects in latent space are significantly above chance. E-F: object localization accuracy of OPPLE for object's polar angle and distance relative to the camera. Each dot is valid object with color representing its segmentation loU. Angle estimation is highly accurate for well segmented objects (red dots). Distance is under-estimated for farther objects. G: objects with failed segmentation (blue dots) mostly far away and occupying few pixels.
RELATED WORK
Our work is on the same tracks as two recent trends in machine learning: object-centric represen- tation (Locatello et al., (020) and self-supervised learning (Chen et al. 2020). We follow the same logic as self-supervised learning: learning to predict part of the data based on another part forces a neural network to learn important structures in the data. However, most of the existing works in self-supervised learning do not focus on object-based representation, but instead encode the entire scene vector. Other works on object-centrie representations overcome this by assigning one representation to each object, as we do. Although works such as MONet (Burgess et al. 2019) IODINE Greff et al. 2019), slot-attention (Locatello et al. 2020). GENESIS (Engelcke et al. 2019) and PSGNet (Bear et al. 2020) can also segment objects and some of them can "imagine" complete objects based on codes extracted from occluded objects draw objects in the correct der consistent with occlusion, few works explicitly infer an object's location in 3D space together with segmentation purely by self-supervised learning, with the exception of closely related work O3V (Henderson & Lampert 2020). Both our works learn from videos. One major distinction that 03V interleaves spatial and temporal convolution, thus it still require video as input at test time. In contrast, our three major networks process each image independently. Therefore, once trained, our network can generalize to single images. Another distinction from Henderson & Lam- per))20200 and many other works is that our model learns from prediction instead of reconstruction. Contrastive-leariigg of a structured world model (Kipf et al. 2019) also learns object masks and predicts their future states by linking each object mask with a node in Graphic Neural Network (GNN). The order of mapping object slot to nodes of GNN is fixed through time. This arrangement may become infeasible with combinatorial number of possible objects as the order of assigning dif- ferent objects to a limited number of nodes may not be consistent across scenes. We solve this by soft matching of object representation between different time points. On the neuroscience side, our work is highly motivated by recent works on predictive learning (O'Reilly et al., (021)) which also yields riew-invariance representation while self-motion signal is available. O Reilly et al. (202T) used biologically plausible but less efficient learning and applied their model to an casier dataset with objects without background, and did not learn object localization. We should note that explicit spatial localization and depth perception were not pursued in previous works on self-supervised object-centric learning, and the images in our dataset have significantly richer texture information than those demonstrated in previous works (Burgess al. 2019 Kipf et al. 2019), making the task more challenging. SAVi++(Elsayed et al., 2022) is closely related to our work but as it requires ex- ternal information of optical flow, cannot make fair comparison here. Although view synthesis is not our central goal, the principle illustrated here can be combined with recent ad advancement in awae image synthesis (Wiles al. 2020

---

Under review as a conference paper at ICLR 2023
REFERENCES
Renee Baillargeon. Object permanence in 31/2-and 41/2-month-old infants. Developmental psy- chology, 23(5):655, 1987.
Daniel M Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Joshua B Tenenbaum, et al. Learning physical graph repre- sentations from visual scenes. arXiv preprint vrXiv:2006.12373 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint 1 /91p99911.119,,,, 2019.
Liang-Chien Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEFE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020.
Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representa- tions for generalizable robot learning. In 2018 IEEE International Conference Roboties and Automation (ICRA), pp. 71 1-71188 IEEE, 2018.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun, Carla: open urban driving simulator. In Conference robot learning, pp. 1-16. PMLR, 2017.
Gamaleldin F Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. arXiv preprint arXiv:2206.07764 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13052, 2019.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Reconstruction bottlenecks in object- centric generative models. arXiv preprint arXiv:2007.06245, 2020.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. arXiv preprint 1xXiv:210400995: 2021.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, al. Neural scene representation and rendering. Science, 36046394):1204-1120,, 2018.
Irwin Feinberg. Efference copy and corollary discharge: implications for thinking and its disorders. Schizophrenia bulletin, 4(4):636, 1978.
John H Flavell. The developmental psychology of jean piaget. 1963.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 2424-2433. PMLR, 09-15 Jun 2019.
Paul Henderson and Christoph H Lampert. Unsupervised object-centric video generation and de- composition in 3d. Advances in Neural Information Processing Systems, 33:3106-311 2020.
10

---

Under review as a conference paper at ICLR 2023
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2910, 2017.
Daniel Kersten, Pascal Mamassian, and Alan Yuille. Object perception as bayesian inference. Annu. Rev. Psychol. 55:271-304 2004.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint IrXiv::412.6900 2014.
Diederik P Kingma and Max Welling. 13 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprint arXiv: 1tt1:vI911.22244 2019.
Thomas Kipf, Gamaleldin Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12544 12594, 2021.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arXiv preprint arXiv:2006. / 15055, 2020.
Li Nanbo, Cian Eastwood, and Robert B Fisher. Learning object-centric representations of multi- object scenes from multiple views. In 34th Conference on Neural Information Processing Sys- tems, 2020.
Randall C O' O'Reilly, Jacob L Russin, Maryam Zolfaghar, and John Rohrlich. Deep predictive learn- ing in neocortex and pulvinar. Journal of Cognitive Neuroscience, 33(6): 1196 2021.
Jean Piaget and Margaret Trans Cook. The origins of intelligence in children. 1952.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28: 91-99, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- asxisted intervention, pp. 234-241. Springer, 2015.
Rishi Veerapaneni, John Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model- reinforcement learning. Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Dequan Wang, Coline Devin, Qi-Zhi Cai, Fisher Yu, and Trevor Darrell. Deep object-centric policies for autonomous driving. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8853-8859, IEEE, 2019.
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7467-7477, 2020.
11

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 PSEUDO CODE OF THE OPPLE FRAMEWORK
Algorithm Developing object-centric representation by predicting future scene
Input: images (tt-1), (+) mx self-motion (t-1) (t-1)
(t)
obs v obs Wabs
Output: prediction (((+)), segmentation #1:K+1 #1:++1+ objects' codes 21.8 =1:K objects'
(t-1) (t)
(t-1)
(t)
(t-1) (t-1) (1)
locations and poses T1:K Pon.
PaLK
for
-= -arr=1
( do scene code e(T) +- U-NetEncoder,, (1(T)
object code (T)
11:K location a Ll.K. pose p di. + (e()))
background code 2K+1 =
depth ( + ha (7))
segmentation mask π (r)
"1:++ f Softmax (U-NetDecoder, fo (((7),2(kk))
end for
object matching scores TMe + RBF(Z (t) K,, +1 for - 1 to do
object motion 01:k W1:K +- rk,,, 26 21(11111 Pox Po: 1: K+1
(t-1) (1) (t-1)
onject-specific optical flow; +- 01:K, Wi:K, Vobs, Whohs D(t)
end for ((++1)
warp + Warp( (1), .optical flow 1low1:K+1)
((+1)) imagine + go (1) @ *I:++ log( (())) ④ π #I:K+1 Dobs, Wabs, 01:K+1 k1:K)
(1)
(1)
final image prediction: (1++1) + -r(tt+1) inegie
imagine: warping weights
warp
update parameters: 0 + [Vo[[ 12 + regularization loss)
A.2 NETWORK TRAINING AND DATASET
We trained the three networks jointly using ADAM optimization Kingma Ba (2014) with a learn- ing rate of 3e - 4, = 1e and other default setting in PyTorch, with batch size of 24. 40 epochs were trained on the dataset. We set >epatial = 1.0, Adepth = 0.1 and L = 0.005.
Images in the dataset rendered in custom Unity environment resolution of 512 x 512 and downsampled to 128 x 128 resolution for training. Images are rendered in sequence of 7 time steps for each scene. In each scene, room with newly selected textures and objects is created, then the objects and camera are placed randomly in the room and checked for possible colliding trajectories, configuration of locations and movement direction of objects and camera found where they don't It collide, we start recording the scene. We do this pre-check to reduce the redundancy of getting bad datasets our recordings. Camera moves with random steps and sways with random rotations between consecutive frames. All possible sequential triplets out of the recorded 7 frames (including frames equally spaced by 0, 1, and 2 frames) form our training samples. For our dataset we use 200,000 scenes which give us 2, 2,400,000 datapoints.
The model was implemented PyTorch and trained on NVidia RTX 6000. We will release the code and dataset upon publication of the manuscript.
A.3 PERFORMANCE ON DEPTH PERCEPTION
We demonstrate a few example images and the inferred depth. Our network can capture the global 3D structure of the scene, although details on object surfaces are still missing. Because background occurs in every training sample, the network appears to bias the depth estimation on objects towards the depth of the walls behind, is also shown in the scatter plot.
12

---

Under review as a conference paper at ICLR 2023
Images
depth
Inferred depth
Ground truth depth
Ground truth depth
Figure 4: Comparison between ground truth depth and inferred depth
A.4 SEGMENTATION PERFORMANCE ON MORE CHALLENGING IMAGES
We further evaluated our model on images of running vehicles in virtual towns generated by Carla environment (Dosovitskiy et al. 2017). following the approach of (Henderson & Lampert (020) but with much more diverse scenes (the traffic dataset used (Henderson & Lampert (2020) were collected in street with camera viewing from side of the vehicle while trained and tested our model across different cities with all possible viewing angles). This dataset is much more challenging it includes more realistic lighting condition (e.g., mirror reflection) and much more complex scene structure. As shown, across different viewing angles, the model can always segment the major car in view correctly, although with some artifact in segmentation mask. One difficult situation is when the road's color is very homogeneous, because the cars mostly move along the road, treating part of the road as object does not increase prediction error by warping and the model turn to make such mistake. Because some cars distant from the camera are very small and barely detectable, we calculate IoU weighted by the size of object, this yields size- weighted IoU of 0.44.
Figure 5: Segmentation performance on images of cars in virtual towns generated by Carla. Top: input images, middle: image weighted by the inferred mask for the object slot containing the car, bottom: image weighted by the inferred mask for the background. Other slots (3 in total in training) neglected.
13

---

Under review as a conference paper at ICLR 2023
A.5
METHOD
A.5.1
PREDICTION BY WARPING
We first describe the prediction of part of the next image by warping the current image. Here consider only rigid objects and the fates of all visible pixels belonging to an object. With depth D E R h (I() of all pixels in a view inferred by the Depth Perception network based on visual features in the image I(t) the 3D location of a pixel at any coordinate (i.j) in the image, where |1| S w-1
2 -1] S h-1, can be determined given the focal length d of the camera as 2
m (e) (1.))
(i.) [r,d.]]. Here, we take the coordinate of the center of an image as (0,0). On V
the other hand, with the inferred (1)) and $ (t-11) the current and previous locations of the object l that the pixel (i belongs to, from (+) and (tt respectively,
xxx
estimate the instantancous
velocity of the object b = (t T (t-1) Similarly, with the inferred the current and previous (1)
pose probabilities of the object, Pix and Pox. we can obtain the likelihood of its angular velocity
(t-1)
plok OK (t-1)
w/ = w)
Σ
plok = -) p( (t-1) 72)
(5)
71.72:7 THE BE (27
By additionally imposing prior distribution (we Von Mises distribution) over w!) that favors slow rotation, we can obtain the posterior distribution of the object's angular velocity (t) (t) (t-1))) and eventually the posterior distribution of the object's next pose p(o (++1)
p(w og 0%
k (t-1)).
Assuming a pixel (i belongs object k using the estimated motion information o6 and p(w ( 0/ th
(t- 1)) of the object, together with the current location and pose of the object and the current
3D location mi (id) of the pixel, we can predict the 3D location m (t-+1) of the pixel at the next k.(i.))
moment as
(t+1) m k.(i.))
= (1 wigh [M(0
(t) (1) (m (5.5) 2 + 1
(()
obs
(t)
(6)
where ( and M 2x are rotational matrices due to the rotation of the observer and the object,
respectively, and wite is the velocity of the observer (relative to its own reference frame at 1). In this way, assuming objects move smoothly most of the time, if the self motion information is known, the 3D location of each visible pixel can be predicted. If a pixel belongs to the background, wx+1 0 and UK+1 (K + 1 the background's index). Given the predicted 3D location, the target coor- dinate (i' (f',j") (f+1) (t+1) of the pixel on the image and its new depth D' (i, (t-1) can be calculated. This prediction of pixel movement allows predicting the image ('(+1) and depth D'((+1) by weighting the colors and depth of pixels predicted to land near each pixel the discrete grid of the next frame, as explained in See 2.4.2.
A.5.2 WARPING CONTRIBUTION WRIGHT
As the object attribution of each pixel is not known but is inferred by fatj (I(0), ) it is represented for every pixel as probability of belonging to each object and the background T.
(1) k = 2, ..... .K+
1. Therefore, the predicted motion of each pixel should be described
probability distribution k,(i,)
over + 1 discrete target locations p(æ" (i.j)
(t+1)) = > (()
kij 8(x (t+1) i.c., pixel (i,j) has a
probability of T to move to location xf k.(i.)) at the next time point, for k = 1,2, ... K 1. With
(2)
(t+1)
such probabilistic prediction of pixel movement for all visible pixel (i, ()),, we can partially predict the colors of the next image at the pixel grids where some original pixels from the current view will land nearby by weighting their contribution:
Eni, wn (
Eesy ww(..j.P.)) otherwise
I (1+1) Warp
(p.g)
En.i we (e(().)))))
(0,
14

---

Under review as a conference paper at ICLR 2023
We define the weight of the contribution from any source pixel (i,j) target pixel (p,
we (t.ji.p..)))
(t) "kis"
3 max - ((((+1)) pl. 0} max - 15'((tt)) g/. 0} (8)
The first term incorporates the uncertainty of which object pixel belongs The second term e ----D" (i.s) resolves the issue of occlusion when multiple pixels are predicted to move close the same pixel grid by down-weighting the pixels predicted land farther from the camera. These last two terms mean that only the source pixels predicted to land within a square of of 2 x 2 pixels centered any target location (p, q) will contribute the color I' 1
(p. q)- The depth Warp can be predicted by the same weighting scheme after replacing (4) (i,j) with each predicted depth (1(++1)) (i.j) assuming the pixel belongs object
map (+++1)
A.6 DEPENDENCY OF SEGMENTATION PERFORMANCE ON OBJECT SIZE AND DISTANCE ACROSS MODELS
Our model (OPPLE)
MONet-122 128
Plads occupied by abjects
10000
Plxcs eccupicd by oqjcct
10000
1000
1000
100
100
1
10
10
0.3
1.0
3.0
1.0 Objectiitance
3.0
Objectdistance
S Slot-attention-128
Genesis
Piaate occupidd by dbjects
10000
10000
1000
by by
1000
0
loU
100
0.3
1.0
3.0
0.3
1.0
Object distance
Object distance
Figure 6: Comparison of segmentation quality with respect the pixel occupancy and distance of the object from the observer.
We observe that for objects further away from the observer which occupies less pixels (because of the 2D perspective projection), the IOU goes down with the increase in the distance. We this for all the compared models. This points to a requirement for robust segmentation to balance between the scales of the object. However, model this would fail as collating them together to the same scale would still be problematic, needing weigh which scale is more important than others. Adding bigger motion the object can help adding extra information for such small objects so that they are not missed during segmentation.
led to believe that even using hierarchical structures to
A.7 COMPARISON WITH THE GENESIS MODEL.
Genesis Engeleke et al. 2019) and the Genesis V2 Engelcke et al.) (2021) models as mentioned the main text are shown to perform very well in their manuscript. However, we were unable to train Genesis V2 with our dataset as we were consistently getting poor results. We decided to reuse the Genesis model pre-trained on the GQN dataset, then fine tuned it on our dataset. We show the segmentation result using this training procedure below. We keep this comparison in the appendix as we are not very confident how much longer it needs to be trained and how much more hyperparameter tuning would need to improve. The comparison results shown in the figure
15

---

Under review as a conference paper at ICLR 2023
Original images
Ground segmentation
Genesis
Our model (OPPLE)
Figure 7: Example of object segmentation by different models
A.8 ILLUSTRATION OF PREDICTION QUALITY
In the figure below, display the first, second image, one of the masked objects, the predicted third image, and the ground truth of the third image, from top to the bottom. We provide reference lines and some circles aid the comparison between images and evaluate the warping quality. Imagination quality can be inspected usually at one side of the predicted images.
16


Reviews:
[{"review_id": "uLHSh0X-zM", "reviewer": "Reviewer_7oBh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper learns to discover objects by video prediction. The method called OPPOLE extracts each frame into object states(inferred object 3D position and 1D pose), and background. The method learns the decomposition in an unsupervised method that renders the object representation into image frames and compares it with GT RGB images and inferred depth. Several additional regularization terms are proposed to avoid the degenerate solutions. The method is compared with per-frame unsupervised object discovery baselines including slot-attention and monet on their proposed synthetic room dataset. \n", "summary_of_the_review": "Overall, the method is clearly presented and written but the key contribution and takeaways are a bit unclear. The empirical results are relatively weak in the sense that 1) the results are conducted on their own datasets and 2) more ablations need to be done in order to understand each components. \n", "strength_and_weaknesses": "Strength:\n+ The model use motion cues (wo explicit supervision) to supervise the object discovery tasks and the object notion emerge via bottle-neck object representation. \n+ The only supervision is RGB images and camera pose and it is interesting to see depth emerging from unsupervised training. \n+ the paper is written clearly and is easy to follow.\n\nWeakness:\n- The biggest claimed contribution is that they combine predictive learning with explicit modeling 3D motion. However, the idea of explicitly modeling 3D is not extremely novel e.g. (Henderson & Lamp NeurIPS 2020, Anciukevicius et al NeurIPS 2022). I do not see a fundamental difference between them.  Although [A] Henderson & Lampert, NeurIPS 20 cannot generalize to a single frame, I wonder what happens if both models are inferred from videos – it may favor [A] as OPPLE is a by-frame prediction, yet it helps to understand what is new in OPPLE. \nAnother important basleines to claim the contribution is to model motions in 2D instead of 3D, but still, use predictive learning to train. \n- More ablations are needed to understand how each component contributes to the tasks. Many designs of the method appear rather arbitrary to me, e.g. imagination network chooses not to include geometry even though the warping network does, the effect of regularization terms in training. \n- The comparisons are shown on the synthetic dataset generated by their own, which looks like a variant of ROOM datasets with more diverse textures and a fixed number of objects. However, I wonder why ROOM/CLEVER benchmark cannot be evaluated. Although the baselines will improve bc of less diverse textures, according to the authors, the proposed method should also work on that dataset. I encourage authors to report results on more widely adopted datasets in addition to their own datasets. This would calibrate their method better.\n- The proposed method does not handle various number of objects like the baselines do.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lW5ZGevPZP", "reviewer": "Reviewer_31jS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "  The paper considers video-based object-centric learning tasks.\n  The proposed model achieves object-centric learning by predicting per-object motion and future frame observation.\n  Experiments show that the proposed method outperforms a collection of single-frame baselines.\n", "summary_of_the_review": "  The method adopted by the paper is solid and has potential.\n  However, I believe the performance can be improved.\n  This work is also related to warping-based depth estimation literature [2][3] (on real-world datasets), which may provide hints on how to improve performance. \n   Priors can be used to regularize the behavior.\n   For example:\n\n   - Encourage points to remain static (to better segment objects from the ground).\n   - Encourage near regions to have similar motion (to reduce over-segmentation within each object).\n   - Bidirectional wrapping\n   - Refine the mask across multiple transitions\n\n[2] The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth\n\n[3] Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency\n", "strength_and_weaknesses": "  - Strength\n    - This work exploits the heuristic that parts belonging to the same object should move in a consistent way.\n    - The proposed model can predict depth.\n    - Compared with baselines that cannot efficiently use motion information, the proposed method can detect objects with non-trivial \ntexture.\n    - The polar angle prediction results clearly demonstrate the potential of this method,\n  - Weakness\n    - While the qualitative results shown in figure 2 is much better than various baselines, the huge gap is not reflected in the quantitative results.\n    - The FG-ARI and IoU are low in number. This score roughly indicates that the model is completely not unusable even on the simply simulated dataset.\n    - If I am not mistaken, slot attention for video (SAVi) supports fully unsupervised training. I am not sure why SAVi is not included as a baseline.\n    - Do all objects in the training set move all the time?\n    - Even though the traditional dataset (especially [1]) is not challenging enough to show off the benefits of the proposed model, it can still provide valuable insights to the research community.\n    - Since the loss comes from the image reconstruction loss, how do you deal with textureless regions (which can probably explain the hole in object centers)?\n\n\n[1] Object-Centric Representation Learning with Generative Spatial-Temporal Factorization\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NJNGBQTXeLi", "reviewer": "Reviewer_GkH5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for learning object-centric representations from single 2D images without supervision. Unlike previous work that learns by reconstructing the current frame, the proposed method learns by predicting object movement in future frames. Specifically, the model is trained from triplets of 2D images that are taken consecutively from a 3D scene by a moving camera. The camera intrinsics and movement are known to the model. At test time, the model can be applied to single images for object segmentation and localization. The experiments are done on a custom dataset featuring complex object texture and background. The model shows comparable or slightly better segmentation performance than SlotAttention.", "summary_of_the_review": "I recommend reject for the paper in its current form, due to potentially limited significance, insufficient discussion of related work, and clarity issues.", "strength_and_weaknesses": "- Strengths\n    - The problem setting is interesting and close to how humans perceive objects.\n    - The custom dataset seems challenging and can be valuable to the community.\n- Weaknesses\n    - The empirical results are a bit discouraging. The proposed method (with access to camera intrinsics and multiple frames during training) is similar to or only slightly better than SlotAttention (with access to only static images) in terms of segmentation performance. The benefit of predicting object movement is not well demonstrated by these results.\n    - It seems the paper aims to tackle (1) complex object texture and (2) object movement in 3D. However, the paper fails to compare with or discuss the related work in these two aspects.\n        - Some recent work (e.g., [GENESIS-v2](https://proceedings.neurips.cc/paper/2021/hash/43ec517d68b6edd3015b3edc9a11367b-Abstract.html) and [SLATE](https://openreview.net/forum?id=h0OYV0We3oh)) tackles complex texture from single 2D images without predicting object movement. The authors did try to train GENESIS-v2 on their dataset, but failed to get good results. Perhaps the authors could try to make some version of the dataset that is similar to those used in GENESIS-v2, and show that predicting object movement leads to better segmentation than using static images alone.\n        - Quite a few papers tackle object-centric 3D representations (e.g., [O3V](https://proceedings.neurips.cc/paper/2020/hash/20125fd9b2d43e340a35fb0278da235d-Abstract.html), [ROOTS](https://www.jmlr.org/papers/v22/20-1176.html), [Crawford & Pineau](https://github.com/oolworkshop/oolworkshop.github.io/blob/master/pdf/OOL_19.pdf), [ObSuRF](https://arxiv.org/abs/2104.01148), etc). In principle, these methods could be applied to the dataset in this paper. In fact, the representations they learn are more comprehensive than this paper (because they capture 3D appearance) while given the same input (2D image + camera parameters).\n        - While not strictly necessary, the paper could also discuss object-centric representation learning from 2D videos (e.g., [SQAIR](https://arxiv.org/abs/1806.01794), [SCALOR](https://arxiv.org/abs/1910.02384), etc). They treat object movement in 2D, but the representation learning also benefits from predicting future frames.\n    - The presentation of the paper can be improved. The paper describes the method at very detailed level, without explaining the high-level intuition/motivation behind the key design choices, making it hard for the reader to follow. In particular, I have the following questions:\n        - Why use discrete yaw angles instead of continuous values?\n        - Why does the imagination network take original image as input rather than the inferred object latents (mentioned in Section 2.3)?\n        - Why use a Von Mises prior distribution for angular velocity?\n        - When doing the soft-matching, why do you only use a subset of the latent instead of the full latent?\n        - Background velocity is set to zero (mentioned in Section 2.4.1). Do you assume static background?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3G-SpvBpVw", "reviewer": "Reviewer_BsMb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Note I have reviewed an earlier version of this paper for last year's conference. I have copied some of the notes from my earlier review but I have read the new version carefully and made sure that I only copied parts that hasn't changed in the new version. Overall, the earlier paper and the current version overlap quite significantly (especially in terms of experimental results).\n\nThis paper presents an unsupervised object-centric scene representation technique that can decompose a scene into multiple objects (and segment the scene) and infer their 3D locations and pose. The overall setup is very similar to earlier models like MONet but this model works on sequences of images, more precisely on 3 consecutive images. It uses the first two images to infer the 3D position and pose of objects and combining this with known camera motion tries to predict the last (third) image. Then it uses an optical flow based method to warp the image at time t using the predicted object location/pose/depth to predict (some of) the pixels in image at time t+1. The model is trained to minimize the difference between this prediction and the true next image.\n\nIn more detail, the object extraction network outputs the location and pose of each object. And a separate depth perception network outputs the depth for each pixel in the image. The location and pose of objects are used to estimate the velocity of each object (e.g., by subtracting the position at t-1 from position at t. note this requires matching each object at time t-1 to object in time t, which they do using a soft-matching approach). These along with the depth information are then used to warp the image at t to predict pixels in image at time t+1. This is possible only for a subset of the pixels so for the rest, they use a separate \"imagination\" network that takes in object information and predicts the color/depth and object masks at t+1. The predictions from warping and imagination network are then combined to form the final predicted color and depth images.\n\nTo train the model, they require images and camera motion, and use a combination of losses: reconstruction loss on predicted and ground truth image, self-supervised losses on object location, pose, and depth.\n", "summary_of_the_review": "Overall, I think the paper is interesting, but unfortunately the empirical evaluation is very limited and this makes it difficult to evaluate the full merit of the proposed approach.", "strength_and_weaknesses": "Overall, as I have said last time, I think this is an interesting paper. The optical flow based warping to predict some subset of the pixels in the next timestep is a nice idea and is perhaps the main novelty of this paper. (However, some recent work has used a similar idea [1].)\n\nI think the writing has improved significantly in the newer version; I found the txt much easier to follow (of course this probably has something to do with the fact that I'm reading paper for a second time.)\n\nI think the main concern with the paper, again like last time, is limited experimental evaluation. The model is evaluated only on a single, rather simple dataset (that was generated by the authors). I would at least have expected a comparison on some more or less standard datasets like CLEVR etc., at least on some of the datasets that other competing models were evaluated on. I understand that the dataset generated by authors have textures, which other techniques find difficult to handle, and their technique can deal with better. But I'd like to see that the proposed technique still works in the \"simpler\" setting that competing methods can handle.\n\nOther notes\n\n- There are better models than MONet and Slot Attention in this literature like PARTS, SLATE etc. It'd be nice to compare to these as well.\n- The authors mention O3V as a similar technique. Why don't you compare against it as well?\n- Did the authors think about applying this technique to longer sequences? Longer sequences may make it more challenging to apply the technique but will enable model to learn since longer sequences should have more information.\n- There are no std deviations in table 1, which makes it difficult to how whether the performance differences are meaningful.\n- Slot attn 128 does worse than smaller slot attn. Do the authors why this is the case? I'd have expected to work at least as well. However, in terms of IoU, it does better. Again, having std dev. here would help to understand what is going on.\n- Typos:\n  - pg 4., top: modificaiton\n  - pg 5., bottom: We -> we\n  - pg 8., top: ground-true -> ground-truth\n\n[1] The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos, https://arxiv.org/abs/2111.06394", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject