Unsupervised Discovery of 3D Physical Objects from Video
Authors: Yilun Du
Kevin A. Smith
Tomer Ullman
Joshua B. Tenenbaum
Jiajun Wu
Venue: ICLR 2021 Poster (2021)
Keywords: unsupervised object discovery
surprisal
scene decomposition
physical scene understanding

Abstract: We study the problem of unsupervised physical object discovery. While existing frameworks aim to decompose scenes into 2D segments based off each object's appearance, we explore how physics, especially object interactions, facilitates disentangling of 3D geometry and position of objects from video, in an unsupervised manner. Drawing inspiration from developmental psychology, our Physical Object Discovery Network (POD-Net) uses both multi-scale pixel cues and physical motion cues to accurately segment observable and partially occluded objects of varying sizes, and infer properties of those objects. Our model reliably segments objects on both synthetic and real scenes.  The discovered object properties can also be used to reason about physical events.

Published as a conference paper at ICLR 2021
UNSUPERVISED DISCOVERY OF 3D PHYSICAL OBJECTS FROM VIDEO
Yilun Du MIT
Kevin Smith MIT
Tomer Ulman Harvard University
Joshua Tenenbaum MIT
Jiajun Wu Stanford University
ABSTRACT
We study the problem of unsupervised physical object discovery. While existing frameworks aim to decompose scenes into 2D segments based off each object's appearance, we explore how physics, especially object interactions, facilitates dis- entangling of 3D geometry and position of objects from video, in an unsupervised manner. Drawing inspiration from developmental psychology, ject Discovery Network (POD-Net) uses both multi-scale pixel cues and physical motion cues accurately segment observable and partially occluded objects of varying sizes, and infer properties of those objects. Our model reliably segments objects on both synthetic and real scenes. The discovered object properties can also be used to reason about physical events.
Physical Ob-
INTRODUCTION
From early in development, infants impose structure on their world. When they look at a scene, infants do not perceive simply an array of colors. Instead, they scan the scene and organize the world into objects that obey certain physical expectations. like traveling along smooth paths or not winking in and out of existence (Spelke & Kinzler, 2007; Spelke et al., 1992). Here we take two ideas from human, and particularly infant, perception for helping artificial agents learn about object properties: that coherent object motion constrains expectations about future object states, and that foveation patterns allow people to sean both small or far-away and large or close-up objects in the same scene.
Motion is particularly crucial in the early ability to segment a scene into individual objects. For instance, infants perceive two patches moving together as single object, even though they look perceptually distinct to adults (Kellman & Spelke, 1983). This segmentation from motion even leads young children to expect that a toy resting on a block is picked up, both the block and the toy will move up if they are single object. This suggests that artificial systems that learn to segment the world could be usefully constrained by the principle that there are objects that move in regular ways.
In addition, human vision exhibits foveation patterns, where only local patch of scene is often visible at once. This allows people to focus on objects that are otherwise small on the retina, but also stitch together different glimpses of larger objects into coherent whole.
We propose the Physical Object Discovery Network (POD-Net). self-supervised model that learns to extract object-based scene representations from videos using motion cues. POD-Net links visual generative model with a dynamics model in which objects persist and move smoothly. The visual generative model factors an object-based scene decompositions across local patches, then aggregates those local patches into a global segmentation. The link between the visual model and the dynamics model constrains the discovered representations be usable to predict future world states. POD-Net thus produces more stable image segmentations than other self-supervised segmentation models, especially in challenging conditions such as when objects occlude each other (Figure 1).
We test how well POD-Net performs image segmentation and object discovery on two datasets: one made from ShapeNet objects (Chang al., 2015), and one from real-world images. We find that POD-Net outperforms recent self-supervised image segmentation models that use regular foreground- background relationships (Greff et al., 2019) or assume that images are composable into object- like parts (Burgess al., 2019). Finally, show that the representations learned by POD-Net can be used to support reasoning in task that requires identifying scenes with physically implausible events
Project page: httpyyylluunnnniguhhhhh/o/oooooe

---

Published as a conference paper at ICLR 2021
With Motion
Without Motion
Images
Color Wheel for Motion
Masks
Figure I: Motion is important for object segmentation from early in development. We combine motion with an approximate understanding of physics discover objects that are physically consistent across time, the video above, motion cues (shown with colored arrows) enable our model to modify our predictions from a single large incorrect segmentation mask to two smaller correct masks.
(Smith et al., 2019). Together, this demonstrates that using motion as a grouping cue to constrain the learning of object segmentations and representations achieves both goals: it produces better image segmentations and learns scene representations that are useful for physical reasoning.
2 RELATED WORK
Developing a factorized scene representation has been core research topic in computer vision for decades. Most learning-bass prior works are supervised, requiring annotated specifications such as segmentations (Janner et al., 2018), patches (Fragkiadaki et al., 2015), or simulation engines (Wu et al., 2017; Kansky et al., 2017). These supervised approaches face two challenges. First, in practical often prohibitively challenging to obtain:
scenarios, annotations geometry, pose, and semantics of every object we encounter, especially for deformable objects such trees. Second, supervised methods may not generalize well out-of-distributio test data such as novel objects or scenes.
cannot annotate the 3D
Recent research on unsupervised object discovery and segmentation in machine learning has attempted to address these issues: researchers have developed deep nets and inference algorithms that learn to ground visual entities with factorized generative models of static (Greff al., 2017; Burgess et al., 2019; Greff et al., 2019; Eslami et al., 2016) and dynamic (van Steenkiste et al., 2018; Veerapaneni al., 2019; Kosiorek et al., 2018; Eslami al., 2018) scenes. Some approaches also learn model the relations and interactions between objects ( (Veerapaneni et al., 2019; Stanić & Schmidhuber, 2019; van Steenkiste al., 2018). The progress in the field is impressive, though these approaches are still mostly restricted to low-resolution images and perform less well small or heavily occluded objects. Because this, they often fail to observe key concepts such as object permanence and solidity. Furthermore, these models all segment objects in 2D, while our POD-Net aims to capture the 3D geometry of objects the scene.
Some recent papers have integrated deep learning with differentiable rendering to reconstruct 3D shapes from visual data without supervision, although they mostly focused on images of a single object (Rezende et al., 2016; Sitzmann al., 2019), require multiview data input (Yan et al., 2016). In contrast, we use object motion and physics to discover objects in 3D with physical occupancy. This allows our model do better in both object discovery and future prediction, captures notions such as object permanence, and better aligns with people's perception, belief, and surprise signals of dynamic scenes. separate body of work utilizes motion cues to segment objects (Brox Malik, 2010; Bideau et al., 2018; Xie et al., 2019; Dave et al., 2019). Such works typically assume single foreground object moving, and aggregate motion information across frames to segment out objects or separate moving parts of objects. Our work instead seeks to distill information captured from motion to discover objects in 3D from images.
Others works have explored 3D object discovery using RGB-D or 3D volumetric inputs (Herbst et al., 2011; Karpathy et al., 2013; Ma & Sibley, 2014). The presence of 3D information, such as depth, is significant difference from work. Such information allows approaches to reliably detect surface orientations and discontinuities (Karpathy et al., 2013; Herbst et al., 201 which significantly reduces the difficulty of discovering objects, especially in the tabletop settings considered.
Our work also related to research in computer vision on unsupervised object discovery from video (Lu et al., 2019; Wang et al., 2019; Yang et al., 2019b). Such works focus detecting objects in
2

---

Published as a conference paper at ICLR 2021
Current Patch
Segments
For each segment
State at time & 11 .sk .9k mmm Physics
3D Translation
Step
3D Size
Lepiyiiss (G, B the mi)
Rotation q
s qi mf) Inferred state at time
Mask
Predicted state time
Step 2
II. backprojection Model
III. Dynamics Model
C2
For each segment
Lamage
Step
Step 4
Generuted
Input
Input Image
Component VAE
Generated segments
image
image
Inference Model
Image Generative Model
Figure 2: POD-Net contains four modules for discovering physical objects from video. (I) An inference model ruto-segresyyvlly infers of candidate object masks and latents describe each patch an image; (II) backprojection model maps each mask to 3D primitive; (III) dynamics model captures the motion of 3D physical objects; and (IV) An image generative model decodes latents and masks to reconstruct the image.
realistic videos, but only focus detecting a single foreground object, instead of all component objects in scene. Furthermore, such approaches rely on supervised information such ImageNet weights, or pretrained segmentation networks for object detection, limiting their applicability to new objects in novel classes. Our our approach also assumes supervised information on the underlying 2D 3D mapping, but does not assume any supervision for object detection. We show that this enables our approach to generalize to novel ShapeNet objects.
METHOD
The Physical Object Discovery Network (POD-Net) (Figure 2) decomposes dynamic scene into set of component 3D physical primitives. POD-Net contains an inference model, which recursively infers a set component primitive descriptions, masks, and latent vectors (Section 3.1). also contains a mooele generative model (Section 3.2). The generative model consists of back- pack-prooection module to infer 3D properties of each component, a dynamics model to predict primitives motions and image generative model in the form VAE (Kingma & Welling, 2014; Rezende et al., 2014) to render primitives onto 2D images. These components ensure that learned primitive representations can reconstruct the original image in a physically consistent manner. Together, these constraints produce a strong signal for self-supervised learning of object-centric scene representations.
3.1 INFERENCE MODEL.
We sequentially infer the underlying masks and latents that represent scene (Figure 2-1). Inspired by MONet (Burgess et al., 2019), we employ attention network Attention(-) to decompose scene into a set of separate masks M = {m,, m2, m.,. We extract corresponding latent z; per mask initialize context
m, In particular, x yet to be explained. At each step, we decode the attention mask m, = 1--1Attention, We iteratively update the corresponding context in the image by c; C;-1 (1 Attention(2:: c,-1)) to ensure that sum of all masks explain the entire image.
1, which we define represent the context in the image
c-11).
We further train a VAE encoder Encode(z mm, æ)), which infers latents z; from each component mask m, We set mg, Zn the first decoded mask and latent - to the background mask m. and latent 25, and define each subsequent mask or latent be object masks and latents.
Sub-pate -atch decomposition. Direct inference of component objects and background from a single image can be difficult, especially when images are complex and when objects are of vastly different sizes. An inference network must learn to pay attention to coarse features in order to segment large objects, and to fine details in the same image order to segment the smaller objects. Inspired by how people solve this problem by stitching together multiple foveations into coherent whole, we train our models and apply inference on overlapping sub-patches an image (Figure 3).
3

---

Published as a conference paper at ICLR 2021
In particular, given image of size H x W. we divide the image into 8 x 8 grid (pic- tured in the left of Figure 3), with each grid element hav- ing size H/8 X W/8. We construct sub-patch for ev- ery 2 2 component sub-grid, leading to a total of 64 differ- ent overlapping sub-patches, We apply inference on each sub-patch. Under this decom- position, smaller objects still appear large in each sub-patch, while larger objects are shared across sub-patch.
Segments from earlier patches
Input Image
Current Patch
Segments from this patch
Merged Segments
Figure 3: Illustration of sub-patch decomposition for image inference. An image is divided a 8x8 grid, with inference applied to each 2 xx2 sub- grid. To generate glohal segmentation mask, object masks are sequentially inferred for each subpatch. Each object mask is either matched to existing object used create
object.
To obtain a global segmentation map, we merge each sub-patch sequentially using a sliding window (Figure 3). At each step, we iterate through each segment given by the inference model from sub-patch, and merge it with segments obtained from previous sub-patches, if there is an overlap in masks above 20 pixels. Every segment that does get merged is initialized new object.
3.2 GENERATIVE MODEL
Our generative model represents a dynamic scene as a set of K different physical objects and the surrounding background at each time step t. Each physical object is represented by its back- projection on 2D, a segmentation mask m E IR "aa of height H and width W, and latent code = E IR of dimension D for its appearance. In addition, the background is captured as a surrounding segmentation mask m E R and code z/, E DR Segmentation masks are defined such that the sum of all masks corresponds the entire image Σn m m 1.
We use a backprojection model to map segmentation masks m to 3D primitive cuboids (Figure 2-I1). Cuboids are a coarse geometric representation that enable physical simulation. We next construct a dynamics model over the physical movement of predicted primitives (Figure 2-III). We further construct a generative model over images x' by decoding latents zt component wise (Figure 2-IV).
Backprojection model. Our backprojection model maps mask m, to an underlying 3D primitive cuboid, represented as a translation tę E 133 size 8y E R2, and rotation qu E RR3 (as a Euler angle) transform on a unit cuboid in a fully differentiable manner. In our case, we primarily pre-train a neural network to approximate the 2D>to-3D projection and use as our differentiable backprojection model. However we show that such a task can also be approximated by assuming the camera parameters and the height of the plane is given (ignoring size and rotation regression), with little reduction in performance (see Appendix A.2 for further details).
Dynamics model. We construct dynamics model over the next state of different physical objects (ti, st gi. m by using first order approximation of velocity/angular velocity of the states of the object. Specifically, our model predicts
fg = 11-1
11
81 =
t
[-1
q = के
(qf o
m = Render (i', $/. qf.))
(2)
Our Render(-) function computes the segmentation mask of a predicted physical object, assuming all other physical objects are rendered. To compute this, we initialize palette Pa = 1, which we define to represent the context in an image that has not been rendered yet. We further utilize separate pre-trained model Project(-) that projects each primitive in 3D to a 2D segmentation mask (the inverse of the backprojection model described above). We then reorder predicted physical objects in increasing distance from the camera (ii., & (..)) We then sequentially render each predicted physical object using Render (f',, 8f., ( = P Project(tt, 8y. gf.)). and update the corresponding palette to be rendered as ps -1 - Project( 8 -f-)).
Given modeled future states, the overall likelihood of a physical object (1), $/ of. mf) is given by
4

---

Published as a conference paper at ICLR 2021
p(t2 s g.. mi) (ith; of) $% o
qe 42 mb),
(3)
where assume Gaussian distributions over translation, sizes, and rotations with 0s 0t
Oy = L. p(-) is the probability of a predicted mask, defined as p(mng mi) = 4m; >u.s + (1
=m r where (()) is the indicator function each individual pixel. Overall, our dynamics model seeks to enforce that objects have zero order motion and maintain shape.
Image generative model. We represent images x² RDD each time step spatial Gaussian mix- ture models, with each mixture model being defined by a segmentation mask m in M (Section 3.1).
Each corresponding latent # is decoded pixel-wise mean μikk and a pixel-wise mask prediction di using a VAE decoder (ecode( de zx)) We assume each pixel i is independent conditioned on 2, so that the likelihood becomes
K
p(
II
Σ
(maNN (ax po zx)) + mwN (2 Pab. of) x po zb)
(4)
i=0
for background component my, 14b, da and object components my, M4g, dg. where pa (dxx z
po(d+ = mak kx), is the probability that decoded mask from the latent matches the ground truth mask for the mixture. We use 0.11 and σ 0.07 to break symmetry between object and background components, encouraging the background to model the more uniform image components (Burgess al., 2019). Our overall loss encourages the decomposition of image into a set of reusable sub-components, as well as a large background.
3.3 TRAINING Loss
Our overall system is trained to maximize the likelihood of both physical object and image generative models. Our loss consists of L(x²) = Phyysiss + Limuge + LKI., maximizing the likelihood of physical dynamics, images, and variational bound. Our image loss is defined to be Limage = - log (p(at I=)).
Ek-n log p (Li, s g m based on Eqn. 3, which enforces that decoded primitives are physically consistent. The KL loss is
based on Eqn. 4. Our physics loss is defined to be Lphysies
K
LKI. =3
KL (Encodde((2)] ) p(=)) + (I(Enoode(z)]] , 11 p(2))) +
(5)
(L (dx/zz) || (a(a [zx)) + (L(g ploo [p(()]=)))
7
enforcing the variational lower bound on likelihood (Kingma et al., 2014), where for brevity, we use (d (a0) to represent the mask generation process in Section 3.1, and p(=) -O0, is the prior.
Our training paradigm consists of two different steps. We first maximize the likelihood of the model under the image generation objective. After qualitatively observing object-like masks (roughly after 100,000 iterations). we switch to maximizing the likelihood of the model under both the generation and physical plausibility objectives. Alternatively, we found that switching loss convergence also worked well. We find that enforcing physical consistency during early stages of training detrimental, as the model has not discovered object-like primitives yet. We use the RMSprop optimizer with a learning rate of 10-4 within the PyTorch framework (Paszke et al., 2019) to train our models.
EVALUATION
We evaluate POD-Net on unsupervised object discovery in two different scenarios: a synthetic dataset consisting of various moving ShapeNet objects, and real dataset of block towers falling. We also test how inferred 3D primitives can support more advanced physical reasoning.
4.1 MOVING SHAPENET
We use ShapeNet objects to explore the ability of POD-Net to learn to segment objects from appearance and motion cues. We also test its ability to generalize to new shapes and textures.
Data. To train models on moving ShapeNet objects, we use the generation code provided in the ADEPT dataset in Smith al. (2019). We generate training set of 1.000 videos. each 100 frames long, of objects (80% of the objects from 44 ShapeNet categories) as well as rectangular occluders. Objects move in either a straight line, back and forth, or rotate, but do not collide with each other. Setup. Videos have a resolution of 1024x 1024 pixels. We apply our model with a patch size of 256x256. We use residual architecture (He et al., 2015) for the attention and VAE components.
5

---

Published as a conference paper at ICLR 2021
Input Image
MONet
POD-Net (No Physics)
POD-Net
Figure 4: Comparisons of unsupervised object segmentation of POD-Net with and without motion and with
MONet on scenes with synthetic objects. MONet is unable to seperate individual instances of objects, but is capable of getting foreground mask of objects in scene. POD-Net (no physics) is able to reliably detect almost all objects, though instances of objects merged together into single object. POD-Net is able reliably detect separate objects even when they are mostly occluded (zoomed images on right).
Model
Multi-Scale Phys
IoU
Detection
3D IOU
3D Recall
MONET
0.289 (0.007) 0.306 (0.005) 0.019 (0.007)
0.057 (0.028)
OP3
0,145 (0.004) 0.121 (0.007) 0.001 (0.001)
0,000 (0.000)
Norm. Cuts
0.634 (0.020) 0.768 (0.029) 0.034 (0.003)
0,042 (0.005)
UVOD
0.129 (0.006) 0.062 (0.006) 0.001 (0.000)
0.000 (0.000)
Crisp Boundary Detection
0.645 (0.012) 0.727 (0.020) 0.080 (0.004)
0.020 (0.001)
POD-Net
No
No
0.314 (0.010) 0.361 (0.007) 0..5210.0022
0.171 (0.012)
POD-Net
No
Yes
0.462 (0.007) 0.512 (0.009)
(0.012)
0.287 (0.016)
POD-Net
Yes
No
0.649 (0.011) 0.709 (0.016) 0.068 (0.011)
0,251 (0.014)
POD-Net (Manual)
Yes
Yes
0.685 (0.017) 0.760 (0.016) 0.090 (0.015) 0.328 (0.016)
POD-Net
Yes
Yes
0.739 (0.011) 0.821 (0.015) 0.095 (0.012) 0.374 (0.017)
Table 1: Average IoU of segmentations on the ADEPT dataset and the proportion of objects detected, where one segmentation mask has greater than 0.5 loU, well average 3D loU and recall. Standard error in parentheses.
Our backprojection model pretrained on scenes a single ShapeNet object, varied across different locations on a plane, with different rotations, translations, and scales. Our backprojection model only serves as a rough relative map from 2D mask to corresponding they are trained utilize separate camera extrinsics/intriniccs than the ADEPT dataset, and also do not exhibit occlusions. To compute the physical plausibility Lphyssce primitives, we utilize the observations from the last three time steps. For efficiency, evaluate physical plausibility on each component sub- sub-patch of image. We train a recurrent model with a total of 5 slots for each image. Image segmentation is trained and evaluated on a per frame basis.
position/size, as the dataset
results, measure the intersection over union (loU) between the pre- dicted segmentation masks and the corresponding ground truth masks. We compute the IoU for each ground truth mask by finding the maximum IoU intersection with predicted mask. We report the average loU across all objects in an image, as well as the percentage of objects detected in an image (with loU 0.5). To measure 3D inference ability, we report the maximum 3D IoU intersection between each ground truth 3D box and our inferred 3D bounding box. We also report the recall of ground truth 3D objects (Georgakis et al., 2016) detected in an image (with 3D IoU threshold > 0.1). We utilize our backprojection model to extract 3D bounding box proposals from 2D segmentations and apply a linear transformation to align coordinate spaces.
Metrics. To quantify
Baselines. We compare with two recent models of self-supervised object discovery, OP3 (Veer- apaneni et al., 2019) and MONet (Burgess al., 2019), as well as three algorithms for object

---

Published as a conference paper at ICLR 2021
Input Images
Inpu: Images
3D Primitives
Primitives
Figure 5: Visualization of discovered 3D primitives in two different scenes (left and right) through time. Our model is able to discover 3D shape that consistent with observed inputs under a perspective map, Furthermore, discovered primitive move coherently through time.
Physical Objecct Displacemenn (.m0.615)
Physical Objject Scale (rw0.345)
Figure 6: Plot of predicted trans- lation of 3D primitive vs ground truth translation of 3D primitives (left) and plot of predicted scale of 3D primitives vs ground scale of 3D primitives (right).
Arttial Desplcceetont
Actual Scale
Predicted, Disphacemens
Predicted Scale
segmentation, Normalized Cuts (Shi & Malik, 2000), Crisp Boundary (Isola et al., 2014), and the recent UVOD (Yang et al., 2019a). We train OP3 with slots, with steps of optimization per mask in the first image, and an additional step of optimization per future timestep. Due to memory constraints, we were only able to train the OP3 model on inputs of size 128 by 128. We train MONet on inputs of size 256 by 256. We apply normalized cuts on region adjacency graph of the 256 by 256 image, and train UVOD in 256 by 256 images. We also compare with ablations of POD-Net: POD-Net applied directly to an image (single-scale) as opposed to across patches (multi-scale), POD-Net without physics, and POD-Net with a hard-coded backprojection model ('Manual').
Results. We quantitatively compare object masks discovered by our model and other baselines in Table 1. We find that OP3 performs poorly, it only discovers limited subset of objects. MONet performs better and is able to discover a single foreground mask of all objects. However, the masks not decomposed into separate component objects in scene (Figure 4, 2nd row). Our scenes consist of a variable set of objects of vastly different scales, making it hard for MONet to learn to assign individual slots for each object. We find that baseline based normalizing cuts/crisp boundary detection also able to segment objects, but is unable to get sharp segmentation boundaries for each object, and often decomposes single object into multiple subobjects (see Appendix A. I for details). Finally, UVOD also only segments a single foreground object.
We find that applying POD-Net (single scale, no physics) improves on MONet slightly, discovering several different masks containing multiple objects, albeit sometime missing other objects. POD-Net (single scale, physics) more reliably segments separate objects, but still misses objects. POD-Net (multi scale, no physics) reliably segments all objects in the scene, but often merges multiple objects into one object, especially when objects are overlapping (e.g., Figure 4, 3rd row). Finally, POD-Net obtains the best performance and segments all objects in the scene and individual objects where multiple objects overlap with each other (Figure 4, 4th row). Utilizing a manually coded backprojection module, POD-Net ('Manual') only leads to slight degradation in performance.
Next we analyze the 3D objects discovered by POD-Net. In Table 1, POD-Net performs the best, achieving the highest 3D average IoU intersection and recall. Crisp boundary detection obtains high average loU but low recall due to a large number of proposals. All IoUS are low due to the challenging nature of the task obtaining high 3D loU requires correct regression of size, position, and depth (using only RGB inputs). Even recent supervised 3D reconstruction approaches using 0.25 IoU thresholds for evaluation (Najibi et al., 2020). Visualizations of the discovered objects in Figure 5 show that POD-Net is able to segment a scene into a set of temporally consistent 3D cuboid primitives. We further find a high correlation r 0.615 between the predicted and ground truth translation of objects and show plots of correlations in Figure 6 of ground truth and predicted objects.
Generalization. Just as young children can detect and reason about new objects with arbitrary shapes and colors, we test how well POD-Net can generalize to scenes with both novel objects and colors. We evaluate the generalization of our model on two datasets: a novel object dataset consisting of 20 new objects and novel color dataset, where each object is split into two colors.
7

---

Published as a conference paper at ICLR 2021
Generalization to New Objects
Generalization to New Colors
Model
Phys
IoU
Detection
Model
Phys
IoU
Detection
No
0.768
0.823
No
0.658
0.716
POD-Net
POD-Net
Yes
0.857
0.922
Yes
0.756
0.843
Figure 7: Generalization to novel objects and colors. Top: POD-Net successfully segments individual objects, except when colors bisect an object (row 2, column 7). Bottom: Evaluation of POD-Net's generalization with or without physical constancy, measured in average IoUs segmentations and in the percentage of objects that detected. Including physics integrates the motion signal and generalizes better in both cases.
Figure 7 shows quantitative analysis of POD-Net applied to both novel objects and colors. We find that in both settings, POD-Net with physical consistency gets better segmentation than without. Performance is higher here compared to that reported on the training set, as both novel datasets contain fewer objects in single scene. Qualitatively, POD-Net performs well when asked to discover novel objects, although it can mistake multicolored novel shape to be two objects.
4.2 REAL BLOCK TOWERS
Next, we evaluate how POD-Net seg- ments and detects objects in real videos.
Model
Multi-Scale Phys
IoU
Detection
MONET
0.521 (0.005) 0.537 (0.003)
Data. We use the dataset in Lerer et al. (2016) with 492 videos of real block towers, which may or may not be falling. Each frame contains 2 to blocks of red, yellow, or green color. Each block has the same 3D shape, although the 2D projections on the camera differ.
OP3
0.311 (0.004)
0.250 (0.007)
Norm. Cuts UVOD
0.652 (0.006)
0.849 (0.018)
0.029 (0.001)
0.0 (0.0)
POD-Net
No
No
0.546 (0.004)
0.523 (0.006)
POD-Net POD-Net
Yes
No
0.734 (0.012)
0.761 (0.008)
Yes
Yes
0.837 (0.004) 0.908 (0.008)
Setup. For our backprojection model, we use a pretrained neural network on scenes of single block at different heights, sizes, and varying distances. Similar Section 4.1, the backprojec- tion model serves rough 2D to 3D model and trained with different cam- and perspective parameters without occlusion. All other settings are the same in Section 4.1.
Input Image
MONet
POD-Net (No Physies)
Results. We compare masks discov- ered by POD-Net and baselines in Fig- ure 8. We find that OP3 and MONet often misses blocks and also groups two blocks into a single object, leading to floating blocks the air (Figure 8, 2nd row). Normalized cuts also suffers from a similar issue of grouping blocks, but suffers an additional issue of overseg- mentation (see Appendix A. UVOD fails predict a segmentation due to limited motion video. POD-Net (sin- gle scale, physies) is able segment
POD-Net
Figure 8: Top: loU segmentation results on real blocks datasct and the percentage of objects detected, Bottom: Qualita- tive comparisons of unsupervised object segmentation of POD- Net with and without physics and with MONet realistic block towers. MONet often groups two blocks similar color (dark blue/green) together and sometimes misses particular blocks. POD-Net without physics reliably detects all blocks, but still groups similar blocks (dark blue/green) into one. POD-Net with physics detects objects and assigns different masks each. Standard error parentheses.
8

---

Published as a conference paper at ICLR 2021
Figure 9: Surprise over time a 'Block' scene. POD-Net has relatively low surprisal through- most of the video. But when the occluder falls and the object appears to "teleport' across the wall, POD-Net recognizes this abnormal shift in position and becomes surprised.
Surprisal
Images
all blocks, but treats the entire stack as a single object. POD-Net (multi-scale, no physics) does better and is able to reliably segment all blocks, though it still groups blocks of similar colors together (Figure 8, 3rd row). Finally, POD-Net with multiple scales and physical consistency performs the best, reliably separating individual blocks in a tower (Figure 8, 4th row).
4.3 JUDGING PHYSICAL PLAUSIBILITY
We whether POD-Net can discover objects reliably enough perform the physical violation de- tection task of Smith et al. (2019), in which videos that have non-physical events (objects disappearing or teleporting) must be differentiated from plausible videos.
We consider two separate tasks: the Overturn (Long) task, which consists of a plane overlaying an object, and the Block task, which consists of physical scenes with a solid wall and an object moving towards the wall, where it may either appear to hit the wall and stop or appear on the other side. To successfully perform the Overturn task, POD-Net must reason about object permanence, while to accomplish the block task, the system must remember object states across large number of timesteps and understand both spatial continuity and object permanence.
Setup. We use POD-Net trained in Section 4.1 to obtain set of physical objects (represented as cuboids) describing an underlying scene. The extracted objects are provided scene description to the stochastic physics engine and particle filter described in Smith et al. (2019). We evaluate our models using a relative accuracy metric (Riochet et al., 2018).
Results. On the Block task, we find that our model achieves a relative accuracy of 0.622. Its performance on a single video can be seen in Figure where it has learned to localize the block well enough that the model is surprised when it appears on the other side of the wall. The model in Smith al. (2019) scores relative accuracy of 0.680; this acts as an upper bound for the performance of our model, since supervised training is used to discover the object masks and recover object properties. In contrast, POD-Net discovers 3D objects in an unsupervised manner, outperforming the baseline generative models studied by Smith et al. (2019) that do not encode biases for objecthood (GAN: 0.44, Encoder-Decoder: 0.52, LSTM: 0.44). On the Overturn (Long) task - the one task where the ADEPT model underperforms baselines our model obtains performance of 0.77, outperforming Smith et al. (2019) (0.73), and equalling or exceeding models that do not encode biases for objects Decoderr 0.61, LSTM: 0.63).
(GAN: 0.81,
limitation of our approach towards discovering 3D object primitives is that across a long video (over 100 timesteps), there may be several spurious extraneous objects discovered. The model in Smith et al. (2019) does not deal well with such spurious detections either, requiring us to tune separate typer-parameters for each task.
CONCLUSION
We have proposed POD-Net, a model that discovers 3D physical objects from video using self- supervision. We show that by retaining principles of core knowledge in our architecture that objects exist and move smoothly - and by factorizing object segmentation across sub-patches, we can learn to segment and discover objects in generalizable fashion. We further show how these discovered objects can be utilized in downstream tasks to judge physical plausibility. We believe further exploration in this direction, such as integration of more flexible representation of physical dynamics (Mrowca et al., 2018; Banchez-Gonzalee et al., 2020), is promising approach towards more robust object discovery and richer understanding of the physical world around us.
Acknowledgements. This work is in part supported by ONR MURI N00014-16 1--12077- the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF- 1-311211166 the Samsung Global Research Outreach (GRO) Program, Toyota Research Institute, and Autodesk. Yilun Du is supported in part by NSF graduate research fellowship.

---

Published as a conference paper at ICLR 2021
REFERENCES
Pia Bideau, Aruni RoyChowdhury, Rakesh R Menon, and Erik Learnd-Millerr The best of both worlds: Combining cnns and geometric constraints for hierarchical motion segmentation. In CVPR, 2018. 2
Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In ECCV, 2010.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. V9X1. 11390, 2019. 1, 3, 6, 14
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. arXiv:1512.03012, 2015.
Achal Dave, Pavel Tokmakov, and Deva Ramanan. Towards segmenting anything that moves. In
ICCV Workshop, 2019. 2
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In NewrIPS, 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 160463944 2018.
Katerina Fragkiadaki, Pablo Arbelaez, Panna Felsen, and Jitendra Malik. Learning to segment moving objects in videos. In CVPR, 2015. 2
Georgios Georgakis, Md Alimoor Reza, Arsalan Mousavian, Phi-Hung Le, and Jana Košecká. Multiview rgb-d dataset for object instance detection. In 3DV, 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In NeurIPS, 2017. 2
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iterative Variational Inference. In ICML,, 2019. 1,2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2015.
Evan Herbst, Peter Henry, Xiaofeng Ren, and Dieter Fox. Toward object discovery and modeling via 3-d scene comparison. In ICRA, 2011. 2
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson. Crisp boundary detection using pointwise mutual information. In ECCV, 2014.
Michael Janner, Sergey Levine, William T Freeman, Joshua B Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. In ICLR, 2018. 2
Ken Kansky, Tom Silver, David A Mély, Mohamed Eldawy, Miguel Lázaro-Gredilla, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with generative causal model of intuitive physics. In ICML, 2017. 2
Andrej Karpathy, Stephen Miller, and Li Fei-Fei. Object discovery in 3d scenes via shape analysis. In ICRA, 2013. 2
Philip Kellman and Elizabeth Spelke. Perception of partly occluded objects in infancy. Cognit. Psychol., 15(4):483 524, 1983.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014. 3
10

---

Published as a conference paper at ICLR 2021
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In NeurIPS, 2014. 5
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In NewrIPS, 2018.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In ICML, 2016.
Xiankai Lu, Wenguan Wang. Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks, In CVPR, 2019. 2
Lu Ma and Gabe Sibley. Unsupervised dense object discovery, detection, tracking and reconstruction. In ECCV, 2014. 2
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Joshua B Tenenbaum, and Daniel L Yamins. Flexible neural representation for physics prediction. In NeurIPS, 2018. 9
Mahyar Najibi, Guangda Lai, Abhijit Kundu, Zhichao Lu, Vivek Rathod, Thomas Funkhouser, Caroline Pantofaru, David Ross, Larry S. Davis, and Alireza Fathi. Dops: Learning detect 3d objects and predict their 3d shapes. In CVPR, 2020. 7
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
Danilo J Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In ICML, 2014. 3
Danilo Jimenez Rezende, SM Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. In NeurlPS, 2016. 2
Ronan Riochet, Mario Ynocente Castro, Mathicu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. 1r03v 2018.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. ICML, 2020.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE TPAMI, 22(8): 888-905, 2000.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-aware neural scene representations. In NeurlPS, 2019. 2
Kevin Smith, Lingjic Mei, Shunyu Yao, Jiajun Wu, Elizabeth Spelke, Josh Tenenbaum, and Tomer Ullman. Modeling expectation violation in intuitive physics with coarse probabilistic object representations. NeurlPS, 2019. 2, 5,9
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Dew. Psychol., 10(1):89-96, 2007. 1
Elizabeth S Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowledge. Psychol. Rev., 99(4),605, 1992.
Aleksandar Stanié and Jürgen Schmidhuber. R-sqair: Relational sequential attend, infer, repeat.
arXiv:1910.05231, 2019. 20199 2
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. ICLR, 2018. 2
11

---

Published as a conference paper at ICLR 2021
Rishi Veerapaneni, John D Co- Co-Reyss, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In CoRL, 2019. 2, 6
Wenguan Wang, Xiankai Lu, Jianbing Shen, David J Crandall, and Ling Shao. Zero-shot video object segmentation via attentive graph neural networks. In ICCV, 2019.
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In NeurIPS, 2017. 2
Christopher Xie, Yu Xiang, Zaid Harchaoui, and Dieter Fox. Object discovery in videos as foreground motion clustering. In CVPR, 2019.
Xinchen Yan, Jimei Yang. Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In NeurIPS, 2016.
Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. CVPR, 2019a.
Zhao Yang, Qiang Wang, Luca Bertinetto, Weiming Hu, Song Bai, and Philip HS Torr. Anchor diffusion for unsupervised video object segmentation. In ICCV, 2019b. 2
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 NORMALIZED CUT/ CRISP BOUNDARY DETECTION QUALITATIVE VISUALIZATION
We provide visualizations of segmentations using normalized cuts in Figure AI. Normalized cuts relies local color similarity and spatial locality to determine segments of objects. Since our objects area relatively similar in color, it is able segment the rough shape of objects. However, compared to POD-Net, normalized cuts results in significantly less sharp boundaries around cach shape and leads to ver-seementttion of an individual object into multiple separate objects. This is due to the fact that shapes still exhibit variations in object color from lighting that causes normalizing cuts to incorrectly segment the object.
Figure Al: Illustration of segmentation using normalized cuts. Normalized cuts often over wver-segments objects and sometimes misses segmentation of small objects.
We further provide visualizations of segmentations using crisp boundary detection in Figure A2. Crisp boundary detection detects the set of edges that determines objects. This enables the approach to segment large objects in scene, where edges are not ambiguous, but fails to accurately segment smaller objects in scene, which have more ambiguous edges. Furthermore, detected edges also sensitive to the lighting of an object, sometimes segmenting an object into multiple separate pieces.
Figure A2: Illustration of segmentation using crisp boundary detection. Crisp boundary detection is able to segment large objects in a scene, but fails accurately segment smaller objects in scene, and sometimes segments objects to multiple small pieces.
A.2 DETAILS ON MANUALLY DESIGNED PROJECTION MODUL.ES
To manually design backprojection and projection model, we assume each physical object (t&, Sk, q has a fixed rotation şu and z-axis length sak, and regress other size and position pa- rameters. For our backprojection model, given a segmentation mask m, of our object, we determine the 2D bounds of the mask Tmin- Tmax- Ymins Ymax in differentiable manner, by taking the segmen- tation mask weighted mean of of boundary pixels (as defined by the 200 most boundary pixels in each direction). We then set t- = 1 + ar/min corresponding to the assumption that higher segmentation masks correspond to further away objects. We compute the remaining coordinates of tę and 8s, by
13

---

Published as a conference paper at ICLR 2021
inverting camera intrinsic matrix on the 2D bounds of the mask and setting the center of an object to be halfway between the extremes the resultant coordinates. We note that our backprojection model is fully differentiable. For projection model, utilize the camera matrix to explicitly project inferred primitives back to the segmentation mask.
A.3 PER CATEGORY SEGMENTATION PERFORMANCE
We report per category segmentation performance on ShapeNet objects below: pillow: 0.554, mug:
0.450, rocket: 0.347, earphone: 0.643, computer keyboard: 0.696, bus: 0.694, camera: 0.725, bowl:
0.565, bookshelf: 0.648, stove: 0.587, birdhouse: 0.544, wine bottle: 0.695, bench: 0.404, microwave:
0.314, lamp: 0.300, pistol: 0.566, chair: 0.465, cabinet: 0.646, bag: 0.602, rifle: 0.497, file: 0,467,
faucet: 0.367, car: 0.594, bathtub: 0.500, microphone: 0.530, ashean: 0.729, basket: 0.752, knife:
0.676, mailbox: 0.578, table: 0.565, printer: 0.660, cap: 0.559, sofa: 0.404, vessel: 0.404, display:
0.771, loudspeaker: 0.646, bicycle: 0.598, remote: 0.720, helmet: 0.368, train: 0.567, telephone:
0.601, jar: 0.663, piano: 0.734, washer: 0.358.
We find that despite having an internal physics representation of a cube, there is relatively little correlation between well-segmented objects and cubeness, with POD-Net performing well on non- cuboid classes such as piano while performing poorly on cuboid classes such as washer. Our physics loss instead encourages segmented objects across time translate uniformly well as maintain size.
A.4 MODEL ARCHITECTURE
We detail our attention model in Table Ala and our component VAE model in Table Alb. In contrast to Burgess et al. (2019), we use a residual architecture for both attention and component VAE networks, with up-sampling of the spatial broadcast layer.
7x7 Conv2D, 32
BatchNorm
3x3 Max Pool (Stride 2)
7x7 Conv2D, 32
ResBlock Down 16
BatchNorm
ResBlock Down 32
3x3 Max Pool (Stride 2)
ResBlock Down 64
ResBlock Down 32
Global Average Pool
ResBlock Down 64
Dense - 256
ResBlock Down 128
256 + 32 (μt, σ)
ResBlock Up 256
Z Z σ)
ResBlock Up 128
Spatial Broadcast z (8x)
ResBlock Up 64
3x3 Conv2d, 256
ResBlock Up 32
ResBlock up 128
ResBlock Up 32
ResBlock up 64
3x3 Conv2D, Output Channels
ResBlock up 32
(a) Attention Model (a.)
ResBlock up 16
ResBlock up 16
3x3 Conv2D, Output Channels
(b) VAE Component Model. (go> Po)
Table A1: Overall Model Architectures used in POD-Net
We detail the architecture of our Backprojection and Projection Models in Table A2.
14

---

Published as a conference paper at ICLR 2021
Dense -> 512
512 + 1024
View 64 4 x 4
ResBlock Up 64
ResNet 18
ResBlock Up 64
Dense 7
ResBlock Up 64
(a) Architecture of Backprojection Model
ResBlock Up 64
ResBlock Up 32
3x3 Conv2d, Output Channels
(b) Architecture Projection Model
Table A2: Overall Model Architectures used in POD-Net
A.5 COMPARISON ON PARTIALLY OCCLUDED OBJECTS
We further explicitly compare the performance of POD-Net on segmenting objects that occlude each other. We evaluate on the ADEPT dataset, but only consider objects such that the bounding boxes intersect. We find that in this dataset of objects, POD-Net (multi-scale, physics) obtains has detection rate of 0.734, with the an average IoU of 0.701 while POD-Net (multi-scale, no physics) obtains a detection rate of 0.601 (loU threshold 0.5) with an average loU of 0.576. This indicates our approach in incorporating physics is able to learn to effectively separate objects that partially occlude each other.
15


Reviews:
[{"review_id": "w3HyGWOdx_q", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a deep network architecture to attack the important problem of discovering physical objects from observing videos only. The architecture is mainly an extension of the MONet architecture with a physics loss and a multi-scale inference scheme. The physics loss encourages the inference module (that masks out the objects) to be more consistent with the observed motion in the scene. Additionally, the multi-scale scheme, motivated by human foveation patterns, shows better empirical performance. The method outperforms reasonable baselines on both a synthetic dataset and a real-world dataset.\n\nThe immediately related works are summarized nicely. However, I believe it might be worth mentioning other methods that discover objects by using motion, such as motion segmentation. Traditional examples include [1], and more recent examples include [2,3,4].\n\nOverall, I think the paper poses an interesting contribution with interesting results. However, the paper is hindered by the confusing equations and text, making it incredibly difficult to reproduce from the paper alone, and what I believe to be false claims of unsupervised learning (see weaknesses for details).\n\nStrengths:\n- Empirical results show that the method significantly outperforms reasonable baselines.\n- The proposed physics loss is well-motivated and leads to improved empirical performance on both datasets. Additionally, the multi-scale scheme, while simple, surprisingly leads to large gains.\n- The connection to violation-of-expectation is interesting, and the method shows pretty decent performance compared to the ADEPT upper bound.\n\nWeaknesses:\n- The equations and the text describing them are very confusingly written, leading to almost irreproducibility of the work from the text alone. See the questions section for my detailed questions on the math.\n- The method claims that the method is unsupervised. However, the unprojection and projection models are pre-trained with supervision, and those models are required in order to evaluate $L_{physics}$. The text claims that the unprojection can be done assuming camera parameters and the plane height, but I don't see how this is the case. Additionally, the projection method, which computes an object mask only from translation/rotation/scale, cannot possibly be unsupervised as the input information lacks shape information. Thus, the claim seems to be false.\n\nQuestions:\n- Questions about the equations:\n    - Please define $alpha_{\\psi}$. I assume it's the same as Attention(). Please keep the notation consistent.\n    - In the dynamics model section, how is \"foreground\" defined for the indicator function? This indicator is supposed to be part of the function that predicts $\\hat{m}^t_k$, which is the predicted foreground mask of object k. Please clarify this.\n    - \"closer than all other objects at the specified pixel location\" <- what does this mean? Can multiple objects be present at the same pixel location? This doesn't appear to be the case when looking at the definition of $\\mathbf{c}_i$ (context of the RNN).\n    - Eq (4):\n        - I believe the LHS is missing notation: $p(\\mathbf{x}_i | \\mathbf{z})$, as this equation should be for a single pixel. Or is it instead missing a product over i? \n        - Where is $k$ in this sum? \n        - The subscript on $m$ should be $k$, shouldn't it? Why is it $i$? Please clarify this.\n        - $\\sigma$ is being overloaded (it was used in the dynamics model section). Please introduce a subscript for clarity.\n        - $c$ is also overloaded; this was context vector for the inference RNN.\n        - Why is the background probability being multiplied, not added? It doesn't seem to be a valid probability otherwise.\n        - Why does probability of $c$ depend on $m$? Shouldn't it depend on $z$? Additionally, it was never defined.\n    - Loss function:\n        - In $L_{image}$, what is Decode($\\mathbf{x}^t$ | $\\mathbf{z}_k^t$)? Is it equal to Eq. (4)? Please clarify this.\n        - In $L_{image}$, what is Decode($\\mathbf{m}^t_k$ | $\\mathbf{z}^t_k$)? Is this meant to be Decode($\\mathbf{c}^t_k$ | $\\mathbf{z}^t_k$)? I'm also assuming $\\mathbf{m}^t_k$ should be multiplied to that term (if so, please fix the parentheses for clarity).\n        - The MONet loss uses a KL term to minimize the loss between the masks produced by the inference module ({$m_k$}) and the masks produced by the VAE decoder ({$c_k$}). According to my deductions of what $L_{image}$ is supposed to be, this now occurs there. Is there a reason for this choice?\n- \"After qualitatively observing object like masks...\" is there a automatic way to do this? Having to baby-sit the training procedure is not ideal. Additionally, how many iterations is this?\n- Is there any reason that POD-Net w/out multi-scale + physics outperforms MONet? The two architectures should be quite similar.\n- What happens if there are inconsistent errors in the segmentation of the subpatches? Does the ordering of the subpatch processing matter?\n\nComments:\n- The physics representation is quite rudimentary, which I imagine will limit the method to simple scenes such as the proposed Moving ShapeNet dataset. More complex scenes will require a higher-fidelity physics representation.\n- If the math/text and claims of no supervision are fixed and/or addressed, I would be willing to increase my score.\n\n\n[1] T. Brox and J. Malik. Object segmentation by long term analysis of point trajectories. In European Conference on Computer Vision (ECCV), 2010.\n\n[2] P. Bideau, A. RoyChowdhury, R. R. Menon, and E. LearnedMiller. The best of both worlds: Combining cnns and geometric constraints for hierarchical motion segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n\n[3] C. Xie, Y. Xiang, Z. Harchaoui, and D. Fox. Object discovery in videos as foreground motion clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.\n\n[4] A. Dave, P. Tokmakov, and D. Ramanan. Towards segmenting anything that moves. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), 2019.\n\nUpdate: Thanks for the authors' response. The authors have clarified the equations, addressed the concerns of claims of no supervision, and provided more experiments and metrics to back the current set of claims. I do believe this paper to be interesting enough for an ICLR paper, and have updated my score accordingly.\nAlso, do note that the motion segmentation works do not assume a single foreground object, they assume an arbitrary amount of them. However, you are correct in that they assume an entire video as input, as opposed to the proposed method which can segment individual frames.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mkWSkpmkzKy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduced an unsupervised approach for 3D physical object segmentation from the video. The proposed algorithm decomposes the scene into several 3D primitive shapes, and learn to generate latent descriptors for a generated model to reason the 3D properties. The system also enforces assumptions that object move according to physics, and eventually produce a scene representation in a self-supervised manner.\n\nThe paper is well motivated and the proposed approach and design are both novel. Experimental results suggest that the proposed approach is effective and the ablation study is helpful too. Moreover, the writings of this paper are clear and the visualizations and diagrams in the paper are informative and easy to follow.\n\nI have some additional comments regarding the paper:\n- In table 1,  the normalized cuts method is a very classical algorithm, but the performance is very good. I was wondering whether authors can provide a few more explanations. \n- Moreover in table 1, why “percentage of objects detected in an image” is so low? (below 1%) I wonder if I missed anything from the descriptions of the paper.\n- In the paper, the approach is under the assumption that each object of the scene is rigid. I understand it's beyond the scope of the paper but the world also consists of many elastic objects and infants are likely to discover those objects well. I wonder if there is a way to extend the system to handle such senarios.\n- Finally, at a high-level, the authors discussed in the related work section that the proposed work is different from existing self-supervised segmentation methods. I think the claims in those section makes sense, but I was wondering if authors can provide some comments on the performance of POD-Net on more challenging scenes (maybe for more complicated backgrounds as a starting point) or if there is a way to compare with existing self-supervised segmentation methods. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "CjoYVZlz7D9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an unsupervised approach for discovering 3D objects from video. Given training videos with objects moving around in a scene, the paper learns to decompose the given images into segments. Each segment is associated with a 3D location (translation, rotation and scale) as well as 3D dynamics (linear and angular velocity), and latent vector capturing the appearance. This appearance is rendered to generate images. A reconstruction loss on the generated images, along with a physics loss on the inferred object locations is used to train the model. The paper conducts experiments on videos rendered using the ShapeNet dataset, and videos of real block tower falling (from Lerer et al.). It measures the accuracy of predicted segments, and compares against other object discovery methods, as well as image segmentation methods. Lastly, the paper reports an experiment for using the proposed model to predict physical plausibility of video stimuli.\n\nStrengths: The proposed unsupervised approach of learning physical properties of 3D objects from videos is novel to the best of my knowledge.\n\nShortcomings: My major issue is with the experimental evaluation. The primary metric used in the paper measures the intersection over union of predicted masks with ground truth masks. This doesn't evaluate the other properties that the paper claims to infer (3D geometry and position of each object). Thus, the current experimental evaluation falls short in evaluating all parts of the proposed model.\n\nFurthermore, even if we limit to evaluation of segmentation masks, I am not sure if the paper is using appropriate baselines. There are a number of papers that tackle unsupervised edge detection (and consequently detection of object segments), see Isola et al. ECCV 2014, Unsupervised Learning of Edges, Li et al. CVPR 2016. A comparison to such unsupervised segmentation techniques should be made.\n\nLastly, the paper is missing details and hard to read: a) at test time, is the segmentation done on a per-frame basis or on the basis of the whole image sequence, b) what does the physics loss capture -- is it trying to induce a first order motion (zero acceleration) on the object, c) why is the learned physics model not used to judge the physical plausibility, d) the paper assumes a parametric form for objects (3D cuboids), but baselines it compares to likely don't (eg: normalized cut doesn't), why are experiments in Figure 7 a fair comparison?\n\nIn summary, I quite like the design of the unsupervised technique to learn about objects and their physical properties, I do not find the experiments convincing.\n\nUpdate: I thank the authors for providing additional 3D metrics, and comparisons to unsupervised 2D segmentation techniques. I have updated my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-MM7Y_YJK72", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed an unsupervised learning model, POD-Net, that learns to discover objects from video. The authors develop an inference model that performs image segmentation and object-based scene decompositions on overlapping sub-patches, and a generative model, which contains an unprojection step, a constant velocity dynamic model and a VAE, to reconstruct the original scene. With the novel dynamic model to predict motions for the 3D object-primitives, the POD-Net learns to segment objects with better physics.\n\n++ Strong points:\nThe paper explores the use of motion cues to train a self-supervised model to extract object-based scene representations from videos. And in the approach that to unproject 2d masks into 3d primitives to compute the motion is novel to me, it allows the proposed approach to better discover objects with physical occupancy on the 2D videos.\n\nOverall, the paper is well written. In particular, the intuition behind the method is described well. The Method section, the POD-Net, is easy to read and understand. And the Evaluation section is well structured, it is clear how the models are setted on different datasets.\n\n++ Concerns:\n\nThe main concern on the paper is that although it claimed itself as a 3D object discovery method, all its evaluations are done on 2D datasets with 2D metrics. Although there are some reasonable improvements shown on these metrics, we do not know what is the capability of this work in terms of recovering 3D segmentation mask and 3D pose. This I consider incomplete for a work that claims its main difference w.r.t. prior work to be getting to 3D object discovery.\n\nThere have been a significant amount of prior work on unsupervised 3D object discovery (many of them on RGB-D) that is missed by the authors:\n\nHerbst et al. Toward Object Discovery and Modeling via 3-D Scene Comparison. ICRA 2011\nKarpathy et al. Object Discovery in 3D scenes via Shape Analysis. ICRA 2013\nMa and Sibley. Unsupervised Dense Object Discovery, Detection, Tracking and Reconstruction. ECCV 2014\n\nDatasets such as \n\nLai et al. A Large-Scale Hierarchical Multi-View RGB-D Object Dataset. ICR 2011\nGeorgakis et al. Multiview RGB-D Dataset for Object Instance Detection. 3DV 2016\n\nexist and they provide ways to evaluate unsupervised 2D-3D object discovery (one can start from RGB and deduct 3D pose and velocity). So I don't think the authors have enough excuses to not show any 3D results.\n\nThe author claims that the POD-Net is an unsupervised method, meanwhile bashing other methods of using pre-training (last paragraph of Section 1). However, their unprojection model and the project model is pre-trained and it is the same kind of supervision as the pre-trained segmentation models in other work.\n\nMinor concerns: \n\n-- In Sec 3.3, \"model surprisal\" doesn't sound to me like proper English, or maybe I'm missing something. If you are introducing a new phrase as a term you probably want to define it first.\n\n-- In Sec 3.1 and 3.2, it is not clear how the author counts object discovery performance w.r.t. the time dimension, e.g. how is it handled if the ground truth mask is completely occluded? Is tracking consistency taken into account?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to do unsupervised discovery of 3D physical objects. The core idea is to decompose the scene into primitives that contain: (a) a segment; (b) 3D position and dynamics; and (c) appearance. These are combined with a physics model and renderer to discover objects/primitives by watching videos; the core supervisory signal used is that one should be able to reconstruct future scenes and that objects/primitives ought to be physically consistent. The system is tested on synthetic data as well as real videos of blocks. 

The reviewers were positive about many aspects but, at the time of submission had a number of concerns. These were, in view of of many of the four reviewers, largely addressed. These are as follows:
- One overarching concern (R3, R4) was the experiments that the paper’s title and motivation focused heavily on 3D but the experiments lacked a 3D experiment of any variety. The authors addressed this by adding 3D IOU and recall. While numbers are low for IOU, this is a challenging area and the AC appreciates this as did R3 and R4.
- Another concern is the data itself (R4,R1). R4 in particular cites the synthetic nature of it as a stumbling block; R1 is similarly concerned about the difficulty of the backgrounds (and the rigidity of the objects). The AC thinks that the data is sufficient for this paper given the overall paper focus, methodological contributions, and particular set of claims. However, the AC is highly sympathetic to R4’s arguments and thinks more realistic real data (beyond the additional data of towers of blocks in front of a white sheet) would substantially improve the impact of the paper and the direction of research.
- The last content-focused concern was disagreement that the system is unsupervised (R2,R4). The authors have addressed this with experiments using a hard-coded system that uses a heuristic based on the bottom coordinate, which obtains good results as well. All reviewers with this concern seem satisfied although the AC would note this assumes a single ground plane, which ties into concerns about the data (although this is a small nitpick).
- R2 had substantial concerns about the legibility and reproducibility of the paper. These have been largely addressed in the revision, as far as the AC can tell.

The paper is an good contribution on a challenging and important problem. While the AC shares some of R4’s concerns about the data (and indeed how data difficulty and method interact), the AC finds the revised paper compelling and recommends acceptance.
Decision: Accept (Poster)