Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposed an unsupervised learning model, POD-Net, that learns to discover objects from video. The authors develop an inference model that performs image segmentation and object-based scene decompositions on overlapping sub-patches, and a generative model, which contains an unprojection step, a constant velocity dynamic model and a VAE, to reconstruct the original scene. With the novel dynamic model to predict motions for the 3D object-primitives, the POD-Net learns to segment objects with better physics.

++ Strong points:
The paper explores the use of motion cues to train a self-supervised model to extract object-based scene representations from videos. And in the approach that to unproject 2d masks into 3d primitives to compute the motion is novel to me, it allows the proposed approach to better discover objects with physical occupancy on the 2D videos.

Overall, the paper is well written. In particular, the intuition behind the method is described well. The Method section, the POD-Net, is easy to read and understand. And the Evaluation section is well structured, it is clear how the models are setted on different datasets.

++ Concerns:

The main concern on the paper is that although it claimed itself as a 3D object discovery method, all its evaluations are done on 2D datasets with 2D metrics. Although there are some reasonable improvements shown on these metrics, we do not know what is the capability of this work in terms of recovering 3D segmentation mask and 3D pose. This I consider incomplete for a work that claims its main difference w.r.t. prior work to be getting to 3D object discovery.

There have been a significant amount of prior work on unsupervised 3D object discovery (many of them on RGB-D) that is missed by the authors:

Herbst et al. Toward Object Discovery and Modeling via 3-D Scene Comparison. ICRA 2011
Karpathy et al. Object Discovery in 3D scenes via Shape Analysis. ICRA 2013
Ma and Sibley. Unsupervised Dense Object Discovery, Detection, Tracking and Reconstruction. ECCV 2014

Datasets such as 

Lai et al. A Large-Scale Hierarchical Multi-View RGB-D Object Dataset. ICR 2011
Georgakis et al. Multiview RGB-D Dataset for Object Instance Detection. 3DV 2016

exist and they provide ways to evaluate unsupervised 2D-3D object discovery (one can start from RGB and deduct 3D pose and velocity). So I don't think the authors have enough excuses to not show any 3D results.

The author claims that the POD-Net is an unsupervised method, meanwhile bashing other methods of using pre-training (last paragraph of Section 1). However, their unprojection model and the project model is pre-trained and it is the same kind of supervision as the pre-trained segmentation models in other work.

Minor concerns: 

-- In Sec 3.3, "model surprisal" doesn't sound to me like proper English, or maybe I'm missing something. If you are introducing a new phrase as a term you probably want to define it first.

-- In Sec 3.1 and 3.2, it is not clear how the author counts object discovery performance w.r.t. the time dimension, e.g. how is it handled if the ground truth mask is completely occluded? Is tracking consistency taken into account?

review_title: A so-called 3D object discovery method without any 3D evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduced an unsupervised approach for 3D physical object segmentation from the video. The proposed algorithm decomposes the scene into several 3D primitive shapes, and learn to generate latent descriptors for a generated model to reason the 3D properties. The system also enforces assumptions that object move according to physics, and eventually produce a scene representation in a self-supervised manner.

The paper is well motivated and the proposed approach and design are both novel. Experimental results suggest that the proposed approach is effective and the ablation study is helpful too. Moreover, the writings of this paper are clear and the visualizations and diagrams in the paper are informative and easy to follow.

I have some additional comments regarding the paper:
- In table 1,  the normalized cuts method is a very classical algorithm, but the performance is very good. I was wondering whether authors can provide a few more explanations. 
- Moreover in table 1, why “percentage of objects detected in an image” is so low? (below 1%) I wonder if I missed anything from the descriptions of the paper.
- In the paper, the approach is under the assumption that each object of the scene is rigid. I understand it's beyond the scope of the paper but the world also consists of many elastic objects and infants are likely to discover those objects well. I wonder if there is a way to extend the system to handle such senarios.
- Finally, at a high-level, the authors discussed in the related work section that the proposed work is different from existing self-supervised segmentation methods. I think the claims in those section makes sense, but I was wondering if authors can provide some comments on the performance of POD-Net on more challenging scenes (maybe for more complicated backgrounds as a starting point) or if there is a way to compare with existing self-supervised segmentation methods. 
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper proposes a deep network architecture to attack the important problem of discovering physical objects from observing videos only. The architecture is mainly an extension of the MONet architecture with a physics loss and a multi-scale inference scheme. The physics loss encourages the inference module (that masks out the objects) to be more consistent with the observed motion in the scene. Additionally, the multi-scale scheme, motivated by human foveation patterns, shows better empirical performance. The method outperforms reasonable baselines on both a synthetic dataset and a real-world dataset.

The immediately related works are summarized nicely. However, I believe it might be worth mentioning other methods that discover objects by using motion, such as motion segmentation. Traditional examples include [1], and more recent examples include [2,3,4].

Overall, I think the paper poses an interesting contribution with interesting results. However, the paper is hindered by the confusing equations and text, making it incredibly difficult to reproduce from the paper alone, and what I believe to be false claims of unsupervised learning (see weaknesses for details).

Strengths:
- Empirical results show that the method significantly outperforms reasonable baselines.
- The proposed physics loss is well-motivated and leads to improved empirical performance on both datasets. Additionally, the multi-scale scheme, while simple, surprisingly leads to large gains.
- The connection to violation-of-expectation is interesting, and the method shows pretty decent performance compared to the ADEPT upper bound.

Weaknesses:
- The equations and the text describing them are very confusingly written, leading to almost irreproducibility of the work from the text alone. See the questions section for my detailed questions on the math.
- The method claims that the method is unsupervised. However, the unprojection and projection models are pre-trained with supervision, and those models are required in order to evaluate $L_{physics}$. The text claims that the unprojection can be done assuming camera parameters and the plane height, but I don't see how this is the case. Additionally, the projection method, which computes an object mask only from translation/rotation/scale, cannot possibly be unsupervised as the input information lacks shape information. Thus, the claim seems to be false.

Questions:
- Questions about the equations:
    - Please define $alpha_{\psi}$. I assume it's the same as Attention(). Please keep the notation consistent.
    - In the dynamics model section, how is "foreground" defined for the indicator function? This indicator is supposed to be part of the function that predicts $\hat{m}^t_k$, which is the predicted foreground mask of object k. Please clarify this.
    - "closer than all other objects at the specified pixel location" <- what does this mean? Can multiple objects be present at the same pixel location? This doesn't appear to be the case when looking at the definition of $\mathbf{c}_i$ (context of the RNN).
    - Eq (4):
        - I believe the LHS is missing notation: $p(\mathbf{x}_i | \mathbf{z})$, as this equation should be for a single pixel. Or is it instead missing a product over i? 
        - Where is $k$ in this sum? 
        - The subscript on $m$ should be $k$, shouldn't it? Why is it $i$? Please clarify this.
        - $\sigma$ is being overloaded (it was used in the dynamics model section). Please introduce a subscript for clarity.
        - $c$ is also overloaded; this was context vector for the inference RNN.
        - Why is the background probability being multiplied, not added? It doesn't seem to be a valid probability otherwise.
        - Why does probability of $c$ depend on $m$? Shouldn't it depend on $z$? Additionally, it was never defined.
    - Loss function:
        - In $L_{image}$, what is Decode($\mathbf{x}^t$ | $\mathbf{z}_k^t$)? Is it equal to Eq. (4)? Please clarify this.
        - In $L_{image}$, what is Decode($\mathbf{m}^t_k$ | $\mathbf{z}^t_k$)? Is this meant to be Decode($\mathbf{c}^t_k$ | $\mathbf{z}^t_k$)? I'm also assuming $\mathbf{m}^t_k$ should be multiplied to that term (if so, please fix the parentheses for clarity).
        - The MONet loss uses a KL term to minimize the loss between the masks produced by the inference module ({$m_k$}) and the masks produced by the VAE decoder ({$c_k$}). According to my deductions of what $L_{image}$ is supposed to be, this now occurs there. Is there a reason for this choice?
- "After qualitatively observing object like masks..." is there a automatic way to do this? Having to baby-sit the training procedure is not ideal. Additionally, how many iterations is this?
- Is there any reason that POD-Net w/out multi-scale + physics outperforms MONet? The two architectures should be quite similar.
- What happens if there are inconsistent errors in the segmentation of the subpatches? Does the ordering of the subpatch processing matter?

Comments:
- The physics representation is quite rudimentary, which I imagine will limit the method to simple scenes such as the proposed Moving ShapeNet dataset. More complex scenes will require a higher-fidelity physics representation.
- If the math/text and claims of no supervision are fixed and/or addressed, I would be willing to increase my score.


[1] T. Brox and J. Malik. Object segmentation by long term analysis of point trajectories. In European Conference on Computer Vision (ECCV), 2010.

[2] P. Bideau, A. RoyChowdhury, R. R. Menon, and E. LearnedMiller. The best of both worlds: Combining cnns and geometric constraints for hierarchical motion segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[3] C. Xie, Y. Xiang, Z. Harchaoui, and D. Fox. Object discovery in videos as foreground motion clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[4] A. Dave, P. Tokmakov, and D. Ramanan. Towards segmenting anything that moves. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), 2019.

Update: Thanks for the authors' response. The authors have clarified the equations, addressed the concerns of claims of no supervision, and provided more experiments and metrics to back the current set of claims. I do believe this paper to be interesting enough for an ICLR paper, and have updated my score accordingly.
Also, do note that the motion segmentation works do not assume a single foreground object, they assume an arbitrary amount of them. However, you are correct in that they assume an entire video as input, as opposed to the proposed method which can segment individual frames.
review_title: Interesting contribution, but confusing equations/text

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an unsupervised approach for discovering 3D objects from video. Given training videos with objects moving around in a scene, the paper learns to decompose the given images into segments. Each segment is associated with a 3D location (translation, rotation and scale) as well as 3D dynamics (linear and angular velocity), and latent vector capturing the appearance. This appearance is rendered to generate images. A reconstruction loss on the generated images, along with a physics loss on the inferred object locations is used to train the model. The paper conducts experiments on videos rendered using the ShapeNet dataset, and videos of real block tower falling (from Lerer et al.). It measures the accuracy of predicted segments, and compares against other object discovery methods, as well as image segmentation methods. Lastly, the paper reports an experiment for using the proposed model to predict physical plausibility of video stimuli.

Strengths: The proposed unsupervised approach of learning physical properties of 3D objects from videos is novel to the best of my knowledge.

Shortcomings: My major issue is with the experimental evaluation. The primary metric used in the paper measures the intersection over union of predicted masks with ground truth masks. This doesn't evaluate the other properties that the paper claims to infer (3D geometry and position of each object). Thus, the current experimental evaluation falls short in evaluating all parts of the proposed model.

Furthermore, even if we limit to evaluation of segmentation masks, I am not sure if the paper is using appropriate baselines. There are a number of papers that tackle unsupervised edge detection (and consequently detection of object segments), see Isola et al. ECCV 2014, Unsupervised Learning of Edges, Li et al. CVPR 2016. A comparison to such unsupervised segmentation techniques should be made.

Lastly, the paper is missing details and hard to read: a) at test time, is the segmentation done on a per-frame basis or on the basis of the whole image sequence, b) what does the physics loss capture -- is it trying to induce a first order motion (zero acceleration) on the object, c) why is the learned physics model not used to judge the physical plausibility, d) the paper assumes a parametric form for objects (3D cuboids), but baselines it compares to likely don't (eg: normalized cut doesn't), why are experiments in Figure 7 a fair comparison?

In summary, I quite like the design of the unsupervised technique to learn about objects and their physical properties, I do not find the experiments convincing.

Update: I thank the authors for providing additional 3D metrics, and comparisons to unsupervised 2D segmentation techniques. I have updated my rating.
review_title: Un-compelling experimental evaluation, missing details

Meta-review: The paper proposes to do unsupervised discovery of 3D physical objects. The core idea is to decompose the scene into primitives that contain: (a) a segment; (b) 3D position and dynamics; and (c) appearance. These are combined with a physics model and renderer to discover objects/primitives by watching videos; the core supervisory signal used is that one should be able to reconstruct future scenes and that objects/primitives ought to be physically consistent. The system is tested on synthetic data as well as real videos of blocks. 

The reviewers were positive about many aspects but, at the time of submission had a number of concerns. These were, in view of of many of the four reviewers, largely addressed. These are as follows:
- One overarching concern (R3, R4) was the experiments that the paper’s title and motivation focused heavily on 3D but the experiments lacked a 3D experiment of any variety. The authors addressed this by adding 3D IOU and recall. While numbers are low for IOU, this is a challenging area and the AC appreciates this as did R3 and R4.
- Another concern is the data itself (R4,R1). R4 in particular cites the synthetic nature of it as a stumbling block; R1 is similarly concerned about the difficulty of the backgrounds (and the rigidity of the objects). The AC thinks that the data is sufficient for this paper given the overall paper focus, methodological contributions, and particular set of claims. However, the AC is highly sympathetic to R4’s arguments and thinks more realistic real data (beyond the additional data of towers of blocks in front of a white sheet) would substantially improve the impact of the paper and the direction of research.
- The last content-focused concern was disagreement that the system is unsupervised (R2,R4). The authors have addressed this with experiments using a hard-coded system that uses a heuristic based on the bottom coordinate, which obtains good results as well. All reviewers with this concern seem satisfied although the AC would note this assumes a single ground plane, which ties into concerns about the data (although this is a small nitpick).
- R2 had substantial concerns about the legibility and reproducibility of the paper. These have been largely addressed in the revision, as far as the AC can tell.

The paper is an good contribution on a challenging and important problem. While the AC shares some of R4’s concerns about the data (and indeed how data difficulty and method interact), the AC finds the revised paper compelling and recommends acceptance.
Decision: Accept (Poster)