Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the problem of predicting the segmentations and poses (position + yaw orientation) of multiple objects, given the image of a scene. The paper introduces a method that is trained without supervision for the segmentations, similar to several other recent object-centric models. In contrast to these existing models, the method proposed in this paper additionally estimates the 3D location of each object by predicting a depth map and classifies the yaw angle by representing the pose domain as equally-spaced bins. To do so, during training the method operates on a short clip of the scene recorded by a moving camera, and uses self-supervision by predicting the scene’s image at the next time step. At test time, the model is able to infer a representation of each object in the scene and segment them given a single image of the scene.
main_review: # Strength

1. The paper tackles the difficult problem of learning to segment objects from an image using no supervision during training.
2. The problem setting and motivation for this task are explained clearly. A detailed description of the method, along with a pseudocode of the learning algorithm is provided in the paper.
3. The paper introduces a new synthetic dataset of images taken from scenes with multiple objects with varying shapes and textures (11 and 15, respectively).
4. Figures 3A-D are very helpful explaining the quantitative performance of the method in relation to the baselines. Figures 3E-G are also helpful showing the failure modes of the proposed method.
 
# Weakness
1. I am not fully convinced that the comparison to the baselines is entirely fair. If I understand correctly, the rest of the methods were trained on single images without having access to previous and next frames. While I appreciate the method’s usage of consecutive frames as part of the supervision, I think this should be stated clearly when comparing with the baselines -- to avoid any overclaims.

    SynSin [1], for example, also predicts the next frame’s RGB and depth images without using additional supervision. Similar to the proposed method in this paper, SynSin synthesizes future images given their camera poses by warping the current frame using differentiable rendering. Combining an object-centric approach (like slot-attention) with such a method that performs future image prediction would make a fairer comparison, in my opinion.

    [1] Wiles, O., Gkioxari, G., Szeliski, R. and Johnson, J., 2020. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7467-7477).

2. I think there are some key references missing in the paper too. For instance, a paper from last year also learns an object-centric representation in an unsupervised fashion to decompose the objects in a scene while estimating their poses in 3D [2]. How does [2] compare to the method presented in this paper? What are the main differences between them?

    [2] Henderson, P. and Lampert, C.H., 2020. Unsupervised object-centric video generation and decomposition in 3D. Advances in Neural Information Processing Systems, 33.

3. The results are reported using only the dataset introduced in the paper. I suggest including results from datasets like the Objects Room or CLEVR, where the existing methods (MONet, Slot-attention, Genesis, [2]) have already reported results on. 

4. There are no ablation studies reported in the paper. Some parts of the loss function seem redundant judging from their current descriptions. I think it would be very helpful either presenting additional results by ablating the loss function / part of the model, or detail in the text why the method needs each of those components.

# Recommended Decision

I think in its current form the paper is not ready to be published. I strongly encourage the authors to clarify the positioning of this paper in relation to the state-of-the-art (see my comment on weaknesses). I vote for [reject, not good enough] for now, but I would be happy to increase my rating once the authors clear up my concerns.

# Additional Feedback

1. It should be \hat(m) instead of \hat(t) in the L_spatial loss’ third term.

2. What is x_rand? If they are randomly sampled point positions then what is their underlying distribution?

3. Have you tested the method on scenes with more than 3 objects? Slot attention, for instance, is able to segment up to 9 objects. Since the paper introduces a new dataset, I would’ve hoped to see a more challenging benchmark with a wider variety of objects and number of instances.

4. Also, the dataset assumes that the camera moves with constant pose change. Have you tested the motion prediction (position, orientation and their time-derivatives) in the presence of different camera velocities? If not, are there any limitations of the method that prevents it?

5. I suggest adding the equivalents of Figure 3G for each baseline in the appendix.

6. What is the number of bins for the yaw angle prediction, b? Have you tried using a continuous representation for the rotation?

7. What does the predicted image (I’) for the next time-step look like? I suggest including more qualitative results to the paper for evaluating the warping function.

8. I think it is a good idea to introduce a more diverse benchmarking dataset for learning object-centric representations. However, I think the dataset proposed in the paper should be further expanded by including more daily life objects, instead of just geometric primitives like spheres, prisms and cones. I suggest taking a look at datasets like YCB or ShapeNet to include more realistic objects to your dataset.

summary_of_the_review: All in all, I think the paper attempts to tackle a very challenging problem. The method looks sound and the results might be interesting to the community in learning object-centric representations. However, there are some major concerns I have, mainly about the position and novelty of this paper with respect to a paper from last year, and the lack of results from datasets the state-of-the-art methods report on.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to learn how to parse 3-frame videos into object-centric representations, which include segmentation masks and 3D positions and yaws for those objects frame-by-frame, as well as an image representation of the background, and an overall depth map. This is accomplished with a depth network, an object network with an LSTM at the bottleneck to iteratively pick out objects and their positions and yaws, and a decoder to provide segmentations, a warping/re-compositing operation that pastes the inferred objects at their estimated positions for the NEXT frame (i.e., with a constant-velocity assumption), and finally an "imagination" network which refines the estimated image. The model learns with a combination of 4 losses, which include image reprojection/prediction, depth consistency, a spatial term that includes consistency and randomness (though I have complaints about this), and finally a penalty term that discourages object probabilities from being zero. The paper also introduces a new synthetic dataset where prior methods do badly, and the proposed method does slightly better. The learned depth maps look good, but this is perhaps expected because camera poses are known.

main_review: Overall, this paper is messily written, and proposes something that only works marginally better than prior methods, on a synthetic toy dataset. The performance in Table 1 illustrates this: the standard deviation of the segmentation IOU (0.34) is about even with the average IOU (0.35)! Looking at the qualitative results in Figure 2, it seems the model often misses objects completely, and produces segmentations that are fractional and have holes. I appreciate that the baselines are doing badly here also, but slot attention had very similar ARI-fg (0.42+-0.35 vs 0.46+-0.39). Statistically maybe these are equivalent. Is it possible at least to show that this method does better on the datasets where those previous papers managed to work (like CLEVR and those deepmind shapes datasets)? In general this area of work seems stuck in a setup where all methods work well on different toy data, but no methods work on real images or videos. 

It seems like the pose estimation is not working at all, judging by Figure 3-E. The discussion mentions this too: "object spatial location is inferred more easily than object pose (which we have not fully investigated), thus the predicted warping relies more on object translation than rotation". Maybe the object pose estimation can be removed from the paper entirely, to make things simpler. 

"our framework can easily generalize to 3D pose" I am sure the formulation can easily be extended to capture this, but I would prefer that the wording here be a bit more careful, to not suggest that the model is expected to work when pitch and roll are unknowns as well. (If you do expect it to work, please try it out and add the results to the paper.)

"At time t, given the location of the camera o(t) ∈ R3 and its facing direction α(t)" Does this mean the camera intrinsics and extrinsics are assumed known? It would be great to say this directly. The discussion section supports this interpretation, saying "additional information required is that of observer’s self-motion, which is available both in the brain as efference copy and easily accessible in vehicles". I do not really buy the argument about the brain, or easy accessibility in vehicles. Self-driving vehicles usually register themselves to a known map; inferring pose from odometry alone causes drift. 

The LSTM reading objects from the full-frame encoding sounds like a very weak part of the model. Why not, for example, use a standard object detector, like MaskRCNN? I know you want to be self-supervised, but then why not self-supervise a well-known architecture that is proven to work, instead of inventing a new one?

"The location prediction is restricted to the range of possible value within the virtual environment by logistic function and linear scaling" I don't know what this means. 

The imagination network is clumsily introduced. The first instance of it is already "the imagination network", and the motivation for it is only written in the caption of Figure1. I found a helpful description later on in page5. These things should be re-arranged.

The description of the unprojection of a 2D coordinate into 3D space looks odd to me. Where is this coming from? Given that i,j are coordinates, what do |i| and |j| represent? Normally we start from x=fX/Z, and then invert this to X=Zx/f, where x is 2D and X is 3D, and f is the focal length. I am also not sure how the first term and second term here are able to multiply, since the first term is a 2-tuple (i,j) and the third term is a 3-tuple (i,d,j). 

I also got lost in the angular velocity equation. It seems the sum is over all \gamma1, all \gamma2, and all differences of the two within 2\pi of \omega. This is too many things to sum over! It seems like you won't end up with a valid probability distribution. I am probably misreading the notation here. In any case, why is this probability distribution useful? The object-based imagination network seems to require a scalar here for the rotational speed. So why not take the expectation of the first pose distribution, expectation of the second, and then take a difference?

It is interesting that you do not use depth supervision, but for toy settings like this I think it is OK to assume depth is known, and focus on other hard parts, like segmentation and tracking.

I need some help understanding L_{spatial}. The second term is described as a contrastive loss, but it's a difference between known positions and random positions. Why is this a good idea, and why is it contrastive? Normally a contrastive loss compares two estimates and pushes them apart (rather than pulling every estimate to random).

I was surprised that the evaluation talked about a model called "OPPLE" ("Only OPPLE shows a bi-modal distribution."). Apparently this is the name of the proposed model, and the place to learn this is the caption of Figure 1! Please do not put critical information exclusively in figure captions. 


Typos: 
- probability mess -> probability mass 
- generated dataset -> generated a dataset
- states-of-art -> state-of-the-art
- network appear to -> network appears to
- intersectin (in fig3) -> intersection
- Figure 1 is never referred to in the text
summary_of_the_review: I think the paper is not quite ready for publication. The method does not work particularly well, a part of the method (object pose estimation) seems to be not working at all, and the evaluation is only on a new toy dataset and does not include evaluation on established datasets. Also the text contains too much notation, and has some parts mixed up (with terms being used before they are introduced), but I think this can be fixed easily.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an unsupervised object-centric scene representation technique that can decompose a scene into multiple objects (and segment the scene) and infer their 3D locations and pose. The overall setup is very similar to earlier models like MONet but this model works on sequences of images, more precisely on 3 consecutive images. It uses the first two images to infer the 3D position and pose of objects and combining this with known camera motion tries to predict the last (third) image. The main contribution here is an optical flow based method to warp the image at time t using the predicted object location/pose/depth to predict (some of) the pixels in image at time t+1.

In more detail, the object extraction network outputs the location and pose of each object. And a separate depth perception network outputs the depth for each pixel in the image. The location and pose of objects are used to estimate the velocity of each object (e.g., by subtracting the position at t-1 from position at t. note this requires matching each object at time t-1 to object in time t, which they do using a soft-matching approach). These along with the depth information are then used to warp the image at t to predict pixels in image at time t+1. This is possible only for a subset of the pixels so for the rest, they use a separate "imagination" network that takes in object information and predicts the color/depth and object masks at t+1. The predictions from warping and imagination network are then combined to form the final predicted color and depth images.

To train the model, they require images and camera motion, and use a combination of losses: reconstruction loss on predicted and ground truth image, self-supervised losses on object location, pose, and depth.


main_review: Overall, I found the paper quite interesting. I think the optical flow based warping to predict some subset of the pixels in the next timestep is in itself an important contribution and this paper would be a good addition to the emerging literature on unsupervised object-centric models.

I think the main concern with the paper is limited experimental evaluation. The model is evaluated only on a single, rather simple dataset (that was generated by the authors). I know these models cannot usually handle complicated datasets so it's fine to have a simple dataset but I'd have liked to see the model evaluated on some of the datasets that other competing models were evaluated on. Some of those datasets might not have camera information etc. but I'm sure there are other datasets that have the necessary information (e.g., see [1], [2] for some potential datasets). 

Also, the authors mention that their technique is the first to infer 3D position of objects while segmenting images. If I'm not mistaken, [1] also does both and would be a great model to compare against.

Other notes

- Std deviations in table 1 are too large. Is this a typo? If not, it looks like all models are doing equally well.
- I don't have any specific recommendations here but I found the model description a bit hard to follow. It might be a good idea to do another pass and see if it can be organized better. For example, the fact that warping and imagination are combined to get the final image can be mentioned earlier so the reader knows where/how the imagination network is used.
- There are many inline equations; these make it difficult to parse the text visually. It would be nice to take these out of the text and split the long section 2.3.1 to subsections and mark these.
- Figure 1 is great but again hard to parse. Perhaps adding variable names (i.e., x, p, z etc.) to the figure might make it easier to understand what goes in and out of each network.

[1] Henderson, Paul, and Christoph H. Lampert. 2020. “Unsupervised Object-Centric Video Generation and Decomposition in 3D.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/2007.06705.
[2] Kabra et al., 2021. "SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition", https://arxiv.org/abs/2106.03849
summary_of_the_review: Overall, I think the paper is quite interesting and would be of interest to the community. However, the empirical evaluation is very limited and this makes it difficult to evaluate the full merit of the proposed approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Inspired by ideas about how humans learn about objects, the authors detail a system to train a neural network to perceive generic objects using image triplets where objects move and the viewer also can move.  The viewer's motion is provided as an input. Object perception by parts of the network operating on the first two time points is rewarded by predicting what is seen at the third time point (the training signal). Having been trained, the object perception part of the system, which is a relatively basic neural network, can segment them from a single image.  This new setup requires different data that what has been used in this space, and the authors contribute a synthetic dataset as well.

main_review: Strengths

I like the general thrust of this paper. While the basic idea has been around in cognitive psychology, which inspired the authors, I am not aware of any significant implementation of it. This is a nice first step. 

To make this work, the authors develop some algorithmic bits that might be useful for followup work. 

The results compare well to others in this space, showing that the training strategy has real promise. In some sense, the methods are not really comparable as the other methods do not have a viable way to make use of the additional training data. (Also see comment about number of objects below). 

Weaknesses

The paper is harder to read than needed. I appreciate there is a lot of stuff going, and I believe I was able to get most of it after a few iterations, so the lack of clarity is not extreme. 

It is not clear how differing numbers of objects are handled. It seems that the number of objects might simply be provided (K=3 in the dataset)? But if the number of objects is known, then the comparison to other work that infers it might not be fair. K is in the pseudo code, but does not seam inferred. Should K be an input. 

A few more details on the LSTM would help.

The paper could use some polishing. Figure 1, which is informative and key, could be tidied up. Also, I am guessing that the "Objects" box is the LSTM. The English could be improved in places, and there are a number of grammar errors (e.g., the first two uses of "pixel" in 2.3.1 should be "pixels", and 'This last two terms mean' on page 5).

The authors do not say whether they will release their code. 

Comments

A clear limitation of this work is that the data is synthetic and very simple (although perhaps more complex than other work in this space). The authors acknowledge this in their discussion. While this might be the standard for this sub-area, real data from a robot or car should be relatively easy to get. Easy block-world-like real data might be better for pushing the work than adding more texture and diverse lighting in synthetic data. 

Using just two time points is both a strength and a weakness. With just two, there is likely to be a lot of ambiguity between translation and rotation, especially if you generalize to more than one pose parameter. Looking at the effect of larger training sequences would be interesting. 

While the part of the system that is deployed at testing is a simple network, there are a number of hand constructed components (e.g., the warping function) that make use of what we know about cameras, it would be more interesting to see those learned.  

summary_of_the_review: This is a good first step in this direction, and should inspire follow up work. The technical innovation is sensible. The results are good.
review_title: null

Meta-review: This paper tackles the difficult problem of learning to segment objects from an image using no supervision during training. The paper is clearly written and a new synthetic dataset is made available. Unfortunately, the reviewers raised a number of issues with the submission (missing citations and comparison to relevant related work / additional baselines  + ablation studies / missing empirical evaluation of the proposed method on standard dataset beyond the toy dataset proposed by the authors). The paper received 1 reject, 2 marginal rejects and 1 accept but even the positive reviewer agreed that these were limitations. The authors also conceded to these limitations and initiated experiments that are starting to address the reviewers' comments. At this time, the results of these experiments remain incomplete and hence most reviewers agree that the paper should go through another round of reviews before it is publishable. I thus recommend this paper be rejected in the hope that a subsequent revision will make it a much stronger contribution.
Decision: Reject