Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: SUMMARY

The paper presents a method to decompose scenes into its constituent objects. This is done with a generative framework that generates both bounding boxes and segmentation masks for each object. It relies on several previously existing technologies. Its main contribution is enforcing consistency between bounding boxes and segmentation masks.

PROS

* Outperforms the baselines.

CONS

* The paper can be hard to read.
* Contributions seem minor.
* Good results, but on two toy datasets only.

COMMENTS

The writing should be improved, as the paper can be hard to follow. One one hand, this includes broken sentences ("Inspired by the observation that foreground segmentation masks and bounding boxes both contain object geometric information and should be consistent with each other."), grammatical errors ("It proves that there are still many useful information can be discovered in those unlabeled data."), and sentences which are just hard to parse ("In the former type of models, the scene is encoded into the object-oriented disentangled spatial and appearance encoding explicitly."). On the other, the authors cite many concepts without introducing them in the paper (stick breaking, spatial broadcast decoder, multi-otsu thresholding, etc) which makes it non self-contained.

The paper presents what seem like engineering improvements over previous works (e.g. combining bounding boxes and segmentation masks) by adding more components to the framework, which is quite convoluted (see Fig. 1: ResNet, FPN, RPN, segmentation, VAEs, etc). It is hard to know where performance comes from, despite the ablation tests.

The experiments are limited to two toy datasets with a fixed number of simple objects (which must be known beforehand), which show no background interference and little occlusion.

In all, I do not think it meets the ICLR bar.

I am not an expert on the topic so I may have missed relevant datasets/baselines.

Detail: "Region of Interest" introduced after ROI has been mentioned several times.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a variation of the MONet model where an additional Region Proposal Network generates bounding boxes for various objects in the scene. An additional loss is introduced during training to make the segmentations produced by the MONet segmenter consistent with the proposed bounding boxes. Results are demonstrated on multi d-Sprites and CLEVR with modest performance gains.

The paper is somewhat middle of the road in most aspects - the proposed method is, in my opinion, only a slight variation on the existing MONet model. Though presented clearly, I don't feel that adding that loss makes the model better in any fundamental way, even if performance numbers are slightly better in some circumstances. Furthermore, though there is some ablation analysis, I feel the level of analysis of the results is sub-par - when a relatively simple variation of a model like here is proposed I would want to see an effort to analyse the contribution beyond how it affects the numbers - do we learn anything new by introducing the variation? does it tell us some fallacy or failure of the original model and if so, does it fix it? These are lacking here.

A few more concrete points:

* In Table 1 - why is the ResNet18 + FPN missing from d-Sprites dataset? This ablation is probably the single most important experiment present in the paper - I would want to see it reported on both datasets.
* In general - I feel the performance on these datasets is quite saturated and I hope to see results on more challenging data in the future - the proposed method included
* There is very little discussion about the choice of hyper-parameters in the paper - how were they chosen? is the system sensitive to these choices?

Post rebuttal comments:

Thank you authors for the detailed response - I think some of of my concerns have been answered - the paper may be a valid contribution to the community and I am raising my score.
review_title: A slight variation of existing models with modest performance boost and lacking analysis.

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors introduce a region-based approach for unsupervised scene decomposition. It extends the previous MONet by introducing the region-based self-supervised training. Instead of purely generating foreground masks in an RNN, they simultaneously predict the bounding boxes and segmentation masks using a Faster-RCNN-based framework. The supervision comes from the object reconstruction loss and the self-supervised loss of classification and regression of anchors in the RPN module. The experiments and comparisons are only conducted on the synthetic CLEVR and Multi-dSprites dataset. 

[Paper strength]
- The paper is well motivated, and the proposed approach seems to be reasonable.
- The self-supervised idea is interesting that uses the segmentation mask to get the pseudo bounding box label for object detection, which could ensure the consistency of object mask and bounding box. 

[Paper Weakness]
1. The self-supervision between segmentation masks and detection bounding boxes is the main contribution. While incorporating the self-supervision into MONet is meaningful and interesting, the overall novelty does not look significant.

2. Clarification of Methods:
- How to learn $m_k$ in a self-supervise way is unclear? MONet uses spatial attention to identify the most salient object one by one, which makes senses. But here you segment all the objects in one step. How could this be possible in an unsupervised way? From the example in Fig. 2, it looks R-MONet could pick out some small and far-away objects first, which is not intuitive.
- In Faster-RCNN, the positive/negative samples are selected by calculating the IoU threshold between the sampled bbox and the ground truth bbox. However, in this self-supervised approach, there is no ground truth bbox. Although the authors proposed to use the pseudo bbox from the segmentation mask $m_k$, how could this be reliable since $m_k$ is likely of poor quality, especially at the initial stage.
- The selected K value is unclear. In the original MONet, the spatial attention network is an RNN-like structure, they decompose the scene step-by-step. Therefore, they define the K steps. However, in this Faster-RCNN-based framework, the objects are selected in one step, how to select the K-1 objects in all proposals?  

3. Results:
- Most of the results are with toy images. There is no result on real images.
- There is no result to really demonstrate the effectiveness of the self-supervised loss. The author should compare their R-MONet(UNet) with the baseline of R-MONet(UNet) w/o the self-supervised loss, i.e. removing the object detection branch. Another missing baseline is MONet(UNet).
- In Table 1, the MONet (ResNet18+FPN) is 10 percent lower than the original MONet. Does this means the network structure has a greater influence on the performance than the self-supervision component.
- In Table 1, the R-MONet(Lite) performs worse. Once again, I guess this poor performance comes from the network structure, as the input image is 64*64, the Resnet downsamples the image to a very low resolution which losses the spatial information.
- The visual results cannot demonstrate the advantage of the proposed approach. For example, in Figure 3, the visual performance of MONet and R-MONet(UNet) are quite similar.

---
---
Update:
In general, I am happy with the authors' responses. They did show the advantage of the introduced self-supervised loss. Although the self-supervised loss is intuitive, incorporating it into MONet is non-trivial and it does outperform MONet. Despite that such self-supervised works are hard to work on real scenes, this paper does have some merits. I am willing to increase my rating.


review_title: An extension of MONet for unsupervised scene decomposition with some interesting ideas but lack of sufficient explanations and experiments.

Meta-review: The paper has good contributions to a challenging problem, leveraging a Faster-RCNN framework with a novel self-supervised learning loss. However reviewer 4 and other chairs (in calibration) considered that the paper does not meet the bar for acceptance. The other reviewers did not champion the paper either, hence i am proposing rejection. 

Pros:
- R1 and R3 agree that the proposed model improves over related models such as MONET.
- The value of the proposed self-supervised loss connecting bounding boxes and segmentations is well validated in experiments.

Cons:
- R4 gives good suggestions that may be useful to reach a broader readership, namely introducing more of the concepts used in the paper., e.g. "stick breaking, spatial broadcast decoder, multi-otsu thresholding" so it becomes more self-contained. R4 also suggests improving the writing more generally.
- R4 still finds the proposed "method quite complex yet derivative" after the rebuttal.
- All reviewers complain about lack of experiments in real data, but the authors did revise their paper and add some coco results in the appendix. These could be part of the main paper in a future version.
Decision: Reject