Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper studies the problem of unsupervised scene decomposition with a foreground-background probabilistic modeling framework. Building upon the idea from the previous work on probabilistic scene decomposition [Crawford & Pineau 2019], this paper further decomposes the scene background into a sequence of background segments. In addition, with the proposed framework, scene foreground-background interactions are decoupled into foreground objects and background segments using chain rules. Experimental evaluations have been conducted on several synthetic datasets including the Atari environments and 3D-Rooms. Results demonstrate that the proposed method is superior to the existing baseline methods in both decomposing objects and background segments.

Overall, this paper studies an interesting problem in deep representation learning applied to scene decomposition. Experimental results demonstrated incremental improvements over the baseline method [Crawford & Pineau 2019] in terms of object detection. However, reviewer has a few questions regarding the intuition behind the foreground-background formulation and the generalization ability to unseen combinations or noisy inputs.

== Qualitative results & generalization ==
The qualitative improvements over the baseline method [Crawford & Pineau 2019] seem not very impressive (Figure 1: only works a bit better with cluttered scenes). First, how does the proposed method perform in real world datasets (Outdoor: KITTI, CItyscape; Indoor: ADE20K, MS-COCO)? Second, the generalization to unseen scenarios are mentioned in the introduction but not really carefully studied or evaluated in the experiments. For example, one experiment would be to train the framework on the current 3D-Rooms dataset but then test on new environments (e.g., other room layout) or new objects (e.g. other shapes such as shapenet objects). 


== Application beyond object detection ==
Equation (4) does not seem to be natural in practice: basically, the background latents depends on the foreground object latents. Alternatively, you can assume them to be independent with each other. It’s better to clarify this point in the rebuttal. As this is a generative model, reviewer would like to know the applicability to other tasks such as pure generation, denoising and inpainting. For example, how does the pre-trained model perform with noisy input (e.g., white noise added to the image)? Also, what’s the pure generation results following the chain rules given by Equation (1), (3) & (4).


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 

In this paper, the authors propose a generative latent variable model, which is named as SPACE, for unsupervised scene decomposition. The proposed model is built on a hierarchical mixture model: one component for generating foreground and the other one for generating the background, while the model for generating background is also a mixture model. The model is trained by standard ELBO with Gumbel-Softmax relaxation of the binary latent variable. To avoid the bounding box separation, the authors propose the boundary loss, which will be combined with the ELBO for training. The authors evaluated the proposed on 3D-room dataset and Atari. 


There are several issues need to be addressed:


1, The organization of the paper should be improved. For example, the introduction to the generative model is too succinct: the spatial attention model did not be introduced in main text. Why this model is call 'spatial attention' is not clear to me. The boundary loss seems an important component, however, it is never explicitly presented. 

2, The parallel inference is due the mean-field approximation, in which the posterior is approximated with factorized model, therefore, the flexibility is restricted. This is a trade-off between flexibility and parallel inference. The drawback of such parametrization should be explicitly discussed. I was wondering is there any negative effect of such the approximated posterior with fully factorized model comparing to the SPAIR?

3, The empirical evaluation is not convincing. The quality illustration in Fig.1, 2 and 3 uses different examples for different methods. This cannot demonstrate the advantages of the proposed model. The quantitative evaluation only shows one baseline, SPAIR, in Table 1, and other baselines (IODINE and GENESIS) are missing. With such empirical results, the performances of the proposed method are not convincing. 

In sum, I think this paper is not ready to be published. 

====================================================================

I have read the authors' reply and the updated version. I will raise my score to 6. 

Although the mean-field inference is standard, the model in the paper looks still interesting and the performances are promising. 

I expect the boundary loss should be specified formally in the final version.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper proposes SPACE: a generative latent variable models for scene decomposition (foreground / background separation and object bounding box prediction). The authors state the following contributions relative to prior work in this space: 1) ability to simultaneously perform foreground/background segmentation and decompose the foreground into distinct object bounding box predictions, 2) a parallel spatial attention mechanism that improves the speed of the architecture relative to the closest prior work (SPAIR), 3) a demonstration through qualitative results that the approach can segment into foreground objects elements that remain static across observations (e.g. the key in Montezuma's Revenge).

The proposed model is evaluated on two sets of datasets: recorded episodes from subsets of the Atari games, and "objects in a 3D room" datasets generated by random placement of colored primitive shapes in a room using MuJoCo.  Qualitative results demonstrate the ability of the proposed model to separate foreground from background in both datasets, as well as predict bounding boxes for foreground objects.  The qualitative results show comparisons against SPAIR, as well as two mixture-based generative models (IODINE and GENESIS), though mostly not for direct comparisons on the same input.  Quantitative results compare the proposed model against the baselines in terms of: gradient step timings, and convergence plots of RMSE of reconstruction against wall clock time, and finally on object bounding box precision and object count error in the 3D room dataset.

The key novelty of the proposed model is that it decomposes the foreground latent variable into a set of latents (one for each detected object), and attends to these in parallel. This leads to improved speed compared to SPAIR, as demonstrated by the gradient step timings. I am convinced that the proposed model is asymptotically faster than SPAIR. However, the fact that the timings are only reported for the gradient step and not more comprehensively for entire training and inference step, is unsatisfying.  I found the qualitative comparisons to be confusing as they were mostly for different input frames, making it hard to have a direct comparison of the quality between the proposed method and baselines.  Moreover, the quantitative results reporting bounding box precision are confusing. Why report precision at exactly IoU = 0.5 and IoU in [0.5, 0.95] instead of the more standard precision at IoU >= 0.5 (and higher threshold values such as 0.95)?  The differences in the reported results seem relatively small and in my opinion, not conclusive given the above unclear points.

Due to the above weaknesses in the evaluation, I am not fully convinced that the claimed contributions are substantiated empirically.  Thus I lean towards rejection.  However, since I am not intimately familiar with the research area, I am open to being convinced by other reviewers and the authors about the conceptual contributions of the model.  As it stands, I don't think this contribution is strong enough to merit acceptance.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Positives:
+The system makes sense and is explained well
+The factoring of scenes into objects and multiple background components is good
+I think overall the experiments are reasonable, although I have a number of questions about whether aspects of them are apples-to-apples

Negatives:
-Some of the experiments do not appear apples-to-apples
-There are a large number of changes, and there aren't any ablations. It's a little hard to follow and verify that the gains are credited properly. 

Overall, I'm favorably inclined towards accepting this paper so long as the experiments are more clearly made apples to apples. Right now, since I'm forced to give a binary decision and I'm not positive about comparisons, I have to lean towards rejection -- I'd peg my actual rating as 4.5.

Method: 
+The method is well-explained and straight-forward (in a good way). 
+The factoring of scenes into objects and multiple background components is good
+The parallelization is good, and the fact that it works far faster than SPAIR with similar results is quite nice

Experiments:
+Overall the experiments are pretty good and compare against the baselines I would expect, and have both qualitative and quantitative results.
+The method appears to do a good job of segmenting the objects, and if Figure 1 is representative, this is quite impressive. 
-Why does Figure 1 show results from different systems on different images? This makes comparison impossible. Paired samples are always more informative.

-It's not clear to me that fair comparisons were done, especially to GENESIS. 
(a) It's never listed how K for genesis was picked -- this should presumably be tuned somewhere to optimize performance. The paper mentions in the 4.2 that it was impossible to run the experiments for GENESIS for more than 256 components -- but the GENESIS paper has numbers more like K=9. If there are an overabundance of components, this might explain some of the object splitting observed in the paper.
(b) Unless I'm missing something, in Figure 5, for 4x4 and 8x8, it doesn't appear that IODOINE or GENESIS have converged at all. Does the validation MSE just then flatten (or go up) there? This is also wall-clock, so I'm not sure why things would stop there. This seems to conflate training speed with performance (although also note that the wall clock times being discussed are pretty small -- the rightmost side of the graph is ~14 hours -- hardly a burdensome experiment).
(c) Similarly, for 16x16 cells, SPAIR seems to be improving consistently. Is it being cut off?-Figure 5 -- The caption for the figure things appears to not make sense: GENESIS is listed as having K = HxW+5 components and SPACE has K=5 listed. Neither make sense to me. Are they out of order?
(d) For SPAIR in Table 1, it's not clear whether it's the slow SPAIR that was mentioned previously or the fast one (e.g., the predicted boxes are described as the same quality as SPAIR -- but is it the slow SPAIR or fast one?). I think the paper would benefit from being a bit clearer about this. I get that the parallel decomposition, in some sense, may be necessary to get any results. But I wish the paper were a bit more explicit.


-There are some reasonable results. I realize that there isn't existing ground truth on atari and other games, but why not label a few hundred frames manually? 
-It would have been nice to have an ablation of some of the components, including the boundary loss. Unfortunately, there's a complex multi-part system and it's not clear how to break off components apart for reuse elsewhere.

Small stuff that doesn't affect my review:
1) Figure 5 -- the figure text size is tiny and should be fixed. 
2) Eqn 3, subscript of the product "i" -> "i=1"
3) Table 1 -- captions on tables go on top
4) Now that the systems work like this, I'd encourage the authors to go and try stuff on more realistic data.
5) I would be a little wary of making a big deal out of the discovery of the Montezuma's Revenge key. I realize this is indeed important, but I don't see why something like the slic superpixel objective, or felzenswalb-huttenlocher wouldn't find it either. I think it's great that there's a setting in terms of network capacity (for fg/bg networks) that yields this result, but this seems to depend heavily on the particular networks used for each of the parts of the method, and not on the general method. Also, it seems largely a function of the fact that they're a small region with a different color.

-----------------------------------------

Post-rebuttal update: I have read the authors' response and I am happy to increase my rating to 6.
review_title: Official Blind Review #2

Meta-review: The paper makes a reasonable contribution to generative modeling for unsupervised scene decomposition.  The revision and rebuttal addressed the primary criticisms concerning the qualitative comparison and clarity, which caused some of the reviewers to increase their rating.  I think the authors have adequately addressed the reviewer concerns.  The final version of the paper should still strive to improve clarity, and strengthen the evaluation and ablation studies.
Decision: Accept (Poster)