Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This submission presents a video scene model that decomposes a scene into object components, and frame components in an unsupervised way. The architecture improves over previous models that are static (MONET, IODINE, etc) and use videos (GENESIS) in that the model performs better and is conceptually simpler. The main idea is the use of a transformer model that stacks all feature representation generated from CNNs for every individual frame, then aggregate to object and frame latents. 

Experimental results are shown on synthetic images only. Comparison are done in two settings: knowing the camera locations (against GQN and Nerf-VAE) and fully unsupervised (compared against MONET and SlotAttention). Empirical results verify the model to be stronger.  
main_review: Well written paper that is an extension of current work on unsupervised models that decompose scenes into its constituents. This model is novel in the way it uses transformers to decompose into frame and object latents. This makes for a conceptually simple model. Experimental results are sufficient, the model is tested in two scenarios (with and without known camera position) and performs better than reasonable chosen baselines. The submission presents some weak evidence of the disentangling of the object/frame latents in that the camera position is not recoverable from the objects but only from the frame representation.  

The main drawbacks are
- This is no fully generative model. The submission comments on this in the future work and limitation section. I concur with the potential extension borrowed from IODINE. 
- Experimental results are shown on synthetic images only. The Nerf-VAE baseline is a fair competitor for this model and this type of date. The question whether these scene models will generalize to more complex environments is still open. This submission improves on the technical side but not on the experimental one.

I recommend acceptance of this paper, I enjoyed reading it and the level of detail is sufficient to understand the details of the method.

Did you experiment with the inclusion of optical flow into the model? Would that not potentially improve on the attention mechanism? What are the main obstacles for using optical flow?
limitations_and_societal_impact: The limitations are sufficiently addressed. Negative societal impact is abstract for this work and is shared with all models that improve on scene understanding in automated ways.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors devise a neural network architecture, SIMONe, for learning unsupervised representations of scenes viewed from multiple camera poses. The latent representation is factored into slots meant to represent time-invariant entities (i.e., physical objects) and time-varying "frame latents," which can capture changes in the camera pose. They show that this method is able to capture the structure of scenes from three synthetic datasets by testing it on two tasks: novel view synthesis and (unsupervised) object instance segmentation. A number of qualitative results suggest that the learned latent representation successfully factors into time-invariant objects and time-varying viewpoint parameters.
main_review: This work fits in the rapidly growing clade of unsupervised scene decomposition methods (also called "object discovery"), specifically those which try to factor scenes into K object slots plus background: MONet, IODINE, GENESIS, Slot-Attention, SPACE, SPAIR, OP3, and plenty of others. These methods are all meant to address a critical problem in AI and computer vision: how to infer and represent the structure of a scene when supervision isn't available. It's widely hypothesized (though not generally confirmed) that such object-centric representations, if they were to be used on downstream tasks, would have stronger generalization properties than unstructured representations. They're also better motivated from the standpoint of human psychology and behavior: most of our attention and goals are directed at _objects_, which can be picked out and referenced independently of the rest of a scene.

Despite this area of research getting a lot of due attention, it seems like there's still a long way to go before any of these methods will be useful for dealing with real, complex scenes. Compared to **supervised** object instance segmentation methods, the object discovery methods work only on datasets and scenes of far lower complexity (and are usually only benchmarked on simple synthetic datasets.) It's been known at least since IODINE was proposed, for instance, that the unsupervised methods fail to identify objects with multiple complex textures and shapes rather than monochromatic spheres, cones, cylinders, and so forth.

To me, the present work mostly addresses a more peripheral deficiency in the previous object discovery methods, rather than tackling the core problems with them head on. The desiderata listed at the end of the Introduction are all surely nice to have, but I think most people who work on robotics, for instance, would rather have an unsupervised object discovery method that works on real videos and objects than one that can handle a sequence of camera poses. 

The authors may have a different view of what's most important or interesting to address, and below I'll try to evaluate the work based on what I think the authors were aiming for. But I think it's important for them to argue more clearly (in the Introduction and Related Work) for taking the approach that they take and adding the enhancements to prior models that they add -- in particular, because other recent works **do** seem to aim more obviously toward the central problems with unsupervised object discovery, recognize what makes them difficult, and design their methods accordingly.

_What makes unsupervised scene decomposition hard, and why doesn't this work try to address it?_

I think it's mainly two things. First, realistic objects have complicated shapes, textures, and multi-part substructures. For this reason, methods that tend to segment objects as things with uniform color or texture or by their simple shape seem destined to badly underfit on realistic scenes and datasets. The present work doesn't really expose this challenge, because even the hardest of the datasets ("Playroom") has mostly monochromatic objects with simple shapes. It's not clear that SIMONe captures even these objects especially well (see below.) So I'm not sure how much of an advance this method represents; looking at more complicated and naturally textured objects (if they can be generated in Playroom) would be one way to show this more clearly.

The second challenging thing about object discovery is that reliable learning signals are hard to come by. Just from looking at a static scene, it's impossible to know for sure what its physical structure is -- i.e., which components ("objects") can move or exist independently of the rest of the scene. Developmental psychologists have long recognized that _object motion_ is a critical source of information about how people decompose scenes into objects, and may be the main cue through which infants _learn_ how to identify physical objects in static scenes [A]. Several recent unsupervised approaches have taken inspiration from this robust literature and prioritized object motion for learning to segment scenes [B,C,D]. This has led to some success on identifying objects in more realistic (or even real) scenes; it's therefore surprising that the authors don't really mention object motion as a critical learning cue or comment on others' progress in using it. Notably, object motion could allow for learning to segment objects without simple, uniform colors and textures, since complex parts could nevertheless be grouped by "Common Fate."

So even if the authors don't think the present work is the appropriate place to solve these issues, I think the work really should at least bring them up and engage more with the literature here. If MONet, IODINE, and the like didn't already exist, there would be some logic in just developing a general approach; but at this point it's critical to identify and speak to the problems that are holding the field back.

_Specific questions and concerns_

1. The main quantitative evaluation presented is on unsupervised instance segmentation (Figure 4 and Table 1.) From what's shown, I don't quite follow how the authors are concluding that SIMONe sets a new "state of the art" on this task. Can SIMONe be tested on Static-ARI-F for direct comparison to MONet, etc. on the task for which the baselines were designed? I can see why the baselines wouldn't do as well on the Video ARI-F metric, since they're not explicitly designed to handle videos (except for S-IODINE -- but why create this "ablation" of OP3 rather than use the model that its authors intended?) Furthermore, the segmentations in Figure 4 from SIMONe look a lot "blobbier" than the ones from MONet shown in the Supplement; can the authors explain why SIMONe nevertheless has higher scores on this dataset?

2. The qualitative analysis of the learned representations is a little cryptic. For example: "The rubber duck does not morph into a chest of drawers because those are always located against a wall. This indicates a well-structured latent representation, which the decoder can adapt to" (L236-241). Why does this indicate a well-structured latent? In other methods, would one expect the rubber duck to morph into a chest of drawers?

3. I didn't really understand section 5.2 ("Temporal abstraction"). The authors seem to be making a point about how, in scenes where one or more object is moving, this motion is captured in the object latent rather than the frame latents, and that object motion is consistent across viewpoints. But what does it mean for an object trajectory to be captured in a "time-invariant code?" Is the idea that the interaction (in the Transformer modules) between the time-averaged object latents and the space-averaged frame latents can give rise to object motion? 

4. The authors point out that their architecture doesn't have an explicit dynamical model (e.g. a recurrent GNN) like OP3 or some others. But that seemed liked a strength of other models -- they could extrapolate object positions into the future (a little.) Would it be possible to do that with SIMONe after some modification?

5. Related to my higher-level concerns above, the section on "Scalability" seems a little misdirected. What makes the authors think that SIMONe will scale to more complex scenes without major changes in the approach?

6. Without at least some ablation studies, it's hard to know how important particular design choices are. Is the Transformer contributing significantly to this architecture, or could other latent state inference modules (like a recurrent GNN) serve just as well?


[A] Spelke, Elizabeth S. "Principles of object perception." Cognitive science 14.1 (1990): 29-56.

[B] Veerapaneni, Rishi, et al. "Entity abstraction in visual model-based reinforcement learning." Conference on Robot Learning. PMLR, 2020.

[C] Bear, Daniel M., et al. "Learning physical graph representations from visual scenes." arXiv preprint arXiv:2006.12373 (2020).

[D] Du, Yilun, et al. "Unsupervised Discovery of 3D Physical Objects from Video." arXiv preprint arXiv:2007.12348 (2020).


limitations_and_societal_impact: Yes, although I think the more relevant limitations are the ones I discussed above (complex objects and learning from object motion) rather than an inability to generate completely novel scenes, for instance.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors propose a structured generative model of video clips. This uses a factored latent space, with per-object latents that are global the video, and temporal latents that are not split between objects. The decoder uses a spatial GMM (as in many prior works), albeit with pixels given by per-coordinate MLPs not a CNN. The encoder uses a transformer on CNN features. Results on synthetic datasets show higher performance than one existing method (S-IODINE) on video segmentation, and higher performance than GQN and NeRF-VAE on novel view synthesis.
main_review: 
Model etc.:

++ The model is clearly described (albeit with fairly major aspects in the supplementary).

++ The model structure is novel -- significantly different to other works in the same area, even though the components (transformers & per-pixel MLP decoding) are well-established elsewhere. Splitting a latent space into temporal and non-temporal components is also not a new idea (see below) -- this should be emphasised better. Overall I feel the novelty is sufficient.

-- It is not clear where object motion is expected to be encoded, and why. The results (Sec.5.2) state post-hoc that encoding them in the object latents is 'correct', but from the model description, they could equally be in the temporal latents.

Related work:

-- "MoCoGAN: Decomposing Motion and Content for Video Generation" [Tukyakov, CVPR 18] is relevant (it models videos and separates latents into static content and per-frame motion) and should be discussed (note various other follow-up works also used a similar factorisation).

-- "Unsupervised object-centric video generation and decomposition in 3D" [Henderson, NeurIPS 20] is relevant (it forms an object-centric latent representation of videos, with temporal information factored) and should be discussed (and/or evaluated against). In particular, it would be expected to handle the case of Fig.8 correctly, as would other 3D-aware generative models.

Experiments:

++ Pixel reconstruction of scenes is very accurate, and camera motion is successfully encoded into (only) the per-frame latents.

++ View-synthesis results are significantly better than baselines.

-- Segmentation results are poor, both quantitatively and qualitatively.

-- The model uses a novel encoder and decoder architectures (per-pixel MLP and transformer), but there is no ablation study, nor attempt to match parameter counts / architectures for the baselines. This means it is difficult to gain useful insights from the paper about what is important (e.g. maybe S-IODINE with a transformer encoder would be better? how much does the MLP decoder help vs. spatial broadcast CNN? how well does the same model work if the latent space is not factored?).

-- Evaluation is only on simple synthetic datasets where a color-based segmentation (rgb clustering with smoothness) would likely work well. Other somewhat-similar works (e.g. RELATE and O3V) use more challenging data; it would be interesting to see how SIMONe performs on their datasets.

-- Please discuss why qualitative GQN results look much worse than the original work.

---

## Post-rebuttal

The authors' response addresses several of my concerns, and I therefore am now mildly in favor of acceptance, provided missing discussion, references, etc. are added in the camera ready. Issues raised by other reviewers also seem to be adequately addressed. Note the authors' discussion in the responses here is more insightful than the original paper in some cases -- it would be good to incorporate as much as possible in the final version. The biggest remaining issues in my view are the simplistic nature of the datasets used in evaluation, and disappointing segmentation performance on them.
limitations_and_societal_impact: Yes, both fine.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes SIMONe, a variational auto-encoder that decomposes a video into object latents and frame latents without any supervision. The object latents correspond to the time-invariant, object-level contents of the scene. The frame latents correspond to the global time-varying elemnts of the scene such as viewpoint. Each pixel is decoded independently by querying a pixel-wise decoder using sampled latents. To infer latents, SIMONe uses a transformer-based encoder that integrates information jointly across space and time. The experiments demonstrate SIMONe's capabilities in view syntheses and instance segmentation. Furthermore, the object latents are shown to be disentangled and summarize the trajectories of the objects while the frame latents are shown to contain viewpoint information.
main_review: Originality: This paper proposes a novel architecture that makes relatively few modeling assumptions about the scene when compared with previous works. The independence assumption in the decoder is also an interesting choice to allow the model to train on smaller inputs than the full image. It may be an interesting ablation to see the training speed tradeoff with this choice.

Quality: The model is sound and makes reasonable assumptions. The claims are well supported in the experiments. In particular, Table 2 demonstrates the camera location information is encoded in the frame latents Table 7 in the supplementary demonstrates the object trajectories are encoded in the object latents. The view synthesis and segmentation results also show that SIMONe outperforms the baselines. Since the authors ran their model on the CATER dataset, it would be interesting to see if the factorized representations can be used in the CATER localization task.

Clarity: The paper is clearly written and well organized. However, I did find that I had to look in the Supplementary to get a full understanding of the model, especially for the decoder. The authors may consider moving parts of the appendix to the main text if space allows. One minor suggestion is to include the spatial pooling between the two transformers in Figure 2, since that is an important operation in obtaining the K object latents. As drawn, it is not clear that K and IJ may be different.

Significance: Given the current state of unsupervised visual scene understanding models, this paper provides a good direction for future research. I would be curious how SIMONe performs on more realistic scenes than the ones shown in the paper.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper presents a method for learning disentanglement of scenes (as seen by an agent) into scene related content and object related content. It is based on transformers and a latent representation separated into time-varying and time-invariant slots.

The paper received 4 expert reviews, and in the discussion phase very quickly a consensus emerged on acceptance.	

Some weaknesses were raised, in particular with experiments, positioning and mid- and long-term objectives of this line work, and the advantages or disadvantages of key design choices (missing ablations). The authors provided a thorough and detail review, which answered most questions related to framing the method, and lead to increased ratings.

All reviewers agreed that this paper is of interest to the community and recommend acceptance.
The AC concurs.
Decision: Accept (Poster)