Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The following work proposes a method for learning the factorized latent structure of a scene. Specifically, while the primary prior work MulMon [28] assume static objects while the viewpoint varies through time, this work allows for both camera and object motion within the training data. To achieve their goal of factorizing object motion from camera motion, they assume high framerate and use clustering to determine cases where the camera is moving quickly and the objects slowly, versus cases where the camera moves slowly but the objects move quickly. Notably, once trained, they are able to resynthesize the scene's dynamic objects from novel viewpoints. 
main_review: Strengths:
- The ability to resynthesize dynamic scenes from novel viewpoints is impressive
- Using clustering to identify cases of fast object slow camera and slow object fast camera seems like a highly practical solution to a difficult problem

Weaknesses/Clarity concerns:
- I think there's a lot of critical detail in the appendix, making the paper a bit difficult to follow. Specifically, I don't think I fully follow the details on how the data clustering for SCFO and FCSO works
- There's a fair amount of prior art being left out here. Specifically there's a lot of works in unsupervised 2D landmark learning that are formulated very similarly.  [1], for example similarly disentangles dynamic objects from videos, falling under the "slow camera fast object" category specified in this work. [2] is another related work in the multi view setting.
- I believe there are a number of additional unstated limitations that should be made clear here. What type of objects can we model here? By using spatial Gaussian mixtures, this typically assumes that the objects can be decomposed into rigid approximately convex shapes. 
- It appears that this model is capable of modeling the background as well. How do the spatial gaussians capture the background, considering the background spatially contains the foreground objects, and I assume the gaussians should not overlap too much?
- What happens if the number of mixture components exceeds the number of objects by a lot? What happens if it's too small?

Overall, I find the work pretty interesting despite the limitations. While it's quite possible that I've missed some critical details during my read-through, I would appreciate if the authors could point me to where the information I missed was, or clarify in their rebuttal.


[1] Lorenz et al. Unsupervised Part-Based Disentangling of Object Shape and Appearance. CVPR 2019

[2] Rhodin et al. Neural Scene Decomposition for Multi-Person Motion Capture. CVPR2019 

Post rebuttal:
I think the authors have adequately addressed most raised concerns and points of confusions. I agree with some other reviewers that the experimental setting is a bit too simple to be convincing, but I think the overall task is novel and the results are pretty interesting. For this reason, I maintain my original rating.
limitations_and_societal_impact: As stated in my main review, I don't think the technical limitaions are adequately addressed. I don't perceive any potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposed a Dynamics-aware Multi-Object Network (DyMON), a model that captures object-centric representations in dynamic settings and enables querying novel viewpoints as well as single objects from the scene. DyMON builds on top of MulMON, a method that works in multi-view static settings. 
main_review: Strengths:
1) The problem setting is novel and has not been tackled from an object-centric representation point of view.

Weaknesses:

2) Is assumption A2 i.e having a large-speed difference between the camera and object realistic? Or is this just a trick to make the model learn? If so, then there must be some results shown on a dataset wherein both scene velocity and observer's velocity are perceivable.

3) Baselines: The baselines used to compare are not fair. For example, T-GQN [Ref1] is a closely related work. This captures the model's dynamics in multi-view dynamic scenes (though not object-centric). It is surprising that the author(s) have missed this.

4) Related works: Another closely related work for multi-view static setting is that of ROOTS [Ref2] which should be discussed in the related works section.

5-a) Experiments: The author(s) claim that DyMON "permits pixel-wise segmentation which the specialized model (GSEWM) cannot". This is false. There are object-level masks in GSWM that essentially are the object segmentation masks. So, I don't see a reason why mIoU is N/A in Table 1(c).

5-b) From my experience, GSWM has a very low MSE (in static view dynamic object scenes). Given that the DRoom dataset is similar to CLEVERER used in GSWM, I expect the reconstructions to be good. Can the author(s) provide the reconstruction comparison (either in the main paper or in the appendix)?

6) Qualitative Results: Comparison against MulMON or GSWM in the qualitative results needs to be shown to visualize where DyMON is doing better. I'd urge the author(s) to include the qualitative comparison and give insights into where other models fail.

Suggestions:
7) The testing algorithm can be included in the main paper to make things clear. $\beta_{\text{FCSO}}$, $\beta_{\text{SCFO}}$ makes the reader think that they are also inferred during the test time, which is not the case.

8) As mentioned in point 6, figures 2 and 3 need to improve. Fig. 2 occupies too much space (can reduce the number of examples and instead put MulMon and GSWM results).

9) Abbreviations for Nv. Seg and Nv. Obs needs to be expanded in the caption of Table 1.

Other minor details (not considered for decision):
10) The paper seems to have some typos, please fix them. Otherwise in general the paper is well written.

References:

[Ref1] Sequential neural processes, G Singh, J Yoon, Y Son, S Ahn - In NeurIPS, 2019

[Ref2] ROOTS: Object-Centric Representation and Rendering of 3D Scenes, Chang Chen, Fei Deng, Sungjin Ahn, JMLR, 2021


Justification for Rating: Lack of proper baselines , comparisons with other methods and some confusion in the assumptions of the model are the reason why I'm currently giving the paper a  reject. I'd be looking forward for the rebuttal of the author(s) and change my score accordingly.

-------

Edit 1 (post rebuttal): The author(s) have addresses some of my concerns and I thank them for it. Hence, I've decided to increase my rating to "5: Marginally below the acceptance threshold" but cannot increase any further due to lack of proper baseline. 

----
Edit 2 (post rebuttal and post T-GQN expt.) : Thanks to the author(s) for including the T-GQN baseline. Having addressed my concerns I'm increasing to the rating to a Weak accept. As mentioned earlier, the problem direction is quite novel and the proposed method is a good step in this direction, I'd hope more works come out which relax the assumptions and/or work in a much more realistic scenarios.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes DyMON, an unsupervised object-centric representation learning model that can be applied to scenes with dynamic objects and multiple viewpoints. DyMON is trained to obtain object latents that are independent of the viewpoints. The model builds off of MulMON, extending it to work with scenes with dynamic objects. Two assumptions are made about the training scenes: 1) a high frame rate assumption and 2) the relative speed difference between the camera and the objects is large -- that is, either the camera moves much faster than the objects or the objects move much faster than the camera. With these assumptions, the problem can be reduced to either a multi-viewpoint, static objects problem or a single-viewpoint, dynamic objects problem. Through experiments, the authors demonstrate that DyMON can be used to query predictions of the scene at arbitrary times and viewpoints.

main_review: This paper is well-motivated, investigating the problem of learning object-centric representations in scenes with dynamic objects and multiple viewpoints without any supervision. This is an area previous works (that I know of) have not explicitly explored and is very important for modeling more realistic scenes.

The authors make several assumptions about the dataset and given those assumptions, the approach they take is reasonable and sound. However, while the authors claim to "resolve" (L127) the temporal entanglement problem, it seems that by making these assumptions, they have instead sidestepped the problem by restricting the dataset to a domain where the problem is not significant.

The experiments are encouraging, especially the ones querying novel viewpoints and times. However, I do not believe the DyMON vs. Dynamic Scenes section is very informative since MulMON is trained on a different dataset than it is evaluated on. I would also be curious how GSWM performs on the multi-view-dynamic-scene dataset. Although it does not explicitly take into account camera viewpoint, it can handle dynamic backgrounds (ie. context), so it may be able to model these scenes. Lastly, in addition to reporting MSE, it would be interesting to see comparisons of the reconstructions of DyMON vs. the baselines to see where the different models qualitatively fail.

My main concern with the paper is with the novelty. If I understand correctly, the model is almost identical to MulMON, but the difference is in the training procedure. While the DyMON training algorithm allows the model to factorize the camera dynamics from the object dynamics, this combined with the aforementioned assumptions made on the dataset limits the contributions of this paper. I encourage the authors to clarify the novelty of their approach over MulMON. I am willing to change my score if I have missed something.

Minor issues / typos:

- GSWM seems to be cited incorrectly (SPACE is cited twice)
- In Equation (4), I'd recommend something other than q for the query since it is overloaded.
- L23: These method -> These methods
- L153: veiw -> view
- L165 -> "we have" seems not necessary

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a method that broadens the scope of multi-view object-centric representation learning to dynamic scenes. The model achieves spatial-temporal factorization by having two weak assumptions: high-frame-rate and large-speed-difference.

main_review: **Strengths:**
1) Overall, this paper is well written, and the technical details are easy to follow.
2) The main idea of unsupervised multi-view object-centric representation learning in the context of dynamic-scene settings is interesting.
3) The factorization of space and time should be important. 

**Weaknesses:**

**Assumptions.** The two assumptions of the high-frame-rate and the large-speed-difference are not holding in real-life scenarios. In real synthetic datasets (e.g., GTA and others), the observer is likely moving while the objects in the scene are also performing a movement. I agree the assumptions are valid on the DRoom and MJC-Arm datasets. However, my main concern is that this work would not generalize into real 3D datasets, unlike NRF, which can generalize novel-view synthesis.

**Experiments.**  Table 1a shows that DYMON does not generalize well compared to MulMON[28] on the novel viewpoints. I find it quite surprising. Can the authors explain why that is? Also, why is there a gap between object reconstructions and segmentations?. Additionally, Table 1b is not clear to me. Why does the MulMON model perform better than DyMON? I would expect from a model that learns dynamics to generalize over static scenes.  


My main concern is I cannot see how this work generalizes over real 3D datasets and helps the field to move forward. I would expect the authors to add at least one real 3D dataset (or at least more realistic environments) since the dynamics play a critical factor in real-life scenarios more than on toy tasks. Furthermore, generalization and understanding of objects and observers' movements are vital for multiple tasks and domains, and thus I believe the spatial-temporal factorization should be leveraged on more realistic scenarios. On the other hand, I liked this paper, its contribution, and its motivation. I am open to the authors' feedback and other reviewers' opinions.


limitations_and_societal_impact: Yes.
review_title: null

Meta-review: - The proposed method is tackling an important novel problem. The reviewers found the approach is reasonable and sound but technically not very novel. 
- Many concerns (including T-GQN) from the reviewers are addressed well by the rebuttal. These results should be included in the updated version.
- The strong assumptions the method takes to work are the main limitations of the approach but as the first work in the direction I think it still has some contribution. 
- Importantly, some key related works like T-GQN and ROOTS are missing and should be discussed with enough emphasis in the revision.

Decision: Accept (Poster)