Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: It looks like authors did not provide a rebuttal or they decided to make their response to other reviewers only visible by those reviewers. Is this the case?
review_title: No response from authors?

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method that learns disentangled representations of objects determined by interaction-relevant and interaction-irrelevant latents. The proposed method is based on OP3, but with a formulation that encourages the desired disentanglement. In experiments, the authors show that the interaction-relevant and interaction-irrelevant latent variables are in fact disentangled by swapping them between two different sequences. They also show that the disentangled representations lead to improved performance in downstream tasks against the baselines.
strength_and_weaknesses: Strengths:
+ New method for disentangled object centric representation learning
+ Experiments showing the disentanglement is present
+ Experiments showing the disentanglement leads to better downstream performance


Weaknesses:

- Intersection over union evaluation:
The authors present an intersection over union evaluation to show that interaction-relevant latents contain location information. However, I am not clear how the authors are able to get locations from the latents, and so, can't really see how this evaluation is possible. Can the authors clarify how this is done? Or point out where in the paper this is described. If this is in fact somewhere in the paper, and I cannot find it, it would be useful if this is revised later on so it's easier to find.



- Pairwise interactions mechanism visualizations:
In page 5, the authors mention that they use a d_{o,o} function to calculate pairwise interactions between objects. However, there is no analysis of how these interactions are happening after a model is trained. If this is possible, it would be good to have some type of visualization showing how these become active when objects are interacting in the pixel space.



- Typos in the writing:
There are many typos throughout the text. For example, in section 3.2, first paragraph, line 7, module is misspelled as moduele. In addition, there is misuse of the verb is. For example, in page 6, 4th paragraph, the authors say "Finally, parameters of a generative model pθ and an inference model qφ is" while it should be "Finally, parameters of a generative model pθ and an inference model qφ are". These type of typos are happening in multiple places in the manuscript. I suggest the authors proof read the manuscript and fix these writing issues.
clarity,_quality,_novelty_and_reproducibility: The paper is fairly clear, but the authors should improve the writing and how they describe things I mentioned in the weaknesses section. The results seem of quality to make the point the paper is about. The method seems novel, and the authors provided details of the architecture in the supplementary material so it should be reproducible.
summary_of_the_review: In conclusion, I find this paper interesting regardless of the toy setup the conceptual contribution is shown. I have a couple of questions that I have stated in the weakness section that I would like the authors to clarify. In the mean time, I am leaning towards accepting this paper, but I am looking forward to discussions to determine the final assessment.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a probabilistic model to learn disentangled object representation in an unsupervised fashion. The generative model is expressed through an observation module and a dynamic module, where the representations of entities are factorized into interaction-relevant relational features and interaction-irrelevant global features. The model is optimized with variational inference by maximizing the ELBO. Latent swapping experiments and downstream tasks are conducted on Block and Weights datasets.
strength_and_weaknesses: Strengths:
- This paper differentiates itself from prior work by disentangling the object-centric representation to interaction-relevant feature and the global feature, which is a promising direction for unsupervised representation learning.
- The experiments show that the model can learn disentangled representations, and the learned representations can facilitate downstream tasks in planning and roll-out reconstruction.

Weaknesses:
The major concern lies in the experiments. 1) the latent swapping experiments are conducted on the block moving datasets, and the involved features are relatively simple, i.e., color for global feature and position as relational feature. Prior work also demonstrates such disentangled representation even without explicit modeling of the dynamic and static features. Without further quantitative comparison with other baselines or in more complex scenarios, it’s hard to justify the significance of the proposed model. 2) The downstream experiments are mainly compared against OP3, which does not disambiguate global or relational features. Are there justifications why not compare to factorized models like STOVE?

clarity,_quality,_novelty_and_reproducibility: Clarity and quality: Although not an expert, I find the paper writing and logic flow clear and easy to follow. The comparison with prior work on object-centric methods is also clear and distinctive.
Typo: section 3.2 dynamic module

Novelty: The proposed model is built upon previous work OP3 and various strategies are applied to infer the parameters. The major novelty lies in the disengagement probabilistic design.

Reproducibility: the paper included implementation details in the supplementary material. Code is not included.

summary_of_the_review: My ratings are given based on my evaluation of the strengths and weaknesses above. The authors are expected to address my concerns over the experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose STEDIE, a new model that disentangles object representations based on interactions, into interaction-relevant relational features and interaction-irrelevant global features without direct supervision. Across different experiments, the authors showcases the model's effectiveness.
strength_and_weaknesses: I want to preface this by saying that I'm not really familiar with the field of object-centric learning.

Strength:
- I think the model is well designed and has some interesting novelties. The authors do a good job explaining the model.

Weakness:
- Equation one \sigma has a (0) superscript, which seems to be a typo.
- I think overall, the downstream task evaluation seems a bit underwhelming to me.
1. I think the evaluation metrics and visualizations are not exactly informative. As far as I can tell from the paper, the action and measurements are done in 3D. The evaluation metrics of accuracy is based off a distance metric that's unspecified in the paper and appendix. Not showing direct measurement in 3D limits my confidence in the model. MSE is calcuated in pixel space, which is weird, because say we move the cube along the camera direction, it results in not so much change in pixel mse, but the 3D location can be far off.
2. From what I can tell, OP3 seems like a model from quite awhile ago, while STOVE seems to be a very relevant baseline. I can't really tell from the paper which reference is STOVE, and why the authors are not comparing with it.
clarity,_quality,_novelty_and_reproducibility: Pretty clear, novel and seems reproducible. However, i think the downstream task's quality is a bit lacking.
summary_of_the_review: Please see my strength and weakness.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a structured world model which learns to disentangle image sequences into interaction-irrelevant global features and interaction-relevant relational features without supervision. The design of the model continues the ideas from IODINE, which iteratively inference latent representations to generative target images. Differently, the global features and relational features are designed to be separated and learned implicitly. The model is evaluated on two downstream tasks: planning and casual understanding, experiments show the effectiveness of the disentangled representation.
strength_and_weaknesses: Strength:
The paper presents a very interesting idea of disentangling representations based on interaction.

Experiments show that the learned representations achieve better performance in planning tasks and understanding causal relationships.

Weakness:
Missing discussion on type of factorization, especially between implicitly factorized representation and explicitly factorized representation.

Please discuss why the performance of OP3 on the Weights dataset is so poor even when doing nothing, because the result on the block moving dataset seems fine.

I would like to see the video prediction performance comparison between OP3 and STEDIE on the non-action-conditioned datasets, like CLEVRER.


clarity,_quality,_novelty_and_reproducibility: There are som typos:
•	(a) (a) (b) in the caption of Figure 1 should be (a) (b) (c).
•	Does a^{(t)} should be a^{(t-1)} in the first row of page 5?
summary_of_the_review: The paper presents a method to disentangle representations based on interaction. The ideas are validated quantitatively and qualitatively. Yet, I am concerned about the insufficient evaluation. I am open to adjusting my rating if the authors do sufficiently address the points raised in weaknesses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of learning a structured, object-decomposed world model of simple synthetic 2D and 3D environments. The novelty lies in how the method disentangles features of objects: it learns to separate object features into interaction-/physics-irrelevant (“global”) properties such as object color and appearance, and dynamic / interaction-relevant properties such as position of an object. This is done via separation of latent variables in the (structured) probabilistic model, based on the OP3 [1] framework. This separation is validated both qualitatively and quantitatively on simple 2D and 3D synthetic environments that involve object-specific actions. The authors further demonstrate quantitative benefits of this separation for planning tasks.

I agree with the reviewers that this paper addresses an interesting and challenging problem (joint action-conditioned video prediction and object decomposition), is fairly well-written and proposes a novel approach to implicitly decompose interaction-relevant and global properties from raw video. The fact that the model can do this without explicitly specifying which features should capture location, size, content etc. of objects is interesting and significant. Concerns are related to the low quality/interpretability of the figures, the limited experimental setup (primarily 2D and 3D toy environments) and the overall technical complexity of the model.

One major concern was that the authors were not comparing against an object-centric baseline with an explicitly factorized latent object representation. In their rebuttal, the authors have added a convincing experimental comparison against STOVE [2] to address this issue. Nonetheless, a comparison against more recent object-centric, action-conditioned video models such as PARTS [3] is missing from the paper and the authors are highly encouraged to at least include a discussion of how their method relates to PARTS [3].

This is very much a borderline paper that would clearly benefit from another round of revisions, but given that the presented idea is both conceptually and technically novel, the experimental evaluation is solid (i.e. claims are verified both qualitatively and quantitatively) but not outstanding, and the claims of the paper are relatively moderate but still significant and of interest for the community, I think it can be considered for acceptance.

As mentioned by the reviewers, the paper would heavily benefit from showing more experimental evidence for disentanglement of global vs. interaction-relevant features beyond just object color and position. Further, the authors are strongly encouraged to improve the quality of the figures (both for model and results) of the paper to make it easier to read and understand.

[1] Veerapaneni et al., Entity Abstraction in Visual Model-Based Reinforcement Learning (CoRL 2019)
[2] Kossen et al., Structured Object-Aware Physics Prediction for Video Modeling and Planning (ICLR 2020)
[3] Zoran et al., PARTS: Unsupervised segmentation with slots, attention and independence maximization (ICCV 2021)
justification_for_why_not_higher_score: This paper is clearly borderline and several concerns around significance of the limited experimental setting (primarily toy 2D and 3D synthetic environments) were raised. Furthermore the quality of presentation could be improved.
justification_for_why_not_lower_score: As mentioned above, the presented idea is both conceptually and technically novel, the experimental evaluation is solid (i.e. claims are verified both qualitatively and quantitatively) but not outstanding, and the claims of the paper are relatively moderate but still significant. Overall, it can be considered for being accepted at ICLR.
Decision: Accept: poster