Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a generative model for scalable sequential object-oriented representation. The paper proposes several improvements based on the method SQAIR (Kosiorek et al. 2018b), (1) modeling the background and foreground dynamics separately; (2) parallelizing the propagation-discovery process by introducing the propose-reject model which reducing the time complexity. Finally, the proposed model can deal with orders of magnitude more objects than previous methods, and can model more complex scenes with complex background.

Accept.
The paper is clearly written and the experimental results are well organized. The results in the paper may be useful for unsupervised multi-objects tracking .I have one concern here, 
（1）As argued in the paper, previous methods are difficult to deal with the nearly a hundred objects situation and there is no direct comparison for these methods. So has the author compared the method SCALOR with previous methods in few objects setting? Does the technical improvements of the method benefit in the few objects setting?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: I thank the authors for the detailed rebuttal, as well as for the updates to the text and several new experiments in the revised version of the paper. Most of my comments are addressed well. I am happy to improve my rating and recommend to accept the paper.

---

The paper proposes an approach for unsupervised detection and tracking of objects in videos. The method continues the "Attend, Infer, Repear" (AIR) and Sequential AIR (SQAIR) line of work, but improves on these previous approaches in terms of scalability and can thus be applied to scenes with tens of objects. The scalability is obtained by replacing, wherever possible, sequential processing of objects by parallel processing. Experiments are performed on three datasets: moving DSprites, moving MNIST, and the real-world "Crowded Grand Central Station" dataset. The method seems to work in all cases, as confirmed by quantitative evaluation on the first two datasets and a qualitative evaluation on all three.

I recommend rejecting the paper in its current state. On one hand, the results look quite good, and the method seems to indeed scale well to many objects. On the other hand, novelty is limited and the experiments are limited in that there are no comparisons with relevant baselines and no clear experiments showing the specific impact of the architectural modifications proposed in this paper. Moreover, the paper is over 9 pages, which, as far as I remember, requires the reviewers apply "higher standards". Overall, I would like the authors to comment on my concerns (below) and may reconsider my rating after that.

Pros:
1) Relatively clear presentation.
2) Judging from the extensive qualitative results and the (limited) quantitative evaluation, the method seems to work.
3) I appreciate the additoinal results on generalization and parallel disovery in the appendix.

Cons:
1) Novelty of the work seems quite limited. The main contribution is in improving the efficiency of object detection/tracking by parallelizing the computation, without much conceptual innovation. This might be a sufficient contribution (in the end, efficiency is very important for actually applying methods in practice), but then a thorough evaluation of the method would be expected (see further comments about it further). Moreover, the previously published SPAIR method by Crawford and Pineau seems very relevant and related, but is not compared against and is only briefly commented upon, despite the code for that method seems to be available online. I would like the authors to clarify the relation to SPAIR and preferably provide an experimental comparison.

2) The experiments are restricted. While there are quite many qualitative results, several issues remain:
2a) No baseline results are reported. It is thus impossible to judge if the method indeed improves upon related prior works. In particular, comparisons to SQAIR and SPAIR would be very useful. If possible, it would be useful to provide even more baselines, for instance some traditional tracking methods. Comparisons can be both in terms of tracking/reconstruction performanc, as well as in terms of computational efficiency. Both can be measured as functions of the number of objects in the scene.
2b) There are few quantitative resutls. In experiments 2-4 of section 5.1 it seems it would be fairly easy to introduce some, in particular, one could compare how do these more difficult settings compare to the "default" one. In section 5.2 given that the ground truth tracks are not available, evaluating tracking is challenging, but one could still compare NLL/reconstruction with appropriate baselines. The provided comparison to a vanilla VAE in therms of NLL is actually somewhat confusing - not sure what it tells the reader; moreover, I would actually expect the NLL of the proposed structured model to be better. Why is it not?
2c) Since the paper is largely about tracking objects through videos, it would be very usefuly to include a supplementary video with qualitative results. 

3) (minor) Some issues with the presentation:
3a) I found the method description at times confusing and incomplete. For instance, it is quite unclear which exactly architectures are used for different components. The details can perhaps bee looked up in the SQAIR paper, but it would still be useful to summarize most crucial points in the supplementary material to make the paper more self-contained.
3b) The use of \citet vs \citep is often incorrect. \citet should be used when the author names are a part of the text, while \ciptp if the paper is cited in passing, for instance: "Smith et al. (2010) have shown that snow is white." vs "It is known that snow is white (Smith et al. 2010)." 
3c) Calling AIR a "seminal work in the field of object detection" is not quite correct - object detection is a well-established task in computer vision, and AIR is not really considered a seminal work in that field. It is a great paper, but not really in object detection.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: UPDATE: My original main concern was the lack of baseline, but during the rebuttal period the authors have conducted the request comparison and addressed my questions satisfactorily. Therefore, I would recommend the paper be accepted.

---
Summary: This paper proposes a generative model and inference algorithm for discovering and propagating object latents in a way that scales to hundreds of objects. The key components of their approach is the parallel discovery and propagation of object latents as well as the explicit modeling of the background. The authors show that the model is able to model and generalize to scenes with hundreds of objects, including a real-world scene of a train station.

Research Problem: This paper tackles the problem of scaling object-oriented generative modeling of scenes to scenes with a large number of objects.

The main weakness of the submission is the lack of a baseline, and mainly for this result I would recommend rejecting the submission at its current state. However, if the authors are able to revise the submission to include such comparisons (with Kosiorek et al. (2018), van Steenkiste et al. (2018), and Alahi et al. (2016), detailed below), then I would highly consider accepting the paper, as the paper makes a novel contribution to modeling scenes with many more objects than previous work, as far as I am aware.

Strengths:
- The authors show that the method can model various synthetic and real-world datasets that show the efficacy of the method
- The method can also generalize to more objects and longer timesteps than trained on.

Weaknesses:
- The main weakness of the submission is the lack of a baseline. It would be important to understand the differences between Kosiorek et al. (2018), which the authors claim is the closest work to theirs, and van Steenkiste et al (2018), which also models objects in a parallel fashion. Alah et al. (2016) also takes an approach of dividing the scene into grid cells and also demonstrate results on modeling human trajectories.
- Motivation: Whereas the authors motivate the benefits for modeling objects, the motivation for specifically scaling to model hundreds of objects is less clear. It would be helpful for the authors to provide examples or arguments for the benefits of modeling so many objects at once. One argument against such a need is that humans only pay attention to a few number of objects at a time and do not explicitly model every possible object in parallel. One argument in favor of such a need is the ability to gain superhuman performance on tasks that could benefit from modeling multiple entities, such as playing Starcraft, detecting anamolies in medical scans, or modeling large scale weather patterns.
- A possible limitation of the method may be in modeling interactions between entities. What mechanism in the propagation step allows for modeling such interactions, and if not, how could such a mechanism be incorporated?
- How would SCALOR behave if the grid cells were smaller than the objects? In this case an object may occupy multiple grid cells. Would the authors provide an experiment analyzing this case? Would SCALOR model a object as multiple entities in this case (because the object spans multiple grid cells), or would SCALOR model the object with a single latent variable?

Question:
- What is the fundamental reason for why the structure of such a generative model would cause the latents to model objects, rather than something else, such as the image patches that show the empty space between objects?

Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., & Savarese, S. (2016). Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 961-971).

Van Steenkiste, S., Chang, M., Greff, K., & Schmidhuber, J. (2018). Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353.

Kosiorek, A., Bewley, A., & Posner, I. (2017). Hierarchical attentive recurrent tracking. In Advances in Neural Information Processing Systems (pp. 3053-3061).
review_title: Official Blind Review #2

Meta-review: After the author response and paper revision, the reviewers all came to appreciate this paper and unanimously recommended it be accepted.  The paper makes a nice contribution to generative modelling of object-oriented representations with large numbers of objects.  The authors adequately addressed the main reviewer concerns with their detailed rebuttal and revision.
Decision: Accept (Poster)