Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: SAVi++ is a successor the SAVi, model, which learns to segment objects in video sequences without segment annotations. Both SAVi and SAVi++ take "hints" about where objects might be on the first frame (such as bounding boxes, though they can be trained without such hints.) They both reconstruct optical flow via a bottleneck of object slots.

SAVi showed several key limitations, in particular its poor performance on complex, realistic (or real) datasets and a general inability to segment static objects. SAVi++ makes progress on both of these fronts by using an additional cue / reconstruction target: depth, which may be supplied in addition to or instead of optical flow. Using depth as an additional cue, SAVi++ can learn to segment complex and realistic scenes better than its predecessor, especially static objects. Using sparse depth signals alone from a real-world autonomous driving dataset, SAVi++ can segment many of the objects despite never receiving ground truth segment annotations.
strengths_and_weaknesses: Strengths:

1. This paper makes progress on a challenging problem -- unsupervised segmentation of objects in realistic or real-world videos. This field has been limited mainly to simple synthetic datasets, so pushing toward real scenes is an important contribution.

2. Both key differences between SAVi++ and SAVi are well-motivated: depth is a useful cue for segmentation, and it makes sense that more expressive architectures (and data augmentations) would be important for scaling to real data.

Weaknesses:

1. The description of the literature on human grouping is a little misleading. It's not known whether perceptual grouping in humans is innate or not (L38). Some ability to group moving objects is present as early as two months, and possibly earlier -- it's nearly impossible to measure. What is clear is that grouping ability **matures** from only segmenting moving objects to being able to segment static objects as well. Whether this maturation is a learning process that depends on experience is unknown too, but that much at least seems plausible via a kind of unsupervised learning (and has been recently modeled to some degree in https://openaccess.thecvf.com/content/CVPR2022/html/Bao_Discovering_Objects_That_Can_Move_CVPR_2022_paper.html and https://arxiv.org/abs/2110.06562 and https://arxiv.org/abs/2205.08515.)

2. SAVI's main limitation is that it can't be trained (in its best form) on raw video datasets that contain nothing but RGB frames: it substantially benefits from (and on challenging datasets, effectively requires) object "hints" on the first frame, and also estimates of optical flow. The latter are pretty easy to come by using pretrained models, but many large-scale video datasets do not have bounding boxes for all objects. Moreover, the authors' motivation for requiring these hints (L141-148) is pretty speculative -- I don't know of evidence that infants require people to point at objects in order to group them. Even if there were such evidence, it wouldn't answer the question of how an infant converts the pointing into something like spatial localization of objects. so the initial hints are a practical limitation, and it's likely that humans can get by without them.

SAVI++ seems to still rely on these hints. There are some qualitative results of models trained without hints on the driving dataset, but I don't see quantification on any of them. It's important to know if the hints are still critical for SAVI++'s success -- it is a valuable contribution either way, but doing away with the need for those hints is a challenging problem in its own right.

3. The driving dataset looks like it's very heavy on cars. Being able to segment real scenes is a huge improvement over most earlier unsupervised models, but as far as video datasets go this one probably doesn't have the diversity of objects that others do (e.g. the DAVIS datasets or Kinetics.) The paper would be much stronger if SAVi++ could be shown to work on one of these more diverse video datasets. Short of that, could the authors provide some evaluation of SAVI++'s performance broken down by category on the driving dataset?

4. The reliance on depth signals restricts the use cases of SAVi++ more than SAVi. With optical flow, there are supervised and unsupervised models (e.g. RAFT and SMURF) that transfer well to real-world video datasets, so reliance on flow isn't much of a limitation. But is that true for depth? If the authors could show that sparse GT depth could be replaced with estimated depth (e.g. from stereo cameras, or a transfer from a pretrained monocular depth model) it would substantially relieve this restriction.


questions: None
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a model to improve on top of the SAVI model. The key differences include: utilising image depth as a signal for supervision and adopting scaling techniques during training to further boost the performance. The model obtains superior results on both a challenging synthetic dataset (MOVi dataset) and also on real-world driving videos (Waymo Open dataset).

strengths_and_weaknesses: Pros:
- With simple modifications, the model obtained clearly superior results compared to the SAVi model.
- The ablation study is detailed and clearly highlights where the improvement comes from
- Obtaining good performance on real-world video is a challenging, missing piece in video object-centric literature, and this paper brings good advancement in this direction.


Cons:
- Even if the model produces strong results, the technical novelty is quite limited. Adding supervision from the depth signal and training techniques such as data augmentation were previously used on video data. However, I agree that clearly isolating the importance of adopting them by the object-centric community is a significant contribution.

questions: - Is there any reason why no metrics are reported for the unconditional settings? There are a couple of qualitative comparisons in the paper but no quantitative ones (for e.g. to compare against unconditional SAVi and SIMONe ). I expect the performance to degrade a lot, but I still consider it important to know if this is a scenario where it is worth applying the proposed model.

- Compared to the baselines, the model requires additional supervision (the depth). The author did a great job in explaining how this signal is often available in the real world. However, experimenting with signals coming from predicted depth rather than ground-truth one would be interesting to explore.

- The results reported in Table 1 seem different from the ones reported in [16]. Is the setup in any way different from that one?

limitations: Since the slots are updated from one step to another, is the model able to “rediscover” an object if it leaves the scene and appears much later? Should we expect the model to attach that object to the original slot, or it will be attached to another random slot? Can this impact the final performance?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Learning the compositional structure (e.g. scene graph) in dynamic visual scenes without labels (e.g. object masks) has proven challenging.  Slot-based models leveraging motion cues have recently made progress in learning to represent, segment, and track objects without direct supervision. However, these methods only work on synthetic data and fails on complex real-world multi-object videos. This paper proposes to exploit depth signals to improve object-centric learning. It introduces SAVi++, an object-centric video model which is trained to predict depth signals from a slot-based video representation. The paper claims that by using sparse depth signals obtained from LiDAR, SAVi++ is able to learn emergent object segmentation and tracking from videos in the real-world Waymo Open dataset.
strengths_and_weaknesses: Strengths

SAVi++ improves on SAVi with predicting depth signals, and utilizing stronger visual backbone architectures in combination with data augmentation.

SAVi++ evaluates on Waymo Open dataset.

Weaknesses

Despite the claims "End-to-End Object-Centric Learning from Real-World Videos", SAVi++ has very limited results on only one real world video dataset, Waymo. The results on Table 2 do not compare with any significant prior work. For more convincing experimental results, the authors are encouraged to consider indoor datasets as well, SUN RGB-D.

The paper ignores a large body of work on monocular depth estimation, e.g. Monocular Depth Estimation Using Relative Depth Maps, CVPR, 2019. 


questions: For Waymo dataset, the authors should compare with SOTA self-supervised instance segmentation methods, e.g. SAVi and GroupViT. The paper should also mention the performance of SOTA supervised methods to clarify on the gaps between these two type of methods. 

Can SAVi++'s depth prediction module benefit from the large body of monocular depth estimation paper? 

It seems that the feature learning can use a strong baseline, e.g.  Self-Supervised Representation Learning from Flow Equivariance, ICCV 2021.
limitations: The paper does not seem to have stated its limitations. The work has several major limitations, e.g. relying on depth information, far from being competitive with SOTA supervised models, and lacking experimental results on real-world videos.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Brief Summary: The paper proposes an extension of a previous work SAVI [30] which used optical flows as a training signal to further include depth maps obtained from RGB-D cameras / LiDAR. The main goal is to learn from videos which have depth-channels but no instance/semantic segmentation ground-truths. 

Experiments on MOVi dataset types C,D, and E which are created synthetically, as well as on Waymo dataset obtained from real world data, show their proposed method SAVI++ outperforms compared baselines.
strengths_and_weaknesses: Pros:

1. With ever increasing 3d input data information, it is imperative that models are able to utilize such information for semantic segmentation tasks. Further, 3d data enables learning about static objects. As such, there is a clear motivation for the problem at hand.

2. The paper provides good visualizations such as Fig3, Fig5, as well those in supplementary material.

Cons:

1. In my opinion, the paper makes too general claims without providing previous citations while their actual technical contributions and evidence is far more limited. For instance, L48 suggests extending previous work is more challenging but doesn't explain why. L63 suggests it is the first model for segmentation without direct supervision or tracking supervision. This is strictly not true, given literature in domain adaption where one trains on synthetic segmentation dataset (such as Synthia or GTA) and domain adaptation transfer to Cityscapes (see [Ref1] for an example). Similar assertion is made in L3, but this ignores literature in video-text pretraining (see [Ref2] as example). 

2. On novelty: the technical novelty in the paper is slightly incremental. given that it mainly suggests using 3d-depth map instead of flow maps used in previous work. 

3. On Model architecture: 

(i) The reason for using Resnet34 is unclear (L193). What metric is being used to decide this ("more capable encoder" is too vague). 

(ii) Why not use something like Vision-Transformer instead, and remove the additional transformer encoder? Design-wise that looks much simpler?

4. On Experiments:

(i) MOVI is synthetic dataset, and Waymo dataset appears to be quite small with only 200 validation videos. More datasets should be tried such as Cityscapes, Cityscapes-VPS, VIPER (see [Ref3]).

(ii) The proposed method performs significantly worse on MOVI-A,B (as shown in Table 1 of supplementary material).  If the reason is indeed due to overfitting on simpler domains as claimed in suppl L26, there should be some experiment with same encoder.

(iii) The compared baselines in Table 1. are quite weak. The authors should compare their model with other methods trained on depth maps, at least some naive extensions of other previous works to use 3d-maps (such as for ODIN [19]).

(iv) The authors should show fine-tune performance on other segmentation tasks such as coco and pascal. 

(v) Some design decisions on the choice of visual encoders should be empirically justified. 

[Ref1]: Hoyer, Lukas, Dengxin Dai, and Luc Van Gool. "HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation." arXiv preprint arXiv:2204.13132 (2022). 

[Ref2]: Miech, A., Alayrac, J. B., Smaira, L., Laptev, I., Sivic, J., & Zisserman, A. (2020). End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9879-9889).

[Ref3]: Kim, Dahun, Sanghyun Woo, Joon-Young Lee, and In So Kweon. "Video panoptic segmentation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9859-9868. 2020.
questions: Q1. How important is the high resolution in waymo dataset? 

Q2. What is the main bottleneck in training time, given that the underlying datasets are not very large? Is it sequential time-steps for filling in the object slots?

Q3. In L100, it is noted that no PGT data is needed. Does using PGT data give improvements, and if so how much?
limitations: Yes. Some more discussions on where the models are likely to fail such as low-data domain, as evidenced by L25 of suppl, could be useful.
review_title: null

Meta-review: metareview: Three out of four reviewers provided positive reviews and scores for this submission. They agreed that SAVI++ makes meaningful improvements over a previously proposed SAVI model. Importantly, while most past approaches evaluate on synthetic data, this submission evaluates the proposed model on a real world dataset. The proposed model clearly improves over the baseline and a clear ablation analysis shows where the improvements come from. 

One reviewer had concerns about the evaluation using just one real world dataset. This was also brought up by other reviewers, who mentioned that the Waymo dataset has less diversity and fewer videos than others. While a more thorough evaluation would make this a stronger submission, the leap from synthetic evaluations to real world evaluations in this line of research is notable and sets the bar for future work. I also note, based on the discussion, that the employed dataset is not trivial and has several challenges for the model. 

Another concern by the reviewer was about missing baselines. The authors did provide additional baselines in their response. While these baselines do not exactly match the ones requested by the reviewer, I think they provide good evidence that the proposed method is able to employ the depth signal effectively. 

Overall, this paper makes solid progress on the problem, provides value to the readers and provides strong results on a real world dataset. Given these reasons, I recommend acceptance.

confidence: Less certain
Decision: Accept