Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a novel method for self-supervised object discovery from video. The authors cast the object discovery problem as optical flow construction problem, where optical flow provided by an off-the-shelf method(RAFT) is used as training signal. The authors demonstrate the effectiveness of the proposed method on several popular datasets includes DAVIS2016, SegTrackv2 and FBMS-59 by comparing to state-of-the-art methods. Compared to other self-supervised object discovery methods exploiting motion cue, the proposed method directly operates on RGB input domain without optical flow computation in the inference stage, thus has advantage on inference speed. 
strength_and_weaknesses: Strength:
1. The proposed method has superior performance on public datasets compared to state-of-the-art methods with similar constraint(Liu et al., 2021). 
2. The proposed method has close or superior performance on public datasets compared to state-of-the-art methods with less constraints, e.g. using optical flow in the inference stage, use synthetic training data. This also proves the effectiveness of the proposed method.
3. The proposed method effectively exploits temporal fusion so motion cue from multiple frames could be used to discover the object. This makes the method being robust in the cases where object is partially occluded, or is static in partial frames. 
4. The proposed method is faster in the inference stage compared to other methods, thanks to the setting of operating on raw RGB video clips.
5. The authors did ablation study to show the impact of each component of the proposed model.

Weaknesses/Questions:
    1. The proposed method is memory intensive. Is this the reason that only up to 7 consecutive frames are used in the experimental setting section?
    2. If the foreground object only appears in part of the video(we don’t have supervisory signal so we don’t have information on this), would the method still work? What would the alpha value for foreground be in those frames where foreground doesn’t exist?
    3. Since optical flow is the training signal, how would the OF quality/failure impact the result? It might be good to evaluate on a synthetic dataset and compare the training with both ground truth OF and estimated/flawed OF.
clarity,_quality,_novelty_and_reproducibility: I would prefer the "related work" section to be put before the modeling section, rather than just before conclusion. But the paper is clearly written and well presented overall. The proposed method is novel in my opinion. 
summary_of_the_review: In this paper the proposed object discovery method is practical and novel. The authors did a good job on modeling the problem, perform experiment on standard datasets and doing ablation study. I would recommend acceptance of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised model for video object discovery. It only takes consecutive RGB frames as input instead of optical flows as in previous works. The model is trained to reconstruct the optical flow between any paired frames that generated through an off-the-shelf optical flow estimator. A temporal consistency loss on motions at different paces is proposed to encourage the model to segment the objects even if they stop moving or are partially occluded at some time point. Experimental results demonstrate its superior performance on several datasets. 
strength_and_weaknesses: Strength: 
- This paper is well-written and easy to follow. 

- The motivation for using RGB as input instead of flow is clear and reasonable, i.e., having cues when static or partially occluded and keeping rich texture information. 

- The proposed model architecture and temporal consistency loss have been aligned with these motivations. I like the idea of directly using the RGB frames as input to segment the object as the optical flow is implicitly included in a sequence of the RGB frames. With the design of temporal fusion and frame comparator module, the model can learn optical flow implicitly instead of precomputing it explicitly. The design of temporal consistency loss help to deal with the static or partially occluded situation.

Weaknesses:
- I read your supplementary material about using DINO for test-time adaptation results, but it is unclear to me how do you manage to use the feature from DINO to do mask prop. If you use the idea of mask propagation. Will the mask error be accumulated along the frame? Why it can boost performance a lot?

- Although the proposed method provides several ablation studies, I still suggest the authors conduct the following ablation studies to enhance the quality of the paper: 

    * Does the number of learnable queries in the slot attention module impact the performance? Since the slot attention module is initially designed for segmenting different objects in a toy dataset. If there are more queries, are we able to tackle the limitation of segmenting multiple objects separately? 

    * Have you tried other optical flow methods such as PWC-Net or ARFlow. It seems that the RAFT optical flow results are very noisy on SegTrackv2 and FBMS-59 datasets using the script provided by motion grouping. Sometimes it is even enabled to identify the object for instance with frame gap =1,2 when the motion changes are too small. Do you think it will hinder your learning?

- A minor question regarding the sliding window during inference. Is that an overlapping sliding window or a non-overlapping sliding window? 

- As you mentioned in the limitation, the proposed method is not able to segment multiple objects. Can you put a figure to visualize how it fails (i.e. only the main object is segmented or multiple objects are partially segmented)?

- Some related work is missing: 

    * Yanchao Yang et al. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In CVPR, 2021.

    * Vickie Ye, et al. Deformable sprites for unsupervised video decomposition. CVPR, 2022.

    * Yangtao Wang, et al. TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut. arXiv preprint arXiv:2209.00383, 2022

clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is well-written and easy to follow.

Quality: Overall, this paper has good quality.

Novelty: This work adds two more modules(i.e., Temporal fusion and frame comparator) on top of Motion Grouping model to get slightly better results, which is not novel enough for me. 

Reproducibility: I believe the work is reproductive. Code was provided but  not carefully checked.
summary_of_the_review: This work on self-supervised learning video object discovery using RGB frames only is well-motivated and designed. However, the proposed model seems to add two more modules on top of the Motion Grouping model and the novelty is limited. Although several empirical results have demonstrated the performance on several datasets, more experiments are still needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
The paper tackles the task of unsupervised moving objects detection from videos. The approach takes as input the video sequence, without flow information.  The network consists of a sequence of known components, ie encoder, transformers, and slot attention decoder, and output a three layer map, ie, the segmentation mask and the bidirectional optical flow. It is only partially un/supervised, given the fact that the main loss function compares the output optical flow to a ground truth one, the lastest one resulting from a fully supervised approach. One of the motivations of the work is to address the problem of 'weak motion', ie when the object might be temporally static in a few frames of the input sequence, or when there are occlusions.  Experimental results are reported on three datasets and compared with related work. 
strength_and_weaknesses: 
The overall goal of the paper touches to one of the key aspect of modern computer vision, ie unsupervised object detection. Using video as input is indeed a key element, as it enables to separate foreground from background. 
One of the interesting observations of the paper (thought not new), is that pairs of frames are not sufficient to accurately detect the motion and hence the object: each frame should be aware of the whole sequence so that the frame-wise segmentation accounts for the whole sequence global motion. This is enabled by the transformer \Phi_temp, which fuses altogether the frames embeddings.

The work attempts to move towards fully unsupervised training, but does in fact use some level of supervision via the OF loss. Similar scenario had been proposed by Choudhury and al (2022). An other choise could have been, instead of supervising the Flow reconstruction loss,  to minimize the image reconstruction loss, after wrapping one of the frames pair to the other. Other proxies to optimize and to constrain the estimated flow could also have been used. 

I am intrigued by the behavior of the segmentation masks. I am not sure what prevent the segmentation masks to be uniform in all frames (all ones and all zeros, for s=1,2 or reciprocally). Uniform masks in all frames would lead to an L_entro and L_cons to be zero, which is a situation which could be favored by the network, unless other constraints are given.

In the experimental section, I am not sure what are the results reported for OLCR.  Xie and al 2022 report results without and with test-time adaptation. Only results without adaptation are reported. However, results with test-time adaptation, which the authors also use, outperform the authors results for SegTrack and FBMS, and are very close for DAVIS2016. 
clarity,_quality,_novelty_and_reproducibility: 
Clarity, quality
The paper is clear, the reading is fluid, the approach and its advantage are well discussed. The whole framework is well presented, including graphs and illustrations. The ablation stutThe appendices gives a level of details sufficient enough for the work to be reproduced. The authors will open  the source code. 
Visual illustration could be improved by adding the images of the difference between the segmentation output and the object mask ground truth. Since the visual improvements are very local and not ovbious, it would facilitate the interpretation of the visual outputs. 
I would personally suggest to put the section Related work after the introduction rather than at the end of the paper. In particular, the work is built upon previous similar and related approach, and it seems to me preferable to mention them at first.  

Novelty
The work motivation, ie unsupervised object detection from video without using OF as input, is a continuation of Choudhury and al (2022). From a technical point of view, the network architecture (and loss functions) mainly reuse components which have already proved to be efficient in similar scenarios. 

summary_of_the_review: The main strength of the paper is certainly the clarity and quality of the work.  It is in the continuation of Choudhury and al (2022) (both for the scenario and the quantitative results).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a method for unsupervised object discovery in videos. In contrast to the previous methods based on the optical flows between adjacent frames, they directly adopted RGB frame sequences. Specifically, the proposed model first encodes RGB frames into visual features, then pass the paired features into frame comparator module that computes relative motion. Finally, dual-layered flow decoder is introduced inspired by slot attention works that outputs two slots, i.e. foreground and background. Experiments are conducted on three benchmarks, surpassing the previous methods.
strength_and_weaknesses: ** Strengths
1. Directly relying on RGB frames to overcome the posed limitations from the usage of optical flows
2. Combining several components from other literature to achieve state-of-the-art performances, e.g. iterative routing for layered representation from MG (Yang et al., 2021a) and test-time adaptation from OCLR (Xie et al., 2022)
3. The submitted code and implementation details in the supplementary material facilitate reproducibility 

** Weaknesses 
1. Mild novelties; The overall framework seems quite similar to the previous idea introduced on MG (Yang et al., 2021a), particularly for dual-layered flow decoder, except for using RGB frames as input and encoding them further through frame comparator. The introduced three loss functions also lack of originalities as they are commonly used in literature for video-based prediction; flow reconstruction as a proxy task for unsupervised learning, temporal smoothness, and contrast of binary masks.
2. Usage of the optical flow; While the authors claim the drawbacks of optical flow in terms of defected supervisory signal, their training object still rely on the pre-computed optical flow, RAFT, that also might suffer from static scenery and occlusions. 
3. The need of supervision; I understand that the proposed method do not require supervision for video object segmentation, but the generated optical flow is yielded by the supervised method, RAFT on MPI sintel and KITTI. I would say the required cost to annotate optical flow ground truth for each pixel is much more than the one of binary segmentation. The authors need to demonstrate their method also fairly works well while using unsupervised optical flow methods; such as ARflow.
clarity,_quality,_novelty_and_reproducibility:  Please see the strength and weaknesses section above.
summary_of_the_review: As the weaknesses outweigh strengths, I lean towards rejection at this point.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: While this paper demonstrated impressive empirical results for the task of self-supervised object discovery, there were concerns regarding the contributions in context of prior work (MG (Yang et al., 2021a) or GWM (Choudhury et al., 2022)). More concretely, compared to prior work, this paper uses a similar overall approach, with the distinction that the input modality varies from optical flow to pairs of RGB images. While the author response clarifying this distinctions with prior works is noted and the empirical benefits of RGB images as input are interesting, the AC agrees with the majority of the reviewers that these are not major contributions. 
justification_for_why_not_higher_score: While the paper demonstrates some  promising empirical results, it makes very minor changes (changing the input modality from flow to RGB image pair) in context of prior work.
Decision: Reject