Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes to decouple the motion and appearance by two pathways: the appearance pathway is designed to learn object segmentation masks, while the motion pathway aims to predict the optical flow per region mask. The whole model is learned in a self-supervised manner by frame-reconstruction loss. While such region-wise, which is in contrast to the typical way of learning pixel-wise dense optical flow, is interesting, the overall novelty is not sufficient for NeurIPS.  
main_review: Originality: The idea that learning the region-wise optical flow to identify primary/moving objects is relatively new. The framework of decoupling the motion and appearance features by two single pathways is not quite novel. Overall, the originality is limited.

Quality: The overall framework of the submission is technically sound. However, since the detailed implementations of each modules (two pathways) are too simple, I doubt its effectiveness.  I doubt that such implementation under the self-supervised learning can only simply segment the primary object. 

Clarity: Overall the submission is written clearly. 

Significance: the novelty is limited.



limitations_and_societal_impact: The whole framework sounds feasible, however, the model is designed coarsely and I doubt whether the full potential of such framework has been exploited by such implementation. For instance, the appearance pathway is designed simply by multi-convolutional layers to predict 'c' masks, which is arduous to learn the segmentations effectively. The motion pathway obtains the region-wise optical flow simply by pooling the predicted pixel-wise motion features. The whole model is supervised by simple frame-reconstruction loss. I doubt its effectiveness. As shown in the paper, the model can only identify the primary object in a scene where there is only one primary moving object. Besides, the strategy for identifying the primary mask by  "This channel can be heuristically identified as the one whose segmentation mask is closest to the image center" is not an elegant solution.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Paper proposes a novel approach to unsupervised object segmenation based on temporal consistency in video sequences.
The authors show good results across a range of video datasets, and that the models trained without supervision generalise well to figure/ground segmenations on image based datasets.

The method itself is conceptually straightforward, decomposing the image into a fixed number of parts so that each part has a low photometric error when propegated to a future frame, using a constant velocity model.
main_review: Paper is well written and the results are compelling, clearly outperforming unsupervised approaches, and compairing well with approaches that make use of supervised components.

The experiments are extensive, and the inclusion of supervised methods that perform better than this approach is welcome, and helps put the overall results in context.

Arguably this paper would be better suited for a CV conference, essentially we have very standard ML choices composed with a standard loss, and limitted novelty from a pure ML perspective. However, I believe that the task itself - completely unsupervised object discovery from video is of strong interest to the wider ML community, and the paper therefore is a good fit for neurips.

I'd characterise it as having sufficient novelty, that's made up for by the relevance of the approach to the community.

Minor comments: 

Line 177 Don't use FCN as a standalone abreviation, it can stand for either fully connected network, or fully convolutional network, and leaves the reader guessing.

non-published -> unpublished

limitations_and_societal_impact: Fine. Work is unlikely to have direct social impact.

 Visualisation of the other segments that do not correspond to objects would be welcome, and help the reader understand the behaviour/limitations better, as would showing some failure cases.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a method that segments the image into a number of segments and estimates an offset vector for each segment. Different from prior two-stream appearance+motion based methods, this approach does not leverage a pre-trained flow network but learns to estimate motion offset vectors implicitly by warping (segmented) regions across frames based on the flow estimates. Reconstruction error (inspired by self-supervised optical flow methods) is used as a loss function and provides a supervisory signal. Thus, the proposed method can be trained using unlabelled video. 
main_review: Pros
-----------

* This paper is tackling a very relevant research problem of learning to segment objects without supervision, leveraging a large video corpus. The way to go! 
* The proposed method is simple but interesting and with the potential to scale very well. 
* The paper is generally well-written, and the methodology is presented clearly. 
* Results do indeed look quite promising: on-par with state-of-the-art on DAVIS, compared to methods that use no supervision. 
* I appreciate that the paper makes an effort to demonstrate the versatility of this method for several tasks, ranging from VOS to object segmentation in images and as unsupervised pre-training for semantic segmentation. 


Cons
-----------

* This paper claims that this method is trained absolutely without any supervision. In principle, this is true: network weights are initialized randomly before training the model on YouTube VOS dataset. However, I would be careful regarding “no supervision” claims because the results demonstrated were obtained via training on a highly curated dataset containing various biases, most notably photographer bias (objects are almost always centered in these videos). This suspicion is pretty much confirmed in L182-184: “We have empirically observed that the moving objects all appear in a particular mask channel across the training videos. This channel can be heuristically identified as the one whose segmentation mask is closest to the image center.” It, therefore, appears to me that the model learns to segment object that appears in the center of the video (or image), based on carefully selected videos that mostly contain one object of interest per clip (objects are present from start to the end of the video). This fundamentally differs from real “in-the-wild” scenarios (e.g., learning from a stream of sensory data that a robot observes). On the other hand, let me also note that most papers that claim to do unsupervised learning do not acknowledge the manual human effort invested in curating datasets that are used for learning. To conclude this thought, the paper would severely benefit from clearly stating assumptions made and limitations. 
* It is not clear to me at all how ResNet50 backbone + a FCN head yield segmentation of an image into multiple segments. Here a crucial piece of information is missing. 
* Optical flow methods (that several baselines use) can be trained in a self-supervised fashion, or using synthetic data. It can be argued that these methods do not use supervision either. I am not saying that it is not “nice” to be able to train the model from scratch, not relying on any external components, but given that baselines (CIS, MG) obtain better or significantly better results on DAVIS, this point should be argued better. 
* While Tab. 1 shows promising results, results and conclusions are also somewhat confusing. Apart from traditional methods, both learning-based baselines perform better on DAVIS (they do, though, use optical flow, but see my comment above). The proposed method performs better than CIS on SegTrack and FBMS, but those two datasets contain stronger photographer bias (correct me on this if I am mistaken). 
* I also have my doubts regarding zero-shot detection results on the DUTS dataset, and I am not sure if I agree with the conclusion. It again seems to me that the method learned to detect exactly the object that is right in the center (looking at images, Fig. 4). A better version of this experiment would be per-frame evaluation on DAVIS, which also contains sequences where the object is not always centered, and occasionally multiple objects are present. 
* I also have concerns regarding using this approach as unsupervised pre-training (which in general is a great idea!). First of all, I would definitely need to see ImageNet pre-training for comparison purposes, otherwise, it is difficult to anchor the results reported wrt. prior work. The same goes for SimSiam and MoCo-v2: ImageNet-trained variants are missing. I do understand why the paper reports these two methods trained on the same data (which indeed should be done in addition to ImageNet trained variants), but I got extremely confused with augmentations here. The core idea behind contrastive learning-based methods such as SimSiam and MoCo is to leverage augmentations to form positive pairs. If those are not used, I do not understand how these methods are trained. Perhaps I misunderstood something.

Tips
-----------

* This paper advertises zero-shot segmentation, but I don't think the terminology is used accurately (wrt. Xian et al., Zero-Shot Learning -- The Good, the Bad and the Ugly, CVPR’17) as the classes that are segmented in the DUTS dataset are present during the training phase, at least most of them, however, they are not labeled. This is rather a case of transductive zero-shot learning. This doesn’t impact my rating, just a remark/tip. 


Justification
-----------

In principle, I like the idea behind this paper: it is simple but has the potential to be quite effective, and I believe that there is a need for such methods for unsupervised representation learning/object segmentation (simple methods that can scale). However, at the moment, I see too many issues with this manuscript to recommend acceptance at this time. The most critical part is the experimental evaluation: in general, I like that the paper makes an attempt to demonstrate the versatility of this approach for several tasks (VOS, object segmentation, unsupervised pre-training). However, no experiment is particularly convincing (see my criticism above), and raises more questions than provides answers. 

I would recommend the following:
* Provide a more thorough discussion on the assumptions and limitations of this approach (see my comments on photographer bias and dataset curation). 
* It would be interesting to see a variant of the proposed method that does leverage optical flow as input (wrt. Tab. 1). However, this experiment is the least problematic in my view. 
* More importantly, claims regarding zero-shot saliency detection and unsupervised pre-training need stronger experimental verification; in particular, detection capabilities should be evaluated in settings where the object is not always centered and potentially multiple objects are present. Otherwise, the experimental setting is too constrained. Furthermore, I cannot judge the performance of this approach for unsupervised pre-training because the results are not discussed and placed well relative to prior work (see my comments above). Ideally, unsupervised pre-training should also be compared to prior work on unsupervised representation learning from video (the first one that comes to mind is Wang et al., Unsupervised Learning of Visual Representations using Videos, CVPR’14). 

Other than that, thumbs up for generally well-written paper and method that appears to me to hold quite a lot of potential! 

**Post rebuttal comments and justification**
It was particularly challenging for me to reach my final recommendation for this paper; as I've mentioned, I like this research direction. The simple yet scalable approach to learning representations from abundant video looks very appealing to me, and I find this work valuable.

In the end, the main question was how convincing is the experimental evaluation and how firm conclusions I can make. As mentioned in my review, I do appreciate that evaluation is versatile, but in the end, I am afraid that I still think that none of the experiments is highly convincing.

VOS evaluation Reviewer zbkz raises the concern that those comparisons are not entirely fair, as the proposed method is the only one to train on YouTube-VOS, which is for order of magnitude larger than DAVIS, SegTrack, and FBMS. Baselines (CIS, MG) were only trained on the target datasets but did utilize pre-trained flow networks. I am afraid I agree that the fairest direct comparison would be to pre-train the network using synthetic data (commonly used to train optical flow networks) and fine-tune it on DAVIS/FBMS/SegTrack.

But perhaps the real question would be: how useful such a pre-trained network would be for downstream tasks, for example, semantic segmentation?

It turns out that this method significantly lags behind SOTA (MoCo-v2, SimSiam, 72.4 mIoU) on VOC2012. I agree with the assessment that ImageNet is better aligned with VOC2012. However, (i) the task itself (moving object segmentation) is more similar to the target task of semantic segmentation, and (ii) there are large automotive datasets out there to which this method could be applied (e.g., pre-train on BDD100k [Yu et al., CVPR'19], train on Cityscapes). I am afraid that the paper did not succeed yet to demonstrate its merit in pre-training for semantic segmentation.

I am afraid I am also not convinced with the results obtained on the DUTS dataset; it appears to me that the network learns to detect anything in the center. I am afraid I disagree with the author's statement that this center bias is present everywhere (it isn't present in robotics/automotive datasets). Another way to show the merit would be to use this approach for pre-training on the instance segmentation task (e.g., on COCO).

I would very be happy to see a future revision of this paper that would present a very convincing experimental evaluation. I would point to [Radford et al., ICLR'21 (CLIP)] for reference (simple method, thorough evaluation) and entirely focus on demonstrating the merit via convincing experimental evaluation.
limitations_and_societal_impact: I see no issues with potential negative social impact, but the limitations were not addressed adequately (details in my review above).
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work develops an approach to unsupervised single-object segmentation. The main idea is to segment the scene into $C$ (hyperparmeter) segments in one frame and to learn a flow field w.r.t. another frame such that it can be reconstructed by translating the segments from the original frame. This simple strategy provides competitive accuracy on standard object video segmentation benchmarks, as well as shows promise in unsupervised saliency detection (DUTS) and representation learning (VOC12). 
main_review: *Originality.*
The idea seems to be original and simple. In fact, so simple that a number of key design points remain unaddressed:
- the model is very sensitive to $C$, the number of masks for prediction, as becomes evident from Tab. 4. 
- selecting the segmentation mask based on its proximity to the centre (l.184) does not appear principled, is highly dataset dependent and puts the fairness of the comparison to prior art into question (esp. for the saliency benchmark).
I also suspect that disocclusions (“holes”) and self-occlusions may pose a problem. In the first case, suppose an object moves and the camera is static. The parts of the image that become disoccluded cannot be reconstructed from the image where there were occluded in the previous frame. How does the method cope with it? For the second point, assume an object (e.g. a ball) rotating around its own axis. Approaches based on dense flow will be able to detect this motion. How would the method handle this scenario?

*Quality.*
The work is well-presented and argued. In particular, the argument that previous work relies on optical flow pre-training whereas the presented method doesn’t, is a good point. However, I would still not overstress it, since the method does use an optical flow network for motion prediction (PWC-Net); and dense optical flow can be learned from synthetic data.
The work also needs to further clarify the following points:
- What is the model accuracy without test-time adaptation? When is it reported with and without this process in the tables?
- How can it be that Fig. 3 shows significant improvements w.r.t. CIS, yet in Tab. 1 CIS actually performs a bit better?
- How is $C$ selected exactly? (l. 140 “empirically” is not concrete). This needs to be clear for every experiment and the dataset. 
- What data (datasets) do the methods in Tab. 2 use for training / hyperparameter selection? This needs to be reported for transparency.

*Clarity.*
The work is generally well-written, but not without a few minor deficiencies:
- notation is not rigorous. For example, Eq. 5 multiples $\mathbb{R}^2$ with $\mathbb{R}^{c \times h \times w}$.
- some missing implementation details in Sec. 3.6: batch size, data augmentation used; how was the termination criterion selected?


*Significance.*
The approach provides tangible improvements w.r.t. previous work, is slightly worse than concurrent work ([6], but it is less of a concern). However, the method also appears much more prone to hyperparameter choice; how those were selected is important for the fairness of the comparisons and needs to be stated unambiguously.


**Post-rebuttal update**

I thank the authors for detailed clarifications. I find the presented results very encouraging in view of the method's simplicity and versatility. However, I still maintain that head-to-head comparisons to related methods (i.e. using the same training data) are essential, whenever technically possible, in order to understand where the approach fits in a larger body of previous work (and even if this use case was not originally intended). One of such experiments in the VOS setting -- pre-training on synthetic data -- is feasible, but not presented.

More generally relating to my point on (self-)occlusions, I'd also like to recommend including a short discussion about the assumptions underlying the approach mainly to clarify: When does it make sense to apply segment flow for the reconstruction loss? This is because such transformation generally does not apply due to the 3D nature of the scene.

Overall, I share the sentiment of Reviewer Y1LA that this work has potential, and encourage the authors to re-submit the manuscript after the revision.
limitations_and_societal_impact: Some of the key model limitations do become apparent after reading the manuscript. Nevertheless, I would recommend offering this discussion more prominently in the main text.
Societal impact is not addressed in sufficient depth (Sec. 3 in supp. “Broader Impact” mentions only “privacy”), hence needs to be elaborated on more in the revision.
review_title: null

Meta-review: Reviewers did not reach a consensus for this paper. Most of them like the idea in the paper and consider that the experimental section is comprehensive, but some of the reviewers are not fully convinced by any of the experiments. 

There was also considerable discussion over if the method depends strongly on photographer bias. I am not too concerned about that in training: one can imagine a curriculum of learning where a method like the proposed is first exposed to videos with photographic bias, and progressively it moves to more complex data. Arguably the proposed model may not scale all the way, also since it has simplifying assumptions like a constant velocity motion model, but it may be able to serve as a good bootstrapping mechanism. There was some criticism in discussion that the model trains on more data than competing methods, but being unsupervised i wouldn't hold that against the method

I join the reviewers in enjoying the simplicity of the approach and understand that the experiments can be improved but think that they are decent already. In any case i encourage the authors to look carefully into the reviews and improve the experiments following the reviewers advice, namely trying to understand how good the segments are in multi-object data. I would be curious if the model still learns on more complex video data, such as imagenet-video or larger action recognition datasets.
Decision: Accept (Poster)