Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a way to segment objects in unsupervised fashion.  The idea is to define a signal to a segmentation model based on the idea that if the object were properly segmented, then the predicted mask could be used to shift the object (by a small translation within the same image) in such a way that this new shifted image looks realistic as judged by a discriminator.  As part of generating this new shifted image of the object, the authors propose to use MAE (masked auto-encoders) to inpaint the missing region which is uncovered by the simulated shift.

The paper demonstrates that this proposed model “MOVE” outperforms comparable SOTA models on unsupervised saliency detection and unsupervised object detection benchmarks.

strengths_and_weaknesses: I enjoyed reading this paper — the results are strong and seem to quite significantly outperform the state of the art.  Given the complexity of the approach, getting this level of performance is quite impressive and points to good execution.  Conceptually, the trick that the authors propose to deal with artifacts generated by MAE (specifically they propose to synthesize these artifacts in images where the object is unshifted which forces model to focus on differences other than the artifacts) is also simple and effective.

The main weakness of the approach is probably the complexity of the model which requires a segmentation model, a separately pre-trained masked auto-encoder and training both in a GAN-like setting, so I suspect that reproducing this model is not easy.

questions: * The cut+paste citation (Remez et al 2018) is slightly mischaracterized — this paper did not require pasting objects to different images (it was always to the same image).  Related to this, the Remez et al approach had an issue of inferring that shadows were part of an object.  What is the behavior of MOVE when there are shadows?
* In Section 5, the authors claim that MOVE does not require training of a generative model —- but this seems somewhat strange — is there a clear line that separates a model such as MAE from generative models?
* How necessary was it to use a DINO pretrained backbone?  Would starting from scratch have worked?

limitations: Yes
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes an unsupervised segmentation approach using the "movability" as the cue to differentiate between foreground and background. The pretext task is to segment the foreground object, shift it into a new location, and compose the new image with inpainted backgrounds. The approach relies heavily on pretrained models, DINO for the segmentation network, and MAE for the inpainting network. A GAN loss is used to encourage the composed image to be realistic. The method assumes that the segmentation network has to segment the foreground object in order to synthesize realistic images. Encouraging results are reported on saliency detection and unsupervised object discovery.
strengths_and_weaknesses: Strengths:
+ The idea of using inpainting to formulate a self-supervised learning pretext task is interesting.

+ The empirical result is encouraging. Notable improvements are obtained against prior works on all benchmarks. 

Weaknesses:
- The method relies heavily on pretrained models, yet without discussing the significance of it. I suspect that the method would not work at all without pretrained DINO & MAE models. The general claim of the method should work even without any pretrained models. I am not convinced about the paper it does not work standalone.

- The writing is poor, especially in the approach section page 4. Description of the method is disorganized. 


questions: - I think it is inappropriate to use the term "movability" to describe such method. It would confuse readers that the method exploit motion cues to segment objects in videos. It is also not correct that any foreground object could be displaced to any location. For example, the monkey could not be displaced to the sky. Also, other background objects or stuffs could be displaced properly. For example, the tree log could be moved left or right. 

- In figure 3, the encoded foreground looks identical to that of the input foreground. Why do we have to encode foreground through MAE?

- Also in figure 3, the inpainted background looks unreal. The grassland behind monkey now becomes homogeneous green region. Why is this a meaningful task? The discriminator would only needs to find these inpainting artifacts in order to discriminate between real and fake. No matter predicted mask is, the inpainted results will always have artifacts which may impede GAN learning. 

limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces a framework for unsupervised image segmentation. The assumption that it exploits is that a slightly shifted foreground segment overlayed on an inpainted background image, whenever the former is meaningful, must look realistic. The work develops and carefully implements this idea using a transformer to generate and inpaint the masks. A discriminator network is tasked to distinguish between the generated images and the real images, where appropriate choices were made to avoid degenerate solutions by augmenting the real image distribution. Elaborate experiments on saliency detection, single object discovery and detection, demonstrate substantially improved accuracy compared to previous state of the art.

Overall, the paper is insightful, original and technically sound. It makes a clear advance in the problem of unsupervised segmentation.

**Post-rebuttal comment.**
I thank the authors for futher elaborations. Regarding the concern of *R-tM6J*: I agree that using pre-trained features (or even depending on it) is a limitation. However, I do not see it as critical. After all, the pre-training is completely unsupervised, and performing accurate segmentation on the representation learned – especially to the degree demonstrated in this work – is highly non-trivial.

In short, this is a nice work that deserves to be presented at the conference, which I strongly recommend.

*To the authors:*
- Please do elaborate on the methodology for tuning the framework, reported in Tab. 1&2, in the revision. This increasingly  becomes a source of confusion in the literature on unsupervised learning, impedes reproducibility and fairness of comparisons.
- The ablation Table 4 needs some revision. Adding some clarification in the caption, or changing the table layout may help.
strengths_and_weaknesses: Strengths:
* clearly written;
* contains a number of technically innovative bits, since implementing this conceptually simple idea is actually prone to degenerate solutions; 
* extensive experimentation with strong results across the board.

A minor weakness:

The main idea exploited here re-implements the “cut-and-paste” principle from previous studies, but in the context of the latest advances in network architectures (ViT), self-supervised learning (DINO) and autoencoders (MAE). However, this is given appropriate credit in the text (l. 29). Also, the focus on small shifts, in contrast to some previous works, also makes sense geometrically, since perspective distortions can be neglected.


A few suggestions:

Fig. 2 and 3 are disconnected and show different parts of the pipeline. Having a unified “overview” diagram, that also depicts 
- what images the “fake” and “real” sets comprise (as the input to the discriminator),
- which network modules remain fixed and which are trained,
would help in accompanying the text in Sec. 2, which is is quite dense.

In Tab. 1-3, it would interesting to see an overview of the computational footprint of those methods (e.g. the backbone / FLOPS) and/or the datasets used for training (e.g. “ImageNet + DUTS-TR”) for this work.

Images in Fig. 4/5 are too small. I’d rather use fewer samples, but in larger size. More images can be always added to the supplemental.

questions: A few open questions:
- Since the framework is completely unsupervised, what stands in the way of training it on larger datasets (e.g. ImageNet)? Would it scale?
- What criteria and data were used to choose hyperparameters?
- How crucial is the ImageNet self-supervised pre-training of the feature encoder?
limitations: Limitations and the societal impact are discussed at appropriate length.
review_title: null

Meta-review: metareview: This paper received mixed scores, with two reviewers recommending acceptance (Strong Accept) and one rejection. The paper was thoroughly discussed by the reviewers and the authors, but the reviewers failed to reach a consensus. Ultimately, after the authors' feedback having addressed most of the reviewers' concerns, the main point of disagreement that remains is the fact that the proposed method cannot be used to train or fine-tune the backbone parameters. While tM6J sees this as evidenced that the method is not sound, 9LXw and tSAq both see sufficient merit in the approach to consider this only as a minor drawback that will eventually be addressed in the future. Considering that the method nonetheless learns some parameters and produces convincing results, the AC agrees with 9LXw and tSAq that one paper does not necessarily need to address all problems. We nonetheless strongly encourage the authors to make this clear in the final version of the paper.
confidence: Certain
Decision: Accept