Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The author presents a method for unsupervised object segmentation by using temporal motion patterns. The paper transforms the segmentation as a minimizing ELBO problem. Some practical tricks of Gumbel-softmax have been applied to make the ELBO differentiable. The evaluations on the physics-related simulation datasets were conducted and outperforms the other baselines. 
strengths_and_weaknesses: This paper converts a classic optimization problem into a differentiable learning problem, the overall idea is very interesting. The experiments have shown the proposed method has achieved the SOTA performance on simulated datasets.

One weakness is no real-world scenario experiments, it would be more convincing if some of the experiments were conducted on real-world videos. 
questions: One question is how well can the methods perform in real-world cases. I’m not asking about adding more results in the rebuttal. It would be helpful if the authors can explain if the methods can apply to real-world cases and what are the potential limitations. For example, the KITTI dataset is one possible option, as it involves mainly rigid motions. But meanwhile, KITTI is a moving camera case, what are the possible concerns if the model is applied to a moving camera case.    

How did the prior distribution p0(m) initialize and how did different prior distributions affect the results?

It will be interesting to see what the authors’ thoughts on the potential to apply the proposed methods to the no-rigid motion are? What are the further requirements to achieve that?

Some other detailed comments:

Line 111: Please elaborate on the assumption “We then assume that the flow depends only the regions”

Line 138: Please justify that “ We assume that regions are statistically independent ” is a reasonable assumption.

Matrix I first appeared in equation (5) but is explained in line 161.

line 2 form -> from.

Ling 156, missing space after as.

limitations: As the author stated, sometimes the motion doesn't provide enough information to segment the object. Maybe adding scene flow (3d flow with depth) can help to improve the performance.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors train a segmentation network that processes still images to produce object masks. Video based supervision is used. The masks predicted for a single image are converted to optical flow using a motion model proposed by the authors. Two different motion models are considered. Hand-crafted post-processing is used to improve final segmentation performance.  

The key novelty is the proposed approach for matching object masks to an optical flow representation. The authors use a motion model (i.e. translation only / affine transform on each object region), introduce an efficient and differentiable implementation of matching, and evaluate the improvements over multiple datasets. Notable performance improvements are on out-of-domain datasets. Also, the method scores relatively better on the mIoU metric, denoting the multi-object capability.    

Ablations highlight the significance of ground truth optical flow for training (low performance without it) as well as better motion model (translation only gives lower results). 

The authors include code in supplementary, and promise to share all code, datasets publicly in future. 
strengths_and_weaknesses: ### Strengths
1. A novel motion model based matching of object masks to optical flow
2. The authors obtain considerable improvements over a range of datasets 

### Weaknesses
1. Using ground-truth optical flow seems crucial for good performance (Table 3.a) - is this method scalable to real world data? Other methods using video supervision can operate on real-world data (e.g. [1]) 
2. Is the method really benefiting from the motion model or simply learning segmentation from the high quality ground-truth optical flow masks? 
3. The authors note how similar models struggle in complex settings, but limit all their evaluations to synthetic datasets: how would this translate to real world settings?  
4. Lack of clarity/simplicity in final loss function: can the authors construct a simpler version of the final loss between masks output from model and optical flow labels (closer to the actual implementation in code)? Many terms in the given equations are constants (or are they learnable params?) 
5. Is comparing to other methods with the additional post-processing fair? The authors do include results without post-processing that are still above SOTA; those may be a better comparison. 

[1] [Discovering Objects that Can Move](https://openaccess.thecvf.com/content/CVPR2022/papers/Bao_Discovering_Objects_That_Can_Move_CVPR_2022_paper.pdf)
questions: 1. Line 114: more explanation for reasoning in ELBO for regularization
2. A direct formulation of the final loss function
limitations: Yes, limitations discussed well.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors approach unsupervised instance segmentation of images by exploiting motion
patterns in videos during training. Given an RGB input frame, a neural network is trained
to predict a segmentation that allows to approximate the optical flow field of that frame
well when using a simple, parametric motion pattern for each segment (e.g., affine
motion). Since the segmentation is predicted based on the RGB frame only, the network
is trained to predict segments that *potentially* move and can be used to segment images
after training. On recent, synthetic multi-object datasets the proposed model is
shown to outperform recent state-of-the-art object-centric models in almost all cases.
strengths_and_weaknesses: Unsupervised segmentation is an active area of research. The general approach of
defining objects as regions with a simple motion pattern when they move is not new; the
respective related work is cited. The particular instantiation of the
approach in this work however is novel and shown to clearly improve over previous models in
almost all cases. A particular strength of the paper is the thorough comparison with
previous models. The authors tested a large range of models on the datasets using
multiple seeds, going beyond most related papers in the field.

On the other hand, the authors only compare to object-centric models that not only aim
at learning to segment images or videos but also to learn a useful representation of
those objects. Another line of works exists that only focusses on unsupervised
segmentation, with models that often make use of optical flow for training (and
sometimes inference) and can be applied to more realistic data. The authors
cite respective works, e.g. Mahendran et al. 2018, Yang et al. 2019, Choudhury et al. 2022.
Empirical comparisons with these models would have been informative and would strengthen
the paper, as these would allow to more directly compare the different approaches for
optical flow based unsupervised segmentation.
questions: - Are the appearance-based models also trained on frames of the video variants of
  CLEVR and CLEVRTex? Or directly on CLEVR/CLEVRTex? In the latter case, an experiment
  testing whether the different training data influences the performance of appearance
  based models should be included.

- Details about the dataset generation are not contained in the supplement,
  although promised in the main paper.

limitations: ## Limitations
The authors discuss limitations of their method adequately in a separate section. One
limitation to add in my view would be the necessity of ground truth optical flow. The
ablation study has shown a substantial performance drop when using optical flow
predicted by SMURF, which probably makes it difficult to apply the method to natural
videos where ground truth optical flow is not available.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors propose to learn unsupervised instance segmentation by grouping regions that tend to produce consistent optical flow. In particular, a network takes an image as input and predicts a distribution over k categories (segments) for every pixel. To compute the loss It also takes ground truth optical flow as input and a geometry-based, probabilistic objective then encourages the optical flow in each resulting region to conform to an affine motion model. The assumption is that each rigid object has a distinct affine motion pattern and thus the regions will correspond to object instances that move in that frame (not that the model itself only sees static frames, and learns the appearance of groups of pixels that tend to move together in the data). Crucially, this approach assumes that the camera in all videos is static, which the authors never explicitly mention. In the experiments on a few toy, synthetic datasets the proposed approach outperforms or performs on par with prior work, especially when evaluated on static images.
strengths_and_weaknesses: Strengths:

The intuition behind the approach is fairly straightforward and sound (grouping pixels in the optical flow field based on flow direction and magnitude does result in segmenting rigidly moving instances, it is the classic approach for motion segmentation). 

The actual formulation of using affine motion model consistency to supervise a grouping method is novel to the best of my knowledge.

The paper is well written with almost no typos or grammatical mistakes.

The approach seems to work well in toy videos with a static camera.

A minimal ablation study is provided.


Weaknesses:

The authors don't discuss prior work on learning image-based, unsupervised object segmentation from motion. In particular, DyStaB of Yang et al., CVPR'21 which operates in a single object regime, and very recent Discovering Objects that Can Move of Bao et al., CVPR'22 which can segment multiple instances. The novelty of the proposed approach compared to these methods is in the loss formulation, but conceptually they are very close.

A major limitation that is not discussed in the paper is that the method relies on the static camera assumption. In fact, segmenting moving objects in the optical flow field with a static camera is trivial. Binary segmentation can be obtained by simple thresholding, but separating instances can also be done with a primitive approach, e.g. by fitting an affine motion model. Going to the more general setting of a moving camera is much harder, however, and would require a very different approach. 

The objective is fairly complex and is not described in sufficient detail (both for the intuition behind the design choices and for the implementation details). In fact, it took me a while to figure out if the model actually predicts optical flow (or some proxy for it), or if the flow is only used to compute the objective function. The latter seems to be the case, making the title misleading. Implementation details are provided in the supplementary, showing that the objective is fairly brittle and requires different hyper-parameters on different datasets.

On the same note, the complexity of the proposed objective seems unnecessary and the authors do not compare to any simpler baselines. For example, a state-of-the-art motion segmentation algorithm could be used to extract moving object segments first and these segments could them be used to supervise the masks directly. The benefit of such a baseline, besides simplicity, is that it would allow the method to generalize to videos with a moving camera, since motion segmentation methods are designed to handle the most general setting.

The authors use a stronger backbone network compared to prior work (Mask2Former vs shallow CNNs used by most baselines), making all the experimental evaluations invalid. It is thus impossible to say if the proposed objective provides and real benefits compared to prior work.

questions: Please evaluate your method on videos with a moving camera.

Please compare to a simpler baseline that computes motion segmentation with a sota method and uses it to supervise the masks.

Please compare to prior work using exactly the same backbone network.

How is k chosen? How robust is your approach to the choice of k?
limitations: The authors completely ignored the main limitation of their work - it's reliance on static camera.
review_title: null

Meta-review: metareview: This paper presents an approach for unsupervised multi-object segmentation. The majority of the reviewers believe the paper contains interesting technical materials that warrants its acceptance. The (only) remaining concern is from Reviewer bRe4, pointing out that the paper uses a more advanced backbone than the baselines. Although the other reviewers also agree to this point, they believe the ablations in the paper are valid and they justify the benefits of the approach. Overall, the ACs recommend the acceptance of the paper.
confidence: Less certain
Decision: Accept