Unsupervised Multi-Object Segmentation by Predicting Probable Motion Patterns
Authors: Laurynas Karazija
Subhabrata Choudhury
Iro Laina
Christian Rupprecht
Andrea Vedaldi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computer Vision
Unsupervised Learning
Object-Centric Learning
Object Decomposition

Abstract: We propose a new approach to learn to segment multiple image objects without manual supervision. The method can extract objects form still images, but uses videos for supervision. While prior works have considered motion for segmentation, a key insight is that, while motion can be used to identify objects, not all objects are necessarily in motion: the absence of motion does not imply the absence of objects. Hence, our model learns to predict image regions that are likely to contain motion patterns characteristic of objects moving rigidly. It does not predict specific motion, which cannot be done unambiguously from a still image, but a distribution of possible motions, which includes the possibility that an object does not move at all. We demonstrate the advantage of this approach over its deterministic counterpart and show state-of-the-art unsupervised object segmentation performance on simulated and real-world benchmarks, surpassing methods that use motion even at test time. As our approach is applicable to variety of network architectures that segment the scenes, we also apply it to existing image reconstruction-based models showing drastic improvement. Project page and code: https://www.robots.ox.ac.uk/~vgg/research/ppmp.

Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns
Laurynas Karazija", Subhabrata Choudhury', Iro Laina, Christian Rupprecht, Andrea Vedaldi
Visual Geometry Group
University Oxford
Oxford, UK
(laurynas, suuuryaas,uuba,,, subha, iro, chrisr, vedaldi)@robots ox ac.
Abstract
We propose a new approach to learn to segment multiple image objects without manual supervision. The method can extract objects form still images, but uses videos for supervision. While prior works have considered motion for segmen- tation, key insight is that, while motion be used identify objects, not all objects are necessarily in motion: the absence of motion does not imply the absence of objects. Hence, model learns predict image regions that likely to contain motion patterns characteristic of objects moving rigidly. It does not predict specific motion, which cannot be done unambiguously from a still image, but a distribution of possible motions, which includes the possibility that object does not move all. We demonstrate the advantage of this approach over its deterministic counterpart and show sate-of-thhe-r unsupervised object segmentation performance on simulated and methods that use motion even at test time. As our approach is applicable to variety of network architectures that segment the scenes, we also apply to existing image reconstruction--assee models showing drastic improvement. Project page and code: https: //www. robots....... ox. ac. uu/ "lk///vg////aeee//// wttgg//eeeacchppping
benchmarks, surpassing
Introduction
Humans have an innate ability to segment individual objects in picture, but learning this capability with an algorithm usually relies on manual supervision. In this paper, we consider the problem of learning to segment objects from visual data only without externally provided labels. Algorithms for this task usually assume that objects are seen in different configurations and in front of different backgrounds. They then exploit cues such as the visual consistency and the co-occurrence of characteristic object parts to learn to discover and segment individual object instances.
Most such methods use still images as input and train with a reconstruction objective [19, 23, 37]. They work well on simple synthetic scenes, but they struggle scenes with more complex visual appearance [27]. This has motivated the development of algorithms that use videos input and can thus observe the motion of the objects as evidence of their presence. A common way of using motion for unsupervised learning is to seek for a compact representation to reconstruct the video itself [24, 26, 33, 53]. Effectively, such methods seek for compressed representation of appearance, but do not sidestep entirely the difficult task of modeling it. This has motivated authors to look instead at reconstructing the video's optical flow [28, 57]. fact, the optical flow measures the motion the objects directly and is much simpler to model than the objects' appearance.
Authors contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this paper, we propose a method that lies in in-between these two classes, i.e. single image- based and video-based. Our model learns to segment objects in still images, and is thus based on appearance, but learns to do so using video as a learning signal, in an unsupervised manner. The learning process can be summarized as follows. Given an image, each pixel is assigned to a slot that represents a certain object. The quality of the assignments is then measured by the coherence of the (unobserved) optical flow within the extracted regions. Because predicting optical flow from a still image is intrinsically ambiguous, the method models distributions of probable flow patterns within each region. The idea is that (rigid) objects generate characteristic flow patterns that can be used to distinguish between them.
Note that, because the segmentation network is based on a single image, it will learn to partition all objects contained the scene, not just the ones that actually move in the video, i.e. solving an object instance segmentation problem, rather than motion segmentation.
We derive closed-form distributions for the flow field generated by rigid objects moving in the scene. We also derive efficient expressions for the calculation of the flow probability under such models. The problem of decomposing the image into a number of regions is cast as a standard image segmentation task and an off-the-shelf neural network can be used for it.
As our method uses videos to train an image-based model, we introduce two new datasets which are straightforward video extensions of the existing image datasets CLEVR [25] and CLEVRTEX [27]. These datasets are built by animating the objects with initial velocities and using physics simulation to generate realistic object movement. Our datasets are constructed with the realistic assumption that not all objects moving at all times. This means that motion alone cannot be used as the sole cue for objectness and reflects scenarios such as a workbench where a person only interacts with a small number of objects time.
Empirically, we validate our model against several ablations and baselines. We compare our approach to existing unsupervised multi-object segmentation methods achieving state-of-the-art performance. We demonstrate particularly strong performance in visually complex scenes even with unseen objects and textures at test time. Our experiments in comparison to image-based models and, in particular, adding our motion- aware formulation to existing models shows substantial improvements, confirming to learn objectness. Furthermore, we show that our learned segmenter,
that motion is an important which operates on still images, produces better segmentation results than current video-based methods that use motion information at test time. Finally, we also apply our method to real-world self-driving scenarios where we show superior performance to prior work.
2 Related Work
Multi-object decomposition. Learning unsupervised object segmentation for static scenes is well-researched problem in computer vision [5, 9, 11, 12, 13, 14, 17, 18, 23, 34, 37, 45, 46, 56). These methods aim to decompose the scene into constituent parts, e.g. the different foreground Hampsee-ased methods [9, 14, 23, 34] find input patches (glimpses) that
objects and the background. contain the objects the scene. These methods Iearn object descriptors that encode their properties (e.g. position, number, size of the objects) using variational inference, composing glimpses into the final picture. More related to ours are approaches that learn per-pixel object masks [5, 11, 12, 13, 37]. MoNet [5] and IODINE [19] employ multiple encoding declodnng steps to sequentially explain the scene as a collection of regions. Slot Attention [37] uses a multi-step soft clustering -liee scheme to find the regions simultaneously. In all cases, learning is posed as an image reconstruction problem. In order to align learnable slots with semantic objects, models have to make efficient use of a limited representation available for each region, such as learning to only explain visual appearance. This principle, however, is difficult to extend to visually complex data [27] and relics on custom specialized architectures. Instead, our method allows for any standard segmentation architecture to be used, which we train to predict regions that are most likely described by rigid motion patterns.
Video-based multi object decomposition. Another line of work extends the unsupervised object decomposition problem to videos [21, 24, 26, 29, 30, 31, 33, 44, 52, 54, 55, 59]. Many of these methods work mainly with simpler datasets [30, 44, 59] and require sequential frames for training. For example, SCALOR [24] a glimpse-based method that discovers and propagates objects across frames to learn intermediate object latents. SIMONe [26) processes the whole video at once, learning both temporal scene representation and time-invariant object representations simultaneously. Slot Attention for Video (SAVi) [28] poses the multi-object problem as optical-flov prediction using
2

---

sequential frames as input. The internal slot-attention mechanism drives the network to learn regions that move simple and consistent manner. Different to our work, it does not assume a specific motion model but relies on directly regressing the flow. It is computationally more expensive and struggles when only one or few frames are available.
Unsupervised video object segmentation. Unsupervised video object segmentation (VOS) is popular problem in computer vision [10, 15, 32, 38, 43, 49, 51, 57,58) 58], that focuses on extracting the most salient object in the scene. Many of the approaches treat the problem as a motion segmentation task as the background typically shows a dominant motion independent of the salient object. Motion Grouping [57] employs the Slot Attention architecture to reconstruct optical flow from itself, avoiding appearance information entirely. In [6] objects are iteratively explained away starting from confident flow segments, which are refined using graph propagation method. Another related line of work [8, 40, 41, 50] employs approximate motion models. These approaches rely on a point estimate of the motion model parameters. In contrast, we adopt more principled probabilistic approach, placing a prior on the motion parameters and integrating them out. To deal with flow outliers that do not conform to rigid motion model, Mahendran et al. [40] use histogram matching- loss and GWM [8] over-segments the scene relying on spectral clustering to produce binary segmentation during inference. Instead, we model the noise in our formulation directly. Finally, Meunier et al. [41] rely on flow as input, limiting the method to videos only.
3 Method
Let frame I E R of a video and its optical flow f E R22 be defined on the H x W lattice. The optical flow a local summary of the motion from one frame to the next. We use it to supervise network & that, given the (single) image I as input, predicts soft assignments of each pixel to up to K different image regions, outputting an H W collection of probability vectors (Z) E Á Hxw C [o, 1] where ÅK is the 1-dimensional simplex. The quality of K the regions is measured based on how likely they contain flow patterns typical of the motion of independent objects.
In more detail, we represent the predicted image regions by a hard K-way pixel assignment (mask) m E A A I C {0, 1]x*Hx where Ax is the space of K-dimensional one-hot vectors. Each mask is a sample from the categorical distribution output by the network, i.e. pp (m [] Categorical (e (Z)). Note that there is one categorical distribution for each pixel and that these are mutually independent.
We then assume that the flow depends only on the regions, in the sense that po (f. m I) = p(f m) P (m I), where p(f m) is model of the distribution of the flow field given the regions. The likelihood of the modeled is bounded by:
log pa (f I) log
E
[p([]
m)]
llgg(f mm]]
(m
mrpp
Furthermore, inspired by ELBO, we regularize the model's prediction ps (m [] by taking its KL divergence from a uniform prior po (m), obtaining the learning objective
E
[-logr( p(f m)] BDDI. (po (m I) po(m)).
(1)
Next, we introduce the closed-form motion model p(f | i)) in Eq. (1) and then explain how the Gumbel--oftman trick can be used train the network.
Approximate motion models for optical flow. We now turn to describing the models of motion used in our work, which play a role in assessing the likelihood of optical flow p(f m). Optical flow measures the coordinate change of pixels between neighboring frames, which arises due to the motion of the camera and objects. We consider rigid-bode body motion of some object k.
Let xf E m.* be the spatial locations of the pixels belonging to region/object at time t. where ny is the number of pixels in the region. For convenience, we stack the coordinates in single vector 2 (xf. E 2R2 The pixels comprising this object undergo coordinate change from 22 to
2 of+1 giving rise to the optical flow for this object as f; = 22 - St. We assume this underlying
3D rigid-body motion can be approximated using linear 2D parametric model HI with parameters
0, so that:
2 =1l(( +E,
fx (7() - 24 + e,
(2)
3

---

where captures the residual error of the approximation. Several forms of models are available (see [1, 4] for an overview). Here, we consider two such models: the translation of an object within the camera plane, and an affine motion, given respectively by linear functions:
IT' (0
+
Le 0
81 8z
1o (N))
X;
Y1
(3)
1 T3k
0
0
X1
Nt
1 BE
06
P
where we use 1,, is vector of ny ones and matrix Ps contains the coefficients of the model.
The affine model supports object rotation, scaling and shearing in addition to translation. It is often sufficient approximation to real-world optical flow, provided the objects are rigid, convex, and mainly rotating in-plane.
We can then use the motion equations (2) to construct the distribution p(f m) by assuming prior the motion parameters and by marginalizing over it. Specifically, denote by my the k-th slice of the tensor m encoding the regions (i.e. the mask of the k-th region). We assume that regions are statistically independent and decompose the likeleiddoood p(f m) as:
log p( m) =
Σ logp(fx m = Σ log d(g, 0y mx)d0k-
Assuming that each object has i.i.d. parameters O2 with a Gaussian prior N y, Σ), and assuming is a zero-mean noise with variance o², Eq. (2) gives marginal optical flow likelihood for segment k:
p(fa m N (fr: (12x) 2s P&EPE + +011)
(5)
where I is the identity matrix. A practical issue with Eq. (5) is that, if segment contains n& = (m,)ss pixels, then the covariance matrix PyEPE + I has dimension 2n 2ng- Inverting such a matrix in the evaluation of the Gaussian log-density is very slow except for very small regions. Furthermore, it is not obvious how relax Eq. (4) to support gradient-based learning, e.g. through the Gumbel-Softmax approximation. We solve these problems in the next section.
Expressions for the likelihood. We now derive expressions for Eq. (5) which are efficient and that lead to a natural relaxation for use in the Gumbel-Soo -Sfftmax sampling. Given the definitions FR fg 11, (S2x) + O4 and A = --11 we rewrite Eq. (5) as:
det Sk det A
p(fx m = (2TT2)-nH
202
= F
(6)
where S - 1/0P, Pk A. The significant advantage of this form is that it involves the computation of the inverse and determinant of matrix Sg, whose size is only 2 2 (for the translation model) or 6 x 6 (for the affine one), instead of the much larger 2n x 2n
We can more explicitly introduce the dependency matrices R₂ E {0, 1] 1m x2n (with n = Σ. n4 = HW) that extract the x and y coordinates of the pixels that belong to the corresponding region, i.e. S2 R2 We can then also write F;- R2FF and Pa = P. Furthermore, the product of the selectors L3 = Rg R& {0, 122 can be written directly as function of the assignment as Lę(m) diag (mg. my). Plugging these back in Eq. (6), we obtain expressions involving L only:
the region assignments m by defining selector
7g = 2 Lx,1, St = 0² -
P= PTLP+ oe
(FT L (P
(7)
Translation-only model. Further simplifications are possible for specific models. For instance, for the translation-only model, assuming that diag(1/7², 1/72) then S₁- diag(n, 1/72 nx 1/72) and, after some calculations, we obtain the expression:
2 RE +
log p( m) n log 2
Σ log
FT
1-
I
mmm
F.
2
202
n&k
m

---

Affine model. For the affine model, the expression for - log m) does not simplify as much. Still, by exploiting the structure of matrix Px we can reduce the calculations to the computation of inverse and determinant of small 3 3 matrices, which can be implemented efficiently in closed form. Please see the Appendix for the derivation. Unless otherwise stated, the mean vector is set to μ (1000 0) centering the prior on the no-motion point.
Gunbel-softmax. In order to train the network using gradient descent, we need a differentiable version of loss (1). To do so, use the re- parametrizable humbel-softtael relaxation [22, 39]. The Gumbel-softmas relaxation replaces categorical samples m A # from the distribution (m Σ) Categorical (Z) with continuous samples m E Å #x from the distribution GumbelSoftmax(f (2)]. We take N = 3 samples from this distribution to evaluate the expected negative agglikelihhod, further reducing variance. Then we simply replace for m in Eq. (1), leading to differentiable quantities.
Post-processing. Eq. (5) naturally encourages the model to form larger regions explain parts of the scene that move in consistent (under the assumed prior) manner. However, we find that this can also lead to the model grouping together objects that only coincidentally move together (e.g. all objects mostly falling due to gravity in one of the datasets). Furthermore, optical flow is ambiguous around object edges and occlusions. To address both the object grouping and occlusion boundary issue, we use a simple post post-processing step. We isolate connected components in the model output, selecting the K largest masks, discarding any that are smaller than 0.1 % of the image area, and combining the left-over and discarded ones with the largest mask overall.
Warp loss. Occasionally, the optical flow used supervise our model can be noisy as it is estimated by other methods. This noise is also unlikely to be isotropic as some surfaces are easier to estimate that others. Rather than supporting heterogencous noise and approximation error (Eq. (2)), we instead prioritize parts of the scene covered by higher-quality flow. To this end, we introduce an additional loss term that simply enforces consistency between adjacent frames Zi, Iz. In particular, warps the predicted mask distributions $ (II,), ((Z)) using the optical flow, weighted by the error of warping the frames themselves, as follows:
waar (I, Iz, fi b2) = fi
(8)
+ w[I,, (2()))
w (Ia, Ib) = 1 norm ITe IT)),
d(p, (x 9) (4 p)/2.
where fi()) indicates warping by forward optical flow f1 (or backward b2). The symmetrized KI. divergence, d(-), measured agreement between predicted and warped mask distributions, weighted by the absolute error of the warped frames normalized in [o, 1]. While the use of this term is not central to our method, we include it to show how tolerance noisy optical flow can be improved. We do not use the warp loss (Eq, (8)) in our experiments, unless otherwise indicated by (WL). that case, the final loss is simply sum of the two terms: Lg + Lwwrr-
4 Experiments
Our method lies in between image-based and video-based segmentation approaches, because it uses videos for supervision, but trains an image segmentation network that operates still images only, We thus evaluate our approach under a number of settings. Firstly, we evaluate how well motion can be used to supervise an object instance segmenter that operates still images. Secondly, compare such a segmenter to tfate-of-the-art object segmentation methods that use motion also at test time (and thus advantaged compared to our model). We conduct further analysis to validate our modelling assumptions and the model's ''s reliance on the quality of the optical flow used for supervision. Finally, we apply method real-world setting.
4.1 Experimental setup
Datasets. We evaluate our method on video and still image datasets. For video-based data, we use the Multi-Object Video (MOVi) datasets, released as part of Kubric [20]. Specifically, we employ MOVi-(A,C,D.E) versions. MOVi-A similar to CLEVR [25] in terms of visual complexity and contains videos of 3-10 falling objects simple, gray background. MOVi-C is significantly more
5

---

Figure Example images and optical flow of the MOVINGCLEVR and MOVINGCLEVRTEX datasets, which extend CLEVR and CLEVRTEX respectively to short videos based on physics simulation. Note that only a subset of objects in motion in each frame.
challenging, as it features scanned, textured, common objects on top of backgrounds textured using HDR images. In MOVi-D, the number of objects is increased up to 23, In MOVi-E, the camera additionaly moving. We use a resolution of 128 x 128 and the provided ground truth optical flow.
To evaluate our method still images, we use CLEVR [25] and CLEVRTEX [27] benchmark suites. Both consist of images depicting 3-10 objects. CLEVR images are simpler with uniformly colored objects with metallic rubbery materials. CLEVRTEX features more diverse objects with complex textures applied. We also use the OOD and CAMO test sets from CLEVRTEX benchmark. OOD contains diftt--diitriittion shapes and textures. CAMO has camouflaged objects where the same texture is sampled for objects and the background.
Since our method requires optical flow during training, extend the implementation of [27] to generate video datasets of CLEVR and CLEVRTEX scenes, where a subser of objects contained in each scene are sliding, rolling and colliding based physies simulation (Fig. 1). We generate 10k sequences for MOVINGCLEVR
and 5k for MOVINGCLEVR where we retain 1000 and 500 sequences, respectively, for validation. Each sequence is 5 frames long. Dataset details can be found in the Appendix. The evaluation is performed on the original CLEVR and CLEVRTEX test sets.
We also evaluate our method the real-world KITTI [16] benchmark which depicts street scenes captured from a moving car. We follow the set up of [3], using 147 videos for training and evaluate on the instance segmentation subset which contains 200 annotated validation frames.
Metrics. Following prior work [27, 28], we measure performance using two metrics. FG-ARI is the Adjusted Rand Index measured on foreground pixels only (selected using the ground-truth segmentation). Mean Intersection-over- Union (mloU), measured through Hungarian matching and averaged across the number of predicted or ground truth components, whichever is higher. When evaluating on videos, we calculate these metrics per-frame.
Network architecture. Our method can employ any image segmentation network architecture and train from scratch. Unless otherwise specified, we use Mask2Former [7], using only its semantic segmentation. Following prior work [28, 37], we use a 6-layer CNN backbone on synthetic datasets and ResNet-18 18 for KITTI. We also experiment with Swin-tiny transformer [36] as the backbone. We use 11 transformer queries which become K = 11 slots on CLEVR, CLEVRTEX, and MOVi-A/C. On MOVi-D/E, we on a single A30 24GB GPU. All training details and hyper- sypeeepparameters are included in the Appendix.
24 and K- 22 on KITTI. The model takes approximately 48h train
4.2 Unsupervised multi-object segmentation in images
In Table I, we evaluate our method on the CLEVR [25] and CLEVRTEX [27] benchmarks and compare to prior work. Our method outperforms image models based on appearance reconstruction on both metrics (mIoU and FG-ARI) and across all datasets. The performance gap increases on the visually complex CLEVRTEX, OOD, and CAMO variants, demonstrating the strong inductive bias that motion provides during training, especially when the objects are camouflaged. Note that, in this setting, our model is advantaged compared to the other models in Table 1, as it can observe (through the loss) the optical flow of the training scenes. For this reason, we also train the optical flow-based, unconditional SAVi [28] model. Nevertheless, we find that despite having access to motion information during training, SAVi does not surpass appearance-only models, likely due to only having access to single frames at test time.
Approx. total compute in this paper: 100 GPU days for our models, 154 GPU days for comparisons.

---

Table 1: Benchmark results on CLEVR, CLEVRTEX, CAMO, and OOD comparing FG-ARI and mloU metrics (see also Appendix for an extended version). Results are a mean of 3 seeds (#r).
Methods above the line are trained on single images, while methods below train on videos. indicates post- posspprocsssin..
CLEVR
CLEVRTEX
0OD
CAMO
Model
FG-ARI:
mloUt
FG-ARIT
miooU
FG-ARIT
mlooUt
FG-ARIf
mlooUt
SPAIR [9]
77.13.1 1.92
65.951
4.02
0.00.1
0.00
0,0010.00
0.00.1
0.00
0.0010.00
0.00.1
0.00
0.00.0.00
SPAIR
77.05.L 1.96
68.871L
9.65
0.00.
0.00
0.00100.00
0.001
0.00
0.00110.00
0.00&L
0.00
0.00-10.00
MN [45]
72.12+ 0.64
0.440 56.81
38.11
0.70
10.46+0.00
37.20+
1.04
12.13=0 19
31.52+
0.87
8.79=0.15
72.08-: 0.62
0.73
10.34 12
37.25 27-28:
1.07
11:97:00.2
0.87
8.77=00115
MONeI [5]
54.47=11.41
30.66=111
14.87
36.66- 0.87
19.78+ 1::+++1:000
32.97
1.00
19.30=0.37
12.44
0.73
10.52=0.38
MONeT
61.30+ 7:33
45.61
64 1.17
23.50+00000
1.46
23.04+0.55
9.94+
11.31=0..00
SA (373
95.89.1 2.37
36.61 21.11.1.55
62. 20.1
2:23
22.58 0...2.1...0
38. 15.1
1.87
20.98 1.38
57.51
1.01
19.83 1.441
2:20
IODINE [19]
93.8111
0.76
15.14 17.85
39.52
2.20
29.17.10.75
53.20 13.20.1
2.55
26 1...5
36.31
2:57
17 10.75
JODINE
DTI-S [42]
89.54+
1.44
2.17 48.74
79.90
1.37
33.73
73.67
0.98
32.55 1.08
72.90-
1.80
27.54 1.55
DTI-S
79.86
30.741
26.30
GNM [23]
65.05.L
4.19
3.72 63.384
59.921
33.37
0.67
12.25:10.18
13.1
0.88
10.8-11..00
15.T3.L
0.89
17.56.10.7
GNM
4.255 65.67
53.38
44.301=0.19
48.44
42.87:00.28
15.72
18.50
SAVI [28]
19.54
31.88
42.68
30.31
42.67
29.60
Ours
91.69.1
0.30
68.7011
0:52
90 80
0.22
55.07
76,011
0.58
16.81110.00 20
72.78
1.31
42.30.1108
Ours
95.94
0.43
84.86
4.08
92.61
0.22
77.67:00225
78.24 0.43
55.54
77.43
0.86
56.43-00.80
CLEVR
CLEVRTEX
OOD
CAMO
RGB
G1
SAW
Ours
Chir
Figure 2: Unsupervised object segmentation on CLEVR and CLEVRTEX benchmarks. Our model is able to segment simple and visually complex scenes. Occasional mistakes around object boundaries and the assignment of different objects to the same component are addressed by post-proe post-processing. indicates post- proceproccssing.
Post-pro prt-eroccssing helps improve results further. shown in Fig. 2, separating connected components in post-processing distinguishes objects that might be assigned to the same mask and suppresses boundary segments that tend group difficult occlusion boundaries. For a fairer comparison, we also test if this post-processing improves the results of other methods but only obtain mixed results.
4.3 Unsupervised multi-object segmentation in video
now evaluate our approach on video segmentation, where motion is available at test time. We report the performance of our method in Table compared to video-based models: SCALOR [24],

---

Table Segmentation results on MOVi datasets. Mean + standard error (5 seeds). We calculate metric for each frame. All values in %. (WL) marks use of warp loss. indicates post-prost post-processing.
MOVIA
MON--C
MOV-D
MOV-F
Model
PG-ARIT
mloU?
FG-ARE
mloU
FG-ARI
mlol
FG-ARI!
mloo!
42.27
42.50
18.74
SCALOR (24)
SAN [28]
88.30
02.69
43.26
31.52
43.45
10.80
17:39
0.70
Ours
=4.01-00.72
6008-11447
G1 84
34.72 17
$5.74=1.002
23 23.00:00.36
G2 62.02200.2
26. 78
65-41:-h
76.19-2.05
65.18=0.94
25.21
63.11-0911
28.59:00200
Frame
Hlow
GI
GWM
SCALOR
SAVi
Ours
Ours
MOVI-C
Figure 3: Qualitative comparisons on MOVi-A and MOVi-C. Our method performs consistently well compared other methods. GWM suffers from oversegmentation where SCALOR has underseg- mentation issue. Among the related methods, SCALOR fails to discover all the objects and SAVi's object boundaries are coarser 1_ indicates post-processing.
unconditional SAVi [28], and GWM [8]; the latter also optical flow supervision. It is important to note that SAVi and SCALOR make predictions jointly over all frames of a video, that allows them actually see the objects in motion at test time. Despite this comparison being unfair to our approach, which operates on a single frame at a time, we achieve competitive results. On the visually simpler MOVi-A, our method has more than 10% lead over SAVi in mloU, but performs slightly worse in terms of FG-ARI. On the more complex MoVi-C/D/E datasets our model shows strong performance, outperforming prior work on both metrics, with the performance gap again increasing with data complexity. Finally, we experiment with a version of the model using deeper backbone (Swin) and the warp loss (Eq. (8)). Though not necessary to achieve state-of-art results, this drastically improves performance on all metrics and dataset versions. In Fig. 3, we also compare these models qualitatively, demonstrating that the different objects are overall better captured by our model with more refined boundaries, which explains the higher mloU.

---

Frame Flow
GT
GWM
SCALOR
SAVI
Ours
Ours'
MONIA
MOViC
Figure 4: Failure cases on MOVi-A and MOVi-C. Our method has difficulty with objects that typically exhibit complex motion. It results in an over-segmentatiin of the object. Due to inherently imprecise optical flow near the boundaries our method also has tendency to segment boundary pixels into a separate mask, which fix with post-processing step. t_ indicates post-processing
Table Model ablations. (3a) replacing flow supervision with an unsupervised flow method, (3b) compares translation-only with the affine flow model, and (3c) adds our motion-based formulation to an appearance-only model. All models with post-processing applied.
(a) Choice of Optical Flow Method
(b) Choice of Motion Model
MOVi-A
MOVi-C
MOVi-A
MOVi-C
Optical Flow
FG-RII
mloU"
FG-ARIf
mloU?
Motion Mdl.
FG-ARI
mloUt
FG-ARI
mioUU
SMURF [47]
80.17
26.3
61.21
28.77
Translation Affine
66.03
59.94
39.77 58.59
32.23
Ground Truch
83.48
72.61
58.59
35.67
83.48
72.61
35.67
(c) Adding motion awareness to appearance-only models
CLEVRTEX
OOD
CAMO
Model
Train data
FG-ARI
mioUt
FG-ARI
mloU1
FG-ARI
mloU*
GNM [23]
CLEVRTEX
53.38
44.30
48.44
42.87
15.72
18.53
GNM [23]
MOVINGCLEVRTRX
18.01
31.47
15.57
15.57
0.21
14.68
GNM+Our Loss
MOVINGCLEVRTEX
63.84
55.26
59.01
48.65
51.00
47.63
SA [37]
CLEVRTEX
62.40
22.58
58.45
20.98
57.54
19.83
[37]
MOVINGCLEVRTEX
61.84
21.44
58.24
20.67
57.30
18.82
SA+Our
MOVINGCLEVRTEE
76.60
38.12
67.01
70.59
33.05
4.4 Model ablations
Optical flow. In Table 3a, we replace the ground-truth optical flow used so far in our experiments with the estimated by SMURF [47]. The additional noise impacts our model's accuracy, as evidenced by the significant drop in mloU (but comparable FG-ARI scores).
Motion model. In Table 3b, we compare our affine motion model to the simpler translation-onn -nlly one. We observe that the ability to describe complex motion patterns with an affine model improves performance over the translation model, which can only represent translation in the camera plane.
Motion awareness. Finally, we investigate the effectiveness of our objective in combination with existing appearance-based methods. The goal of this experiment to understand the advantage of using motion information during training (if available) and decouple the effect of our formulation from the choice of architecture. To this end, we employ our objective on top of two models based on appearance reconstruction, GNM [23] and SA [37], with no other modifications. We train GNM and SA with and without our loss on videos (MOVINGCLEVRTEN and evaluate on the corresponding dingleimaae test sets of the CLEVRTEX suite. In Table 3c, we compare these models respectively to the original methods trained on static images (CLEVRTEX)) We find that when trained on video data without our loss, both GNM and SA struggle. We attribute this to the reduced number of scenes in MOVINGCLEVRTER compared to CLEVRTEX, However, we note that using our loss significantly improves the performance of the appearance methods, suggesting the effectiveness of exploiting motion information through our formulation.

---

4.5 Segmentation on real-world data
We now turn to assessing our model's performance in real- world setting. We follow the setting of Bao et al. [3] and evaluate on KITTI [16]. using RAFT [48] to estimate optical flow and ResNet- 18 as the backbone of our model, trained from scratch, We lower the input resolution for our model from 368 x 2428 [3] to 288 x 960, which enables us to fit on single GPU. We evaluate at 96 x 320 resolution.
Table 4: Real-world segmenta- tion results on KITTI. Baseline results from [3]. Bao et al. [3] and our method use RAFT for optical flow. Models above the line use ResNet-18 backbone. (WL) marks use of warp loss.
As we show in Table 4, our method outperforms prior work on the challenging real-world setting. In the same table, we also con- sider our method with an additional warp loss term, which further boosts performance. We also experiment with a transformer- based backbone (Swin) which is also pre- trained using self sole--upeeviiion Although, not necessary to show state-of-art result, this signifi- cantly improves real-world performance.
KITTI
Model
FG-ART
SA [37]
13.8
MONet
14.9
SCALOR [35]
21.1
S-IODINE [19]
111-1
MCG [2]
40.9
Bao et al. [3]
47.1
Ours
30.8
Ours (WL.
51.9
4.6 Limitations
Ours (Swin WL)
58.3
Motion is sometimes insufficient to distinguish different objects, for instance because they do not move or because they move similarly. In principle, this should not matter if sufficient motion diversity is observed in the training data as whole; in practice, model occasionally merges different object at test time, which we address partly in post-processing. Further improvements could be obtained by choosing more informative prior po(m) in Eq. (1), to capture other desirable properties of objects, such as compactness and connectedness.
Figure 4 shows some failure cases where the affine motion model struggles to capture strong perspec- tive effects caused by non-smooth depth changes in the object geometry. This could be addressed by modeling the object geometry (depth) and the ensuing complex flow patterns. Alternatively, this could be dealt with using a hierarchical segmentation model that can account for geometric discontinuities and seff-occlusions. Hierarchical segmentation would also help with pronounced non-rigid motion (e.g. humans dancing or animals running) as motion could be explained at the level of object parts.
Conclusions
We have presented method that bridges the gap between image- image-based and video-based scene decomposition, in that it requires only a single image as input, yet exploits motion cues available in videos during training. In comparison to prior work on image-based multi-object segmentation, our approach shows that motion provides useful objectness cues, especially as the visual complexity of a scene increases. Different from aideo-based approaches, however, our model operates still images and does not rely on motion to detect or refine objects, which makes it more generally applicable. Finally, we deviate from the common objective of image flow reconstruction and, instead, model the problem by only predicting regions likely to contain affine flow patterns. This does not require specialized architecture, thus any segmentation network is suitable for this task. Our approach achieves state-of the-art performance on multiple image and video benchmarks, in simulated and real-world settings, validating the paradigm of training image models using motion.
Broader impact. Our work introduces a principled method for unsupervised multi-object seg- mentation. The work is mainly evaluated on 3D simulated datasets that do not contain people or personal information. Additionally, we evaluate on KITTI, a real-world self-driving dataset, which occasionally contains images of pedestrians. Consent cannot be obtained in this case, but we follow the KITTI terms of usage. We build on top of open source projects, respecting licenses and release all code, trained models and datasets for research purposes. Currently, the application of this approach is mainly limited to simulated imagery. Although the results on KITTI show promise, the immediate broader impact of our work in real-world scenarios, beyond the research community, is limited.
Acknowledgements L. K. is funded by EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems EP/S024050/1. S. C. is supported by scholarship sponsored by Facebook. I. I.,, C. R. and A. V. are supported by European Research Council (ERC) grant 2020- 101-101001212 UNION, I. L. and C. R. are also funded by EPSRC grant VisualAI EP/T028572/1.
10

---

References
[1] Gilad Adiv. Determining three-dimensional motion and structure from optical flow generated by several moving objects. IEEE transactions on pattern analysis and machine intelligence, (4):384-40) 1985.
Pablo Arbelaez, Jordi Pont-Tuset, Jonathan T. Barron, Ferran Marques, and Jitendra Malik. Multiscale com- binatorial grouping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
[3] Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu- Xiong Wang, Adrien Gaidon, and Martial Hebert. Discov- ering objects that can move. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 17899-1798,, June 2022.
[4] James Bergen, Patrick Anandan, Keith Hanna, and Rajesh Hingorani. Hierarchical model- motion estimation, In European conference on computer vision, pages 237-252. Springer, 1992.
[5] Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 11390, 2019.
[6] Honglin Chen, Rahul Venkatesh, Yoni Friedman, Jiajun Wu, Joshua Tenenbaum, Daniel LK Yamins, and Daniel M Bear. Unsupervised segmentation in real-world images via spelke object inference. In European Conference on Computer Vision, pages 719-735. Springer, 2022.
[7] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1290-1299 2022.
[8] Subhabrata Choudhury, Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Guess What Moves: Unsupervised video and image segmentation by anticipating motion. In British Machine Vision Conference (BMVC), 2022.
[9] Eric Crawford and Joelle Pincau. Spatially invariant unsupervised object detection with convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3412-3420, 2019.
[10] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Leamingg to combine motion and appearance for fully automatic segmentation of generic objects in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3664-3673, 2017.
[11] Patrick Emami. Pan He. Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized inference for leamiig symmetric and disentangled multi-object representations. In Proceedings of the 38th International Conference on Machine Learning, pages 2970- 2970-2981. PMLR, 2021.
[12] Martin Engeleke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020.
[13] Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object represen- tations without iterative refinement. In Advances in Neural Information Processing Systems, volume 34, 2021.
[14] S. M. Ali Eslami. Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu. and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Proceedings of the 30th International Conference on Neural Information Processing Systems, page 3233-3241, 2016.
[15] Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
[16] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3354-3361 IEEE, 2012.
[17] Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuher. Binding via reconstruction clustering. ArXiv, abs/1511 064418, 2015.
[18] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Advances in Neural Information Processing Systems, pages 4484-4492, 2016.
[19] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning. pages 2424 2433. PMLR, 2019.

---

[20] Klaus Greff,, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herraann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mchdi S. M. Sajadi,, Matan Scla, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable datasct generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[21] Zhen He, Jian Li, Daxue Liu, Hangen Hc, and David Barber. Tracking by animation: Unsupervised learning of multi-object attentive trackers. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1318-1327, 2019.
[22] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR, 2017.
[23] Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. Advances in Neural Information Processing Systems, volume 33, pages 12572-12582, 2020.
[24] Jindong Jiang, Sepehr Janghorbani,, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Representations, 2020.
[25] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2017.
[26] Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew M. Botvinick, Alexander Lerehner, and Christopher P. Burgess. SIMONe: View-invariant, caaporrlly-abstractee object representations via unsupervised video decomposition. volume 34, pages 4611419999, 2021.
[27] Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevitex: A texture-rich benchmark for unsuper- vised multi-object segmentation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[28] Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski. Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In International Conference Learning Representations, 2022.
[29] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer. repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018.
[30] Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, and Kristian Kersting. Structured object- aware physics prediction for video modeling and planning. In Proceedings of the International Conference on Learning Representatiinss 2020.
[31] Nanbo Li, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi-object scenes from multiple views. Advances in Neural Information Processing Systems, 33:5656-5666, 2020.
[32] Siyang Li, Bryan Seybold, Alexey Vorobyov, Alireza Fathi, Qin Huang, and C-C Jay Kuo. Instance embedding transfer to unsupervised video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6526-6535, 2018
[33] Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pages 6140-616 6149. PMLR, 2020.
[34] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.
[35] Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang. Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[36] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[37] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dasovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems, volume 33, pages 11525-11553 2020.
[38] Xiankai Lu, Wenguan Wang. Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3623 2632, 2019.
12

---

[39] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR, 2017.
[40] Aravindh Mahendran, James Thewlis, and Andrea Vedaldi. Self-supervised segmentation by grouping optical-flow In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018.
[41] Etienne Meunier, Anais Badoual, and Patrick Bouthemy. EM-driven unsupervised learning for efficient motion segmentation. arXiv preprint arXiv:2201.0200 020074, 2022.
[42] Tom Monnier, Elliot Vincent, Jean Ponce, and Mathicu Aubry. Unsupervised layered image decomposition into object prototypes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8640-8650, 2021.
[43] Anestis Papazoglou and Vittorio Ferrari. Fast object segmentation in unconstrained video. In Proceedings of the IEFE International Conference on Computer Vision (ICCV), December 2013.
[44] Gautam Singh, Skand Peri, Junghyun Kim, Hyunscok Kim, and Sungjin Ahn. Structured world belicf for reinforcement learning in POMDP. In International Conference on Machine Learning, pages 9744-9755.
PMLR, 2021.
[45] Dmitriy Smirnov, Michael Gharbi, Matthew Fisher, Vitor Guizilini, Alexei A Efros, and Justin Solomon. Marionette: supervied sprite learning. In Advances in Neural Information Processing Systems, volume 34, 2021.
[46] Karl Stelzner. Robert Peharz. and Kristian Kersting. Faster attend attend-infer-repeal with tractable probabilistic models. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 5966-5975. PMLR, 2019.
[47] Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. Smurf: Self- teaching frami-frame unsupervised raft with full- mull-image warping. 2021 IEEE/CVF Conference on Computer Vision and Pottern Recognittin (CVPR), pages 3886-3895, 2021.
[48] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV), 2020.
[49] Pavel Tokmakov, Cordelia Schmid, and Karteek Alahari. Learning to segment moving objects. International Journal of Computer Vision, 127(3)2222-301, mar 2019.
[50] Philip H. S. Torr. Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 356:1321 1340. 1998.
[51] Yi-Ilsuan Tsai, Ming-Isuan Yang, and Michael J. Black. Video segmentation via object flow. In 2016
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3899-3908, 2016.
[52] Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353 2018.
[53] Rishi Vecrapaneni, John D Co-Reyes, Michacl Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement lcarning. In Conference on Robot Learning, pages 1439-1456. PMLR, 2020.
[54] Marissa A. Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S. Ecker. Benchmarking unsupervised object representations for video sequences. Journal of Machine Learning Research, 22( 183):11611, 2021.
[55] Yizhe Wu, Oiwi Parker Jones, Martin Engeleke, and Ingmar Posner. Apex: Unsupervised, object-centric scene segmentation and tracking for robot manipulation. 2021 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 3375-3382, 2021.
[56] Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Multi-object generation with amortized structural regularization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., 2019.
[57] Charig Yang. Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-superrised video object segmentation by motion grouping. In Proceedings of the IRFF//VF International Conference on Computer Vision, pages 7177-7188, 2021.
[58] Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[59] Polina Zablotskaia, Edoardo A. Dominici, Leonid Sigal, and Andreas M. Lehrmann. Provide: a probabilistic framework for unsupervised video decomposition. In Proceedings of the Thirty-Seventh Conference on PMLR, 2021.
Uncertainty in Artificial Intelligence, volume 161, pages
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Longer form deriva- tions are included in the supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] We include instruction and specification of our methods. See project page https: / wwww.roboos. ox ac wt/p/g////vv//// www.gggg//eer..ggeee rew.arch/ppppp for code and checkpoints.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See supplementary material
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See e.g. Table and Table 2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you eite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We provide the code for dataset creation and examples of the dataset in the supplementary material (due to size). See project page https: ac. htt//"vggreeeearcche
ox.
for data.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We include one experiment on the KITTI dataset following their terms of usage. This data contains accidentally-captured pedestrians, from which consent cannot be obtained. These are ignored in the evaluation.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See Section 5.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "qNh5ckeQ8UJ", "reviewer": "Reviewer_bRe4", "summary": "The authors propose to learn unsupervised instance segmentation by grouping regions that tend to produce consistent optical flow. In particular, a network takes an image as input and predicts a distribution over k categories (segments) for every pixel. To compute the loss It also takes ground truth optical flow as input and a geometry-based, probabilistic objective then encourages the optical flow in each resulting region to conform to an affine motion model. The assumption is that each rigid object has a distinct affine motion pattern and thus the regions will correspond to object instances that move in that frame (not that the model itself only sees static frames, and learns the appearance of groups of pixels that tend to move together in the data). Crucially, this approach assumes that the camera in all videos is static, which the authors never explicitly mention. In the experiments on a few toy, synthetic datasets the proposed approach outperforms or performs on par with prior work, especially when evaluated on static images.", "questions": "Please evaluate your method on videos with a moving camera.\n\nPlease compare to a simpler baseline that computes motion segmentation with a sota method and uses it to supervise the masks.\n\nPlease compare to prior work using exactly the same backbone network.\n\nHow is k chosen? How robust is your approach to the choice of k?", "limitations": "The authors completely ignored the main limitation of their work - it's reliance on static camera.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe intuition behind the approach is fairly straightforward and sound (grouping pixels in the optical flow field based on flow direction and magnitude does result in segmenting rigidly moving instances, it is the classic approach for motion segmentation). \n\nThe actual formulation of using affine motion model consistency to supervise a grouping method is novel to the best of my knowledge.\n\nThe paper is well written with almost no typos or grammatical mistakes.\n\nThe approach seems to work well in toy videos with a static camera.\n\nA minimal ablation study is provided.\n\n\nWeaknesses:\n\nThe authors don't discuss prior work on learning image-based, unsupervised object segmentation from motion. In particular, DyStaB of Yang et al., CVPR'21 which operates in a single object regime, and very recent Discovering Objects that Can Move of Bao et al., CVPR'22 which can segment multiple instances. The novelty of the proposed approach compared to these methods is in the loss formulation, but conceptually they are very close.\n\nA major limitation that is not discussed in the paper is that the method relies on the static camera assumption. In fact, segmenting moving objects in the optical flow field with a static camera is trivial. Binary segmentation can be obtained by simple thresholding, but separating instances can also be done with a primitive approach, e.g. by fitting an affine motion model. Going to the more general setting of a moving camera is much harder, however, and would require a very different approach. \n\nThe objective is fairly complex and is not described in sufficient detail (both for the intuition behind the design choices and for the implementation details). In fact, it took me a while to figure out if the model actually predicts optical flow (or some proxy for it), or if the flow is only used to compute the objective function. The latter seems to be the case, making the title misleading. Implementation details are provided in the supplementary, showing that the objective is fairly brittle and requires different hyper-parameters on different datasets.\n\nOn the same note, the complexity of the proposed objective seems unnecessary and the authors do not compare to any simpler baselines. For example, a state-of-the-art motion segmentation algorithm could be used to extract moving object segments first and these segments could them be used to supervise the masks directly. The benefit of such a baseline, besides simplicity, is that it would allow the method to generalize to videos with a moving camera, since motion segmentation methods are designed to handle the most general setting.\n\nThe authors use a stronger backbone network compared to prior work (Mask2Former vs shallow CNNs used by most baselines), making all the experimental evaluations invalid. It is thus impossible to say if the proposed objective provides and real benefits compared to prior work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fwPnWdGy-L", "reviewer": "Reviewer_9r13", "summary": "The authors approach unsupervised instance segmentation of images by exploiting motion\npatterns in videos during training. Given an RGB input frame, a neural network is trained\nto predict a segmentation that allows to approximate the optical flow field of that frame\nwell when using a simple, parametric motion pattern for each segment (e.g., affine\nmotion). Since the segmentation is predicted based on the RGB frame only, the network\nis trained to predict segments that *potentially* move and can be used to segment images\nafter training. On recent, synthetic multi-object datasets the proposed model is\nshown to outperform recent state-of-the-art object-centric models in almost all cases.", "questions": "- Are the appearance-based models also trained on frames of the video variants of\n  CLEVR and CLEVRTex? Or directly on CLEVR/CLEVRTex? In the latter case, an experiment\n  testing whether the different training data influences the performance of appearance\n  based models should be included.\n\n- Details about the dataset generation are not contained in the supplement,\n  although promised in the main paper.\n", "limitations": "## Limitations\nThe authors discuss limitations of their method adequately in a separate section. One\nlimitation to add in my view would be the necessity of ground truth optical flow. The\nablation study has shown a substantial performance drop when using optical flow\npredicted by SMURF, which probably makes it difficult to apply the method to natural\nvideos where ground truth optical flow is not available.\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Unsupervised segmentation is an active area of research. The general approach of\ndefining objects as regions with a simple motion pattern when they move is not new; the\nrespective related work is cited. The particular instantiation of the\napproach in this work however is novel and shown to clearly improve over previous models in\nalmost all cases. A particular strength of the paper is the thorough comparison with\nprevious models. The authors tested a large range of models on the datasets using\nmultiple seeds, going beyond most related papers in the field.\n\nOn the other hand, the authors only compare to object-centric models that not only aim\nat learning to segment images or videos but also to learn a useful representation of\nthose objects. Another line of works exists that only focusses on unsupervised\nsegmentation, with models that often make use of optical flow for training (and\nsometimes inference) and can be applied to more realistic data. The authors\ncite respective works, e.g. Mahendran et al. 2018, Yang et al. 2019, Choudhury et al. 2022.\nEmpirical comparisons with these models would have been informative and would strengthen\nthe paper, as these would allow to more directly compare the different approaches for\noptical flow based unsupervised segmentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MO4lZd71BE", "reviewer": "Reviewer_9Fas", "summary": "The author presents a method for unsupervised object segmentation by using temporal motion patterns. The paper transforms the segmentation as a minimizing ELBO problem. Some practical tricks of Gumbel-softmax have been applied to make the ELBO differentiable. The evaluations on the physics-related simulation datasets were conducted and outperforms the other baselines. ", "questions": "One question is how well can the methods perform in real-world cases. I’m not asking about adding more results in the rebuttal. It would be helpful if the authors can explain if the methods can apply to real-world cases and what are the potential limitations. For example, the KITTI dataset is one possible option, as it involves mainly rigid motions. But meanwhile, KITTI is a moving camera case, what are the possible concerns if the model is applied to a moving camera case.    \n\nHow did the prior distribution p0(m) initialize and how did different prior distributions affect the results?\n\nIt will be interesting to see what the authors’ thoughts on the potential to apply the proposed methods to the no-rigid motion are? What are the further requirements to achieve that?\n\nSome other detailed comments:\n\nLine 111: Please elaborate on the assumption “We then assume that the flow depends only the regions”\n\nLine 138: Please justify that “ We assume that regions are statistically independent ” is a reasonable assumption.\n\nMatrix I first appeared in equation (5) but is explained in line 161.\n\nline 2 form -> from.\n\nLing 156, missing space after as.\n", "limitations": "As the author stated, sometimes the motion doesn't provide enough information to segment the object. Maybe adding scene flow (3d flow with depth) can help to improve the performance. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper converts a classic optimization problem into a differentiable learning problem, the overall idea is very interesting. The experiments have shown the proposed method has achieved the SOTA performance on simulated datasets.\n\nOne weakness is no real-world scenario experiments, it would be more convincing if some of the experiments were conducted on real-world videos. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BLr5UzOaPZr", "reviewer": "Reviewer_iFdZ", "summary": "The authors train a segmentation network that processes still images to produce object masks. Video based supervision is used. The masks predicted for a single image are converted to optical flow using a motion model proposed by the authors. Two different motion models are considered. Hand-crafted post-processing is used to improve final segmentation performance.  \n\nThe key novelty is the proposed approach for matching object masks to an optical flow representation. The authors use a motion model (i.e. translation only / affine transform on each object region), introduce an efficient and differentiable implementation of matching, and evaluate the improvements over multiple datasets. Notable performance improvements are on out-of-domain datasets. Also, the method scores relatively better on the mIoU metric, denoting the multi-object capability.    \n\nAblations highlight the significance of ground truth optical flow for training (low performance without it) as well as better motion model (translation only gives lower results). \n\nThe authors include code in supplementary, and promise to share all code, datasets publicly in future. ", "questions": "1. Line 114: more explanation for reasoning in ELBO for regularization\n2. A direct formulation of the final loss function", "limitations": "Yes, limitations discussed well. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. A novel motion model based matching of object masks to optical flow\n2. The authors obtain considerable improvements over a range of datasets \n\n### Weaknesses\n1. Using ground-truth optical flow seems crucial for good performance (Table 3.a) - is this method scalable to real world data? Other methods using video supervision can operate on real-world data (e.g. [1]) \n2. Is the method really benefiting from the motion model or simply learning segmentation from the high quality ground-truth optical flow masks? \n3. The authors note how similar models struggle in complex settings, but limit all their evaluations to synthetic datasets: how would this translate to real world settings?  \n4. Lack of clarity/simplicity in final loss function: can the authors construct a simpler version of the final loss between masks output from model and optical flow labels (closer to the actual implementation in code)? Many terms in the given equations are constants (or are they learnable params?) \n5. Is comparing to other methods with the additional post-processing fair? The authors do include results without post-processing that are still above SOTA; those may be a better comparison. \n\n[1] [Discovering Objects that Can Move](https://openaccess.thecvf.com/content/CVPR2022/papers/Bao_Discovering_Objects_That_Can_Move_CVPR_2022_paper.pdf)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept