Segmenting Moving Objects via an Object-Centric Layered Representation
Authors: Junyu Xie
Weidi Xie
Andrew Zisserman
Venue: NeurIPS 2022 Accept (2022)
Keywords: motion segmentation
multi-object video segmentation
object-centric representation
layered representation
Transformer
Sim2Real generalisation

Abstract: The objective of this paper is a model that is able to discover, track and segment multiple moving objects in a video. We make four contributions: First, we introduce an object-centric segmentation model with a depth-ordered layer representation. This is implemented using a variant of the transformer architecture that ingests optical flow, where each query vector specifies an object and its layer for the entire video. The model can effectively discover multiple moving objects and handle mutual occlusions; Second, we introduce a scalable pipeline for generating multi-object synthetic training data via layer compositions, that is used to train the proposed model, significantly reducing the requirements for labour-intensive annotations, and supporting Sim2Real generalisation; Third, we conduct thorough ablation studies, showing that the model is able to learn object permanence and temporal shape consistency, and is able to predict amodal segmentation masks; Fourth, we evaluate our model, trained only on synthetic data, on standard video segmentation benchmarks, DAVIS, MoCA, SegTrack, FBMS-59, and achieve state-of-the-art performance among existing methods that do not rely on any manual annotations. With test-time adaptation, we observe further performance boosts.

Segmenting Moving Objects via an Object-Centric Layered Representation
Junyu Xie!
Weidi Xie
Andrew Zisserman'
Visual Geometry Group, Department of Engineering Science, University of Oxford, UK
Coop.. Medianet Innovation Center, Shanghai Jiao Tong University, China Kjyx,weide wew.wiiiia,zzrobtt.t az)@robots o...ttts..........oo....ooomm... ax.ac.uk
Abstract
The objective of this paper is model that is able discover, track and segment multiple moving objects in a video. We make four contributions: First, we introduce object-centric segmentation model with depth--rdered layer representation. This is implemented using a variant of the transformer architecture that ingests optical flow, where each query vector specifies an object and its layer for the entire video. The model can effectively discover multiple moving objects and handle mutual occlusions; Second, we introduce scalable pipeline for generating multi-object synthetic training data via layer compositions, that is used to train the proposed model, significantly reducing the requirements for ichour-intessive annotations, and supporting Sim2Real generalisation; Third, we conduct thorough ablation studies, showing that the model is able to learn object permanence and temporal shape consistency, and is able to predict amodal segmentation masks; Fourth, we evaluate our model, trained only on synthetic data, on standard video segmentation benchmarks, DAVIS, MoCA, SegTrack, FBMS-59, and achieve state- of-the-art performance among existing methods that not rely on any manual annotations. With test-time adaptation, observe further performance boosts.
Introduction
Humans have the ability to discover and segment moving objects videos. This ability is present even in young infants, together with notions of object permanence through occlusions, and temporal shape constancy
Achieving this ability by machine has long been goal of the field [Z] 68 and modern methods are now able to reliably discover and segment single moving objects using both appearance (RGB) and motion (optical flow) streams of the video. However, success is limited in the case of multiple moving objects and occlusion.
50
54
9 [1 27
33
45
58
59
Our goal in this paper develop and study model that can discover multiple moving objects and also handle their mutual occlusions. To this end, we propose a layered object-centric model that predicts amodal segmentation masks 773 for each object throughout the video. Note that, given amodal segmentations, the depth ordering of the object layers can be determined from occlusions in the observed (modal) frames when the objects overlap. See Figure Our innovation is build this mechanism into the representation: when objects overlap in frame, the model has to maintain the amodal shapes through temporal consistency, and can thus infer depth ordering.
To achieve this goal we use only optical flow as our primary information stream. There are benefits and drawbacks this choice. The benefits are fold: first, the information that the segmentation masks, are directly available from discontinuities in the flow field; second, a key advantage of using solely optical flow is that the domain adaptation problem is largely avoided [15] 32 and this means that the model can be trained on synthetic sequences and applied directly to real sequences without significant Sim2Real disparity. However, the drawbacks are also two fold: first,
after,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Amodal Masks Layer
RGB B
Optical Flow
Layer 0
Layer 2
Modal Masks
te
}
41
to
Figure I: Multiple-object Discovery and Segmentation. We develop a layered model for segmenting and tracking multiple objects by their motion, even under mutual occlusion. The model takes only optical flow as its input, and predicts an amodal segmentation (whole physical structure) for each object, and associating an object to the same layer throughout the video. The resultant modal masks (visible parts) can be constructed by overlaying layers based on the inferred depth ordering (in this case, layer 0-1-2 from front to back). From left to right: RGB sequence, estimated (imperfect) optical flows, amodal mask inferences from our model and resultant modal masks.
unlike the appearance stream, objects can be invisible in flow when they are not moving, thereby requiring the model build concept of object permanence; and second, but more subtly, flow not like appearance in that we cannot expect constancy even when the object is moving (for example, due to camera shake that affects the flow not the appearance), and thus the model cannot strongly use flow coherence as a cue but must instead learn to model the segmentation mask shape.
We make four contributions: (i) we introduce an Object-Centee Layered Representation (OCLR) model for discovering and segmenting multiple moving objects and inferring their mutual occlusions from optical flow alone. The architecture of the model is based on a transformer, with DETR like learnable queries, where each query determines the amodal segmentation mask for an object and its layer order throughout the video. (ii) we introduce a layer-based based synthetic pipeline generate multi- object training sequences that cover the variability of the target real sequences in terms of: multiple deforming objects, overlapping objects some frames, moving cameras, and non-stann son-static backgrounds. (iii) We demonstrate that the model trained on flow from the synthetic videos is able to learn object permanence and temporal shape constancy even if an object is partially occluded in many frames, the model is still able to predict an amodal mask. (iv) Finally, we evaluate the model (trained only on synthetic sequences) on multiple real video segmentation benchmarks, DAVIS, MoCA, Seg Track, and FBMS- 59; and demonstrate strong Sim2Real generalisation, achieving superior segmentation methods. The addition of a test-time adaptation
performance compared to other human-label to include appearance from self-super DINO model boosts the performance further, outperforming some supervised approaches that are finetuned on real data sequences.
2
The Object-Centric Layered Representation (OCLR) Model
This section describes our object-centric layered representation model that ingests optical flow for the frames video, and outputs the segmentation masks and tracks of the moving objects it contains. The model also determines the number of moving objects and their depth order. Specifically, the model predicts amodal (í.e. complete) segmentation masks and associates each object with a layer. The layers are composed to form the modal segmentations in the observed optical flow. The model is trained on synthetic sequences to learn temporal coherence and maintain the object shape even under occlusion, and as a consequence, develops notion of object permanence.
2.1 Layered representation
For given video with T frames, we can represent its motion by optical flows, Vnow {Fi, F,,..., }r). we aim to segment the moving objects in videos by adopting layered model, where each object segmented and consistently associated to one layer throughout the video:
((A (Ar, fT)] (Vooo)))
(I)
2

---

Layer depth f
Layer ordering
Optical flows
Amodal masks
encoders
Depth predictor
decoders
A
maxpooling object queries
dec
Transformer encader
Transformer decoder
attention map
Vrr-enc
W Tilec
FT
AT
enc
dec
K learnt queries
Figure 2: OCLR Model architecture. The model is U-Net architecture with Transformer bottleneck. takes optical flow as input and extracts spatial features by CNN encoders. A transformer encoder jointly processes spatio-temporal features across all frames, followed by transformer decoder that determines the layer representations. Each learnable query vector in the transformer decoder is associated to object and used to infer its layer depth. Additionally, the cross-attention maps from the last transformer decoder are extracted and upsampled by CNN decoders to infer amodal segmentation for the moving objects. Note, the skip connections from the CNN encoders to decoders are not shown.
where A, E #xx refers to the binary amodal segmentation of the i-th frame, together with predicted layer depth é R H, W refer the input height and width, and K denotes the number of object layers. Note that, the layer depth value here does not have true meaning, but only indicates layer ordering. By definition, at time steps, the same layer would remain empty or always bind to the same object throughout the video, i.e. objects can be effectively tracked by the considered layered representation.
Modal mask generation. Given the predicted amodal segmentation and layer depth, we describe the composition procedure to generate the modal segmentation for moving objects. In specific, we permute the index of layers (amodal segmentations) based on their depth values, such that, *-1 is always in front of A Note that, pixels amodal masks defined here are binary-valued, i.e. either or 1. For the i-th frame, its modal segmentation thus be computed via an iterative, front-to-back blending procedure:
M = (1 at) @ A
(2)
where of = clip(a, + M -0,, 1), denoting an accumulated opacity layer with a
2.2 Architecture
In order to accommodate the layered representation, we introduce an architecture that takes optical flows as input, and outputs a set of layers and their depths, see Figure We adopt U-Net architecture with Transformer- based bottleneck, the queries in transformer decoder play the role layers in our proposed representation, each is either empty or encoding the amodal shape of one moving object along the video.
CNN backbone. Given sequence of optical flow inputs, U-Net encoder is used to compute frame-wise features, V {o1,.... 3 = = eTn (Fi)...... Den (Fr)), where n; E Rhxxxe refers the feature maps, with height h, width w and number of feature channels c. Following [67]. use Instance Normalisation (IN) [60] after each convolution, which encourages the separation of foreground motions from background.
Transformer encoder. We flatten the outputs from the ConvNets in both temporal and spatial dimensions, and inject the positional information by adding of learnable spatio-temporal embeddings. As a consequence, the video features are converted into a sequence of vectors, and passed into Transformer Encoder model the temporal relationship across multiple frames.
Transformer decoder. pass the vector sequence to transformer decoder, where K learnable embeddings used as object queries for individual layers. We broadcast the queries along the

---

temporal dimension, and supply positional information with learnable temporal embeddings, Robj E RTTxe, The queries that only differ in temporal embeddings will be associated to the same moving object along the video.
As outputs from transformer decoder, we extract the last cross-attention maps and recover their spatial dimensions, p E xx2 which will be progressively upsampled with skip connections from the ConvNets encoders, resulting in binary amodal segmentation for the object queries in each frame, (AI...., Ar} = (p).
Layer ordering. Apart from amodal segmentations, the model will also infer the depth of the layers. We assume global layer ordering that remains fixed across frames. As shown in Figure 2 output object queries from transformer decoder maxpooled along the temporal dimension and passed through two feed forward layers, to get the output E IR RxIx indicating the depth for each of the K layers. Intuitively, this is equivalent reason the layer depth based on one selected key occlusion frame from the whole sequence. By defining the lowest depth value as corresponding to the top layer, then obtain depth order prediction between layers. Note that, for layers that are empty or contain non- internttag objects, the ordering is not strictly defined, thus any prediction should not be penalised.
2.3 Training objectives
In this section, describe the training procedure for the proposed architecture, with groundtruth supervision on both amodal segmentation and layer ordering.
Amodal segmentation loss. In addition to the conventional pixelwise binary classification (Lrce) on each the K layers, we also adopt a pixelwise classification loss defined only on a strip region around object boundaries, termed the boundary loss Loound This additional loss further emphasises the boundary regions, forcing the model to maintain shape information. During training, since the predicted amodal segmentations are permutation invariant, we use Hungarian matching to match their corresponding ground truth [1], and then compute the amodal loss:
K
T
aamodal
1
Abcee Loce (A, Ab) + Abound Loound (A) At))
KT
where Abee and Abound are loss weights.
Layer ordering lass. For layers with mutual occlusions, we train the order prediction module with ranking losses on all K! pairs of layers, formally,
(o( Fj) T
Londer
Σ log A
KI
where a is the sigmoid function with temperature factor т, and the ground-truth ordering indicates a relative depth relationship A; > fg between layer and j.
Total loss. The overall loss for training is a combination of amodal Lamodal and ordering loss Conter:
Liotal Lamodaal + Aanter Londer
where Awddr is the weight factor for the layer ordering loss.
Discussion. As discussed in early sections, our model is trained infer amodal (i.e. complete) object segmentations and layer depth, even though the object itself may sometimes be invisible in the flow field due to being temporarily static, or partially occluded by other objects. As a result, the model is forced to always maintain the object shape internally, i.e. notion of object permanence, as well as infer mutual occlusions from the visible flows. In the next section, we will introduce a pipeline for simulating video sequences, where groundtruth amodal segmentations and layer orderings can be generated at scale to train our model.
Synthetic dataset generation
We introduce scalable pipeline synthesize videos to train the proposed layered object-centric model. The pipeline builds on the method Lamdouar al. (32), but extends it to simulating videos with multiple objects and complex inter-object occlusions through a layer composition.

---

Background Layer
Homography
Thin-plate Spline
Layer 0
Layer 1
Layer2
RGB
Transformation
overlay
Frame 0
Optical Flow
Polygon
object
Motion Simulation
Overlay
Frame
Figure 3: Overview of the synthetic data pipeline. Left: Motion simulations of polygon and real object sprites based on homography transformations (including spatial translations) and thin-plate spline. Right: Layer composition of frames with independently moving objects.
As shown in Figure 3 we first simulate independent motions for foreground objects and backgrounds within each layer. To construct frame, overlay the layers via an iterative binary blending procedure, and an occlusion occurs when objects in different layers reside in the same spatial position, similar the idea discussed in Sect. 2.1 The proposed pipeline yields rich information at frame level, including RGB appearances, groundtruth optical flows, and modal and amodal object segmentations. Additionally, can also compute optical flows for the RGB sequences with off-the-shell flow estimator, for example, RAFT [57]. In the following sections, we describe some key procedures for video simulation, and leave more technical details to the Supplementary Material.
Foreground objects. We generate opaque 2D sprites with random shapes and RGB textures as foreground objects. In detail, we adopt two major shapes, namely polygons, and real object sprites. The polygon shapes can be either convex or non-convex, with different numbers of sides. A minimum distance between vertices is also defined to better support vertex-based deformations. On the other hand, the real object masks are directly sourced from the silhouettes in the YouTube dataset [66]. These generated objects then applied with textures sampled from the PASS dataset 3 which is large-scale image dataset with human identifiable information removed.
Non-rigid object transformations. To simulate non-rigid motions, apply both homography transformations and thin plate splines to the objects. Thin plate splines define a non- mon-linear coordinate transformation with set of control points. For polygon-shapee objects, we consider their vertices as control points, and spatially perturb subset of the vertices by distances d, E (0, Dz/2). where D, min (PP; l. Vj). defines the minimum distance between the perturbed point P; and any other points P,. For real object silhouettes, a uniform grid is distributed over the object, and the grid points are defined control points, with D; being the distance between adjacent grid points.
Backgrounds and artificial occluders. To simulate the camera motions, we randomly sample images from the PASS dataset, and apply random homography transformations and color jitterings. Additionally, we replace backgrounds with real videos cropped from copyright-free sources to simulate the complex background motions and depth variations. For cases with objects being occluded, we introduce artificial occluders superimposed as the top layer. These occluders are designed to have the same motion the background.
Summary. The proposed simulation pipeline enables an arbitrary number of videos be generated, together with multiple types of groundtruth annotation; for example, amodal segmentation and layer ordering. We prepare 4k synthetic sequences (around 120k frames) for training, the video sequences contain 1, 2, 3 objects in equal proportions. In the following, we train all models on this synthetic video dataset unless otherwise specified.
4 Experiments
4.1 Datasets
To evaluate our lulti-layer model, we benchmark on several popular datasets for video object segmen- tation tasks. A brief overview of the datasets given below, with full details in the Supplementary Material.
5

---

For single object video segmentation, we evaluate the model on DAVIS2016 49), SegTrackys [34]. FBMS-59 [46] and MoCA [3]. Note that, despite the fact that multiple objects may annotated, the community often treats Seg Track22 and FBMS-59 as a benchmark for single object segmentation 25 by grouping objects in the foreground.
68
To benchmark motion-based segmentation for multiple objects, we introduce synthetic validation dataset (Syn-val) and a curated dataset (DAVIS201 7-motion). The former is generated with the same parameters as our synthetic training set (Sect. 3 containing over 300 multi-object sequences (around 10k frames) with 1,2, objects at equal proportions, and controllable occlusions for evaluating modal and amodal segmentations in the ablation studies. Moreover, as objects in common motion cannot be distinguished purely from motion cues, we re-annotate the original DAVIS2017 [51] dataset by grouping jointly moving objects to form a new DAVIS201 7-motion dataset. We will release our modified annotations for future research.
4.2 Evaluation metries
Following the common practice [49] we adopt region similarity (J), aka. intersection-over in single object segmentation tasks. While for multiple object segmentation, we additionally consider the contour similarity (F) 44). As for MoCA dataset, since only bounding box annotations provided, we follow the same metric used in (37 67 and report mean object detection success rate averaged over different IoU thresholds (0.5, 0.6, 0.7, 0.8, 0.9..
4.3 Implementation details
Training details. As for pre- pre-processing. we use RAFT to estimate optical flows at 11 frame gaps, and resize the obtained flows to 128 x 224. During training, we split the video sequences into 30 frames per sample, each input frame is first encoded by U-Net encoder into feature map with 1 1/16 of its original spatial resolution, and passed to the transformer bottleneck. We use K learnable object queries, associating to 3 independent foreground layers. The model is trained by the Adam optimizer [28) with a learning rate linearly warmed up to x 10-5 during 40k iterations, and decreased by half every 80k iterations.
RGB-based test-time adaptation. To alleviate the drawbacks from using purely flow informa- tion (e.g. stationary objects over long period of time), we investigate the benefits of test-time adaptation with appearance features. For this we use the self-supervised DINO- EENO--petrained vision transformer (13] (VIT-S/8, patch sizes 8 x 8). The model is adapted by using the per-frame object masks obtained from the flow-based model predictions as noisy annotations. These annotations are used to finetune the last two layers of the vision transformer by supervised contrastive learning.
The finetuned model for each sequence is then applied for mask propagation in the same manner as (24). Instead of propagating from the first frame of the sequence, we formulate a heuristic selection process that picks single key frame based on temporal coherence between OCLR segmentation predictions. Object masks in this selected frame are then bi-directionally propagated across the whole sequence. Apart from the starting frame information, now-predicted masks other frames are also selectively introduced during the mask propagation process as form of dynamic refinement. Finally, we adopt CRF as a post-processing step to refine the resultant mask predictions. More technical details can be found in the Supplementary Material.
4.4 Ablation study
Here, we present a series of ablation studies on training details and pipeline for data simulation however, due to the space limitation, we can only summarise some key findings, we refer the reader to Supplementary Materials for all details.
As shown in Table I we can make the following observations: First, Instance Normalisation in CNN encoder is indispensable for training the network, as indicated by Ours-A vs. Ours-C: Second, training on only modal mask degrades model performance on all datasets, especially on Syn-Val with multiple objects and occlusions, suggesting that explicit amodal supervision are critical for learning object permanence, as indicated by Ours-B vs. Ours-D; Third, longer temporal input (Ours- vs. Ours-D) tends to result in slightly higher overall performance; Fourth, applying boundary loss (Ours-C C vs. Ours-Ei E) leads to noticeable performance boost. This validates assumption that

---

Table 1: Settings for training parameters. IN: Instance Normalisation: Amodal: Training on
amodal mask (vs. modal mask); Abound weight on boundary loss; number of input frames. Syn-Val Val (MLA) corresponds to modal and amodal results on synthetic dataset.
Training Settings
I (Mean)
Model
IN
Amodal
Abaund
Syn-Val (M I.)
DAVIS2016
Ours-A
X
0.2
30
83.5 13.0
67.6
48.7
Ours-B
0.2
30
81.1 76.9
69.2
50.5
Ours-C
v
0.2
30
85.6 84.7
72.1
54.5
Ours-D
0.2
82.8 83.0
71.3
53.5
Ours-E
0
30
80.9 81.6
71.5
54.1
focusing on object boundaries can help the model learn the information regarding object shapes and layer orders from optical flows.
4.5 Single object video segmentation
In this section, we compare our model with of-e-off-hhe-ar methods on various single object segmen- tation benchmarks. Note that, we mainly compare with the self-supervised approaches, both lines share the same spirit in the sense that training does not require any manual annotation, nor fine-tuning on the target dataset. As shown in Table 2 our flow-only model demonstrates superior performance over all other human-label- free approaches. Figure provides qualitative illustrations of the model, in comparison with other state-of-the- art methods. It can be seen that our predictions temporally consistent and not affected by noticeable background distractors flow signals. Furthermore, our model is capable of handling complex scenarios including heavy object deformations and occlusions. The other methods are not able to maintain the object shape consistently, and consequently their performance is weaker.
Table 2: Quantitative comparison on single object video segmentation benchmarks. "HA" stands for human annotations, and "SR" refers to the detection success rate on MoCA. In column Sup. (supervision), "None", "Syn.", "Real" represent self-supervision, synthetic data supervision, and real data supervision, respectively. Bold represents the state-of-the-arr performance (excluding our test-time adaptation results, which are labelled blue instead).
Training Settings
J (Mean)) 1
SR (Mcan)
Model
HA
Sup.
RGB
Flow
DAVIS2016
SegTrackv2
FBMS
MoCA
NLC T7
None
55.1
67.2
CIS (w. post-pracess.)
68
X
None
/
71.5
62.0
63.5
0.363
Motion Grouping
67
X
None
X
68.3
58.6
53.1
0.484
SIMO 32
Syn.
V
67.8
62.0
0.566
OCLR (flow-only)
Syn.
V
72.1
67.6
65.4
0,599
OCLR (test. adap.)
Syn.
80.9
72.3
69.8
0.559
FSEG
70.7
61.4
68.4
COSNet
Real
80.5
49.7
75.6
0.417
MATNet
Real
82.4
50.4
76.1
0.544
3D
Real
85.5
The RGB-based test-time adaptation gives a further performance boost on most of the benchmarks, supervised approaches that have been finetuned on the target video
and is even competitive data. Note that the test-time adaptation is actually detrimental to performance on MoCA. This is not unexpected though, as MoCA has many camouflage sequences where the objects are not visually distinguishable in appearance from their background environment, and motion provides crucial cues for discovering them.
4.6 Multiple object video segmentation
To the best of our knowledge, no existing unsupervised approaches have reported performance segmenting multiple objects purely based on optical flow. Apart from re-running the original self-supervised Motion Grouping [67] method (with three foreground queries) as baseline
7

---

RGB
Flow
Mation MATNE Groaping
No prediction
Ours (flow)
Ours (last
actap.)
GT
Figure 4: Qualitative results of single object video segmentation on DAVIS2016 and MoCA.
DAVIS2017-motio we also train variant of the Motion Grouping model directly supervised by our synthetic dataset, namely Motion Grouping (sup.). As an additional baseline, we train Mask R-CNN [22 model with only optical flow inputs on synthetic data, and refer to this as Mask R-CNN (flow-olly). We also compare with existing semi-supervised approaches. where the first-frame ground-truth segmentation provided, and the model only needs to propagate the masks through the video, thus an easier task than ours which requires to simultancously discover the objects and track them.
As shown in Table our proposed flow-only model outperforms both the Motion Grouping and Mask R-CNN baselines. This is because our OCLR model exploits a layered representation to maintain object shape through the video, which enables to segment the objects that are under occlusion or having unnoticeable motion. In contrast, Motion Grouping only predicts per-frame segmentations, iee. forced to preserve temporal relations between objects. This conjecture can also be confirmed by the qualitative results in Figure 5 for example, despite the person and one dog (4th column) not visible in the flow, our model still correctly recovers them. Morcover, after RGB-based test-time adaptations, the performance can be further boosted both quantitatively and qualitatively, for example, the boundary the pig mask in 1st column, the person in the 5th column.
Table 3: Quantitative comparison of multi-object video segmentation on DAVIS201 DAVIS2017-motion 7-motion. Note that, the compared methods here are trained without using any human annotations during training, in particular, Motion Grouping (sup.), Mask R-CNN (flow-only) and OCLR models supervised by only synthetic data, and other approaches are trained with self-supervision. Bold represents the state-of-the-an performance (excluding our test-time adaptation results, which are labelled blue instead).
Training settings
DAVIS2017-motion performance
Model
1st-frame-GT
RGB
Flow
J&FF
J (Mean)
F (Mean) 1
Motion Grouping 67
X
35.8
38.4
33.2
Motion Grouping (sup.)
X
V
39.5
44.9
34.2
Mask R-CNN (flow-only)
X
V
50.3
50.4
50.2
OCLR (flow-only)
X
V
55.1
54.5
55.7
OCLR (test. adap.)
/
V
64.4
65.2
63.6
CorrFlow
X
54.0
54.2
53.7
UVC [6
X
65.5
66.2
64.7
MAST 30
/
70.9
71.0
70.8
CRW 24
XX
73.4
72.9
74.1
DINO
78.7
77.7
79.6

---

RGB
Flow
Ours (test-time Ours adap.) (flow)
GT
Figure 5: Qualitative results of multi-object video segmentation on DAVIS2017-motian
Related Work
Video object segmentation has been longstanding task in computer vision, two protocols have attracted increasing interest in the recent literature 66
18 19 27
30 31
4 45 48
51 59 61
62, 66 [69], namely, semi-supervised video object segmentation (semi-supervised VOS), and unsupervised video object segmentation (unsupervised VOS). The former aims to re-localize one multiple targets that are specified in the first frame of video with pixel-wise masks, and the latter considers automatically segmenting the object interest (usually the most salient one) from the background in video sequence. Despite being called unsupervised VOS, in practice, the popular methods address such problems by supervised training on scage external datasets, this in contrast to proposed approach that does not rely on human annotations whatsoever.
Motion segmentation focuses discovering the moving objects in videos. In the literature, 9 27
45 (65 proposed to cluster the pixels with similar motion patterns; [14 58 59 train deep networks to map the motions to segmentation masks. In [68), adversarial training was adopted to leverage the independent motions between the moving object and its background; In 33), the authors propose to highlight the independently moving object by compensating for the background motion, either by registering consecutive frames, or explicitly estimating camera motion. In [67], a Transformer like architecture is used reconstruct the input flows, and the segmentation masks can be generated as a side product. In contrast to the existing approaches that primarily focus on single moving object segmentation, we focus segmenting multiple moving objects,
under mutual occlusions.
Layered representation was originally proposed by Wang and Adelson 663 to represent video as a composition layers with simpler motions. Recently, the layer decomposition ideas have been adopted for novel view synthesis [55] 71), separating reflections and other semi-transparent effects [ 2 20 40 41, or burgrround/bcckgruund estimation (20]. Unlike these approaches that primarily focus graphics applications, propose a layered representation to handle the occlusion problems multi-object discovery purely from their motions.
Object-centric representation decomposes the scenes into "objects", normally, visual frame re- construction has been widely used as training objective, for example, IODINE (21] uses iterative variational inference to infer a set of latent variables recurrently, with each representing one object in steps,
image. Similarly, MONet [10] and GENESIS [16] also adopt multiple sicoding-decodie propose slot attention mechanism, which enables the iterative binding procedure. this paper, we also adopt Transformer variant, but focus discovering objects in videos by motions.
29 39
Amodal segmentation refers to segmenting the complete shape of objects, including the invisible parts due to possible occlusions. Some existing approaches address this problem by applying human- or synthetic 23 supervision, while other works generate training datasets with
estimated [52 156
64
38 70 Although not specifically targeting the amodal segmentation task, our work leverages the idea of amodal perception and grants the trained model a notion of object permanence via synthetic amodal mask supervision.
synthetic occluders 35

---

6 Discussion
We have achieved our design goal, in that the model is able to handle multiple objects, and their occlusions, and to predict depth ordering of their layers. Also, the model demonstrates superior perfor- mance, both quantitatively and qualitatively, over prior methods that rely on zero human annotations. Nevertheless, there of course some limitations and room for further improvements: First, the current method may fail in very challenging real-world scenarios such as heavy object deformations, complex mutual interactions, etc. Probably more sophisticated data simulation pipeline with highly articulated objects or even 3D sprites would help to further reduce this Sim2Real domain gap. Second, our test-time adaptation has demonstrated a remarkable performance improvement benefiting from the combination between flow inferences and RGB correlations. This overcomes, to an extent, the problem of the flow-only model 'forgetting" the shape of objects when they are stationary for multiple frames. A further direction could naturally be to incorporate RGB information into our network in pre-training. Third, the model currently uses global depth ordering for the sequence, so it cannot handle situations such as order altering and mutual occlusions. Potential future studies could work on a more sophisticated layered model by focusing on pair-wise relationships between objects.
Despite these limitations, the approach has convincingly demonstrated both the value of inferring amodal segmentation masks, in order to handle occlusions, and the possibility of training such models entirely on synthetic sequences.
Acknowledgements
This research is funded by EPSRC Programme Grant VisualAI EP/T028572/1, a Royal Society Research Professorship RPARIN19113 132, and a Clarendon Scholarship. We thank Tengda Han for proof-reading.
References
[1] Jean-Baptiste Alayrac, João Carreira, and Andrew Zisserman. The visual centrifuge: Model-free layered
video representations. In CVPR, 2019.
[2] Jean-Baptist Alayrac, Joao Carreira, Relja Arandjelovic, and Andrew Zisserman. Controllable attention for structured layered video decomposition. In /CCV, 2019.
[3] Yuki M Asano, Christian Rupprecht, Andrew Zisserman, and Andrea Vedaldi. Pass: An imagenet replace-
ment for self-supervised pretraining without humans. In NeurlPS Truck on Datasets and Benchmarks,
2021.
Renée Baillargeon and Julie DeVos. Object permanence in young infants: Further evidence. Child
Development, 1991.
[5] Renée Baillargeon, Elizabeth S. Spelke, and Stanley Wasserman. Object permanence in five-month-old infants. Cognition, 1985.
[6] Pia Bideau Erik Learned-Miller. A detailed rubric for motion segmentation. arXiv preprint
arXiv:1610. 10033, 2016.
[7] Pia Bideau and Erik Learned-Miln It's moving! a probabilistic model for causal motion segmentation in moving camera videos. In ECCV, 2016.
[8] Pia Bideau, Aruni RoyChowdhury. Rakesh R. Menon, and Frik LcameedMMlllrr. The best of both worlds: Combining cnns and geometric constraints for hierarchical motion segmentation. In CVPR, June 2018.
[9] Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In
ECCV, 2010.
[10] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised
decomposition and representation. arXiv preprint
arXiv:1901.13990 2019,
[11] Sergi Caclles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Lue Van Gool. Onc-shot vidco object segmentation. In CVPR, 2017.
[12] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[13] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[14] Achal Dave, Pavel Tokmakov, and Deva Ramanan. Towards segmenting anything that moves. In ICCV workshops, 2019.
[15] Carl Doersch and Andrew Zisserman. Sim2real transfer learning for 3d human pose estimation: motion to the rescue. NeurIPS, 2019.
[16] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones,, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2020.
10

---

[17] Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In BMVC, 2014.
[18] Deng-Ping Fan, Wenguan Wang, Ming-Ming Cheng, and Jianbing Shen. Shifting more attention to vidco salient object detection. In CVPR, 2019.
[19] Katerina Fragkiadaki, Geng Zhang. and Jianho Shi. Video segmentation by tracing discontinuities in
trajectory embeddingg In CVPR. 2012.
[00] Yossi Gandelsman, Assaf Shocher, and Michal Irani. "Double-DIP"; Unsupervised image decomposition
via coupled peepiimage-priors. In CVPR, 2019.
[21] Klaus Greff, Raphael Lopez Kaufmmnn, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic
Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object representation learning with iterative
variational inference. In ICML, 2019.
[22] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. ICCV, 2017.
[23] Yuan- Ting Hu, Hong-Shun Shuo Chen, Kexin Hui, Jia-Bin Huang, and Alexander G. Schwing. vail-vos: Semantic amodal instance level video object segmentation - a synthetic dataset and baselines. In CVPR,
2019.
[24] Allan Jabri, Andrew Owens, and Alexei A Efros. Space-time correspondence as a contrastive random walk.
In NeurlPS, 2020.
[25] Suyoe Dutt Jain. Bo Xiong, and Kristen Crauman. Fusionseg: Learning to combine motion and appearance
for fully automatic segmentation of generic objects in videos. In CVPR, 2017.
[26] Yeong Jun Koh and Chang-Su Su Kim. Primary object segmentation in videos based on region augmentation
and reduction. In CVPR, 2017.
[27] Margret Keuper, Bjoemn Andres, and Thomas Brox. Motion trajectory segmentation via minimum cost
multicuts. In ICCV, 2015.
[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICI.R, 2015.
[29]
Thomas Kipf, Gamalcldin F. Élsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold,
Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centte Learning from Video. In ICLR, 2022.
[30] Zihang Lai, Erika Lu, and Weidi Xie. Mast: A memory-augmentedd self-supervised tracker. CVPR,
2020.
[31]
Zihang Lai and Weidi Xie. Self-xupervised learning for video correspondence flow. In BMVC, 2019.
[32] Hala Lamdouar, Weidi Xie, and Andrew Zisserman. Segmenting invisible moving objects. In BMVC,
2021.
[33] Hala Lamdouar, Charig Yang. Weidi Xie, and Andrew Zisserman. Betrayed by motion: Camouflaged object discovery via motion segmentation. In ACCV, 2020.
[34] Fuxin Li. Taeyoung Kim, Ahmad Humayun, David Tsai, and James M. Rehg. Video segmentation by tracking many figure-ground segments. In ICCV, 2013.
[35] Ke Li and Jitendra Malik. Amodal instance segmentation. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, 2016.
[36] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint-task self superviedd learning for temporal correspondence. In NeurlPS, 2019.
[37] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zimick. Microsoft coco: Common objects in context. In ECCV, 2014.
[38] Huan Ling. David Acuna, Karsten Kreis, Seung Wook Kim, and Sanja Fidler. Variational amodal object completion. In NeurlPS, 2020.
[39] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
[40] Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie. Andrew Zisserman, David Salesin, William T Freeman, and Michael Rubinstein. Layered neural rendering for retiming people in vidco. In SIGGRAPH, 2020.
[41] Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William T Freeman, and Michael Rubinstein.
Omnimatte: Associating objects and their effects in video. In CVPR, 2021.
[42] Xiankai Lu. Wenguan Wang. Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In CVPR, 2019.
[43] Kevis-Kokitsi Maninis. Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taixé Daniel Cremers. and Lue Van Gool. Video object segmentation without temporal information. TPAMI, 2018
[44] D.R. Martin, C.C. Fowlkes, and J. Malik. Learning to detect natural image boundaries using local brightness, color, and texture cues. TPAMI, 2004.
[45] Peter Ochs and Thomas Brox. Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions. In ICCV, 2011.
[46] P. Ochs, J. Malik, and T. Brox. Segmentation of moving objects by long term video analysis. TPAMI, 2014.
[47] Scoung Wug Oh, Young Lee. Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In JCCV, 2019.
[48]
Papazoglou and Vittorio Ferrari. Fast object segmentation unconstrained video. ICCV, 2013. [49] Federico Perazzi, Jordi Pust-Tuset, Brian McWillians, Luc Van Gool, Markus Gross, and Alexander Dorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.
[50]
Jean Piaget. The construction of reality in the child. New York: Basic Books, 1954,
[51]
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Sorkine-Hornung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: 170.:070440067,, 2017.
11

---

[52]
Lu Qi, Li Jiang. Shu Liu, Xiaoyong Shen, and Jiaya Jia. Amodal instance segmentation with kins dataset. In CVPR, 2019.
[53]
Christian Schmidt, Ali Athar, Sabarinath Mahadovan, and Bastian Leibe. D2conv3d: Dynamic dilated convolutions for object segmentation in vidcos. In WACV, 2022.
[54]
ELIZABETH Spelke, Principles of object perception. Cognitive science, 1990.
[55]
Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely.
Pushing the boundaries of view extrapolation with multiplane images. In CVPR, 2019.
[56]
Yihong Sun, Adam Kortylewski, and Alan Yuille. Amodal segmentation through out-of-task and out-of-
distribution generalization with bayesian model. In CVPR, 2021.
[57]
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020. Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid. Learning motion patterns in videos. In CVPR,
[58]
2017.
[59]
Pavel Tokmakov, Cordelia Schmid, and Karteek Alahari. Learning to segment moving objects. IJV, 2019. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprini (XXix:6607.0000
[60]
2016.
[61]
Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang Chich Chen. Feelvos: Fast end-to-end embedding learning for video object segmentation. In CVPR, 2019. Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In ECCV, 2018.
[62]
[63]
John YA Wang and Edward H Adelson. Representing moving images with layers. The IEEE Transactions
on Image Processing Special Issue: Image Sequence Compression, 1994.
[64]
Yuting Xiao, Yanyu Xu, Ziming Zhong, Weixin Luo, Jiawei Li, and Shenghua Gao. Amodal segmentation
based on visible region segmentation and shape prior. In AAAI, 2021.
[65]
Christopher Xie, Yu Xiang, 7aid Harchaoui, and Dieter Fox. Object discovery videos as foreground motion clustering. In CVPR, 2019.
[66]
Ning Xu, Linjie Yang. Yuchen Fan, Dingeheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale vidco object segmentation benehmark. In ECCV, 2018. Charig Yang, Hala Lamdouar, Erika Lu,, Andrew Zisserman, and Weidi Xie. Self-supervised video object
[67]
segmentation by motion grouping. In ICCV, 2021.
[68]
Yanchao Yang. Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object
detection via contextual information separation. In CVPR, 2019.
[69]
Zhao Yang. Qiang Wang, Luca Bertinetlo, Song Bai, Weiming Hu, and Philip H.S. Torr. Anchor diffussio
for unsupervised video object segmentation. In ICCV, 2019.
[70] Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy. supe-supe
scene de-ncclusion. In CVPR, 2020.
[71] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification:
Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
[72] Tianfei Zhou, Shunzhou Wang, Yi Zhou, Yazhou Yao, Jianwu Li, and Ling Shao. Motion-attentivy transition for zero-shot vidco object segmentation. In AAAI, 2020.
[73] Yan Zhu, Yuandong Tian, Dimitris Mexatas, and Piotr Dollár, Semantic amodal segmentation. In CVPR,
2017.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
12

---

[Yes] In Abstract and Section I we have clearly included the intuitions, motivations and contributions in our work, and briefly introduced a depth-ordered layered model for multi-object motion segmentation by following Sim2Real training procedure. (b) Did you describe the limitations of your work?
[Yes] In Section 6 we reflect on several limitations and propose possible further research directions.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] In the Supplementary Material, we have included section discussing if our work leads to possible negative social consequences. Overall, we reckon that our work does not lead to noticeable detrimental effects to human rights, security, environment, human livelihood, economics, etc.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] In the Supplementary Material, provided detailed explanation on how our work conforms ethics review guidelines, including our effort to eliminate potential bias and personally identifiable information in the dataset.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] There are no theoretical results introduced our work.
(b) Did you include complete proofs of all theoretical results?
[N/A] There are no theoretical results introduced our work.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)?
have provided the details in the curated DAVIS2017-motin dataset, together with some demonstration videos of our synthetic dataset. The full datasets and codes for training and evaluation will be released before the publication.
[No] In the Supplementary Material,
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Section we have described the dataset, evaluation metrics and implemen- tation details in our work. We also give more training details and settings in the Supplementary Material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)?
[Yes] In the Supplementary Material, we have reported the main results with error bars from repeated experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] In the Supplementary Material, we have reported the amount of computation to train, test and test-time adapt our model, together with details of the adopted GPU and approximate time taken.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] In Section
4 we have described the datasets we used with references to the
original source.
(b) Did you mention the license of the assets?
[Yes] the Supplementary Material, we have mentioned that dataset set used and curated follows CC-BY license with references to the original source provided. Real video backgrounds for synthetic data generation are from http://pixabay.com/iiiii
All selected video follows Pixabay License which states that videos are free for any commercial and non-commercial uses with no attribution required.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] In the Supplementary Material, we have explained in detail how we curate DAVIS2017 dataset to new DAVIS2017-motion dataset, and also provided more
13

---

information about our synthetic dataset. All generation code and resultant datasets will be released.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating?
[Yes] In the Supplementary Material, we have discussed that the datasets used for curation and synthetic data generation are DAVIS2017, YouTube and PASS, which follow CC-BY license. We have included references for all these sources.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content?
[Yes] In the Supplementary Material, there a discussion on how we eliminate personally identifiable information and offensive content in our curated dataset and synthctic dataset.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A] There are no crowdsourcing or conducted research with human subjects in our work.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable?
[N/A] There are no crowdsourcing or conducted research with human subjects in our work.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation?
[N/A] There are crowdsourcing or conducted research with human subjects in our
work.
14


Reviews:
[{"review_id": "yzWUioelOY", "reviewer": "Reviewer_mEda", "summary": "This paper considers the problem of segmeting multiple moving objects in a scene by taking optical flow as inputs. The solution is composed of two key ideas: First, the inputs are optical flow, so they can employ simulated videos for training, and the learned model is able to generalize to real videos. Second, a depth-oredered layered representation is used to handle mutual occlusion. Experimental results show improvements upon methods with non-layered representation on single/multiple object segmentation datasets.", "questions": "There might be some hard cases for layered representation. In some cases, objects have mutual occlusion with each other, say a part of object A occludes object B, meanwhile a part of object B occludes object A. Can the proposed method handle such cases? If not, is there any solutions to such cases under the layered representation framework?", "limitations": "The authors have discussed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nGood. The key innovations are three fold: (1) object-centric representation; (2) layered representation; (3) amodel representation. Although all three techniques have already been explored in previous literature, there is no such an attempt before to combine them for video object segmentation. Furthermore, most relevant works experiment on simulated datasets, while in this work all experiments are conducted on real world videos, which is much more difficult and more convincing. \n\nQuality:\n\nGood. The proposed method is easy to understand and shows good performance on all dataset considered, surpassing the most relevant method, Motion Grouping, by a significant margin.\n\nClarity:\n\nGood but can be improved. In general the paper reads smoothly and is easy to follow. But, perhaps due to limited space, some important details are defered to the supplementary material, which from my point of view, would be better to be placed in the main text. For example, test-time adaptation seems bring large improvements and is important to the final performance, it would be clearer to put relevant details in the main text.\n\nSignificance:\n\nThe sutdied topic is of great importance. Object-centric, layered repersentation shows promise to become the next generation of vision paradigm.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "p7UDmTcBFpN", "reviewer": "Reviewer_2xHR", "summary": "This paper proposes a U-net for generating a motion segmentation of a given motion field. In particular the method assumes T snapshots of the field and generates T amodal segmentation masks. The novelty of the paper lies in combining the U-net with a Transformer, i.e. the Transformer receives the latent embeddings of the U-Net encoder and generates output embeddings from which the segmentations masks and an ordering of these masks is inferred. The authors follow the idea of layered motion and assume that each motion segment has unique depth by that the ordering is interpreted as a depth ordering. The model is limited to a fixed size of layers (queries), in this paper three layers. The method is then trained on new synthetic flow data without any manual labelling and applied to a variety of existing datasets and also to new data.", "questions": "How would your method behave in the case of multiple persons walking frontoparallel in the same direction?\nIs your method influences by the choice of the optical flow method?\nYou assume three layers / three motion segments. How did you come to this assumption of three?\nWhat is your idea how to handle this dilemma between appearance and flow for camouflaged cases in your method (line 238)?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written, very well structured. The methodological approach sounds appropriate and the presentation of the results is well done. I am not working on this specific problem but I believe the authors that the proposed neural network model is novel and not published before. It shows how this generative problem can be elegantly solved by combining a U-Net for the task of segmentation together with a Transformer for the task of inferring spatiotemporal correlations to guide the segmentation. The evaluation seems properly done with partially very good results, although some statements go to far, e.g. in line 227 where it is argued “…large margin …”, although improvements are not that large e.g.for FBMS-59.\nI believe that it has been shown that motion segmentation is basically learnable from synthetic data only. If so, one of the co-reviewers more expert than I will definitely refer to this. \nWhat I miss in the paper is a clear distinction between motion segmentation and occlusion reasoning (2.1D sketch). As the model can only infer motion boundaries, it might miss some of the occlusion boundaries especially for surfaces with interior occlusion boundaries. It is also not clear to the reader how the model behaves for corrupted flow inputs or for more than three motions, e.g. also for a moving camera. Finally, the reader might also be interested to understand why longer sequences give better results or why object boundaries can help improving the layer ordering (line 223).\nMonet is also a new method to infer occlusion regions and motion boundaries (H. Kim et al., BMVC’21). I suggest to clarify this in the paper. I also suggest to add X. Zhan et al.’s, CVPR’20 paper to the related work that shows a nice way to compute amodal maps.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "e1XT1k7hHi", "reviewer": "Reviewer_zSVP", "summary": "**Problem**: The paper addresses the problem of discovering and segmenting multiple foreground objects in videos without using supervision. \n\n**Solution**: The proposed solution involves training on synthetic data and only leveraging optical flow (not RGB) to facilitate easier sim2real generalization. The paper introduces a novel model architecture which tasks as input a sequence of optical flow frames and produces $K$ amodal segmentation masks each associated with an estimated depth ordering. These amodal segmentation maps can be combined using the ordering information to form the final estimated multi-object segmentation map. The architecture involves a transformer based model inspired from DETR which uses K learned queries to produce the K unique outputs.\n\n ", "questions": "- Please discuss the question on unsupervised vs synthetic supervised presented in the Weakness above which is my only major concern.", "limitations": "Yes, the limitations have been discussion fairly well. \nThere is no discussion on the societal impact. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n### Writing\nThe writing of the paper is very clear and easy to follow. The paper is organized very well to facilitate easier understanding of the details of the model and proposed data generation pipeline. The details of the proposed model is presented in a concise manner covering most necessary aspects. The paper includes detailed discussions of the pros and cons of most design choices. These discussions make it much easier to follow and verify the claims. \n\n### Proposed Model \nThe paper proposes a novel architecture that is interesting and might be applicable to a wider range of domains. The model makes several interesting design choices:\n- The choice of leveraging optical flow only from synthetic data is not novel and has been used in other domains. However, the application to the problem of multi-object segmentation is unique. \n- The model goes beyond segmenting objects in individual frames and produces *amodal* segmentation maps for each frame. This is an interesting design choice that seems to be very effective (see more about this in weakness below). \n- The core of the model is heavily inspired from DETR, but the ideas of using this architecture to estimate layer depth and amodal  masks is still novel and interesting. In addition to modifying and adopting the objective of DETR for estimating the masks, the paper introduces a layer ordering loss to accurately estimate the depth order of each object. \n\n### Synthetic Data Generation\nThe data generation pipeline proposed in this work is novel. The utility of this data beyond the task of multi-object segmentation is unclear. However, I believe this pipeline could be adopted by other researchers in this domain. \n\n### Evaluation\nThe experimental evaluation in this work exhaustively covers standard benchmarks and the necessary ablative studies to verify the claims.\nAcross two tasks (single object and multi-object segmentation), the proposed model outperforms existing unsupervised learning methods. The ablative studies show the benefits of using amodal segmentation maps as the intermediate output which is a key design choice of the proposed model.\n\n## Weakness\n\n### Supervision and Comparisons\n- The paper claims to be an **unsupervised** method for video object segmentation. I'm not sure if this is true based on the conventional usage of the term unsupervised. The proposed method uses synthetic supervision. This can obviously be corrected in the text. However, the bigger issue is the comparisons to existing work. If this work were to be categorized as a supervised learning method, the single object results are only as good as other supervised learning methods and in the multi-object case there is no comparison to supervised learning methods.\n- Since the proposed method uses synthetic supervision, it is also *somewhat* unfair to compare to a model trained with real-world human supervision. But I think it is at least important to demonstrate the benefit of synthetic data *i.e.* scalability. Since generating synthetic data is not expensive, if the proposed model can scale in performance with volume of synthetic data and outperform supervised methods that use limited supervision, that would be a compelling result for adopting the proposed model. \n\n### Amodal Evaluation\nOne of the interesting aspects of the proposed model is the choice of producing amodal maps as the intermediate step. However, the evaluation of this output is limited to the ablation of comparing to a model using modal maps. It would have been interesting to see how well this model performs on the amodal segmentation task. \nSee the following for evaluation protocol:\n>Zhan, Xiaohang, et al. \"Self-supervised scene de-occlusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n> Xiao, Yuting, et al. \"Amodal segmentation based on visible region segmentation and shape prior.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 4. 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NRtoGO-4-7r", "reviewer": "Reviewer_FcJQ", "summary": "The authors propose a new model for amodal segmentation of videos. Given optical flow,\na Transformer-based neural network predicts an amodal (i.e., unoccluded) segmentation\nmask and depth for each object in the input frame. The model is trained using ground\ntruth supervision on a synthetic dataset created by the authors. When evaluated on\ncommon video instance segmentation benchmarks, the model shows promising results without\nrequiring further training. The proposed model clearly outperforms previous unsupervised\nsegmentation methods and even outperform some supervised methods trained on the\nrespective benchmarks.", "questions": "What is your notion of an \"unsupervised\" model? It is stated that the proposed model\n\"achieve[s] state-of-the-art unsupervised segmentation performance\". The model however\nis trained with ground truth supervision, so in my view the model cannot be termed\nunsupervised and the comparison to truly unsupervised methods is misleading. Instead, it\nshould be made clearer that the proposed method does \"unsupervised transfer\".\n\nI cannot completely follow the computation of the model segmentation:\n- The first accumulated opacity layer $\\alpha^0$ should probably be all zeros. As it is\n  stated now, it follows that $\\alpha^1=2\\cdot \\hat{M}^0$.\n- The modal segmentation masks are added to obtain the opacity layers. This however\n  means that in the case of occlusions it might be $\\alpha^k > 0$, leading to the\n  amodal mask being multiplied with a negative weight. Did you take any means to adress\n  this problem?", "limitations": "One limitation of the model arises due to using optical flow as primary input, which\nmakes it impossible to segment non-moving objects. This limitation is discussed openly\nby the authors and subjected to future work, what I believe is justified.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Video object segmentation is an active area of research. Predicting layered amodal segmentation masks that can be combined into a modal\nsegmentation is an uncommon but very natural approach. With this approach, the model is tasked to not\nonly learn about visible object parts but complete objects. Occlusions arise naturally\nwhen masks are combined. Compared to the more common approach of only reasoning about\nvisible object fragments this may result in a more useful object representation.\n\nThe main weakness of the paper in my view is the comparison with other models. The\nproposed model is trained supervisedly on synthetic data and then transferred without\nsupervision. None of the other models is trained in this way, but either completely\nunsupervised or trained supervisedly on the respective benchmark. The performance\nimprovements on the benchmark are therefore not necessarily due to the proposed model\narchitecture, but might as well arise due to the different data seen during training.\n\nA fair comparison would be to also train the other methods on the synthetic dataset and\nevaluate on the benchmarks. The key question then would be whether the transfer\nperformance of the proposed model is better than that of previous models, which would be\nespecially interesting for other models using optical flow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept