Motion-inductive Self-supervised Object Discovery in Videos
Authors: Shuangrui Ding
Weidi Xie
Yabo Chen
Rui Qian
XIAOPENG ZHANG
Hongkai Xiong
Qi Tian
Venue: Submitted to ICLR 2023 (2023)
Keywords: Video Object Segmentation
Motion Segmentation
Object Discovery

Abstract: In this paper, we consider the task of unsupervised object discovery in videos. Previous works have shown promising results via processing optical flows to segment objects. However, taking flow as input brings about two drawbacks. First, flow cannot capture sufficient cues when objects remain static or partially occluded. Second, it is challenging to establish temporal coherency from flow-only input, due to the missing texture information. To tackle these limitations, we propose a model for directly processing consecutive RGB frames, and infer the optical flow between any pair of frames using a layered representation, with the opacity channels being treated as the segmentation. Additionally, to enforce object permanence, we apply temporal consistency loss on the inferred masks from randomly-paired frames, which refer to the motions at different paces, and encourage the model to segment the objects even if they may not move at the current time point. Experimentally, we demonstrate superior performance over previous state-of-the-art methods on three public video segmentation datasets (DAVIS2016, SegTrackv2, and FBMS-59), while being computationally efficient by avoiding the overhead of computing optical flow as input.

Under review as a conference paper at ICLR 2023
MOTION-INDUCTIV SELF-SUPERVISED OBJECT DIS- COVERY IN VIDEOS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we consider the task of unsupervised object discovery in videos. Previous works have shown promising results via processing optical flows seg- ment objects. However, taking flow as input brings about two drawbacks. First, flow cannot capture sufficient cues when objects remain static or partially oc- cluded, Second, it is challenging to establish temporal coherency from flow-only input, due to the missing texture information. To tackle these limitations, we propose a model for directly processing consecutive RGB frames, and infer the optical flow between any pair of frames using a layered representation, with the opacity channels being treated as the segmentation. Additionally, to enforce ob- ject permanence, we apply temporal consistency loss on the inferred masks from randomly-paired frames, which refer to the motions at different paces, and en- courage the model to segment the objects even if they may not move at the current time point. Experimentally. we demonstrate superior performance over previous fate-of-theeare methods on three public video segmentation datasets (DAVIS2016, 6, SegTrackv2, and FBMS-59) while being computationally efficient by avoiding the overhead of computing optical flow as input.
INTRODUCTION
Representing the visual scene with objects as the basic elements has long been considered fun- damental cognitive ability of the intelligent agent, for it enables understanding and interaction with the world more efficiently, for example, combinatorial generalization in novel settings (Tenenbaum al. 201T). )) Although remains somewhat obscure the level of neurophysiology on exactly how humans discover the objects in a visual scene in the first place, it is consensus that motion seems to play an indispensable role in defining and discovering the objects from the scene. For example, in 1923, Wertheimer introduced the common fate principle that elements moving together tends to be perceived as a group (Wertheimer 1923); while later Gibson claimed the independent motion has even been treated one attribute define object visually (Gibson & Carmichael [966). Grounded on the above assumptions, the recent literature has witnessed numerous works with dif- ferent models proposed for segmenting the moving objects via unsupervised learning YYng al.,
2019 2021b
Liu et al. 2021).
Exploiting optical flows for object discovery naturally incurs critical limitations: First, objects in videos may stop moving or be partially occluded at any time point, leaving no effective cues for their existence in the flow field; Second, computing optical flow from pair of frames refers lossy encoding procedure, that poses significant challenge for establishing temporal coherence, due the lack of effective texture information. In contrast, adopting RGB frame sequences poses a few clear advantages. The most obvious one is that, while objects do not necessarily move all the time, the property of temporal coherence in RGB space naturally guarantees a preliminary understanding of object permanence; Additionally, the rich textures in the appearance stream give more distinctive patterns than those in motion, allowing to better identify and distinguish the different objects. Last but not least, processing RGB streams still enables faster processing speed than using optical flow.
In this paper, our goal is to train a video segmentation model that can discover the moving objects within sequence of RGB frames, in the form of segmentation. In specific, our proposed model first encodes consecutive frames independently, into a set of frame-wise visual features, that is followed by temporal fusion with Transformer encoder. To localise the moving objects, we randomly

---

Under review as a conference paper at ICLR 2023
RGB frame
Optical flow
Flow-only prediction
Our prediction
Time
Figure I: Illustration about Bicycle Motocross (BMX) sequence on SegTrackv2 Li et al. 2013 The red boxes and the yellow boxes refer to the arm and the back of the player, respectively. Flow- only method (Yang et al. (021a) fails to track the same region in temporal consistent fashion since it derives the foreground region directly from current optical flow. However, methodology of processing a RGB video clip develops a sense of object permanence and solves the issue.
pair the visual features from two frames and pass them into a frame comparator module, effectively establishing the relative motion between frames. Inspired by Yang et al. (2021a), we decode the motion features into optical flows with a dual-layered representation, with the opacity weight of each layer treated as the segmentation mask. At training time, we exploit off- optical flow estimator, e.g., RAFT (Teed & Deng 2020), as the induction for flow reconstruction. To develop the property of object permanence, masks, which encourages the model to mine effective texture information from the RGB sequence and keep track of the objects even if they may be static at the current time point.
enforce a temporal consistency on the inferred segmentation
In short, we summarize the contributions in this paper: First, we introduce the Motion-inductive Object Discovery (MOD) model, simple architecture for discovering the moving objects in videos, by directly processing a set of consecutive RGB frames. Second, we propose self-supervised proxy task that is used to train the architecture without relying upon any manual annotation. To overcome the challenge from flow-based methods, i.e., objects may stay static or move slowly, we adopt a random- random-paired policy and restrain the temporal consistency. Third, we conduct series of ablation studies to validate each key component of our method, such as the temporal consistency of random- paired flow. While evaluating three public benchmarks, we demonstrate superior performance over existing approaches on DAVIS2016 (Perazzi et al.) (2016), SegTrackkv2 (Li et al.) (013)), and FBMS- 59 (Ochs et al. 2013), with considerable speed-up during the inference procedure.
2 RELATED WORK
Video Object Segmentation. How to segment objects coherently in one video sequence has tended the topic of instance segmentation in the image. There is a great amount of work about video object segmentation (VOS) in recent decades (Caelles et al.) 2017 Hu et al. [017]] Fan al.]
2019
Dutt Jain et al.] 2017; Lai & Xie 2019 Maninis et al. 2018 Oh et al.] 2019:] Voigtlaender
al. 2019: Caelles al. 2017 Perazzi al. 2017: Hu et al. 2018 Li Loy] 2018 Bao al. Voigtlaender et al. 2019:Johname
2018
et al. 2019)) Recently, the research on getting rid of the dense annotation and designing more effective self-supervised algorithms has attracted more and more interest in the computer vision community including VOS (Xu & Wang 2021 Jabri et al.)
2020 Lai et al. 2020: Li et al. 2019; Vondrick et al., 2018 Lu et al. 2020 Wang et al., 2019 Kipf et al. 2022). For VOS, there are two mainstream protocols to evaluate the learned model. One is semi-supervised video object segmentation, the other is unsupervised video object segmentation.
Given the first- frammf mask of the objects of interest, semi-supervised VOS tracks those objects in subsequent frames, while unsupervised VOS directly segments the most salient objects from the background without any reference. These two protocols are defined in the inference phase, meaning
2

---

Under review as a conference paper at ICLR 2023
methods could leverage ground truth annotations in the training stage. In this paper, don't 't use any kinds of manual annotations for either training or evaluation.
Motion Segmentation. As the name suggests, the aim of motion segmentation is to discover moving objects. One line of the work (Brox & Malik, 2010; Fragkiadaki et al. 2012 Ochs & Brox,
2012 Ochs al., 2013 Lezama al.) 2 Keuper al. 2013)) formulates motion as the point trajectory to take advantage of long-range temporal information so that segmentation results can be acquired by grouping the trajectories. Later. deep learning methods take over the area (Tokmakov et al. 2017b Xie et al.] 2019 Yang et al.
al.] 2019, 2021a Choudhury et al., 2022 Yang et al. [22 Ye et al.j 2022a Wang et al. 2022). [Tokmakov et al. (2017b) adopts two-stream network that ingests both RGB and optical flow. Then they realize memory mechanism by the convolutional
recurrent unit to enhance the visual cues. CIS (Yang et al., 2019) achieves fully unsupervised motion segmentation which discards the supervision of annotated masks during training. By formulating a min-max game of mutual information, the generator is asked to create foreground segments that the warp synthesis error to train appearance and motion pathways without any supervision. The most similar work to ours is MG (Yang al.) 202Ta), which solely leverages the optical flow to separate the pixels via cross attention mechanism (Locatello et al. 2020). Compared to MG (Yang) al. 202 we keep reserved on the module design and the training recipe to demonstrate the improvement brought by our method is purely from taking consecutive RGB frames instead of optical flow. Recent work GWM (Choudhury et al. 2022) also utilizes RGB images and adopt the supervision from optical flow but their model ingests single image to segment the foreground and fails to consider temporal coherency.
unrelated possible to the background. AMD (Liu et al.] (02I) minimizes
Object Discovery. There is rich literature on identifying salient objects without explicit super- vision, known object discovery. There exist series of works that aim to learn object-centric representations in images (Lccatell et al.
Lin et al. 2020 Jiang & Ahn 2020; Greff et al. 2019 Crawford Pineau| 2019: Engelcke et al. 2019
2020
2019 Emami al. 2 Burgess al.
2021 Typically, TODINE (Greff et al. 2019) develops iterative variational inference to separate
different objects. Locatello al. (2020) proposes slot attention iteratively update latent object
representations. Further, a line of works (Zablotskaia et al. Min
Min et al. 2022 Kosiorek et al.)
2018 Kipf et al.. 2022 Jiang et al. 2 Kabra et al., 2021 Crawford & Pineau 2020 Beshinar
& Frossard 2021 Bear et al. 2020: Bao et al.) [022 Ye et al. 2022b Yang et al. (222aa) extend
object-centric learning to video domain. Most of these approaches incorporate motion cues into the
reconstruction task and perform well in moving object segmentation but perform poorly in process- ing static objects. While in this work, we adopt slot attention to form optical flow reconstruction bottleneck and perceive both dynamic and static instances through RGB clip input in temporally consistent way.
3 MOTION-INDUCTIVE OBJECT DISCOVERY (MOD) MODEL
In this section, we detail our MOD model, which processes a set of consecutive RGB frames and automatically discovers the moving objects in the form of segmentation. An overview of the training procedure can be seen in Figure 2 where the visual features computed from individual frames are temporally fused, and randomly paired together, for decoding the optical flows between corre- sponding frames. Taking inspiration from the motion grouping (Yang et al. (021a)), we also adopt dual-layered representation for the output flow, with the foreground and background flows being reconstructed separately, and later composited with the inferred opacity masks (soft segmentation). In the following section, we will detail each key component in our proposed architecture.
3.1 SPATIAL-TEMPORAL VISUAL ENCODER
To start with, our model takes short video clip as input, i.e., f. FT U E RT'xHxxx3 consisting of a set of RGB frames, the frame-wise visual representations are computed with a shared visual encoder &enc- Formally, the output feature map or at timestamp (1 < < T) can be obtained:
0: B (xte)
TR. hxxxd
3

---

Under review as a conference paper at ICLR 2023
Visual Eacoder
Temporal Fusion
Frame Comparator
Iterative Routing
Flow Decoder
Handenjar
Framestle Vonal Reprosstation
Random
Frame Comparator
Iterative Routing
Choroienn
Figure 2: Model architecture. Our model first extracts spatial features of consecutive frames by the visual encoder. To jointly model the temporal relation, we aggregate and interact among the multi-frame features with late-fusion. We randomly pair frames' visual representations and pass them into the frame comparator to encode relative motion. Then, we decode the flow from random-paired frames. Through iterative routing, we adopt dual-layer representation for the flow reconstruction, i.e. outputting the foreground and background flows separately, and composing them with inferred opacity weights. The whole training procedure does not require supervision from any mask annotations.
where h, w, d denotes the dimension of the height, width, and channel, respectively. Till this point, build the temporal dependency between multiple visual frames, a global fusion module is introduced, via a standard Transformer Encoder layer:
fō ) Prmp ppp.. or + per)),
(2)
where pe refers to the learnable spatial-temporal positional encodings, and the output ā E Rexwxd remains the same dimension as input.
By taking the multiple RGB frames as input, our proposed visual encoder can explicitly consider the temporal coherence within the video clip. Note that, such seemingly simple design poses two critical differences from previous work on motion-driven object discovery (Yang et al.) 2019 202 where only frame-wise optical flow adopted: First, using RGB frames as input can drastically reduce the computation latency at inference time. The throughput of our model without computing dense optical flow reaches round 100 fps on a standard 32GB Tesla V100 GPU while prior works need to calculate flow at first. RGB provides more semantic information than flows for the model to exploit, i.e., including not only the object's shape but also its texture. Second, processing multiple frames contributes to the development of sense of object permanence within the video clip, i.e., the understanding that items or people still exist even when they cannot be perceived explicitly. Therefore, though the objects in videos may stop moving be partially occluded at any time point, they can still be effectively segmented with the temporal cues.
3.2 RANDOM-PAIRES FRAME COMPARATOR
Till this point, we consider building the relative motion between any visual frames within the video clip, from reference frame to target frame j. i.e., fi-tj- In specific, we select the visual
representation of the two corresponding frames, i.e, o, and oj. concatenate them along the feature dimension, and feed it into comparator module:
fi-55 = Peom (concatc,,,,,( ūj)) E hxwxd
where Pemmp 2d >> Rd consists multiple deformable convolutional layers (Zhu et al. 2019 followed by series of Transformer Encoders, that dynamically construct the feature representation for later estimating relative motion between frames while reducing the feature dimensions at the same time.
4

---

Under review as a conference paper at ICLR 2023
Discussion. With such design of mundom-paired frame comparator, MOD is capable of mod- eling the relative motion between any two randomly sampled frames out of the video clip (T' frames), accounting for total of T frame pairs for forward, backward, single, multi-step motions. Note that, we do not distinguish the case of i * j and i = j. Specifically, the former encourages the model to discover the relative motion between two different frames, while the latter refers case that neither objects nor camera is moving, and no motion cues are available, thus enforcing the model to discover objects via temporal coherence, i.e., objects that moves in any frame along the video should also be discovered in static frames.
extreme
3.3 DUAL-LAYERED FLOW DECODER
To decode the features into the form of optical flow, we adopt several slot attention layers (Locatello) al. (020)) with two learnable queries, i.e., termed as slot vectors, iteratively attending the output visual features from the comparator, and decoded into the optical flow between any two frames with dual-layered representation.
detail, slot attention module acts similarly to a Transformer Decoder, with the only exception being that the normalisation is computed along the slot side, thus each slot competes take over the pixels. each iteration, given two slot vectors as S E R2xd and visual feature maps o Rhwxd, we use three linear projections to compute the query, key and va lulue, i.e. Q 2x K, V E Rewxxd Thereafter, we can obtain the weights matrix W E 22x and normalise along the slot dimension, i.e.,
W exp(Wx m/ Σ exp (Wi,), where W
OOK.
vd
Then, we gain the next iteration's 's slot vectors by aggregating the values V and passing them into Gated Recurrent Unit (GRU) (Cho et al., 2014), i.e.
W ,s
S = GRUninptts = AV, states = 5), where A., = Σ W.
E R 2xxx))
(5)
We iterate the whole routing process for N times. In this way. the entities of similar RGB and flow patterns are grouped together and distinct pixels are separated by two slots. Eventually, we broadcast the final outputted slots into G {G' Rhxwxdj2 J==1 added with learnable spatial positional embeddings to construct the optical flow. flow decoder & consisting Transformer encoders and up-sampling layers takes the slot grids input, and outputs dual layers of optical flows (7 Hx x and their opacity weights {ar E HxHxxx =
(7 o2,0 ) (G),,
where a* E [0,1] is normalized across two slots via softmax function. Noted that thanks to the softmax function, the alpha value for foreground could be close to zero when the foreground object is missing part of the video. For given frame pair (i,j), their relative flow I,., can be computed via:
T, = E 8 P:20
where @ denotes the element-wise multiplication. At inference time, we adopt the binarized opacity weights a as the object segmentation masks.
3.4 TRAINING
In this section, we describe the training procedure for the proposed model, on the raw videos without using manual annotations for the object segmentations. In general, the training loss is composed of three components, namely, flow reconstruction, temporal consistency, and entropy minimisation.
Note that following the common practise, also output 3-channel RGB images, that refers transfor- mation from the traditional 2-channel optical flow hased on the color wheel proposed in Sun al. (201)..
5

---

Under review as a conference paper at ICLR 2023
Flow Reconstruction. As the main objective for optimisation, where we adopt an off-the-shelf optical flow estimator, for example, RAFT (Teed & Deng 2020), to estimate the flow between any two frames in the video. We minimise the discrepancy between the dual-layer flow reconstruction and the output from the existing flow estimator:
the flow reconstruction,
Σ (1--(u)- NER
(8)
recon
[22
where st - (1,... (h.r
W represents the spatial lattice, and a 1- denotes the L2 norm. In practice, the reconstruction constraint for zero flow when i j dropped due to the distribution gap between dynamic flow and static flow, resulting in the divergence of the training. Note that, flow estimator is only used for model training, inference time, our proposed architecture directly processes the RGB video clips.
Temporal Consistency. In order to build up temporal consistency within the input video, the pair of motion embeddings fi+tj fi that starts from the same reference frame i, are passed through the flow decoder reconstruct the optical flow between two corresponding frames. Note that, as 1 S j. k < T are randomly sampled at every training iteration, the output flow will refer to the motion at different pace. However, the predicted alpha weights for flow composition denote the soft segmentation for the same objects in the i-th frame, thus remaining consistent. In specific, the two inferred masks (j {a are enforced to pull closer by minimising mean- squan- error Ceons, i.c.,
T
2
Lcons
Σ
Coons
where L
Σ MEl
a aa- (u) - at (u))2
2/122
Entropy Minimisation. Lastly, impose pixel-wise entropy regularisation inferred masks, that zero the alpha channels are one-hot, and maximum when they are of equal probability. Intuitively, this helps encourage the masks to be binary, which aligns with goal in obtaining segmentation masks:
1
Cento
Σ
a (wx
(10)
2/521
Total Loss. Accordingly, we rewrite aforementioned reconstruction loss Liecon and entropy regu- latization Learr in summed version:
T
Crocon
Ci+ji + Litkk
(11)
2T
2m1
T
1
Lentro
Ci+jj + Link
(12)
2T
i=1
The total loss for training our model can thus be computed as:
Lut XrCrecon + Ag + Xc
(13)
where we set X, 100, A = de 0.01 at the beginning of the training. We notice the model to be fairly robust to these hyper-paramettes.
EXPERIMENTAL SETUP
In the experiments setup, we first introduce the benchmarks and then elaborate on implementation details.
4.1 DATASETS
We benchmark on three popular datasets designed for video object segmentation. DAVIS2016 (Per-
azzi al. 2016) consists of 50 high quality videos, 3455 frames in total. Every frame is annotated

---

Under review as a conference paper at ICLR 2023
with a aixel-aceurate segmentation mask. SegTrackv2 (Li et al. 2013) contains 14 sequences and 947 fully-annotated frames. Each sequence involves 1-6 moving objects and presents challenges including motion blur, appearance change, complex deformation, occlusion, slow motion, and in- teracting objects. FBMS-59 (Ochs et al. 2013) has 59 sequences with greatly varied resolution and annotates every 20th frame. Many sequences contain multiple moving objects. Following previous evaluation metric (Yang et 2019, Xie et al. 2022), we merge objects of SegTrackv2 and FBMS- 59 into one single object for video object segmentation. We evaluate the pixel-wise segmentation through Jaccard index J. also called Intersection over Union (IoU). Following prior arts (Yang et al.) 2019 a)) we compute the mean per frame over the test set and merge multi-object annotation into single unified segmentation.
4.2 IMPLEMENTATION DETAILS
For data input, we sample T = consecutive frames as the input clip. Each frame resized to 192 384 and the estimated optical flow is computed by RAFT (Teed & Deng 2020). which is pre-trained on the synthetic dataset (Mayer et al. (2016).
To compute spatial-temporal visual representation, we adopt the first three stages of a Swin as the frame encoder Pene. which is then followed by a standard Transformer Encoders with heads (Vaswani al. 2017) as temporal fusion module Premp. For the frame comparator Dommp. use two deformable convolutional layers in the company with three standard Transformer Encoders with 8 heads to process the pixel transformation. Then, choose N = iteration in total for the iterative routing in slot attention. Lastly, we utilize three stages of V2 blocks with the linear patch expanding layers as the flow decoder &atc (Cao et al. 2021).
As for training, we adopt AdamW optimizer (Loshchilov & Hutter 2018) with learning rate x 10-5, 5, The model is trained from scratch without any pretrained weights, for total of 300k itera- tions. At inference time, we adopt the overlapping temporal sliding window to ensemble the segmen- tation masks. We average the resultant masks obtained by the whole temporal segments. We propose two protocols to evaluate our results. Besides measuring the masks without any post-processing, we also apply test-time adaptation with the help of the self-supe DINO- preerrrrai ViT (Caron al., (021). Without any fine-tuning, the pretrained VIT can propagate the masks as noisy anno- tations to whole frames in the same manner as CRW (Jabri et al.) 2020). We refine the masks further with CRF (Lafferty et al. 2001). For more detailed technical information, please refer to the supplementary materials.
RESULTS
In this section, compare primarily with several por-performing approaches trained without human annotations, for example, OCLR (Xic et al. [2022). MG (Yang et al. 2021a), CIS (Yang et al.) 2019). etc. However, as the architecture, modality, input resolution, and post-processing protocols are all different, we try our best to conduct the comparison as fairly as possible.
5.1 ABLATION STUDY
We conduct all ablation studies on DAVIS2016 and vary one variable each time, as shown in Table [
Temporal Fusion & Ptemp and Frame Comparator & comp. As shown by Ours-A and Our-C, the performance degrades significantly without temporal fusion, demonstrating the importance of build- ing up global temporal dependency. Also, indicated by Ours-B, we find the model fails to converge when removing the component of frame comparator Deomp- It meets expectations because frame & &ommp is the sole module in charge of relative motion estimation. Without it, the model cannot reconstruct optical flow thus leading to divergence.
comparator
Number of Frames T. While comparing Ours-A, Ours-D, and Ours-E, there is clear trend that increasing the frame number boosts the segmentation quality, which coincides with our intuition that incorporating wider temporal receptive field can enhance the sense of temporal coherence and object permanence. Due to the limited computational memory, we only set T as the maximum frame number in the paper. A promising performance is expected when inputting more frames.

---

Under review as a conference paper at ICLR 2023
Model
temp
crmpp
Leons
Lentro
DAVIS(J 1)
Ours-A
73.9
Ours-B
x
7
V
V
fail
Ours-C
X
V
68.3
Ours-D
66.4
Ours-E
v
V
68.2
Ours-F
/
7
X 60.4
Ours-G
V
65.6
Ours-H
X
69.5
Table 1: Ablation studies on temporal fusion (9 (Premp), frame comparator ( (Pcmm)), the number of input frames (T), temporal consistency (Leoms) and entropy loss (Lento).
Training
Inference
I (Mcan) 1
Model
Sup.
RGB
Flow
p.p.
DAVIS2016
SegTrackv2
FBMS
SAGE (Wang et al.
2017
None
x
42.6
57.6
61.2
NLC (Faktor & Trani
2014
None
4
55.1
67.2
51.5
CIS
Yang
2019
None
/
V
71.5
62.5
63.5
AMD [Liu al.. 2021
None
X
X
57.8
57.0
47.5
SIMO Lamdouar al 2021
Syn.
V
67.8
62.0
MG Yang ct al.
202
a
None
X
V
X
68.3
58.6
53.1
OCLR [Xie et al. 2022
Syn.
X
X
72.1
67.6
65.4
MOD (w/o post-processing
None
X
73.9
62.2
61.3
MOD (test-time adaptation)
None
x
79.2
69.4
66.9
FSEG (Dutt Jain et 2017
70.7
61.4
COSNet (Lu ct al. 2019
GT
x
80.5
49.7
75.6
MATNet (Zhou et al 2020
GT
82.4
50.4
76.1
DdCon33d Schmidt et
2022
85.5
Table 2: Quantitative comparison on unsupervised video object segmentation. We compare our method three standard datasets, DAVIS2016, SegTrackv2, and FBMS-59. Sup. refers to the supervision, including None, Synthetic (Syn.), and Ground Truth (GT). p.p. is short for post- processing (e.g., CRF (Lafferty al. 2001)).
Lastly, comparing Ours-G and Ours-A, observe that the performance increases considerably with temporal consistency. It man- ifests the validity of our design motivation, temporal consistency conduces to persistently tracking the object. The entropy regularisation is also indispensable shown by Ours-H and Ours-A.
Temporal Consistency Lcons and Entropy Regularisation Centro-
5.2 COMPARISON WITH STATE- SE-TE-OF-THE-ARE
We show the comparison with state-of-the-an art Table 2
On DAVIS2016, MOD achieves 73.9%
mIOU without any post-processing, exceeding MG (Yng et al., 2021a) by large margin (+5.8%).
Compared to the latest method OCLR (Xie et al. 2022) which fabricates a synthesized dataset to train its model, our method still surpasses it only using the information of the DAVIS dataset itself. On another two benchmarks Seg Trackv2 and FBMS-59, MOD also beats MG (Yang et al. la), which only leverages single-step flows decompose foreground and background, by +3.6% and +8.2%, respectively. The superior experimental results demonstrate that our methodology of multi- frame reasoning benefits moving object discovery. Furthermore, equipped with DINO-pretrained VIT, further performance gain is observed on all three benchmarks, which is even more competitive with current supervised approaches.
5.3 QUALITATIVE RESULTS
In Figure 3)
we present several qualitative illustrations of the model. can seen that our results are robust the noticeable background flow signal (drift-chicane sequence in the second column) and
8

---

Under review as a conference paper at ICLR 2023
estimate accurate boundaries when single-step foreground flow cannot represent exact object shape (breakdance and dance-twirl sequences in middle) compared to MG (Yang et al. 2021a). demonstrates inferring the masks by associating bunch of RGB features well resolves the limitation of the usage of flow. Moreover, in virtue of temporally consistent cues, our model handles occlusion well shown in the libby sequence the rightmost column, which could be hard for the flow-only method to maintain the object shape constantly.
RGB
Flow
MG
Ours
Ours
GT
Figure 3: Qualitative results of object video segmentation on DAVIS2016. MG refers Yang et al. 2021a). Red boxes outline the corresponding difference.
5.4 LIMITATIONS
Though we demonstrate that associating multiple frames stimulates the comprehensive sense of objectness, which is proved by the superior experimental performance across the prior arts, there still exist limitations and room for improvement. First, our method uses number of consecutive frames, which challenges computational memory. How to utilize pretrained features for reducing training expenses would be meaningful. Second, how to segment multiple objects remains unresolved. We display a preliminary result in supplementary material. will be promising to exploit the semantic information from RGB to discriminate different foreground objects. We leave them the feature work. Despite these limitations, the approach has convincingly manifested the value of considering textural information and processing RGB frames whole.
CONCLUSION
In this paper, propose self-supervised model for video object discovery. The model takes a set of consecutive RGB frames as input and generates the segmentation mask for the moving objects in the video. At training time, the model tasked to reconstruct the optical flow between any pair of frames, through a layered representation with the opacity channels being treated as the segmentation. To encourage the model capture the objects even when they may be static certain time point, a temporal consistency loss is enforced on the inferred masks on the randomly- paired frames. As a consequence, we demonstrate superior performance over previous state-of-the art methods on three public video segmentation datasets (DAVIS2016, SegTrackv2, and FBMS-59), while being computationally efficient by avoiding the overhead of computing optical flow.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Linchao Bao. Baoyuan Wu, and Wei Liu. Can in mrf Video object segmentation via inference in cnn-based higher-order spatio-temporal mrf. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5977-5986, 2018
Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang. Adrien Gaidon, and Martial Hebert. Discovering objects that can move. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11789-11798, June 2022.
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li F Fei-Fei, Jiajun Wu, Josh Tenenbaum, al. Learning physical graph representations from visual scenes. Advances in Neural Information Processing Systems, 33:6027-6033 2020.
Beril Besbinar and Pascal Frossard. Self-supervision by prediction for object discovery in videos. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 1509-1513. IEEE, 2021.
Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In European conference on computer vision, pp. 282-295, Springer, 2010.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint ww0iiv1901.... 2019.
Sergi Caelles,
Maninis, Jordi Pont-Tuset Laura Taixe, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In Proceedings of the IEFE conference on computer vision and paitern recognition, pp. 221-230, 2017.
Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. arXiv preprint arXiv:2105.05537. 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehree Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP. 2014.
Subhabrata Choudhury, Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Guess what moves: Unsupervised video and image segmentation by anticipating motion. arXiv preprint arXiv:2205. wwxvv22005.07844, 2022.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 3412-3420, 2019.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3684- 3692, 2020.
Suyog Dutt Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Learning to combine motion and appearance for fully automatic segmentation of generic objects in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3664-3673 2017.
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized in- ference for learning symmetric and disentangled multi-object representations. In International Conference on Machine Learning, pp. 2970-2981. PMLR, 2021.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint 13055 2019.
10

---

Under review as a conference paper at ICLR 2023
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems, 34:8085-8094, 2021.
Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In BMVC, volume 2, pp. 8, 2014.
Deng-Ping Fan, Wenguan Wang, Ming-Ming Cheng, and Jianbing Shen. Shifting more attention to video salient object detection. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 8554-8564, 2019.
Katerina Fragkiadaki. Geng Zhang, and Jianbo Shi. Video segmentation by tracing discontinuities in trajectory embedding. In 2012 IEEE Conference Computer Vision and Puttern Recognition, pp. 1846-1853. IEEE, 2012.
James Jerome Gibson and Leonard Carmichael. The senses considered as perceptual systems, vol- ume 2. Houghton Mifflin Boston, 1966.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. International Conference on Machine Learning, pp. 2424-2433, PMLR, 2019.
Yuan-Ting Hu, Jia-Bin Huang, and Alexander Schwing. Maskrnn: Instance level video object seg- mentation. Advances in neural information processing systems, 30, 2017.
Yuan-Ting Hu, Jia-Bin Huang, and Alexander G Schwing. Videomatch: Matching based video object segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 54-70, 2018.
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. Advances in neural information processing systems,
2020.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. Advances in Neural Informa- fion Processing Systems, 13:12222-12582 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint arXiv: 2019.
Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg- generative appearance model for end-to-end video object segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8953-8962, 2019.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loie Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerchner, and Chris Burgess. Simone: View-invariant, temporally-aastraated object representations via unsupervised video decomposition. Advances Neural Information Process-
ing Systems, 34:201 46 20159, 2021.
Margret Keuper, Bjoem Andres, and Thomas Brox. Motion trajectory segmentation via minimum cost multicuts. In Proceedings of the IEEE international conference on computer vision, pp-
3271-3279, 2015.
Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In International Conference on Learning Representations, 2022. URL
https: htttps:/cppeneviiee net/ 1
aD7uesX1CF
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data. 2001.
11

---

Under review as a conference paper at ICLR 2023
Z. Lai and W. Xie. Self-supervised learning for video correspondence flow. In BMVC, 2019.
Zihang Lai, Erika Lu, and Weidi Xie. Mast: A mmemory-augmented self-supervised tracker. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 6479- 6488, 2020.
Hala Lamdouar, Weidi Xie, and Andrew Zisserman. Segmenting invisible moving objects. British Machine Vision Association, 2021.
José Lezama, Karteck Alahari, Josef Sivic, and Ivan Laptev. Track to the future: teatio-temporl video segmentation with long-range motion cues. In CVPR 2011, pp. 3369-3377 IEEE, 2011.
Fuxin Li, Taeyoung Kim, Ahmad Humayun, David Tsai, and James M Rehg. Video segmentation by tracking many figure-ground segments. In Proceedings of the IEEE international conference on computer vision, pp. 2192-2199, 2013.
Xiaoxiao Li and Chen Change Loy. Video object segmentation with joint re-identification and attention-aware mask propagation. In Proceedings the European conference computer vision (ECCV), pp. 90-105, 2018.
Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming- Hsuan Yang. Joint-task self-supervised learning for temporal correspondence. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems. volume 32. Curran Asso- ciates. Inc., 2019. URL [attps::/ mew.:eeeeeddnn.......e..ss...e.e pctppaperr/211//iieee.
pdf
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:2001.022 024007, 2020.
Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. Learning by analogy: Reliable supervision from transfor- mations for unsupervised optical flow estimation. In IEEE Conference on Computer Vision and Pattern Recognition( CVPR), 2020.
Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin. The emergence of objectness: Learning zero-shot segmentation from videos. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems. volume 34, pp. 13137-13152. Curran Associates, Inc.. 2021. URL wtt///tttp////rrreeeoooooo .ew.eeeee.ddnggn.nn..s...... neurips. cc/ Ec/paper/2021/
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33:11525-11538. 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3623- 3632, 2019.
Xiankai Lu, Wenguan Wang, Jianbing Shen, Yu- Yu-Wing Tai, David Crandall, and Steven H. Hoi. Learning video object segmentation from unlabeled videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
K-K Maninis, Sergi Caelles, Yuhua Chen, Jordi
Laura Leal-Taixé, Daniel Cremers, and Lue Van Gool. Video object segmentation without temporal information. IEEE transactions on pattern analysis and machine intelligence, 1555-1555 2018.
12

---

Under review as a conference paper at ICLR 2023
N. Mayer, E. Ilg. P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estima- tion. In IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2016. URL http:://mbt informatik...................... .nnntikk.iiii wre-freebuugge MIFDB16 arXiv:151..02134.
weweibbrgddeeeee .ww.gddee/ee Pale/Palliii http:/abliiaaaiiio cat onntiion//0/0
Cheol-Hui Min, Jinseok Bae, Junho Lee, and Young Min Kim. Gatsbi: Generative agent-centric spatio-temporal object interaction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3074-3083, 2021.
Peter Ochs and Thomas Brox. Higher order motion models and spectral clustering. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 614-621. IEEE, 2012.
Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term video analysis. IEEE transactions on pattern analysis and machine intelligence. 3666):1187-1200, 2013.
Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9226-9235, 2019.
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine- Hornung. A benchmark dataset and evaluation methodology for video object segmen- tation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 724-732, 2016.
Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine- Hornung. Learning video object segmentation from static images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2663-2672, 2017.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Christian Schmidt, Ali Athar, Sabarinath Mahadevan, and Bastian Leibe. D2conv3d: Dynamic dilated convolutions for object segmentation in videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1200-1209, 2022.
Deqing Sun, Xiaodong Yang, Ming-Yi Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8934-8943, 2018.
Zachary Teed and Jia Deng- Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 402-419. Springer, 2020.
Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow mind: Statistics, structure, and abstraction. science, (2022) 2011.
Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid. Learning motion patterns in videos. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3386-3394, 2017a.
Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid. Learning video object segmentation with visual memory. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4481-4490, 2017b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang-Chiet Chich Chen. Feelvos: Fast end-to-end embedding learning for video object segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9481-9490, 2019.
13

---

Under review as a conference paper at ICLR 2023
Carl Vondrick, Abhinay Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Track- ing emerges by colorizing videos. In Proceedings of the European conference on computer vision (ECCV), pp. 391-408, 2018.
Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li. Unsupervised deep tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1018-131,,00 2019.
Wenguan Wang, Jianbing Shen, Ruigang Yang, and Fatih Porikli. Saliency-aware video object segmentation. IEEE transactions on pattern analysis and machine intelligence, 40(1):20-33, 2017.
Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. TokenCut: Segmenting Objects in Images and Videos with Self- supervised Transformer and Normalized Cut. working paper or preprint, 2022. URL https:
7/hal. www.aaiisessueeeeeeeeeeee ouvertes. fr/haa-031654222
Max Wertheimer. Untersuchungen zur lehre von der gestalt. ii. Psychologische forschung, 4(1): 301-350, 1923.
Christopher Xie, Yu Xiang, Zaid Harchaoui, and Dieter Fox. Object discovery in videos fore- ground motion clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9994 10003, 2019.
Junyu Xie, Weidi Xie, and Andrew Zisserman. Segmenting moving objects via object-centric layered representation. In Advances in Neural Information Processing Systems, 2022.
Jiarui Xu and Xiaolong Wang. Rethinking self- sulf-supe correspondence learning: A video frame- level similarity perspective. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 10075-10085, 2021.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 7177-7188, October 2021a.
Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2019.
Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
June 2021b.
Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsupervised video decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2657-2666, 2022a.
Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsupervised video decomposition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022b.
Polina Zablotskaia. Edoardo A Dominici, Leonid Sigal, and Andreas M Lehrmann. Unsupervised video decomposition using spatio-temporal iterative inference. arXiv preprint 1XX2:20006.4727,, 2020.
Tianfei Zhou, Shunzhou Wang, Yi Zhou, Yazhou Yao, Jianwu Li, and Ling Shao. Motion-attentive transition for zero-shot video object segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13066-13073, 2020.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 9308-93011 93189, 2019.
14

---

Under review as a conference paper at ICLR 2023
A MORE IMPLEMENTATION DETAILS
In this section, we list the architecture details and training settings. Codes and models will be released publicly.
A.1
VISUAL ENCODER
The visual encoder contains the first three stages of Swin-Tiny V2. We tabulate the workflow in Table 3
stage input
operation
output sizes
x 192 x 384
PatchEmbed
x 4, stride 4, 96
96 x 48 x 96
SwinBlock1
96 x 48 96
12
DownSamplel
192
192 24 x 48
SwinBlock2
x
192 X 24 x
12
DownSample2
384
384 12 x 24
h 12
SwinBlock3
x 6
384 X 12 x 24
ws = 12
Table Architecture of visual encoder. h stands for the number of attention heads while ws refers to window size.
A.2 FRAME COMPARATOR
The frame comparator possesses two deformable convolutional layers, with ReLU operation in be- tween and three Transformer Encoder blocks. We tabulate the workflow in Table
stage
operation
output sizes
input
768 x 12 x 24
DeformConvl
3 x3, 768
768 X 12 24
ReLU
768 X 12 x 24
DeformConv2
3 x3, 384
384 x 12 24
h 8
Transformer
384 X 12 24
384
Table 4: Architecture frame comparator. h represents the number attention heads
A.3
FLOW DECODER
The frame comparator consists of three stages SwinV2 block added with linear expanding layers. We tabulate the workflow in Table 5
A.4 TRAINING DETAILS
For all datasets, we train with a batch size of 2. To train more efficiently, we sample three flow pairs + j) for a given frame one static replication i j. another two are motion pair i of j. We apply temporal consistency first the masks from two motion pairs, then pull the static mask to the average of two dynamic masks closer. We linearly warm up the learning rate for the first Ik iterations. Besides, for every 100k iterations,
decay the learning rate by half and increase the scale temporal consistency X, and entropy regularisation X, by the factor of In the default setting, train for about 3 days on standard Tesla V100 GPUs with 32GB memory each.
15

---

Under review as a conference paper at ICLR 2023
stage input
operation
output sizes
384 12 24
h 12
SwinBlock1
x 2
384 12 x 24
w0 = 12
PatchExpandl
768
192 24 x
SwinBlock2
192 24 x 48
ws = 12
PatchExpand2
384
96 x 48 x 96
SwinBlock3
96 x 48 x 96
12
PatchExpand3
1536
96 X 192 x 384
outConv
5 x 5, stride 1,4
x 192 x 384
Table 5: Architecture of flow decoder. h stands for the number of attention heads while ws refers to window size.
A.S TEST-TIME ADAPTATION
adopt test-time adaptation based RGB sequence to enhance appearance consistency. In detail, we follow existing works self-supervised tracking (Lai al. 2020; Jabri al.) 2020) propagate object masks across time span. The whole process consists of three steps. First, we extract RGB features of each frame with DINO DINO-pretrained ViT encoder. Then, we select key frames for object mask propagation. Finally, we calculate the affinity matrix between frames and perform mask propagation.
Inspired by OCLR (ie al. 2022
DINO Feature Extraction. Given a video sequence v (s1,,.., TT), RRTxxxXx3 we use DINO pretrained ViT-Small encoder with patch size 8 x 8 to extract features:
-....., fr] {((#)) ((TT)), Se Rx
(14)
where h H//8 and W//8. The extracted features will be used in the mask propagation step.
Key Frame Selection. Given video v, our model predicts object mask of each frame as m = fai,..., Ory
TxxxxxI Since the video frames continuous along the temporal di- mension, it is practical to propagate object masks between neighboring frames. The propagation operation the same Jabri et al. (2020), the only difference is that have no ground-truth mask for reference. Therefore, we need to design a mechanism to select object masks of high confidence. To do this, we measure the temporal coherence of predicted object masks for key frame selection. Specifically, for each timestamp t E {3...., T -2,, we can calculate four propagated masks as:
à; = (Mass--rop(aa--0e Mask ppoppp::ppppipp
Mask-prop(aas)
Mask (onkkk-rop((((2)))) (24+))] (15)
where 'Mask-prop' denotes the propagation operation. Then we calculate the average IoU between the original mask and propagated masks the confidence score:
loU(á,[0), ox) loU(á, + (oU(á;;())o)) (oU((x(())or,)
St
(16)
The calculated s, measures the coherency between a and its neighbors. Our empirical studies show that it serves as reliable signal for key frame selection.
Object Mask Propagation. We select timestamps with Top-k% confidence score as the key ref- erence frames (A: = 15 on DAVIS2016, k 25 on SegTrackv22 A = 10 on FBMS-59). Then we iteratively propagate the object masks with a neighbor temporal window size n. 7. Compared to conventional semi-supervised object segmentation which only relies on the first frame as key frame, we have multiple key frames on different temporal positions to correct the accumulated propagation enror. In this way, the propagation enhances object permanence across time and boosts performance.
B RESULTS BREAKDOWN
We include a specific result breakdown this section. We show the per-sequence results on DAVIS2016 in T SegTrackv2 in Table and FBMS-59 in Table
16

---

Under review as a conference paper at ICLR 2023
T (Mean)
Sequence
w/o post proc.
test-time time adap.
dog
80.7
87.4
cows
87.2
88.8
goat
47.5
80.6
camel
85.6
86.1
libby
72.5
77.7
parkour
72.9
87.9
soapbox
84.6
86.5
blackswan
48.9
46.9
bmx-trees
50.1
55.8
kite-surf
55.9
62.6
car-shadow
87.9
86.9
breakdance
82.6
76.0
dance-twirl
82.5
85.4
scooter-black
80.2
80.3
drift-chicane
78,6
82.2
motocross-jun jumpp
68.4
88.9
horsejump-hie
78.0
84.3
drift-straight
69.2
80.0
car-roundabout
87.7
83.9
paragliding-
62.1
62.8
frame avg.
73.9
79.2
Table 6: Sequence-wise results on DAVIS2016.
J (Mean) f
Sequence
w/o post proc.
test-time adap.
drift
41.7
40.7
birdfall
38.2
61.5
girl
76.5
82.3
cheetah
18.4
30.1
worm
52.5
74.3
parachute
90.2
92.0
monkeydog
14.3
31.1
hummingbird
61.2
58.8
soldier
66.3
58.8
bmx
73.7
78.8
frog
80.5
76.3
penguin
63.5
62.7
monkey
46.8
77.4
bird of paradise
85.3
85,4
frame avg.
62.2
69.4
Table 7: Sequence-wis wise results SegTrackv2.
17

---

Under review as a conference paper at ICLR 2023
J (Mean) f
Sequence
w/o post proc.
test-time adap.
camel01
25.8
66.9
cars
66.1
88.3
carsl0
31.6
33,9
cars4
72.9
83,2
cars5
81.2
82.5
cats01
80.6
79.2
cats03
62.0
63,4
cats06
40.1
38.7
dogs01
70.6
61.2
dogs02
62.8
82.2
farm01
86.7
88.9
giraffes01
38.6
52.2
goats01
44.8
45.3
horses02
64.4
77.6
horses04
68.5
73.5
horses05
43.7
49.0
lion01
60.1
71.5
marple12
74.7
80.3
marple2
65.0
71.4
marple4
79.1
91.6
marple6
76.0
85.1
marple7
72.5
55,2
marple9
87.5
97.9
people03
76.5
48.5
people1
72.4
80.8
people2
80.9
83.0
rabbits02
50.2
58.9
rabbits03
39.5
55.7
rabbits04
47.0
53,0
tennis
63.1
71.1
frame avg.
61.3
66.9
Table 8: Sequence- wise results FBMS-59.
18

---

Under review as a conference paper at ICLR 2023
Flow Estimator
DAVIS2016
SegTrackv2
FBMS-59
RAFT ARFlow
73.9
62.2
61.3
59.2
51.1
50.0
Table 9: Ablation studies on flow estimator.
RGB
Flow
Slot 1
Slot 2
Slot
Slot
Slot 5
Figure 4: Per-slot visualization on DAVIS 2016.
C MORE QUANTITATIVE RESULTS
Besides RAFT flow estimator, we consider a fully unsupervised model ARFlow (Liu et al. 2020) shown in Table g The inferior result brought from ARFlow demonstrates that a good optical flow quality lays strong foundation for the success of the object discovery, which aligns with our motiva- tion.
D MORE QUALITATIVE RESULTS
We visualize the segmentation results when initializing 5 slots vectors in Figure 4 The results are promising, where each slot groups similar part under the only supervision of motion. We believe more supervision like semantics (Caron al. 2021 and depth (Ranfil et al. 2021 would lead to better segmentation. We show more qualitative results of Seg TrgTTrakv22 and FBMS- in Figure and and Figure 6 Besides,
19

---

Under review as a conference paper at ICLR 2023
RGB
Flow
MG
Ours
yow
Ours
Figure 5: Qualitative results SegTrackv2. MG refers Yang al. 2021a). Red boxes outline the corresponding difference.
RGB
Flow
C2
MG
Ours
Ours
GT
Figure 6: Qualitative results on FBMS-59, MG refers to Yang et al. 202 Red boxes highlight the corresponding difference.
20


Reviews:
[{"review_id": "s6_NRdZJfs2", "reviewer": "Reviewer_7qqQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel method for self-supervised object discovery from video. The authors cast the object discovery problem as optical flow construction problem, where optical flow provided by an off-the-shelf method(RAFT) is used as training signal. The authors demonstrate the effectiveness of the proposed method on several popular datasets includes DAVIS2016, SegTrackv2 and FBMS-59 by comparing to state-of-the-art methods. Compared to other self-supervised object discovery methods exploiting motion cue, the proposed method directly operates on RGB input domain without optical flow computation in the inference stage, thus has advantage on inference speed. ", "summary_of_the_review": "In this paper the proposed object discovery method is practical and novel. The authors did a good job on modeling the problem, perform experiment on standard datasets and doing ablation study. I would recommend acceptance of this paper.", "strength_and_weaknesses": "Strength:\n1. The proposed method has superior performance on public datasets compared to state-of-the-art methods with similar constraint(Liu et al., 2021). \n2. The proposed method has close or superior performance on public datasets compared to state-of-the-art methods with less constraints, e.g. using optical flow in the inference stage, use synthetic training data. This also proves the effectiveness of the proposed method.\n3. The proposed method effectively exploits temporal fusion so motion cue from multiple frames could be used to discover the object. This makes the method being robust in the cases where object is partially occluded, or is static in partial frames. \n4. The proposed method is faster in the inference stage compared to other methods, thanks to the setting of operating on raw RGB video clips.\n5. The authors did ablation study to show the impact of each component of the proposed model.\n\nWeaknesses/Questions:\n    1. The proposed method is memory intensive. Is this the reason that only up to 7 consecutive frames are used in the experimental setting section?\n    2. If the foreground object only appears in part of the video(we don’t have supervisory signal so we don’t have information on this), would the method still work? What would the alpha value for foreground be in those frames where foreground doesn’t exist?\n    3. Since optical flow is the training signal, how would the OF quality/failure impact the result? It might be good to evaluate on a synthetic dataset and compare the training with both ground truth OF and estimated/flawed OF.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aATjREZhlD", "reviewer": "Reviewer_NSCm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised model for video object discovery. It only takes consecutive RGB frames as input instead of optical flows as in previous works. The model is trained to reconstruct the optical flow between any paired frames that generated through an off-the-shelf optical flow estimator. A temporal consistency loss on motions at different paces is proposed to encourage the model to segment the objects even if they stop moving or are partially occluded at some time point. Experimental results demonstrate its superior performance on several datasets. ", "summary_of_the_review": "This work on self-supervised learning video object discovery using RGB frames only is well-motivated and designed. However, the proposed model seems to add two more modules on top of the Motion Grouping model and the novelty is limited. Although several empirical results have demonstrated the performance on several datasets, more experiments are still needed. ", "strength_and_weaknesses": "Strength: \n- This paper is well-written and easy to follow. \n\n- The motivation for using RGB as input instead of flow is clear and reasonable, i.e., having cues when static or partially occluded and keeping rich texture information. \n\n- The proposed model architecture and temporal consistency loss have been aligned with these motivations. I like the idea of directly using the RGB frames as input to segment the object as the optical flow is implicitly included in a sequence of the RGB frames. With the design of temporal fusion and frame comparator module, the model can learn optical flow implicitly instead of precomputing it explicitly. The design of temporal consistency loss help to deal with the static or partially occluded situation.\n\nWeaknesses:\n- I read your supplementary material about using DINO for test-time adaptation results, but it is unclear to me how do you manage to use the feature from DINO to do mask prop. If you use the idea of mask propagation. Will the mask error be accumulated along the frame? Why it can boost performance a lot?\n\n- Although the proposed method provides several ablation studies, I still suggest the authors conduct the following ablation studies to enhance the quality of the paper: \n\n    * Does the number of learnable queries in the slot attention module impact the performance? Since the slot attention module is initially designed for segmenting different objects in a toy dataset. If there are more queries, are we able to tackle the limitation of segmenting multiple objects separately? \n\n    * Have you tried other optical flow methods such as PWC-Net or ARFlow. It seems that the RAFT optical flow results are very noisy on SegTrackv2 and FBMS-59 datasets using the script provided by motion grouping. Sometimes it is even enabled to identify the object for instance with frame gap =1,2 when the motion changes are too small. Do you think it will hinder your learning?\n\n- A minor question regarding the sliding window during inference. Is that an overlapping sliding window or a non-overlapping sliding window? \n\n- As you mentioned in the limitation, the proposed method is not able to segment multiple objects. Can you put a figure to visualize how it fails (i.e. only the main object is segmented or multiple objects are partially segmented)?\n\n- Some related work is missing: \n\n    * Yanchao Yang et al. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In CVPR, 2021.\n\n    * Vickie Ye, et al. Deformable sprites for unsupervised video decomposition. CVPR, 2022.\n\n    * Yangtao Wang, et al. TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut. arXiv preprint arXiv:2209.00383, 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3ctUVkUzc5", "reviewer": "Reviewer_HWKn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a method for unsupervised object discovery in videos. In contrast to the previous methods based on the optical flows between adjacent frames, they directly adopted RGB frame sequences. Specifically, the proposed model first encodes RGB frames into visual features, then pass the paired features into frame comparator module that computes relative motion. Finally, dual-layered flow decoder is introduced inspired by slot attention works that outputs two slots, i.e. foreground and background. Experiments are conducted on three benchmarks, surpassing the previous methods.", "summary_of_the_review": "As the weaknesses outweigh strengths, I lean towards rejection at this point. ", "strength_and_weaknesses": "** Strengths\n1. Directly relying on RGB frames to overcome the posed limitations from the usage of optical flows\n2. Combining several components from other literature to achieve state-of-the-art performances, e.g. iterative routing for layered representation from MG (Yang et al., 2021a) and test-time adaptation from OCLR (Xie et al., 2022)\n3. The submitted code and implementation details in the supplementary material facilitate reproducibility \n\n** Weaknesses \n1. Mild novelties; The overall framework seems quite similar to the previous idea introduced on MG (Yang et al., 2021a), particularly for dual-layered flow decoder, except for using RGB frames as input and encoding them further through frame comparator. The introduced three loss functions also lack of originalities as they are commonly used in literature for video-based prediction; flow reconstruction as a proxy task for unsupervised learning, temporal smoothness, and contrast of binary masks.\n2. Usage of the optical flow; While the authors claim the drawbacks of optical flow in terms of defected supervisory signal, their training object still rely on the pre-computed optical flow, RAFT, that also might suffer from static scenery and occlusions. \n3. The need of supervision; I understand that the proposed method do not require supervision for video object segmentation, but the generated optical flow is yielded by the supervised method, RAFT on MPI sintel and KITTI. I would say the required cost to annotate optical flow ground truth for each pixel is much more than the one of binary segmentation. The authors need to demonstrate their method also fairly works well while using unsupervised optical flow methods; such as ARflow.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1qLZfmNu_c", "reviewer": "Reviewer_fGvA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe paper tackles the task of unsupervised moving objects detection from videos. The approach takes as input the video sequence, without flow information.  The network consists of a sequence of known components, ie encoder, transformers, and slot attention decoder, and output a three layer map, ie, the segmentation mask and the bidirectional optical flow. It is only partially un/supervised, given the fact that the main loss function compares the output optical flow to a ground truth one, the lastest one resulting from a fully supervised approach. One of the motivations of the work is to address the problem of 'weak motion', ie when the object might be temporally static in a few frames of the input sequence, or when there are occlusions.  Experimental results are reported on three datasets and compared with related work. ", "summary_of_the_review": "The main strength of the paper is certainly the clarity and quality of the work.  It is in the continuation of Choudhury and al (2022) (both for the scenario and the quantitative results). ", "strength_and_weaknesses": "\nThe overall goal of the paper touches to one of the key aspect of modern computer vision, ie unsupervised object detection. Using video as input is indeed a key element, as it enables to separate foreground from background. \nOne of the interesting observations of the paper (thought not new), is that pairs of frames are not sufficient to accurately detect the motion and hence the object: each frame should be aware of the whole sequence so that the frame-wise segmentation accounts for the whole sequence global motion. This is enabled by the transformer \\Phi_temp, which fuses altogether the frames embeddings.\n\nThe work attempts to move towards fully unsupervised training, but does in fact use some level of supervision via the OF loss. Similar scenario had been proposed by Choudhury and al (2022). An other choise could have been, instead of supervising the Flow reconstruction loss,  to minimize the image reconstruction loss, after wrapping one of the frames pair to the other. Other proxies to optimize and to constrain the estimated flow could also have been used. \n\nI am intrigued by the behavior of the segmentation masks. I am not sure what prevent the segmentation masks to be uniform in all frames (all ones and all zeros, for s=1,2 or reciprocally). Uniform masks in all frames would lead to an L_entro and L_cons to be zero, which is a situation which could be favored by the network, unless other constraints are given.\n\nIn the experimental section, I am not sure what are the results reported for OLCR.  Xie and al 2022 report results without and with test-time adaptation. Only results without adaptation are reported. However, results with test-time adaptation, which the authors also use, outperform the authors results for SegTrack and FBMS, and are very close for DAVIS2016. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject