The Emergence of Objectness: Learning Zero-shot Segmentation from Videos
Authors: Runtao Liu
Zhirong Wu
Stella Yu
Stephen Lin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Self-supervised learning
zero-shot learning
video object segmentation
image segmentation
saliency estimation
video representation learning

Abstract: Humans can easily detect and segment moving objects simply by observing how they move, even without knowledge of object semantics. Inspired by this, we develop a zero-shot unsupervised approach for learning object segmentations. The model comprises two visual pathways: an appearance pathway that segments individual RGB images into coherent object regions, and a motion pathway that predicts the flow vector for each region between consecutive video frames. The two pathways jointly reconstruct a new representation called segment flow. This decoupled representation of appearance and motion is trained in a self-supervised manner to reconstruct one frame from another.
When pretrained on an unlabeled video corpus, the model can be useful for a variety of applications, including 1) primary object segmentation from a single image in a zero-shot fashion; 2) moving object segmentation from a video with unsupervised test-time adaptation; 3) image semantic segmentation by supervised fine-tuning on a labeled image dataset. We demonstrate encouraging experimental results on all of these tasks using  pretrained models.

The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos
Runtao Liul.2+
Zhirong Wu
Stella X. Yu
Stephen Lin
Microsoft Research Asiat
John Hopkins
University2
UC Berkeley ICSI
runtao2190gmail. com stellayu@berkeley edu
[wuzhiron, wtw.eiioooommooooooooooooooooooe
Abstract
Humans can easily segment moving objects without knowing what they are. That objectness could emerge from continuous visual observations motivates to model segmentation and movement concurrently from unlabeled videos. Our premise is that a video contains different views of the same scene related by moving components, and the right region segmentation and region flow allow view synthesis which can be checked on the data itself without any external supervision.
Our model first deconstructs video frames in two separate pathways: appearance pathway that outputs fcature-based region segmentation for a single image, and motion pathway that outputs motion features for pair of images. It then binds them a conjoint region flow feature representation and predicts segment flow that provides gross characterization of moving regions for the entire scene. By training the model to minimize view synthesis errors based on segment flow, our appearance and motion pathways learn region segmentation and flow estimation automatically without building them up from low-level edges or optical flow respectively.
Our model demonstrates the surprising emergence of objectness in the appearance pathway, surpassing prior works on 1) zero-shot object segmentation from single image, 2) moving object segmentation from a video with unsupervised test-time adaptation, and 3) semantic image segmentation with supervised fine-tuning. Our work is the first truly end-to-end learned zero-shot object segmentation model from unlabeled videos. not only develops generic objectness for segmentation and tracking, but also outperforms image-based contrastive representation learning without augmentation engineering.
Introduction
Contrastive learning [1-3] has recently become a powerful method for obtaining high-level visual representations from images [4]. While these representations are shown to be more generalizing. there remain two practical limitations: 1) Hand-crafted augmentations such image cropping and color jittering [5] are critical for achieving invariant recognition, and yet they fall short of capturing more complex object deformations and 3D viewpoint changes. 2) Additional labeled data are required at the downstream for representation fine-tuning, preventing standalone applications.
Our goal here is to overcome these limitations of contrastive representation learning by developing object segmentation models automatically from unlabeled videos without any supervision. Unlike single static images, videos contain sequences of dynamic images that could reveal not only moving objects from their backgrounds, but also their internal part organizations with articulated movements. Once fguureground segregation occurs automatically in raw videos, object semantics can be readily discovered from those foreground segmentations.
"Equal contribution. Work done when Runtao was StarBridge intern at MSRA.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

segment flows
work
images
segments
dense flows
past work
Figure I:
Our zero-shot object segmentation is learned from an unsupervised factorization of images into segments and their motions, whereas past work segments objects based on dense pixel-wise optical flows, which are brittle in the presence of noise, articulated movement, and abrupt motion.
Several observations motivate us to explore such zero-shot learning of object segmentation. 1) Humans can easily segment moving objects without knowing what they are. In biological vision, newborn chicks raised in a controlled visual world rapidly develop more accurate object representations when presented with temporally slow and smooth objects, generalizing from very limited viewpoints [6, 7J. 3) Invariant recognition can be developed by seeking slowly varying features from temporally varying signals [8]. disentangling object identity and object location unsupervisedly.
We model segmentation and movement concurrently from unlabeled videos. Our premise is that a video contains different views of the same scene related by moving components, and the right region segmentation and region flow would allow mutual view synthesis between frames that can be checked on the data itself. That is, if we know how regions of frame are moved from regions of frame i, we can synthesize frame by copying regions from frame i and paste them according to how they move. Comparing the synthesized frame j with the actual frame j provides feedback how improve both region segmentation and region flow estimation without needing any supervision.
View synthesis has been frequently adopted as self-supervised criterion for learning dense optical flows [9], monocular depth [10], and multi-plane image representation [11] ctc from images. Unlike prior works that focus low-level visual correspondences,
work tackles object segmentation for mid- to high-level visual recognition directly. Specifically, as illustrated in Figure 1, instead of deriving dense optical flows between successive frames and supplying additional cues for image- based object segmentation in a bottom-up manner, we seek a top-down factorized representation that provides a gross characterization of moving regions for the entire scene.
Our model first deconstructs video frames by processing them two separate pathways: an appear- ance pathway that models what is moving and outputs feature-based region segmentation given single image, and a motion pathway that models how it moves and outputs motion features given a pair of images. It then binds them in conjoint region flow feature representation, based on which we predict segment flow as the common fate [12] or piecewise constant movement of all the pixels in the same region. By training the model to minimize view synthesis errors based on segment flow, our appearance and motion pathways learn region segmentation and flow estimation automatically without building them up from low-level edges or optical flows respectively.
After training our segmentation and flow features for view synthesis on unlabeled videos, our model demonstrates the surprising emergence of objeciness in particular feature channel of the appearance pathway. That is, our model can be directly applied to novel images and videos for segmenting foreground objects: Our appearance pathway can perform zero-shot object segmentation on single image, whereas our overall model can perform zero-shot moving object segmentation on a single video with test-time adaptation. Our image feature learned from unlabeled videos can be further fine-tuned on small labeled dataset for semantic segmentation. Experimentally, we demonstrate strong performance on all three applications, with considerable gains over baselines.
To summarize, our work makes the following contributions. 1) We develop the first truly end-to- end learned zero-shot object segmentation model from unlabeled videos, assuming no low-level computation such as edges or optical flow. 2) We bypass the traditional low-level dense optical flow and propose to compute novel mid-level segment flow directly. 3) Our model not only develops

---

generic objectness for segmentation and tracking, but also outperforms prevalent amage-based contrastive learning methods without augmentation engineering. Our code is available at https: // http:u/..oooott//1/oaaee http:///ttttt/eeeeeeeeeeee-eee- of of-oojecteess.
2 Related Works
Video object segmentation. Segmentation of moving objects requires finding correspondences across time. A major line work assumes that an object mask given the initial frame, and the goal is to propagate the mask to future frames based on the similarity of learned visual representation. Such a representation can be trained from pixel-level object masks with long-term relations in videos [13, 14], or from self-supervised criteria such as colorization [15] and cycle-consisteney [16].
Fully unsupervised video object segmentation without initial masks has received less attention. NLC [17] and ARP [18] segment moving objects based on temporal clustering. but they rely on edge and saliency annotations, and are thus not completely unsupervised. FTS [19] performs segmentation by obtaining motion boundaries from optical flow. SAGE [20] takes into account edges, motion segmentation, and image saliency for video object segmentation. Contextual information separation [21] segments moving objects by exploiting independence between the foreground motion and the background motion. concurrent work based on motion grouping [22] clusters pixels by similar motion vectors. These works rely off-the-shelf optical flow results, which may be trained with [23, 24] or without [9] supervision. Our work does not assume any known low-level features such edges optical flow, and learns the right feature to extract completely from scratch.
Motion Segmentation. Classical methods for motion segmentation [25-27] find regions of distinc- tive motion based on two-frame optical flow. Supervised learning approaches [28, 29] map optical flow segmentation masks. Ideally, these methods require dense and accurate optical flow. In practice, the low-level differential flow is often present along edges; it is neither dense nor smooth within a region, often inhomogeneous for deformable and articulated objects [30], and sensitive to varying scene depths and camera motion [3] 1-133,, 33). Motion segmentation is shown to be less brittle when examined over a large time interval [30]. Trajectory clustering [34] tracks point trajectories over hundreds of frames, extracts descriptors for the trajectories, and clusters them to obtain segmentation. Such a global approach is computationally demanding.
In contrast, our segmentation is based not motion between two frames, but on image appearance in single image, which provides rich cues such as color, texture, and edges for pixel grouping and segregation. While our segmentation model does not need dense pixel correspondences between frames, learned to in sync with region-wise correspondences for best view synthesis.
Unsupervised learning for segmentation. Human annotation of pixel-wise segmentations is not only time-consuming but also often inaccurate along boundaries. Learning segmentation without labels is thus of great interest in practice. SegSort [35] predicts segmentation by learning to group super-pixels of similar appearance and semantic context from static images. Later work [36] contrasts holistic mask proposals obtained by traditional bottom-up grouping.
A related line of work focuses on learning part segmentation from images and videos of the same ob- ject category, such as humans and faces. SCOPS [37] is a representative co-part segmentation method, learned in supf-supe fashion; its general idea follows unsupervised landmark detection [38], leveraging geometric invariance, representation equivariance, and perceptual reconstruction. [39] explores motion cues in videos discover object part organization and dynamics. Motion-supervised co-part segmentation [40] models part motion hetween adjacent frames using affine parameters. A similar idea is implemented with capsule networks [41]. In contrast, our work is not restricted to videos in single category and learns object segmentation from a collection of generic videos.
Learning objectness from data. Segregating foreground objects from background is central problem in visual recognition. Prior works on images first generate hierarchical segmentations [42- 44] using low-level visual cues such colors and boundaries [45], and then rank these candidate regions according to a certain criterion [46, 47]. These approaches often generate many overlapping redundant individual object instance proposals.
Videos of slowly moving objects [8] are shown to enable development of newborn vision [6, 7]. Linear models [48, 49] can factorize sequence of images into foreground and background layers,
3

---

assuming independent motion among them. Layered representations are also used for optical flow estimation [50. 27, 51], niew-interpolation. and time retargeting [52-55].
Our work adopts an unordered layered representation with multiple segmentation channels, each identifying region of common motion. While our view synthesis objective during training does not differentiate foreground or background in these channels, surprisingly, objectness emerges automatically in a particular channel after we train our model from a collection of raw videos.
Image representation learning from videos. Motion reveals the location, shape, and part bierarchy of moving objects. Motion segmentation has thus been used to supervise learning of image- object representations [56]. Motion propagation [57] predicts dense optical flow from sparse optical flow, conditioned on the color image. Unlike prior works, the single image representation produced by our model is not learned from motion supervision, but concurrently learned with between-framee region flow by-product of our moving object segmentation from unlaheled videos.
3 Segmentation by Appearance-Motion Decomposition
We would like model to segment moving objects without necessarily knowing what and how many they are. Our model is trained on a collection of unlabeled generic videos, and can be directly deployed novel image (video) to produce (moving) object segmentation. No human annotations are required during either training or testing.
frame
reconstructed frame
segment flows
i+j
motion
frame
segmentation
X
X
warp
Figure 2: We learn single-image segmentation network and a dual-frame motion network with an unsupervised image reconstruction loss. We sample two frames, and j. from video. Frame goes through the segmentation network and outputs a set of masks, whereas frames i and j go through the motion network and output a feature map. The feature is pooled per mask and a flow is predicted. All the segments and their flows are combined into a segment flow representation from frame i - which are used to warp frame into j. and compared against frame j to train the two networks.
Illustrated in Figure 2, our so-called appearance-motion decomposition (AMD) model first decon- structs a pair of video frames, X; and Xj. in two separate pathways. The bottom appearance pathway fA takes in a single image X; and outputs a feature-based segmentation, whereas the top motion pathway fM takes in both frames (X;, Xj) and outputs the flow features between them. The two pathways then come together to construct a conjoined segment flow representation F, which is used to synthesize frame X, by warping X,. The overall model can be trained to minimize the reconstruction eтror on frame X, over sampled image pairs the training dataset.
1) Appearance pathway for feature-based segmentation. We adopt fully convolutional neural network for segmenting a single RGB image size hxw into e regions, where a hyper-paramett Formally, video frame X; E I 3xxxx is transformed by fa into segmentation S with soft masks:
S fA(Xi) E xexxx
Σ S(p) = 1,
p=
h X w.
(2)

---

The above normalization equation reflects that values of S are the probabilities of pixel p belonging to e regions. Empirically, we choose c= 5 by default. Our ablation study later shows that a larger e may lead to over-segmentation, whereas smaller e may lack the spatial resolution to locate objects.
Note that our segmentation network is based on the RGB appearance in single frame instead of optical flow between two frames. Since it is designed to operate on static images, it can be transferred to downstream image-based vision tasks. Our segmentation network trained on unlabeled videos can be directly used to segment not only moving objects from novel videos (Section 4.2), but also salient objects from single images zero-shot fashion (Section 4.1). can be further fine-tuned on a labeled image dataset for semantic segmentation (Section 4.3).
2) Motion pathway for correspondences. We adopt PWC-Net [23] for extracting pixel-wis, motion features between a pair of images. PWC-Net is originally designed for predicting dense optical flow, and the feature for each pixel in one frame captures its similarity to spatial neighbors in the other frame. Formally, video frames (X,, Xj) are transformed by fM into motion correspondence feature V of dy dimensions:
V (X, E Rdd xkxw.
Note that we only adopt the network architecture not the trained weights of PWC-Net in [23], and our choice among alternative architectures such as FlowNet [58]. FlowNet2 [59], SpyNet [60], and RAFT [24] is based on conceptual simplicity and light-weight model size.
3) Segment flow representation. We now construct a conjoined segment flow representation from both pathways to enable view synthesis. Specifically, we pool the pixel-wise correspondence feature V in the motion pathway according to the image segmentation S in the appearance pathway, resulting in an average d. dd-dimensional motion feature per segment, i.c., Vn for the m-th mask Sm
hux (p) x 5m > 1
Vm
E GR m=
(4)
Ehxr (p)
We then predict a single common 2D flow vector Fm for the entire m-th segment Sm based on its average motion feature Vn, using two-layer multilayer perceptron (MLP) for the head function g:
Fn g( (m R², m= mm=l....c..
(5)
So far, we deconstruct pair of video frames (X,, xx,x,) into segmentation masks and their associated flow vectors {(Sm, Fm) : m = 1..... m). assuming one common motion for pixels within the same segment. This piece-wise constant motion assumption simplifies flow estimation and provides a gross charactcrization of movement in the scene. While it may not hold for deformable and articulated objects, when trained over a collection of videos, our model with appearance feature- based segmentation is able to aggregate smoothly moving pieces in a wholesome segment.
We then compose these moving components into novel flow representation F for the entire image:
F(p) = Σ Fm x Sp))), p= h x W.
(6)
m==
We call segment flow, as its values indicate the overall displacement at the segment level. This conjoined representation allows motion and segmentation to cross- cross-supervise each other. Given a between-frame flow vector, the segmentation network could be supervised to find pixels of this offset. Given a segmentation mask, the motion network could be supervised to find the flow for this mask.
Our approach to image segmentation with motion inputs is fundamentally different from motion segmentation methods: 1) Our segmentation mask is predicted from static image appearance that does not require dense and accurate flow for supervision; 2) Our flow estimation is at the segment level, which can be inferred from sparse and noisy pixel-level flow estimates.
4) Reconstruction objective for view synthesis. How do we validate our segment flow F, con- joined representation from both appearance and motion pathways? Intuitively, the right segmentation and motion Fm would allow the synthesis of frame X, from frame X; according to their segment flow F, and the reconstruction X, should be close to the actual frame Xy:
X,()) = (2 + F(p)), p == I,...,h h X w
L= D(X, Xz).
(8)
5

---

where D is metric measuring the distance between two images. We adopt the pixel-wise photometric loss SSIM [61] for simplicity, among alternatives such deep-feature matching losses [62, 63] and contrastive losses [64]. The warping loss L is the only self self- supervisevi our model receives.
does necessarily mean that the segmentation and flow correct, but correct segmentation and flow must result in a small reconstruction error. That is, this reconstruction objective is a necessary condition for correct segmentation and flow estimation.
Note that small reconstruction
Compared to traditional optical flow estimation, our model also derives motion flow from brightness constancy between two frames, but the optical flow computation assumes pixel-wise local displace- ments that are independent of each other, whereas our segment flow assumes all the pixels in the same segment have common displacement. In addition, segmentation is determined from the image appearance, instead of fixed-size patches assumed in the Kanade optical flow method [65].
Our model has thus two essential bottlenecks: One is the number of segments, and the other is piecewise constant segment flow. Both are important for directly delivering a mid-level organization without building it up from low-level vision such edges and optical flow.
5) The emergence of objectness. The appearance pathway in model only segments an image into regions, one in each of its e channels. Our reconstruction objective is only concerned with collective view synthesis from these e channels, and does not designate any channels for moving objects or background. That is, any channel could contain the moving foreground object for particular video.
Surprisingly, observe empirically that objects in training videos tend to concentrate in the same channel, with a relatively sharp and uniform mask in the center of the image (Figure 3). We conduct analysis to understand the emergence of objectness in particular channel, the channel activated by the feature that seems to capture generic objects against their backgrounds.
Input
channel
channel
channel 2
channel
channel 4
average plesinise mask entropy per channe)
avedage pleoklaiee recomds.ction ertor channel
a) segmentation uncertainty
b) reconstruction eгror
c) mask center and spread
Figure Top) Our model shows the surprising emergence objectness particular channel. Note that the index of the channel (channel 0 here) could be random in different training runs, but there always a channel concentrated with objects from all the training videos. Bottom) Channel-wise statistics over 17,500 sample training frames of our segmentation network reveal that our objectness channel has a) the least segmentation uncertainty (measured by the entropy of S.)). b) the largest reconstruction training етror (measured by SSIM), and e) mostly central locations (the average of the mean and standard deviation of mask centers marked by the channel number and the small black circle) and relatively focused areas (the half of average mask spread shown the color-shaded disk).
We calculate three types of per-pixel statistics for each segmentation channel from 17,500 sampled frames across training videos: 1) segmentation uncertainty measured by the entropy of mask value
Sm.. 2) reconstruction error between Xj and X, measured by SSIM, and 3) the mean and standard deviation of the mask center and the average mask spread. Figure 3 shows that the objectness channel has the least segmentation uncertainty, the largest reconstruction training error, mostly central locations and relatively focused areas.
Our conjecture is that three factors contribute to the emergence of objectness in our model: 1) Training videos tend track moving objects in the center field of view; 2) Our piece-wise wise constant motion assumption holds better for the background; 3) Motion of object pixels tends to be independent of

---

that of background pixels, whereas motion of background pixels could be interpolated from other pixels scattered in multiple background regions.
6) Limitations. Our current model develops a general sense of objectness from a collection of unlabeled videos. It can only segment foreground objects without differentiating between object instances or between semantic classes. It is not guaranteed to segment out all the objects or full objects. Like most data-driven learning methods, our model performance heavily depends on the properties of training videos, the coverage of object categories, and the object motion as well as camera motion. Nevertheless, we still find amazing that our model is able to generalize the concept of objectness across a variety of datasets in zero-shot fashion, moving a step closer to human vision.
Experiments
Tasks. We train our AMD model on unlabeled videos and test it on three downstream applications. 1) Zero-shot object segmentation. We directly apply segmentation network to static images for salient object detection. 2) Zero-shot moving object segmentation. We apply our AMD model to segment moving objects in novel videos with zero human labels. 3) Fine-tuning for semantic segmentation. We fine-tune our appearance pathway on labeled images for semantic segmentation.
Training data. Our training videos come from Youtube- VOS [66]. large object-centric video dataset. Its training split contains about 4,000 videos covering 94 categories of objects. The total duration of the dataset is 334 minutes. We sample video frames at 24 frames per second, without using any segmentation labels provided in Youtube-VOS.
Implementation details. We train our model from scratch without external pretraining, For the segmentation network, we use ResNet50 [67] as our backbone followed by fully convolutional head containing two convolutional blocks. For the motion network, we use PWC-Net [23]. We resize the shorter edge of the input image 400 pixels, and randomly crop a square image of size 384 x 384 with random horizontal flipping augmentation. No other augmentations are used. We adopt the symmetric reconstruction loss that considers either frame the target frame and sums the two reconstruction errors. We use the Adam optimizer with learning rate of 1 x 10-1 and weight decay of x 10-6 We train AMD on 8x V100 GPUs, with each processing two pairs sampled adjacent frames. The network is optimized for 400K iterations.
4.1 Zero-Shot Saliency Detection
We directly evaluate our Coutube-VOS trained segmentation network DUTS [74], a salient object detection benchmark which contains 5,019 test images with pixel-level ground truth annotations. We follow two widely used metrics: the F3 score and the per-pixel mean squared errors (MAE). F3
(1+8")PXI
defined as the weighted harmonic mean of the precision (P) and recall (R) scores: Fg 8*P+R
with 3 0.3. MAE simply the per-pixel averaged error of the soft prediction scores.
Experimental results. We compare our saliency estimation results against several traditional methods based on low-level cues and various priors: background priors [68], objectness [70, 72], and color contrast [75]. Table shows that our method achieves F3 score 60.2 and an MAE score of 0.13, outperforming baselines by sizable margins. Note that AMD not designed specifically for this task or this dataset, and its strong performance demonstrates the generalization power of our model.
Table Salient object detection perfor- mance on the DUTS dataset. Our model outperforms traditional low-level meth- ods by notable margins.
Table 2: Transfer performance for semantic segmenta- tion on VOC2012. Our method outperforms TimeCycle and compares favorably with contrastive methods.
Model
Data
Aug.
mloU
Model
F93
MAE
Scratch
48.0
RBD[68]
51.0
0.20
TimeCyle[73]
VLOG
light
52.8
HS[69]
52.1
0.23
MoCo-v2[2]
YTB
light
61.5
MC[70]
52.9
0.19
AMD
YTB
light
62.0
DSR[71]
55.8
0.14
MoCo-v2[2]
YTB
heavy
62.8
DRFI[72]
55.2
0.15
AMD
YTB
heavy
62.1
AMD
60.2
0.13
MoCo-v2[2]
IMN
heavy
72.4
7

---

movable objects
stationary objects
Figure 4: Sample salient object detection results. We directly apply our pretrained segmentation network to novel images in DUTS without any finetuning. Surprisingly, we find that the model pretrained on videos to segment moving objects can generalize to detect stationary unmovable objects in static image, e.g. the statue, the plate, the bench and the tree in the last column.
In related unsupervised learning of saliency detection [76-78], the priors of traditional low-level methods are ensembled. Though they do not use saliency annotations, their models are pretrained for ImageNet classification and even semantic segmentation with pixel-level annotations. These methods thus fully unsupervised and omitted from comparisons.
Figure 4 shows sample results on salient object detection. Surprisingly, we find that our model trained on Youtube-VOS to segment moving objects not only detects movable objects in single images, but also stationary unmovable objects such as statues, benches, trees and plates. These results suggest that our model learns generic objectness from unlabeled videos.
To quantify this observation, we manually label objects from the DUTS dataset into movable objects and stationary objects. The Fl score of movable objects and stationary objects are 63.0 and 57.9 respectively, without significant performance gap. We hypothesize that our model could also learn objectness from camera motion, which causes objects and backgrounds at various depths to have different 2D optical flow even though the objects are static.
4.2 Zero-shot Video Object Segmentation
Since our method does not require any labels, we apply our AMD model to object segmentation in novel videos using test-time adaptation: Given novel video, optimize the training objective in Eq. on pairs frames sampled from the test video. The adaptation takes 100 iterations per video.
We evaluate zero-shot video object segmentation on three datasets. DAVIS 2016 [79] contains 20 validation videos with 1,376 annotated frames. SegTrackv2 [80] contains 14 videos with 976 annotated frames. Following prior works, we combine multiple foreground objects in the annotation into a single object for evaluation. FBMS59 [30] contains 59 videos with 720 annotated frames. The dataset is challenging
the object may be static for period of time. We pre-process ground-truth labels as [21]. For evaluation, we report the Jaceard score, which equivalent the intersection over union (IoU) between the prediction and the ground truth segmentation.

---

boundaries, whereas Table 3: Unsupervised video object segmentation performance on DAVIS 2016, SegTrackv2 and FBMS59 datasets, measured in terms of Jaccard score. The table is split into traditional non-leamnng: based and recent self-suuervised learning methods. Results which rely on other kinds of human supervisions (Sup.) are grayed. Dependence for pretrained dense flow method is also listed for each model. MG's results on SegTraa SegTrackv2 and FMBS59 using ARFlow are reproduced by ours and marked with *, We evaluate AMD in two settings: appearance pathway only and both pathways with test time adaptation. AMD performs favorably to CIS on DAVIS 2016, while showing large gains on the other two benchmarks.
Model
c2e
Sup.
Flow
DAVIS 2016
SegTrackv2
FBMS59
SAGE[81]
X
X
LDOF[82]
42.6
57.6
61.2
traditional
NLC[17]
X
edge
SIFTFIow
55.1
67.2
51.5
CUT[34]
X
X
LDOF[82]
55.2
54.3
57.2
FTS[19]
X
X
LDOF[84]
55.8
47.8
47.7
ARP[18]
X
saliency
CPMFlow[85]
76.2
57.2
59.8
CJS[21]
X
PWC[23]
59.2
45.6
36.8
learning
MG[22]
X
X
ARFlow[9)
53.2
37.8*
50.4*
AMD (per-img)
J
X
X
45.7
28.7
42.9
AMD (per-vid)
V
X
X
57.8
57.0
47.5
Experimental results. We consider baseline methods claiming to be unsupervised for the full pipeline: traditional non-learning-basee
approaches [81, 17, 19, 34, 18] and recent self-supervised learning methods [21, 22]. Table 3 summarizes results for all the methods on the three datasets. Note that NLC [17] actually relies on an edge model trained with numan-annotann
ARP [18] depends on segmentation model trained on a shade their entries in gray. For the traditional methods, since the original papers do not report results on most of these benchmarks, we simply provide their performance reported in CIS [21].
saliency dataset. We thus
We evaluate AMD with and without test-time adaptation. No adaptation boils down to per-image saliency estimation using only the appearance pathway, whereas adaptation fine-tunes both appearance and motion pathways. On DAVIS 2016, our method achieves Jaccard score of 57.8%, surpassing all traditional unsupervised models. For CIS [21]. their best performing model uses significant amount of post-processing, including model ensembling, multi multi-crop, temporal and spatial smoothing. We thus refer to their performance obtained from single model without post-processing. Our model is slightly worse than CIS on DAVIS, by 1.4%. However, on SegTrackv2 and FBMS59, our method outperforms CIS by large margins of 11.4% and 10.7% respectively. Motion grouping [22] is motion segmentation approach that relies on an off-the-shelf
work concurrent with ours. pre-computed dense optical flow model. Motion grouping performs worse than our method on DAVIS2016 and SegTrackv2 when a low-performance unsupervised optical flow model ARFlow is used [9]. With state-of-the-arr supervised optical flow model [24] which is trained on ground truth flow, their performance improves significantly. Among all the discussed methods, ours the first end-to-end self-supervised learning approach which does not require pretrained optical flow model.
Figure shows result comparisons with CIS [21]. For most of these examples, our segment flow only coarsely reflects the true pixel-level optical flow. However, our segmentation results are significantly better and less noisy, insensitive to the flow quality. the first and the third examples, our model produces high-quality object segmentations even though the object motion cues are weak.
4.3 Semantic Segmentation
Since our Youtube- VOS trained segmentation network can already segment generic objects, we further examine its modeling power of semantic segmentation Pascal VOC 2012 [86]. which contains 20 object categories with 10,582 training images and 1,449 validation images. We finetune our AMD model on the PASCAL VOC training set and evaluate it the validation set. The finetuning takes 40,000 iterations with batch size 16 and the initial learning rate 0.01. The learning rate undergoes polynomial decay with power parameter of 0.9.
Experimental results. Our baselines are an image-based contrastive model, MoCo-v2 [2], and self-supervised video pretraining model, TimeCycle [73]. TimeCycle is pretrained on the VLOG

---

dense
flow
CIS [21]
segment flow
AMD
Figure 5: Comparisons with motion segmentation method CIS [21]. CIS segmentation is sensitive to noise, articulated motion, and camera motion in its dense flow. By disentangling appearance from motion, our AMD is less prone to these vulnerabilities, resulting in better and more robust results.
dataset, which is larger than Youtube- VOS dataset. For MoCo-v2, we also pretrain the contrastive model on Youtube- VOS, to ablate the role pretraining datasets. Since our method does not utilize heavy augmentations as in contrastive models, we also study the effects of data augmentations. Table 2 shows that our method outperforms the video pretrained TimeCycle significantly by 9.2%. With light augmentation (resizing, cropping), model slightly outperforms MoCo-v2 by 0.5%. However, with heavy data augmentation (color jitter, grayscale, blurring), our method underperforms MoCo-v2 by 0.7%. The reason that our model does directly relate augmentations, and thus cannot build invariance effectively across augmentations. MoCo-v2 performs much stronger when pretrained on ImageNet, possibly because the semantic distribution of ImageNet well aligned with that of VOC2012. Overall, our model outperforms prior self-supervised video model TimeCycle and compares favorably a contrastive model MoCo-v2 under the same training data setting.
4.4 Ablation Study
The number of segmentation channels, is an important syper-parammter of our model. Figure 6 shows our model predictions trained for с 5, 6, 8: training becomes unstable when 4. A larger tends to lead to over-segmentation: The car and the swan are split into multiple regions even when the motion is very similar between separated regions. The model trained with c segments full object, while the model trained with c 8 separates the object into parts. Quantitatively. the video object segmentation performance on DAVIS2016 drops as we increase the number of segments.
#segments
DAVIS (J)
5
57.8
= 6
45.3
41.0
c=5
6
c= 5
C=6 6
Figure 6: Ablation study on e with different numbers segments. Left) Two sample results with segmentation masks and segment flows. Right) Jaccard scores DAVIS2016. As increases, the object region is oversegmented, decreasing the video object segmentation performance.
Summary. We show that objectness emerges from our AMD model trained on unlabeled videos. Our model first deconstructs video frames into appearance and motion, and then binds them into a conjoined segment flow representation for view synthesis. While prior works rely on accurate method leams from scratch
dense optical flow for object segmentation, While our segment flow is a coarse characterization of motion, our object segmentation is in fact more robust. Validated on several segmentation benchmarks, our AMD model is the first end-to-end learning approach for zero-shot object segmentation without using any pretrained modules.
pixel observations.
Acknowledgements. This work was supported, in part, by Berkeley Deep Drive to SY.
10

---

References
[1] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[2] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[4] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Man- nat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.0198% 2021.
[5] Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural net- works. IEEE transactions on pattern analysis and machine intelligence, 38(9):1344-174 2015.
[6] Justin N Wood and Samantha MW Wood. The development of newborn object recognition fast and slow visual worlds. Proceedings of the Royal Society B: Biological Sciences, 283(1829):20101166 2016.
[7] Justin N Wood. smoothness constraint on the development of object recognition. Cogmition, 153:140-145, 2016.
[8] Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
[9] Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang. Jilin Li, and Feiyue Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6489-6498, 2020.
[10] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3828-3833 2019.
[11] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09877 2018.
[12] Max Wertheimer, Untersuchungen zur lehre von der gestalt. ii Psychologische forschuug. 4(1):301-350, 1923.
[13] Seoung Wug Oh, Joon- Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9226-9235, 2019.
[14] Yizhuo Zhang, Zhirong Wu, Houwen Peng. and Stephen Lin. A transductive approach for video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6949-6958, 2020.
[15] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In Proceedings of the European conference on computer vision (ECCV), pages 391-408, 2018.
[16] Allan Jabri, Andrew Owens, and Alexei A Efros. Space-time correspondence as a contrastive random walk. arXiv preprint 24Xv:2006..46633 2020.

---

[17] Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In BMVC, volume 2, page 8, 2014.
[18] Yeong Jun Koh and Chang- Su Kim. Primary object segmentation in videos based on region augmentation and reduction. In 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 7417-7425. IEEE, 2017.
[19] Anestis Papazoglou and Vittorio Ferrari. Fast object segmentation in unconstrained video. In Proceedings of the IEEE international conference on computer vision, pages 1777-1784, 2013.
[20] Wenguan Wang, Jianbing Shen, and Fatih Porikli. Salieney-aware geodesic video object seg- mentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages -395-3402, 2015.
[21] Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 879-888, 2019.
[22] Charig Yang. Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Sulf-super video object segmentation by motion grouping. arXiv preprint arXiv:2104.07766 2021.
[23] Deqing Sun, Xiaodong Yang, Ming-Yu Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934-8943, 2018.
[24] Zachary Teed and Jia Deng. Raft: Recurrent all- all-pairs field transforms for optical flow. In European Conference on Computer Vision, pages 402-419. Springer, 2020.
[25] Jianbo Shi and Jitendra Malik. Motion segmentation and tracking using normalized cuts. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pages 1154-1160.
IEEE, 1998.
[26] M Pawan Kumar, Philip HS Torr, and Andrew Zisserman. Learning layered motion segmenta- tions of video. International Journal of Computer Vision, 76(3):301-319, 2008.
[27] Deqing Sun, Erik B Sudderth, and Michael J Black. Layered segmentation and optical flow estimation over time. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages
IEEE, 2012.
[28] Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid. Learning motion patterns in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3386-3394, 2017.
[29] Pavel Tokmakov, Cordelia Schmid, and Karteek Alahari. Learning to segment moving objects. International Journal of Computer Vision, 127(3):282-301, 2019.
[30] Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term video analysis. IEEE transactions on pattern analysis and machine intelligence, 36(6):1187- 1200, 2013.
[31] Manjunath Narayana, Allen Hanson, and Erik Leamed-Miller. Coherent motion segmentation in moving camera videos using optical flow orientations. International Conference on Computer Vision, 2013.
[32] Erik Learned-Mlllrr Pia Bideau. It's moving! probabilistic model for causal motion segmenta- tion in moving camera videos. In European Conference on Computer Vision, 2016.
[33] Pia Bideau, Aruni RoyChowdhury, Rakesh R Menon, and Erik Learned-Miller The best of both worlds: Combining enns and geometric constraints for hierarchichal motion segmentation.
In CVPR, 2018.
[34] Margret Keuper, Bjoern Andres, and Thomas Brox Motion trajectory segmentation via minimum cost multicuts. In Proceedings of the IEEE international conference on computer vision, pages 3271-3279, 2015.
12

---

[35] Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Hlang-Chiet Chen. Segsort: Segmentation by discriminative sorting of segments. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 7334-7344, 2019.
[36] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Un- supervised semantic segmentation by contrasting object mask proposals. arXiv preprint arXiv:2102.06191, 2021.
[37] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz. Scops: Self-supervised co-part segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 869-878, 2019.
[38] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. arXiv preprint 1rXiv:1806.0 07823, 2018.
[39] Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Unsupervised discovery of parts, structure, and dynamics. arXiv preprint 1rXiv:1903.05136, 2019.
[40] Aliaksandr Siarohin, Subhankar Roy, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Motion-supervised co-part segmentation. arXiv preprint wrXiv:2004.03234, 2020.
[41] Sara Sabour, Andrea Tagliasacchi, Soroosh Yazdani, Geoffrey E Hinton, and David J Fleet. Unsupervised part representation by flow capsules. arXiv preprint arXiv:2011.13920, 2020.
[42] Derek Hoiem, Alexei A Efros, and Martial Hebert. Recovering occlusion boundaries from an image. International Journal of Computer Vision, 91(3):328-346, 2011
[43] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898-916, 2010.
[44] Eitan Sharon, Meirav Galun, Dahlia Sharon, Ronen Basri, and Achi Brandt. Hierarchy and adaptivity in segmenting visual scenes. Nature, (217700))810-8111 2006.
[45] Ian Endres and Derek Hoiem. Category-indepeeddnt object proposals with diverse ranking. IEEE transactions on pattern analysis and machine intelligence, 36(2):222 2013.
[46] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal
journaal of computer vision, 104(2):154-177 2013 2013.
[47] Sudheendra Vijayanarasimhan and Kristen Grauman. Efficient region search for object detection. In CVPR 20/ I, pages 1401-1408. IEEE, 2011.
[48] John YA Wang and Edward H Adelson. Layered representation for motion analysis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 361-366.
IEEE, 1993.
[49] John YA Wang and Edward H Adelson. Representing moving images with layers. IEEE transactions on image processing, 3(5):625-638, 1994.
[50] Deqing Sun, Jonas Wulff, Erik B Sudderth, Hanspeter Pfister, and Michael J Black. fully- connected layered model of foreground and background flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2013.
[51] Deqing Sun, Erik Sudderth, and Michael Black. Layered image motion with explicit occlusions, temporal consistency, and depth ordering. Advances in Neural Information Processing Systems, 23:2226-222 2234,2222222 2010.
[52] Gabriel Brostow and Irfan A Essa. Motion based decompositing of video. In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 1, pages 8-13. IEEE, 1999.
13

---

[53] Jean-Baye -aan-Baptiste Alayrac, Joao Carreira, and Andrew Zisserman. The visual centrifuge: Model-free layered video representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2457-2466, 2019.
[54] C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele. Simon Winder, and Richard Szeliski. High-quality video view interpolation using a layered representation. ACM transactions on graphics (TOG), 23(3):600 608, 2004.
[55] Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William T Freeman, and Michael Rubinstein. Layered neural rendering for retiming people video. arXiv preprint arXiv:2009.000 07833, 2020.
[56] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan, Learning features by watching objects move. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2701-2710, 2017.
[57] Xiaohang Zhan, Xingang Pan, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self-supervised learning via conditional motion propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1881-1889, 2019.
[58] Alexey Dosovitskiy, Philipp Fischer, Eddy llg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. 2015 IEEE International Conference on Computer Vision, pages 2758-2766, 2015.
[59] E. Ilg. N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE Conference on Computer Vision and Puttern Recognition, Jul 2017.
[60] Anurag Ranjan and Michael J. Black. Optical flow estimation using a spatial pyramid network. In IEEE Conference on Computer Vision and Puttern Recognition, 2017.
[61] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, (3()):000- 612, 2004.
[62] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
[63] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[64] Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for unpaired image-to-image translation. In European Conference on Computer Vision, pages 319-345. Springer, 2020.
[65] Bruce Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. International Joint Conference on Artificial Intelligence, 1981
[66] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang. and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint wrXiv:1009.03327 2018.
[67] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[68] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun. Saliency optimization from robust background detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2814-2821, 2014.
14

---

[69] Wenbin Zou and Nikos Komodakis. Harf: detection. In Proceedings of the IEEE international conference on computer vision, pages 406-41 414,,14,200 2015.
rich features for salient object
[70] Bowen Jiang, Lihe Zhang, Huchuan Lu, Chuan Yang, and Ming- Hsuan Yang. Saliency detection via absorbing markov chain. In Proceedings of the IEFE international conference on computer vision, pages 1665-1672, 2013.
[71] Xiaohui Li, Huchuan Lu, Lihe Zhang, Xiang Ruan, and Ming- Hann- Yang. Saliency detection via dense and sparse reconstruction. In Proceedings of the IEFE international conference on computer vision, pages
2013.
[72] Huaizu Jiang, Jingdong Wang. Zejian Yuan, Yang Wu, Nanning Zheng, and Shipeng Li. Salient object detection: A discriminative regional feature integration approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2083-2090, 2013.
[73] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2576, 2019.
[74] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng. Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 136-145, 2017.
[75] Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu. Global contrast based salient region detection. IEEE transactions on pattern analysis and machine intelligence, 37(3):569-582, 2014.
[76] Jing Zhang, Tong Zhang, Yuchao Dai, Mehrtash Harandi, and Richard Hartley. Deep unsuper- vised saliency detection: A multiple noisy labeling perspective. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9029-9038, 2018.
[77] Yu Zeng, Yunzhi Zhuge, Huchuan Lu, Lihe Zhang, Mingyang Qian, and Yizhou Yu. Multi- source weak supervision for saliency detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6074-6083, 2019.
[78] Due Tam Nguyen, Maximilian Dax, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo. Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox. Deepusps: Deep robust unsupervised saliency prediction with suff-supervisoon arXiv preprint PexXiv:1909 33055, 2019.
[79] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine- A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Puttern Recognition, 2016.
[80] Fuxin Li, Taeyoung Kim, Ahmad Humayun, David Tsai, and James M Rehg. Video segmen- tation by tracking many figure-ground segments. In Proceedings of the IEEE International Conference on Computer Vision, pages 2192-2199, 2013.
[81] Wenguan Wang, Jianbing Shen, Ruigang Yang, and Fatih Porikli. Saliency-aware video object segmentation. IEEE transactions on pattern analysis and machine intelligence, 40(1):20-33, 2017.
[82] Thomas Brox and Jitendra Malik. Large displacement optical flow: variational motion estimation. IEEE Iransactions pattern analysis and machine intelligence, 33(3):500-513, 2010.
descriptor matching in
[83] Ce Liu et al. Beyond pixels: exploring new representations and applications for motion analysis. PhD thesis, Massachusetts Institute of Technology, 2009.
[84] Narayanan Sundaram, Thomas Brox, and Kurt Keutzer. Dense point trajectories by gpu- accelerated large displacement optical flow. In European conference on computer vision, pages 438-451 Springer, 2010.
15

---

[85] Yinlin Hu, Rui Song, and Yunsong Li. Efficient coarse-to- fine patchmatch for large displacement optical flow. In CVPR, 2016.
[86] Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-333 2010.
Checklist
1. For all authors...
(a) the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] All data used in the paper public; our code will be published.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Unfortunately, we did not provide the error bars due to resource constraints. We will make this up in the future.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited all video and image datasets used this paper.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "q69YnpnUys2", "reviewer": "Reviewer_Y1LA", "summary": "This paper presents a method that segments the image into a number of segments and estimates an offset vector for each segment. Different from prior two-stream appearance+motion based methods, this approach does not leverage a pre-trained flow network but learns to estimate motion offset vectors implicitly by warping (segmented) regions across frames based on the flow estimates. Reconstruction error (inspired by self-supervised optical flow methods) is used as a loss function and provides a supervisory signal. Thus, the proposed method can be trained using unlabelled video. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n-----------\n\n* This paper is tackling a very relevant research problem of learning to segment objects without supervision, leveraging a large video corpus. The way to go! \n* The proposed method is simple but interesting and with the potential to scale very well. \n* The paper is generally well-written, and the methodology is presented clearly. \n* Results do indeed look quite promising: on-par with state-of-the-art on DAVIS, compared to methods that use no supervision. \n* I appreciate that the paper makes an effort to demonstrate the versatility of this method for several tasks, ranging from VOS to object segmentation in images and as unsupervised pre-training for semantic segmentation. \n\n\nCons\n-----------\n\n* This paper claims that this method is trained absolutely without any supervision. In principle, this is true: network weights are initialized randomly before training the model on YouTube VOS dataset. However, I would be careful regarding “no supervision” claims because the results demonstrated were obtained via training on a highly curated dataset containing various biases, most notably photographer bias (objects are almost always centered in these videos). This suspicion is pretty much confirmed in L182-184: “We have empirically observed that the moving objects all appear in a particular mask channel across the training videos. This channel can be heuristically identified as the one whose segmentation mask is closest to the image center.” It, therefore, appears to me that the model learns to segment object that appears in the center of the video (or image), based on carefully selected videos that mostly contain one object of interest per clip (objects are present from start to the end of the video). This fundamentally differs from real “in-the-wild” scenarios (e.g., learning from a stream of sensory data that a robot observes). On the other hand, let me also note that most papers that claim to do unsupervised learning do not acknowledge the manual human effort invested in curating datasets that are used for learning. To conclude this thought, the paper would severely benefit from clearly stating assumptions made and limitations. \n* It is not clear to me at all how ResNet50 backbone + a FCN head yield segmentation of an image into multiple segments. Here a crucial piece of information is missing. \n* Optical flow methods (that several baselines use) can be trained in a self-supervised fashion, or using synthetic data. It can be argued that these methods do not use supervision either. I am not saying that it is not “nice” to be able to train the model from scratch, not relying on any external components, but given that baselines (CIS, MG) obtain better or significantly better results on DAVIS, this point should be argued better. \n* While Tab. 1 shows promising results, results and conclusions are also somewhat confusing. Apart from traditional methods, both learning-based baselines perform better on DAVIS (they do, though, use optical flow, but see my comment above). The proposed method performs better than CIS on SegTrack and FBMS, but those two datasets contain stronger photographer bias (correct me on this if I am mistaken). \n* I also have my doubts regarding zero-shot detection results on the DUTS dataset, and I am not sure if I agree with the conclusion. It again seems to me that the method learned to detect exactly the object that is right in the center (looking at images, Fig. 4). A better version of this experiment would be per-frame evaluation on DAVIS, which also contains sequences where the object is not always centered, and occasionally multiple objects are present. \n* I also have concerns regarding using this approach as unsupervised pre-training (which in general is a great idea!). First of all, I would definitely need to see ImageNet pre-training for comparison purposes, otherwise, it is difficult to anchor the results reported wrt. prior work. The same goes for SimSiam and MoCo-v2: ImageNet-trained variants are missing. I do understand why the paper reports these two methods trained on the same data (which indeed should be done in addition to ImageNet trained variants), but I got extremely confused with augmentations here. The core idea behind contrastive learning-based methods such as SimSiam and MoCo is to leverage augmentations to form positive pairs. If those are not used, I do not understand how these methods are trained. Perhaps I misunderstood something.\n\nTips\n-----------\n\n* This paper advertises zero-shot segmentation, but I don't think the terminology is used accurately (wrt. Xian et al., Zero-Shot Learning -- The Good, the Bad and the Ugly, CVPR’17) as the classes that are segmented in the DUTS dataset are present during the training phase, at least most of them, however, they are not labeled. This is rather a case of transductive zero-shot learning. This doesn’t impact my rating, just a remark/tip. \n\n\nJustification\n-----------\n\nIn principle, I like the idea behind this paper: it is simple but has the potential to be quite effective, and I believe that there is a need for such methods for unsupervised representation learning/object segmentation (simple methods that can scale). However, at the moment, I see too many issues with this manuscript to recommend acceptance at this time. The most critical part is the experimental evaluation: in general, I like that the paper makes an attempt to demonstrate the versatility of this approach for several tasks (VOS, object segmentation, unsupervised pre-training). However, no experiment is particularly convincing (see my criticism above), and raises more questions than provides answers. \n\nI would recommend the following:\n* Provide a more thorough discussion on the assumptions and limitations of this approach (see my comments on photographer bias and dataset curation). \n* It would be interesting to see a variant of the proposed method that does leverage optical flow as input (wrt. Tab. 1). However, this experiment is the least problematic in my view. \n* More importantly, claims regarding zero-shot saliency detection and unsupervised pre-training need stronger experimental verification; in particular, detection capabilities should be evaluated in settings where the object is not always centered and potentially multiple objects are present. Otherwise, the experimental setting is too constrained. Furthermore, I cannot judge the performance of this approach for unsupervised pre-training because the results are not discussed and placed well relative to prior work (see my comments above). Ideally, unsupervised pre-training should also be compared to prior work on unsupervised representation learning from video (the first one that comes to mind is Wang et al., Unsupervised Learning of Visual Representations using Videos, CVPR’14). \n\nOther than that, thumbs up for generally well-written paper and method that appears to me to hold quite a lot of potential! \n\n**Post rebuttal comments and justification**\nIt was particularly challenging for me to reach my final recommendation for this paper; as I've mentioned, I like this research direction. The simple yet scalable approach to learning representations from abundant video looks very appealing to me, and I find this work valuable.\n\nIn the end, the main question was how convincing is the experimental evaluation and how firm conclusions I can make. As mentioned in my review, I do appreciate that evaluation is versatile, but in the end, I am afraid that I still think that none of the experiments is highly convincing.\n\nVOS evaluation Reviewer zbkz raises the concern that those comparisons are not entirely fair, as the proposed method is the only one to train on YouTube-VOS, which is for order of magnitude larger than DAVIS, SegTrack, and FBMS. Baselines (CIS, MG) were only trained on the target datasets but did utilize pre-trained flow networks. I am afraid I agree that the fairest direct comparison would be to pre-train the network using synthetic data (commonly used to train optical flow networks) and fine-tune it on DAVIS/FBMS/SegTrack.\n\nBut perhaps the real question would be: how useful such a pre-trained network would be for downstream tasks, for example, semantic segmentation?\n\nIt turns out that this method significantly lags behind SOTA (MoCo-v2, SimSiam, 72.4 mIoU) on VOC2012. I agree with the assessment that ImageNet is better aligned with VOC2012. However, (i) the task itself (moving object segmentation) is more similar to the target task of semantic segmentation, and (ii) there are large automotive datasets out there to which this method could be applied (e.g., pre-train on BDD100k [Yu et al., CVPR'19], train on Cityscapes). I am afraid that the paper did not succeed yet to demonstrate its merit in pre-training for semantic segmentation.\n\nI am afraid I am also not convinced with the results obtained on the DUTS dataset; it appears to me that the network learns to detect anything in the center. I am afraid I disagree with the author's statement that this center bias is present everywhere (it isn't present in robotics/automotive datasets). Another way to show the merit would be to use this approach for pre-training on the instance segmentation task (e.g., on COCO).\n\nI would very be happy to see a future revision of this paper that would present a very convincing experimental evaluation. I would point to [Radford et al., ICLR'21 (CLIP)] for reference (simple method, thorough evaluation) and entirely focus on demonstrating the merit via convincing experimental evaluation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I see no issues with potential negative social impact, but the limitations were not addressed adequately (details in my review above). ", "review_text": "", "extra_scores": {"ethical_concerns": "I see no ethical concernes. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "ktTyBMvIlD_", "reviewer": "Reviewer_zbkz", "summary": "This work develops an approach to unsupervised single-object segmentation. The main idea is to segment the scene into $C$ (hyperparmeter) segments in one frame and to learn a flow field w.r.t. another frame such that it can be reconstructed by translating the segments from the original frame. This simple strategy provides competitive accuracy on standard object video segmentation benchmarks, as well as shows promise in unsupervised saliency detection (DUTS) and representation learning (VOC12). ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality.*\nThe idea seems to be original and simple. In fact, so simple that a number of key design points remain unaddressed:\n- the model is very sensitive to $C$, the number of masks for prediction, as becomes evident from Tab. 4. \n- selecting the segmentation mask based on its proximity to the centre (l.184) does not appear principled, is highly dataset dependent and puts the fairness of the comparison to prior art into question (esp. for the saliency benchmark).\nI also suspect that disocclusions (“holes”) and self-occlusions may pose a problem. In the first case, suppose an object moves and the camera is static. The parts of the image that become disoccluded cannot be reconstructed from the image where there were occluded in the previous frame. How does the method cope with it? For the second point, assume an object (e.g. a ball) rotating around its own axis. Approaches based on dense flow will be able to detect this motion. How would the method handle this scenario?\n\n*Quality.*\nThe work is well-presented and argued. In particular, the argument that previous work relies on optical flow pre-training whereas the presented method doesn’t, is a good point. However, I would still not overstress it, since the method does use an optical flow network for motion prediction (PWC-Net); and dense optical flow can be learned from synthetic data.\nThe work also needs to further clarify the following points:\n- What is the model accuracy without test-time adaptation? When is it reported with and without this process in the tables?\n- How can it be that Fig. 3 shows significant improvements w.r.t. CIS, yet in Tab. 1 CIS actually performs a bit better?\n- How is $C$ selected exactly? (l. 140 “empirically” is not concrete). This needs to be clear for every experiment and the dataset. \n- What data (datasets) do the methods in Tab. 2 use for training / hyperparameter selection? This needs to be reported for transparency.\n\n*Clarity.*\nThe work is generally well-written, but not without a few minor deficiencies:\n- notation is not rigorous. For example, Eq. 5 multiples $\\mathbb{R}^2$ with $\\mathbb{R}^{c \\times h \\times w}$.\n- some missing implementation details in Sec. 3.6: batch size, data augmentation used; how was the termination criterion selected?\n\n\n*Significance.*\nThe approach provides tangible improvements w.r.t. previous work, is slightly worse than concurrent work ([6], but it is less of a concern). However, the method also appears much more prone to hyperparameter choice; how those were selected is important for the fairness of the comparisons and needs to be stated unambiguously.\n\n\n**Post-rebuttal update**\n\nI thank the authors for detailed clarifications. I find the presented results very encouraging in view of the method's simplicity and versatility. However, I still maintain that head-to-head comparisons to related methods (i.e. using the same training data) are essential, whenever technically possible, in order to understand where the approach fits in a larger body of previous work (and even if this use case was not originally intended). One of such experiments in the VOS setting -- pre-training on synthetic data -- is feasible, but not presented.\n\nMore generally relating to my point on (self-)occlusions, I'd also like to recommend including a short discussion about the assumptions underlying the approach mainly to clarify: When does it make sense to apply segment flow for the reconstruction loss? This is because such transformation generally does not apply due to the 3D nature of the scene.\n\nOverall, I share the sentiment of Reviewer Y1LA that this work has potential, and encourage the authors to re-submit the manuscript after the revision.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some of the key model limitations do become apparent after reading the manuscript. Nevertheless, I would recommend offering this discussion more prominently in the main text.\nSocietal impact is not addressed in sufficient depth (Sec. 3 in supp. “Broader Impact” mentions only “privacy”), hence needs to be elaborated on more in the revision.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "czYvghcptpe", "reviewer": "Reviewer_pZ87", "summary": "Paper proposes a novel approach to unsupervised object segmenation based on temporal consistency in video sequences.\nThe authors show good results across a range of video datasets, and that the models trained without supervision generalise well to figure/ground segmenations on image based datasets.\n\nThe method itself is conceptually straightforward, decomposing the image into a fixed number of parts so that each part has a low photometric error when propegated to a future frame, using a constant velocity model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper is well written and the results are compelling, clearly outperforming unsupervised approaches, and compairing well with approaches that make use of supervised components.\n\nThe experiments are extensive, and the inclusion of supervised methods that perform better than this approach is welcome, and helps put the overall results in context.\n\nArguably this paper would be better suited for a CV conference, essentially we have very standard ML choices composed with a standard loss, and limitted novelty from a pure ML perspective. However, I believe that the task itself - completely unsupervised object discovery from video is of strong interest to the wider ML community, and the paper therefore is a good fit for neurips.\n\nI'd characterise it as having sufficient novelty, that's made up for by the relevance of the approach to the community.\n\nMinor comments: \n\nLine 177 Don't use FCN as a standalone abreviation, it can stand for either fully connected network, or fully convolutional network, and leaves the reader guessing.\n\nnon-published -> unpublished\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Fine. Work is unlikely to have direct social impact.\n\n Visualisation of the other segments that do not correspond to objects would be welcome, and help the reader understand the behaviour/limitations better, as would showing some failure cases. ", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "J1jAvir_fr_", "reviewer": "Reviewer_5AXj", "summary": "This paper proposes to decouple the motion and appearance by two pathways: the appearance pathway is designed to learn object segmentation masks, while the motion pathway aims to predict the optical flow per region mask. The whole model is learned in a self-supervised manner by frame-reconstruction loss. While such region-wise, which is in contrast to the typical way of learning pixel-wise dense optical flow, is interesting, the overall novelty is not sufficient for NeurIPS.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The idea that learning the region-wise optical flow to identify primary/moving objects is relatively new. The framework of decoupling the motion and appearance features by two single pathways is not quite novel. Overall, the originality is limited.\n\nQuality: The overall framework of the submission is technically sound. However, since the detailed implementations of each modules (two pathways) are too simple, I doubt its effectiveness.  I doubt that such implementation under the self-supervised learning can only simply segment the primary object. \n\nClarity: Overall the submission is written clearly. \n\nSignificance: the novelty is limited.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The whole framework sounds feasible, however, the model is designed coarsely and I doubt whether the full potential of such framework has been exploited by such implementation. For instance, the appearance pathway is designed simply by multi-convolutional layers to predict 'c' masks, which is arduous to learn the segmentations effectively. The motion pathway obtains the region-wise optical flow simply by pooling the predicted pixel-wise motion features. The whole model is supervised by simple frame-reconstruction loss. I doubt its effectiveness. As shown in the paper, the model can only identify the primary object in a scene where there is only one primary moving object. Besides, the strategy for identifying the primary mask by  \"This channel can be heuristically identified as the one whose segmentation mask is closest to the image center\" is not an elegant solution.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers did not reach a consensus for this paper. Most of them like the idea in the paper and consider that the experimental section is comprehensive, but some of the reviewers are not fully convinced by any of the experiments. 

There was also considerable discussion over if the method depends strongly on photographer bias. I am not too concerned about that in training: one can imagine a curriculum of learning where a method like the proposed is first exposed to videos with photographic bias, and progressively it moves to more complex data. Arguably the proposed model may not scale all the way, also since it has simplifying assumptions like a constant velocity motion model, but it may be able to serve as a good bootstrapping mechanism. There was some criticism in discussion that the model trains on more data than competing methods, but being unsupervised i wouldn't hold that against the method

I join the reviewers in enjoying the simplicity of the approach and understand that the experiments can be improved but think that they are decent already. In any case i encourage the authors to look carefully into the reviews and improve the experiments following the reviewers advice, namely trying to understand how good the segments are in multi-object data. I would be curious if the model still learns on more complex video data, such as imagenet-video or larger action recognition datasets.
Decision: Accept (Poster)