The Emergence of Objectness: Learning Zero-shot Segmentation from Videos

1 Introduction . Contrastive learning [ 1–3 ] has recently become a powerful method for obtaining high-level visual representations from images [ 4 ] . While these representations are shown to be more generalizing , there remain two practical limitations : 1 ) Hand-crafted augmentations such as image cropping and color jittering [ 5 ] are critical for achieving invariant recognition , and yet they fall short of capturing more complex object deformations and 3D viewpoint changes . 2 ) Additional labeled data are required at the downstream for representation fine-tuning , preventing standalone applications . Our goal here is to overcome these limitations of contrastive representation learning by developing object segmentation models automatically from unlabeled videos without any supervision . Unlike single static images , videos contain sequences of dynamic images that could reveal not only moving objects from their backgrounds , but also their internal part organizations with articulated movements . Once figure-ground segregation occurs automatically in raw videos , object semantics can be readily discovered from those foreground segmentations . ∗Equal contribution . Work done when Runtao was a StarBridge intern at MSRA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Several observations motivate us to explore such zero-shot learning of object segmentation . 1 ) Humans can easily segment moving objects without knowing what they are . 2 ) In biological vision , newborn chicks raised in a controlled visual world rapidly develop more accurate object representations when presented with temporally slow and smooth objects , generalizing from very limited viewpoints [ 6 , 7 ] . 3 ) Invariant recognition can be developed by seeking slowly varying features from temporally varying signals [ 8 ] , disentangling object identity and object location unsupervisedly . We model segmentation and movement concurrently from unlabeled videos . Our premise is that a video contains different views of the same scene related by moving components , and the right region segmentation and region flow would allow mutual view synthesis between frames that can be checked on the data itself . That is , if we know how regions of frame j are moved from regions of frame i , we can synthesize frame j by copying regions from frame i and paste them according to how they move . Comparing the synthesized frame j with the actual frame j provides feedback on how to improve both region segmentation and region flow estimation without needing any supervision . View synthesis has been frequently adopted as a self-supervised criterion for learning dense optical flows [ 9 ] , monocular depth [ 10 ] , and multi-plane image representation [ 11 ] etc from images . Unlike prior works that focus on low-level visual correspondences , our work tackles object segmentation for mid- to high-level visual recognition directly . Specifically , as illustrated in Figure 1 , instead of deriving dense optical flows between successive frames and supplying additional cues for imagebased object segmentation in a bottom-up manner , we seek a top-down factorized representation that provides a gross characterization of moving regions for the entire scene . Our model first deconstructs video frames by processing them in two separate pathways : an appearance pathway that models what is moving and outputs feature-based region segmentation given a single image , and a motion pathway that models how it moves and outputs motion features given a pair of images . It then binds them in a conjoint region flow feature representation , based on which we predict segment flow as the common fate [ 12 ] or piecewise constant movement of all the pixels in the same region . By training the model to minimize view synthesis errors based on segment flow , our appearance and motion pathways learn region segmentation and flow estimation automatically without building them up from low-level edges or optical flows respectively . After training our segmentation and flow features for view synthesis on unlabeled videos , our model demonstrates the surprising emergence of objectness in a particular feature channel of the appearance pathway . That is , our model can be directly applied to novel images and videos for segmenting foreground objects : Our appearance pathway can perform zero-shot object segmentation on a single image , whereas our overall model can perform zero-shot moving object segmentation on a single video with test-time adaptation . Our image feature learned from unlabeled videos can be further fine-tuned on a small labeled dataset for semantic segmentation . Experimentally , we demonstrate strong performance on all three applications , with considerable gains over baselines . To summarize , our work makes the following contributions . 1 ) We develop the first truly end-toend learned zero-shot object segmentation model from unlabeled videos , assuming no low-level computation such as edges or optical flow . 2 ) We bypass the traditional low-level dense optical flow and propose to compute novel mid-level segment flow directly . 3 ) Our model not only develops generic objectness for segmentation and tracking , but also outperforms prevalent image-based contrastive learning methods without augmentation engineering . Our code is available at https : //github.com/rt219/The-Emergence-of-Objectness . 2 Related Works . Video object segmentation . Segmentation of moving objects requires finding correspondences across time . A major line of work assumes that an object mask is given in the initial frame , and the goal is to propagate the mask to future frames based on the similarity of learned visual representation . Such a representation can be trained from pixel-level object masks with long-term relations in videos [ 13 , 14 ] , or from self-supervised criteria such as colorization [ 15 ] and cycle-consistency [ 16 ] . Fully unsupervised video object segmentation without initial masks has received less attention . NLC [ 17 ] and ARP [ 18 ] segment moving objects based on temporal clustering , but they rely on edge and saliency annotations , and are thus not completely unsupervised . FTS [ 19 ] performs segmentation by obtaining motion boundaries from optical flow . SAGE [ 20 ] takes into account edges , motion segmentation , and image saliency for video object segmentation . Contextual information separation [ 21 ] segments moving objects by exploiting independence between the foreground motion and the background motion . A concurrent work based on motion grouping [ 22 ] clusters pixels by similar motion vectors . These works rely on off-the-shelf optical flow results , which may be trained with [ 23 , 24 ] or without [ 9 ] supervision . Our work does not assume any known low-level features such as edges or optical flow , and learns the right feature to extract completely from scratch . Motion Segmentation . Classical methods for motion segmentation [ 25–27 ] find regions of distinctive motion based on two-frame optical flow . Supervised learning approaches [ 28 , 29 ] map optical flow to segmentation masks . Ideally , these methods require dense and accurate optical flow . In practice , the low-level differential flow is often present along edges ; it is neither dense nor smooth within a region , often inhomogeneous for deformable and articulated objects [ 30 ] , and sensitive to varying scene depths and camera motion [ 31–33 , 33 ] . Motion segmentation is shown to be less brittle when examined over a large time interval [ 30 ] . Trajectory clustering [ 34 ] tracks point trajectories over hundreds of frames , extracts descriptors for the trajectories , and clusters them to obtain a segmentation . Such a global approach is computationally demanding . In contrast , our segmentation is based not on motion between two frames , but on image appearance in a single image , which provides rich cues such as color , texture , and edges for pixel grouping and segregation . While our segmentation model does not need dense pixel correspondences between frames , it is learned to be in sync with region-wise correspondences for best view synthesis . Unsupervised learning for segmentation . Human annotation of pixel-wise segmentations is not only time-consuming , but also often inaccurate along boundaries . Learning segmentation without labels is thus of great interest in practice . SegSort [ 35 ] predicts segmentation by learning to group super-pixels of similar appearance and semantic context from static images . Later work [ 36 ] contrasts holistic mask proposals obtained by traditional bottom-up grouping . A related line of work focuses on learning part segmentation from images and videos of the same object category , such as humans and faces . SCOPS [ 37 ] is a representative co-part segmentation method , learned in a self-supervised fashion ; its general idea follows unsupervised landmark detection [ 38 ] , leveraging geometric invariance , representation equivariance , and perceptual reconstruction . [ 39 ] explores motion cues in videos to discover object part organization and dynamics . Motion-supervised co-part segmentation [ 40 ] models part motion between adjacent frames using affine parameters . A similar idea is implemented with capsule networks [ 41 ] . In contrast , our work is not restricted to videos in a single category and learns object segmentation from a collection of generic videos . Learning objectness from data . Segregating foreground objects from background is a central problem in visual recognition . Prior works on images first generate hierarchical segmentations [ 42– 44 ] using low-level visual cues such as colors and boundaries [ 45 ] , and then rank these candidate regions according to a certain criterion [ 46 , 47 ] . These approaches often generate many overlapping redundant individual object instance proposals . Videos of slowly moving objects [ 8 ] are shown to enable development of newborn vision [ 6 , 7 ] . Linear models [ 48 , 49 ] can factorize a sequence of images into foreground and background layers , assuming independent motion among them . Layered representations are also used for optical flow estimation [ 50 , 27 , 51 ] , view-interpolation , and time retargeting [ 52–55 ] . Our work adopts an unordered layered representation with multiple segmentation channels , each identifying a region of common motion . While our view synthesis objective during training does not differentiate foreground or background in these channels , surprisingly , objectness emerges automatically in a particular channel after we train our model from a collection of raw videos . Image representation learning from videos . Motion reveals the location , shape , and part hierarchy of moving objects . Motion segmentation has thus been used to supervise learning of image-level object representations [ 56 ] . Motion propagation [ 57 ] predicts dense optical flow from sparse optical flow , conditioned on the color image . Unlike prior works , the single image representation produced by our model is not learned from motion supervision , but concurrently learned with between-frame region flow as a by-product of our moving object segmentation from unlabeled videos . 3 Segmentation by Appearance-Motion Decomposition . We would like our model to segment moving objects without necessarily knowing what and how many they are . Our model is trained on a collection of unlabeled generic videos , and can be directly deployed on a novel image ( video ) to produce ( moving ) object segmentation . No human annotations are required during either training or testing . Illustrated in Figure 2 , our so-called appearance-motion decomposition ( AMD ) model first deconstructs a pair of video frames , Xi and Xj , in two separate pathways . The bottom appearance pathway fA takes in a single image Xi and outputs a feature-based segmentation , whereas the top motion pathway fM takes in both frames ( Xi , Xj ) and outputs the flow features between them . The two pathways then come together to construct a conjoined segment flow representation F , which is used to synthesize frame Xj by warping Xi . The overall model can be trained to minimize the reconstruction error on frame Xj over sampled image pairs in the training dataset . 1 ) Appearance pathway for feature-based segmentation .. We adopt a fully convolutional neural network for segmenting a single RGB image of size h×w into c regions , where c is a hyper-parameter . Formally , video frame Xi ∈ R3×h×w is transformed by fA into segmentation S with c soft masks : S = fA ( Xi ) ∈ Rc×h×w , ( 1 ) c∑ m=1 S ( p ) = 1 , p = 1 , . . . , h× w. ( 2 ) The above normalization equation reflects that values of S are the probabilities of pixel p belonging to c regions . Empirically , we choose c=5 by default . Our ablation study later shows that a larger c may lead to over-segmentation , whereas a smaller c may lack the spatial resolution to locate objects . Note that our segmentation network is based on the RGB appearance in a single frame instead of optical flow between two frames . Since it is designed to operate on static images , it can be transferred to downstream image-based vision tasks . Our segmentation network trained on unlabeled videos can be directly used to segment not only moving objects from novel videos ( Section 4.2 ) , but also salient objects from single images in a zero-shot fashion ( Section 4.1 ) . It can be further fine-tuned on a labeled image dataset for semantic segmentation ( Section 4.3 ) . 2 ) Motion pathway for correspondences . We adopt PWC-Net [ 23 ] for extracting pixel-wise motion features between a pair of images . PWC-Net is originally designed for predicting dense optical flow , and the feature for each pixel in one frame captures its similarity to spatial neighbors in the other frame . Formally , video frames ( Xi , Xj ) are transformed by fM into motion correspondence feature V of dv dimensions : V = fM ( Xi , Xj ) ∈ Rdv×h×w . ( 3 ) Note that we only adopt the network architecture not the trained weights of PWC-Net in [ 23 ] , and our choice among alternative architectures such as FlowNet [ 58 ] , FlowNet2 [ 59 ] , SpyNet [ 60 ] , and RAFT [ 24 ] is based on conceptual simplicity and light-weight model size . 3 ) Segment flow representation . We now construct a conjoined segment flow representation from both pathways to enable view synthesis . Specifically , we pool the pixel-wise correspondence feature V in the motion pathway according to the image segmentation S in the appearance pathway , resulting in an average dv-dimensional motion feature per segment , i.e. , Vm for the m-th mask Sm : Vm = ∑h×w p=1 V ( p ) × Sm ( p ) ∑h×w p=1 Sm ( p ) ∈ Rdv , m=1 , . . . , c. ( 4 ) We then predict a single common 2D flow vector Fm for the entire m-th segment Sm based on its average motion feature Vm , using a two-layer multilayer perceptron ( MLP ) for the head function g : Fm = g ( Vm ) ∈ R2 , m=1 , . . . , c. ( 5 ) So far , we deconstruct a pair of video frames ( Xi , Xj ) into c segmentation masks and their associated flow vectors { ( Sm , Fm ) : m = 1 , . . . , m } , assuming one common motion for pixels within the same segment . This piece-wise constant motion assumption simplifies flow estimation and provides a gross characterization of movement in the scene . While it may not hold for deformable and articulated objects , when trained over a collection of videos , our model with appearance feature-based segmentation is able to aggregate smoothly moving pieces in a wholesome segment . We then compose these moving components into a novel flow representation F for the entire image : F ( p ) = c∑ m=1 Fm × Sm ( p ) , p=1 , . . . , h× w. ( 6 ) We call F segment flow , as its values indicate the overall displacement at the segment level . This conjoined representation allows motion and segmentation to cross-supervise each other . Given a between-frame flow vector , the segmentation network could be supervised to find pixels of this offset . Given a segmentation mask , the motion network could be supervised to find the flow for this mask . Our approach to image segmentation with motion inputs is fundamentally different from motion segmentation methods : 1 ) Our segmentation mask is predicted from static image appearance that does not require dense and accurate flow for supervision ; 2 ) Our flow estimation is at the segment level , which can be inferred from sparse and noisy pixel-level flow estimates . 4 ) Reconstruction objective for view synthesis . How do we validate our segment flow F , a conjoined representation from both appearance and motion pathways ? Intuitively , the right segmentation S and motion Fm would allow the synthesis of frame Xj from frame Xi according to their segment flow F , and the reconstruction X̂j should be close to the actual frame Xj : X̂j ( p ) = Xi ( p+ F ( p ) ) , p = 1 , . . . , h× w ( 7 ) L = D ( Xj , X̂j ) , ( 8 ) whereD is a metric measuring the distance between two images . We adopt the pixel-wise photometric loss SSIM [ 61 ] for simplicity , among alternatives such as deep-feature matching losses [ 62 , 63 ] and contrastive losses [ 64 ] . The warping loss L is the only self-supervision our model receives . Note that a small reconstruction error does not necessarily mean that the segmentation and flow are correct , but correct segmentation and flow must result in a small reconstruction error . That is , this reconstruction objective is a necessary condition for correct segmentation and flow estimation . Compared to traditional optical flow estimation , our model also derives motion flow from brightness constancy between two frames , but the optical flow computation assumes pixel-wise local displacements that are independent of each other , whereas our segment flow assumes all the pixels in the same segment have a common displacement . In addition , our segmentation is determined from the image appearance , instead of fixed-size patches assumed in the Lucas–Kanade optical flow method [ 65 ] . Our model has thus two essential bottlenecks : One is the number of segments , and the other is piecewise constant segment flow . Both are important for directly delivering a mid-level organization without building it up from low-level vision such as edges and optical flow . 5 ) The emergence of objectness . The appearance pathway in our model only segments an image into c regions , one in each of its c channels . Our reconstruction objective is only concerned with collective view synthesis from these c channels , and does not designate any channels for moving objects or background . That is , any channel could contain the moving foreground object for a particular video . Surprisingly , we observe empirically that objects in training videos tend to concentrate in the same channel , with a relatively sharp and uniform mask in the center of the image ( Figure 3 ) . We conduct analysis to understand the emergence of objectness in a particular channel , the channel activated by the feature that seems to capture generic objects against their backgrounds . We calculate three types of per-pixel statistics for each segmentation channel from 17,500 sampled frames across training videos : 1 ) segmentation uncertainty measured by the entropy of mask value Sm , 2 ) reconstruction error between Xj and X̂j measured by SSIM , and 3 ) the mean and standard deviation of the mask center and the average mask spread . Figure 3 shows that the objectness channel has the least segmentation uncertainty , the largest reconstruction training error , mostly central locations and relatively focused areas . Our conjecture is that three factors contribute to the emergence of objectness in our model : 1 ) Training videos tend to track moving objects in the center field of view ; 2 ) Our piece-wise constant motion assumption holds better for the background ; 3 ) Motion of object pixels tends to be independent of that of background pixels , whereas motion of background pixels could be interpolated from other pixels scattered in multiple background regions . 6 ) Limitations . Our current model develops a general sense of objectness from a collection of unlabeled videos . It can only segment foreground objects without differentiating between object instances or between semantic classes . It is not guaranteed to segment out all the objects or full objects . Like most data-driven learning methods , our model performance heavily depends on the properties of training videos , the coverage of object categories , and the object motion as well as camera motion . Nevertheless , we still find it amazing that our model is able to generalize the concept of objectness across a variety of datasets in a zero-shot fashion , moving a step closer to human vision .

Summary: This work develops an approach to unsupervised single-object segmentation. The main idea is to segment the scene into $C$ (hyperparmeter) segments in one frame and to learn a flow field w.r.t. another frame such that it can be reconstructed by translating the segments from the original frame. This simple strategy provides competitive accuracy on standard object video segmentation benchmarks, as well as shows promise in unsupervised saliency detection (DUTS) and representation learning (VOC12).