The Emergence of Objectness: Learning Zero-shot Segmentation from Videos

1 Introduction . Contrastive learning [ 1–3 ] has recently become a powerful method for obtaining high-level visual representations from images [ 4 ] . While these representations are shown to be more generalizing , there remain two practical limitations : 1 ) Hand-crafted augmentations such as image cropping and color jittering [ 5 ] are critical for achieving invariant recognition , and yet they fall short of capturing more complex object deformations and 3D viewpoint changes . 2 ) Additional labeled data are required at the downstream for representation fine-tuning , preventing standalone applications . Our goal here is to overcome these limitations of contrastive representation learning by developing object segmentation models automatically from unlabeled videos without any supervision . Unlike single static images , videos contain sequences of dynamic images that could reveal not only moving objects from their backgrounds , but also their internal part organizations with articulated movements . Once figure-ground segregation occurs automatically in raw videos , object semantics can be readily discovered from those foreground segmentations . ∗Equal contribution . Work done when Runtao was a StarBridge intern at MSRA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Several observations motivate us to explore such zero-shot learning of object segmentation . 1 ) Humans can easily segment moving objects without knowing what they are . 2 ) In biological vision , newborn chicks raised in a controlled visual world rapidly develop more accurate object representations when presented with temporally slow and smooth objects , generalizing from very limited viewpoints [ 6 , 7 ] . 3 ) Invariant recognition can be developed by seeking slowly varying features from temporally varying signals [ 8 ] , disentangling object identity and object location unsupervisedly . We model segmentation and movement concurrently from unlabeled videos . Our premise is that a video contains different views of the same scene related by moving components , and the right region segmentation and region flow would allow mutual view synthesis between frames that can be checked on the data itself . That is , if we know how regions of frame j are moved from regions of frame i , we can synthesize frame j by copying regions from frame i and paste them according to how they move . Comparing the synthesized frame j with the actual frame j provides feedback on how to improve both region segmentation and region flow estimation without needing any supervision . View synthesis has been frequently adopted as a self-supervised criterion for learning dense optical flows [ 9 ] , monocular depth [ 10 ] , and multi-plane image representation [ 11 ] etc from images . Unlike prior works that focus on low-level visual correspondences , our work tackles object segmentation for mid- to high-level visual recognition directly . Specifically , as illustrated in Figure 1 , instead of deriving dense optical flows between successive frames and supplying additional cues for imagebased object segmentation in a bottom-up manner , we seek a top-down factorized representation that provides a gross characterization of moving regions for the entire scene . Our model first deconstructs video frames by processing them in two separate pathways : an appearance pathway that models what is moving and outputs feature-based region segmentation given a single image , and a motion pathway that models how it moves and outputs motion features given a pair of images . It then binds them in a conjoint region flow feature representation , based on which we predict segment flow as the common fate [ 12 ] or piecewise constant movement of all the pixels in the same region . By training the model to minimize view synthesis errors based on segment flow , our appearance and motion pathways learn region segmentation and flow estimation automatically without building them up from low-level edges or optical flows respectively . After training our segmentation and flow features for view synthesis on unlabeled videos , our model demonstrates the surprising emergence of objectness in a particular feature channel of the appearance pathway . That is , our model can be directly applied to novel images and videos for segmenting foreground objects : Our appearance pathway can perform zero-shot object segmentation on a single image , whereas our overall model can perform zero-shot moving object segmentation on a single video with test-time adaptation . Our image feature learned from unlabeled videos can be further fine-tuned on a small labeled dataset for semantic segmentation . Experimentally , we demonstrate strong performance on all three applications , with considerable gains over baselines . To summarize , our work makes the following contributions . 1 ) We develop the first truly end-toend learned zero-shot object segmentation model from unlabeled videos , assuming no low-level computation such as edges or optical flow . 2 ) We bypass the traditional low-level dense optical flow and propose to compute novel mid-level segment flow directly . 3 ) Our model not only develops generic objectness for segmentation and tracking , but also outperforms prevalent image-based contrastive learning methods without augmentation engineering . Our code is available at https : //github.com/rt219/The-Emergence-of-Objectness . 2 Related Works . Video object segmentation . Segmentation of moving objects requires finding correspondences across time . A major line of work assumes that an object mask is given in the initial frame , and the goal is to propagate the mask to future frames based on the similarity of learned visual representation . Such a representation can be trained from pixel-level object masks with long-term relations in videos [ 13 , 14 ] , or from self-supervised criteria such as colorization [ 15 ] and cycle-consistency [ 16 ] . Fully unsupervised video object segmentation without initial masks has received less attention . NLC [ 17 ] and ARP [ 18 ] segment moving objects based on temporal clustering , but they rely on edge and saliency annotations , and are thus not completely unsupervised . FTS [ 19 ] performs segmentation by obtaining motion boundaries from optical flow . SAGE [ 20 ] takes into account edges , motion segmentation , and image saliency for video object segmentation . Contextual information separation [ 21 ] segments moving objects by exploiting independence between the foreground motion and the background motion . A concurrent work based on motion grouping [ 22 ] clusters pixels by similar motion vectors . These works rely on off-the-shelf optical flow results , which may be trained with [ 23 , 24 ] or without [ 9 ] supervision . Our work does not assume any known low-level features such as edges or optical flow , and learns the right feature to extract completely from scratch . Motion Segmentation . Classical methods for motion segmentation [ 25–27 ] find regions of distinctive motion based on two-frame optical flow . Supervised learning approaches [ 28 , 29 ] map optical flow to segmentation masks . Ideally , these methods require dense and accurate optical flow . In practice , the low-level differential flow is often present along edges ; it is neither dense nor smooth within a region , often inhomogeneous for deformable and articulated objects [ 30 ] , and sensitive to varying scene depths and camera motion [ 31–33 , 33 ] . Motion segmentation is shown to be less brittle when examined over a large time interval [ 30 ] . Trajectory clustering [ 34 ] tracks point trajectories over hundreds of frames , extracts descriptors for the trajectories , and clusters them to obtain a segmentation . Such a global approach is computationally demanding . In contrast , our segmentation is based not on motion between two frames , but on image appearance in a single image , which provides rich cues such as color , texture , and edges for pixel grouping and segregation . While our segmentation model does not need dense pixel correspondences between frames , it is learned to be in sync with region-wise correspondences for best view synthesis . Unsupervised learning for segmentation . Human annotation of pixel-wise segmentations is not only time-consuming , but also often inaccurate along boundaries . Learning segmentation without labels is thus of great interest in practice . SegSort [ 35 ] predicts segmentation by learning to group super-pixels of similar appearance and semantic context from static images . Later work [ 36 ] contrasts holistic mask proposals obtained by traditional bottom-up grouping . A related line of work focuses on learning part segmentation from images and videos of the same object category , such as humans and faces . SCOPS [ 37 ] is a representative co-part segmentation method , learned in a self-supervised fashion ; its general idea follows unsupervised landmark detection [ 38 ] , leveraging geometric invariance , representation equivariance , and perceptual reconstruction . [ 39 ] explores motion cues in videos to discover object part organization and dynamics . Motion-supervised co-part segmentation [ 40 ] models part motion between adjacent frames using affine parameters . A similar idea is implemented with capsule networks [ 41 ] . In contrast , our work is not restricted to videos in a single category and learns object segmentation from a collection of generic videos . Learning objectness from data . Segregating foreground objects from background is a central problem in visual recognition . Prior works on images first generate hierarchical segmentations [ 42– 44 ] using low-level visual cues such as colors and boundaries [ 45 ] , and then rank these candidate regions according to a certain criterion [ 46 , 47 ] . These approaches often generate many overlapping redundant individual object instance proposals . Videos of slowly moving objects [ 8 ] are shown to enable development of newborn vision [ 6 , 7 ] . Linear models [ 48 , 49 ] can factorize a sequence of images into foreground and background layers , assuming independent motion among them . Layered representations are also used for optical flow estimation [ 50 , 27 , 51 ] , view-interpolation , and time retargeting [ 52–55 ] . Our work adopts an unordered layered representation with multiple segmentation channels , each identifying a region of common motion . While our view synthesis objective during training does not differentiate foreground or background in these channels , surprisingly , objectness emerges automatically in a particular channel after we train our model from a collection of raw videos . Image representation learning from videos . Motion reveals the location , shape , and part hierarchy of moving objects . Motion segmentation has thus been used to supervise learning of image-level object representations [ 56 ] . Motion propagation [ 57 ] predicts dense optical flow from sparse optical flow , conditioned on the color image . Unlike prior works , the single image representation produced by our model is not learned from motion supervision , but concurrently learned with between-frame region flow as a by-product of our moving object segmentation from unlabeled videos . 3 Segmentation by Appearance-Motion Decomposition . We would like our model to segment moving objects without necessarily knowing what and how many they are . Our model is trained on a collection of unlabeled generic videos , and can be directly deployed on a novel image ( video ) to produce ( moving ) object segmentation . No human annotations are required during either training or testing . Illustrated in Figure 2 , our so-called appearance-motion decomposition ( AMD ) model first deconstructs a pair of video frames , Xi and Xj , in two separate pathways . The bottom appearance pathway fA takes in a single image Xi and outputs a feature-based segmentation , whereas the top motion pathway fM takes in both frames ( Xi , Xj ) and outputs the flow features between them . The two pathways then come together to construct a conjoined segment flow representation F , which is used to synthesize frame Xj by warping Xi . The overall model can be trained to minimize the reconstruction error on frame Xj over sampled image pairs in the training dataset . 1 ) Appearance pathway for feature-based segmentation .. We adopt a fully convolutional neural network for segmenting a single RGB image of size h×w into c regions , where c is a hyper-parameter . Formally , video frame Xi ∈ R3×h×w is transformed by fA into segmentation S with c soft masks : S = fA ( Xi ) ∈ Rc×h×w , ( 1 ) c∑ m=1 S ( p ) = 1 , p = 1 , . . . , h× w. ( 2 ) The above normalization equation reflects that values of S are the probabilities of pixel p belonging to c regions . Empirically , we choose c=5 by default . Our ablation study later shows that a larger c may lead to over-segmentation , whereas a smaller c may lack the spatial resolution to locate objects . Note that our segmentation network is based on the RGB appearance in a single frame instead of optical flow between two frames . Since it is designed to operate on static images , it can be transferred to downstream image-based vision tasks . Our segmentation network trained on unlabeled videos can be directly used to segment not only moving objects from novel videos ( Section 4.2 ) , but also salient objects from single images in a zero-shot fashion ( Section 4.1 ) . It can be further fine-tuned on a labeled image dataset for semantic segmentation ( Section 4.3 ) . 2 ) Motion pathway for correspondences . We adopt PWC-Net [ 23 ] for extracting pixel-wise motion features between a pair of images . PWC-Net is originally designed for predicting dense optical flow , and the feature for each pixel in one frame captures its similarity to spatial neighbors in the other frame . Formally , video frames ( Xi , Xj ) are transformed by fM into motion correspondence feature V of dv dimensions : V = fM ( Xi , Xj ) ∈ Rdv×h×w . ( 3 ) Note that we only adopt the network architecture not the trained weights of PWC-Net in [ 23 ] , and our choice among alternative architectures such as FlowNet [ 58 ] , FlowNet2 [ 59 ] , SpyNet [ 60 ] , and RAFT [ 24 ] is based on conceptual simplicity and light-weight model size . 3 ) Segment flow representation . We now construct a conjoined segment flow representation from both pathways to enable view synthesis . Specifically , we pool the pixel-wise correspondence feature V in the motion pathway according to the image segmentation S in the appearance pathway , resulting in an average dv-dimensional motion feature per segment , i.e. , Vm for the m-th mask Sm : Vm = ∑h×w p=1 V ( p ) × Sm ( p ) ∑h×w p=1 Sm ( p ) ∈ Rdv , m=1 , . . . , c. ( 4 ) We then predict a single common 2D flow vector Fm for the entire m-th segment Sm based on its average motion feature Vm , using a two-layer multilayer perceptron ( MLP ) for the head function g : Fm = g ( Vm ) ∈ R2 , m=1 , . . . , c. ( 5 ) So far , we deconstruct a pair of video frames ( Xi , Xj ) into c segmentation masks and their associated flow vectors { ( Sm , Fm ) : m = 1 , . . . , m } , assuming one common motion for pixels within the same segment . This piece-wise constant motion assumption simplifies flow estimation and provides a gross characterization of movement in the scene . While it may not hold for deformable and articulated objects , when trained over a collection of videos , our model with appearance feature-based segmentation is able to aggregate smoothly moving pieces in a wholesome segment . We then compose these moving components into a novel flow representation F for the entire image : F ( p ) = c∑ m=1 Fm × Sm ( p ) , p=1 , . . . , h× w. ( 6 ) We call F segment flow , as its values indicate the overall displacement at the segment level . This conjoined representation allows motion and segmentation to cross-supervise each other . Given a between-frame flow vector , the segmentation network could be supervised to find pixels of this offset . Given a segmentation mask , the motion network could be supervised to find the flow for this mask . Our approach to image segmentation with motion inputs is fundamentally different from motion segmentation methods : 1 ) Our segmentation mask is predicted from static image appearance that does not require dense and accurate flow for supervision ; 2 ) Our flow estimation is at the segment level , which can be inferred from sparse and noisy pixel-level flow estimates . 4 ) Reconstruction objective for view synthesis . How do we validate our segment flow F , a conjoined representation from both appearance and motion pathways ? Intuitively , the right segmentation S and motion Fm would allow the synthesis of frame Xj from frame Xi according to their segment flow F , and the reconstruction X̂j should be close to the actual frame Xj : X̂j ( p ) = Xi ( p+ F ( p ) ) , p = 1 , . . . , h× w ( 7 ) L = D ( Xj , X̂j ) , ( 8 ) whereD is a metric measuring the distance between two images . We adopt the pixel-wise photometric loss SSIM [ 61 ] for simplicity , among alternatives such as deep-feature matching losses [ 62 , 63 ] and contrastive losses [ 64 ] . The warping loss L is the only self-supervision our model receives . Note that a small reconstruction error does not necessarily mean that the segmentation and flow are correct , but correct segmentation and flow must result in a small reconstruction error . That is , this reconstruction objective is a necessary condition for correct segmentation and flow estimation . Compared to traditional optical flow estimation , our model also derives motion flow from brightness constancy between two frames , but the optical flow computation assumes pixel-wise local displacements that are independent of each other , whereas our segment flow assumes all the pixels in the same segment have a common displacement . In addition , our segmentation is determined from the image appearance , instead of fixed-size patches assumed in the Lucas–Kanade optical flow method [ 65 ] . Our model has thus two essential bottlenecks : One is the number of segments , and the other is piecewise constant segment flow . Both are important for directly delivering a mid-level organization without building it up from low-level vision such as edges and optical flow . 5 ) The emergence of objectness . The appearance pathway in our model only segments an image into c regions , one in each of its c channels . Our reconstruction objective is only concerned with collective view synthesis from these c channels , and does not designate any channels for moving objects or background . That is , any channel could contain the moving foreground object for a particular video . Surprisingly , we observe empirically that objects in training videos tend to concentrate in the same channel , with a relatively sharp and uniform mask in the center of the image ( Figure 3 ) . We conduct analysis to understand the emergence of objectness in a particular channel , the channel activated by the feature that seems to capture generic objects against their backgrounds . We calculate three types of per-pixel statistics for each segmentation channel from 17,500 sampled frames across training videos : 1 ) segmentation uncertainty measured by the entropy of mask value Sm , 2 ) reconstruction error between Xj and X̂j measured by SSIM , and 3 ) the mean and standard deviation of the mask center and the average mask spread . Figure 3 shows that the objectness channel has the least segmentation uncertainty , the largest reconstruction training error , mostly central locations and relatively focused areas . Our conjecture is that three factors contribute to the emergence of objectness in our model : 1 ) Training videos tend to track moving objects in the center field of view ; 2 ) Our piece-wise constant motion assumption holds better for the background ; 3 ) Motion of object pixels tends to be independent of that of background pixels , whereas motion of background pixels could be interpolated from other pixels scattered in multiple background regions . 6 ) Limitations . Our current model develops a general sense of objectness from a collection of unlabeled videos . It can only segment foreground objects without differentiating between object instances or between semantic classes . It is not guaranteed to segment out all the objects or full objects . Like most data-driven learning methods , our model performance heavily depends on the properties of training videos , the coverage of object categories , and the object motion as well as camera motion . Nevertheless , we still find it amazing that our model is able to generalize the concept of objectness across a variety of datasets in a zero-shot fashion , moving a step closer to human vision .

Summary: This paper presents a method that segments the image into a number of segments and estimates an offset vector for each segment. Different from prior two-stream appearance+motion based methods, this approach does not leverage a pre-trained flow network but learns to estimate motion offset vectors implicitly by warping (segmented) regions across frames based on the flow estimates. Reconstruction error (inspired by self-supervised optical flow methods) is used as a loss function and provides a supervisory signal. Thus, the proposed method can be trained using unlabelled video.