Dense Unsupervised Learning for Video Segmentation

1 Introduction . Unsupervised learning of visual representations has recently made considerable and expeditious advances , in part already outperforming even supervised feature learning methods [ 8 , 11 , 14 ] . Most of these works , however , require substantial computational resources [ 8 , 11 ] , and only few accommodate one of the most ubiquitous types of visual data : videos [ 13 , 33 ] . In contrast to image sets , video data embeds ample information about typical transformations occurring in nature . Exploiting such cues may allow systems to learn more task-relevant invariances , instead of relying only on hand-engineered data augmentation [ 30 ] . In this work , we propose to learn such invariances with a novel framework for the task of video object segmentation ( VOS ) . In contrast to previous works [ 17 , 19 ] , we learn dense representations efficiently in a fully convolutional manner , previously considered prone to degenerate solutions [ 17 ] . This is achieved by leveraging the assumption of feature equivariance to similarity transformations ( e. g. , multi-scale cropping ) applied to the input frame . Fig . 1 presents an overview of our approach . For each video sequence , we designate one of the frames as the reference frame . We sample a set of features , produced from the reference frame by a fully convolutional network , on a spatially uniform grid , and refer to them as anchors . Using a contrastive formulation [ 12 ] , our approach learns to represent the temporally proximate frames to the reference in terms of these anchors . Importantly , this representation is learned to be equivariant to similarity transformations . Towards this goal , we devise a self-training objective [ 20 ] that generates pseudo labels by leveraging the second , transformed view of the original video sequence . These pseudo labels contain dominant assignments of the features from the second view to the anchors . Following the self-training mechanism , we learn to assign the features in the original view consistently with the transformed view . Our self-supervised loss further disambiguates the anchors themselves spatially and between independent video sequences , while also ensuring their transformation equivariance . Implementing this process in a novel framework , we attain a new state of the art in video segmentation accuracy , with compelling computational and data efficiency . Our framework can be trained on Code ( Apache-2.0 License ) available at https : //github.com/visinf/dense-ulearn-vos . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a single commodity GPU , yet achieves higher segmentation accuracy than previous work , while requiring orders of magnitude less data . Improving the accuracy even further with more data , our approach also exhibits favourable scalability in a more comprehensive testing scenario , which has not yet been shown in prior work [ 17 , 38 ] . 2 Related work . The research domains most relevant to our work are video object segmentation ( VOS ) without supervision and representation learning . VOS . The inference setting of semi-supervised VOS , which is the task considered here , is to densely track object masks provided in the first frame . It is typically approached by learning dense feature representations to establish temporal correspondences for propagating the semantic labels . While most previous methods are supervised by pixel-wise mask annotations [ e. g. , 27 ] , an emerging direction is to leverage unsupervised learning . Our work contributes to these research efforts . The Contrastive Random Walk ( CRW ) of Jabri et al . [ 17 ] and its predecessor [ 39 ] exploit a time-cycle consistency constraint , akin to the forward-backward consistency in unsupervised optical flow estimation [ 24 ] . Other methods [ 18 , 19 ] learn the label propagation process at training time by substituting the target mask ( which is unavailable at training time ) with the image , an idea inspired by video colourisation [ 37 ] . The algorithm for label propagation itself significantly affects the VOS accuracy . Mask propagation with naive pixel-wise correspondences usually benefits greatly from leveraging more sophisticated region-level propagation [ 19 , 21 ] , which is not the focus here . Representation learning . Although previous work on unsupervised representation learning pursued spatially view-invariant objectives [ 5 , 8 , 11 , 14 , 15 , 25 , 36 ] , a concurrent study [ 10 ] finds that equipping these methods with temporally-invariant , or temporally-persistent constraints , yields sizeable gains in terms of accuracy on action recognition benchmarks . These findings align well with previous and concurrent work tailored specifically to learning spatio-temporal representations [ 3 , 9 , 13 , 32 , 33 ] for video recognition and retrieval tasks . These works additionally distinguish the representations of frames from different videos ( or , at different timesteps ) . The limitation of these works in the context of VOS is that they do not learn dense , but global feature representations at the image or video level , and typically require considerable computational budgets . By contrast , Pinheiro et al . [ 31 ] learn dense representations and exploit the equivariance constraint , as in our work . However , their approach is limited to image sets of static scenes , whereas we address videos of ( potentially ) dynamic scenes here . Semi-supervised representation learning . Recent works [ 13 , 41 ] leverage optical flow [ 34 ] to ensure the representation similarity of temporally corresponding features . Such an objective can be alternatively supervised by motion segments [ 28 ] . Spatial equivariance has been also explored in part co-segmentation [ 16 ] , a setup that requires semantically aligned image pairs , as well as in weakly supervised semantic segmentation [ 40 ] and unsupervised domain adaptation [ 1 ] . In contrast to this body of work , we learn to establish temporal correspondences via a suitable feature representations learned in a completely unsupervised way . Clustering . Contrastive learning methods are inherently expensive due to the size of pairwise feature comparisons they consider in training [ 8 , 14 ] . Clustering provides a more computationally efficient alternative [ 6 ] , since pairwise distances need to be computed only w. r. t. a ( moderately sized ) set of pre-defined clusters . Combining these ideas , Caron et al . [ 5 ] propose to cluster the embeddings and then enforce consistency between the cluster assignments by contrastive learning . The main challenge in this approach is to impose an expressive prior on the cluster assignments . For example , a uniform prior prevents trivial solutions where feature assignments collapse to a single cluster . Here , we neither explicitly define the clusters nor the prior . Instead , we sample attractor points , called anchors , directly from the available feature representations using a spatial grid sampling strategy . 3 Method . Learning feature representations from unlabelled data is an inherently ill-posed problem and requires specifying assumptions about what constitutes a useful feature embedding with a downstream task in mind . We begin with outlining the assumptions guiding our unsupervised learning method for video segmentation , before delving into the technical specifics of their implementation . We then look in more detail into the mechanisms that our method implements to comply with those assumptions . 3.1 Assumptions . Assumption 1 : Non-local feature diversity . We assume that the distinctness of visual artefacts in a natural scene exhibits spatial correlation : the more distant two visual phenomena are on the pixel grid , the more likely they are to belong to different semantic entities , and vice versa . Dense representations of the images should mirror this property , i. e. be spatially distinguishable . Note that this assumption is weaker than that of spatial smoothness , which is problematic at object boundaries , where it would encourage the same representation of the object and the background . In contrast , non-local feature diversity is agnostic to the spatial arrangement of the objects and the background in the scene . On a feature grid , where every cell corresponds to its own receptive field in the image , we can discriminate the representation on all levels of the semantic hierarchy ( provided sufficient grid resolution ) : between the objects and the background , as well as between object parts . Jabri et al . [ 17 ] implicitly relied on this assumption to spatially distinguish the nodes of the space-time graph . Assumption 2 : Temporal coherence . We posit that the feature representation extracted from one frame in a video should be closer in the embedding space ( e. g. , in terms of the cosine similarity ) to a feature vector from another temporally close frame of the same sequence , rather than to a feature embedding derived from another video . The premise for this assumption is the temporal coherence in videos : the changes in appearance within a video sequence are seldom as significant as between two independent video shots . Also leveraged by Wang et al . [ 38 ] , this is a desirable property for dense tracking , which is our focus in this work . Assumption 3 : Temporal persistence of semantic content . We further assume that the semantic content of video clips remains unchanged , at least for a short time span . Specifically , if we represent a given reference frame with a set of distinct features ( following Assumption 1 ) , the semantic content of temporally close frames can be faithfully represented with the same feature set . Note that the contrastive random walk ( CRW ) [ 17 ] makes a somewhat stronger assumption that every frame in a sequence must contain the same feature set as the reference frame . By contrast , under our assumption every frame may comprise only a ( proper ) subset of the reference feature set . Although the edge dropout technique improves model robustness to partial occlusions in [ 17 ] , following our assumption allows to handle full occlusions , provided the occluded object is visible in the reference frame . Assumption 4 : Equivariance to similarity transformations . Finally , we require the feature representation be equivariant to similarity transformations , i. e. scaling or flipping of an input video frame should produce a correspondingly scaled or flipped feature embedding . This assumption has not been yet explored in dense unsupervised learning from videos , and is in contrast to the common practice of unsupervised learning from static image sets , where two views produced by a random similarity transform need to yield the same global , hence invariant representation [ 8 ] . While Assumptions 1 and 2 have been studied independently in prior work [ 17 , 38 ] , we investigate their combination here , and further complement them with our occlusion-aware variant of Assumption 3 and the yet unexplored Assumption 4 on equivariance . 3.2 Overview . The core of our training procedure is a self-supervised loss imposing four assumed properties on the feature representation . First , we learn to densely represent every video frame in terms of spatially discriminative features ( Assumption 1 ) . Second , the set of discriminative features representing a video is learned to be distinguishable from the set of another video sequence ( Assumption 2 ) . Third , our method learns to represent every frame in a video sequence as a composite of discriminative features extracted from a single reference frame of the same sequence ( Assumption 3 ) . Last , we learn a feature representation satisfying these properties under the equivariance constraint ( Assumption 4 ) . Specifically , we minimise the distance between every pair of spatially corresponding features extracted from two views and related by a similarity transform w. r. t. non-corresponding pairs .

Summary: This paper aims at unsupervised feature learning from videos. The learned representation is tested in a video segmentation/tracking setting on the DAVIS and the Youtube-VOS datasets. Although the paper describes many principles for the learning strategy, I could not find any of them to be new and also the results do not indicate that something fundamental was added.