Dense Unsupervised Learning for Video Segmentation

1 Introduction . Unsupervised learning of visual representations has recently made considerable and expeditious advances , in part already outperforming even supervised feature learning methods [ 8 , 11 , 14 ] . Most of these works , however , require substantial computational resources [ 8 , 11 ] , and only few accommodate one of the most ubiquitous types of visual data : videos [ 13 , 33 ] . In contrast to image sets , video data embeds ample information about typical transformations occurring in nature . Exploiting such cues may allow systems to learn more task-relevant invariances , instead of relying only on hand-engineered data augmentation [ 30 ] . In this work , we propose to learn such invariances with a novel framework for the task of video object segmentation ( VOS ) . In contrast to previous works [ 17 , 19 ] , we learn dense representations efficiently in a fully convolutional manner , previously considered prone to degenerate solutions [ 17 ] . This is achieved by leveraging the assumption of feature equivariance to similarity transformations ( e. g. , multi-scale cropping ) applied to the input frame . Fig . 1 presents an overview of our approach . For each video sequence , we designate one of the frames as the reference frame . We sample a set of features , produced from the reference frame by a fully convolutional network , on a spatially uniform grid , and refer to them as anchors . Using a contrastive formulation [ 12 ] , our approach learns to represent the temporally proximate frames to the reference in terms of these anchors . Importantly , this representation is learned to be equivariant to similarity transformations . Towards this goal , we devise a self-training objective [ 20 ] that generates pseudo labels by leveraging the second , transformed view of the original video sequence . These pseudo labels contain dominant assignments of the features from the second view to the anchors . Following the self-training mechanism , we learn to assign the features in the original view consistently with the transformed view . Our self-supervised loss further disambiguates the anchors themselves spatially and between independent video sequences , while also ensuring their transformation equivariance . Implementing this process in a novel framework , we attain a new state of the art in video segmentation accuracy , with compelling computational and data efficiency . Our framework can be trained on Code ( Apache-2.0 License ) available at https : //github.com/visinf/dense-ulearn-vos . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a single commodity GPU , yet achieves higher segmentation accuracy than previous work , while requiring orders of magnitude less data . Improving the accuracy even further with more data , our approach also exhibits favourable scalability in a more comprehensive testing scenario , which has not yet been shown in prior work [ 17 , 38 ] . 2 Related work . The research domains most relevant to our work are video object segmentation ( VOS ) without supervision and representation learning . VOS . The inference setting of semi-supervised VOS , which is the task considered here , is to densely track object masks provided in the first frame . It is typically approached by learning dense feature representations to establish temporal correspondences for propagating the semantic labels . While most previous methods are supervised by pixel-wise mask annotations [ e. g. , 27 ] , an emerging direction is to leverage unsupervised learning . Our work contributes to these research efforts . The Contrastive Random Walk ( CRW ) of Jabri et al . [ 17 ] and its predecessor [ 39 ] exploit a time-cycle consistency constraint , akin to the forward-backward consistency in unsupervised optical flow estimation [ 24 ] . Other methods [ 18 , 19 ] learn the label propagation process at training time by substituting the target mask ( which is unavailable at training time ) with the image , an idea inspired by video colourisation [ 37 ] . The algorithm for label propagation itself significantly affects the VOS accuracy . Mask propagation with naive pixel-wise correspondences usually benefits greatly from leveraging more sophisticated region-level propagation [ 19 , 21 ] , which is not the focus here . Representation learning . Although previous work on unsupervised representation learning pursued spatially view-invariant objectives [ 5 , 8 , 11 , 14 , 15 , 25 , 36 ] , a concurrent study [ 10 ] finds that equipping these methods with temporally-invariant , or temporally-persistent constraints , yields sizeable gains in terms of accuracy on action recognition benchmarks . These findings align well with previous and concurrent work tailored specifically to learning spatio-temporal representations [ 3 , 9 , 13 , 32 , 33 ] for video recognition and retrieval tasks . These works additionally distinguish the representations of frames from different videos ( or , at different timesteps ) . The limitation of these works in the context of VOS is that they do not learn dense , but global feature representations at the image or video level , and typically require considerable computational budgets . By contrast , Pinheiro et al . [ 31 ] learn dense representations and exploit the equivariance constraint , as in our work . However , their approach is limited to image sets of static scenes , whereas we address videos of ( potentially ) dynamic scenes here . Semi-supervised representation learning . Recent works [ 13 , 41 ] leverage optical flow [ 34 ] to ensure the representation similarity of temporally corresponding features . Such an objective can be alternatively supervised by motion segments [ 28 ] . Spatial equivariance has been also explored in part co-segmentation [ 16 ] , a setup that requires semantically aligned image pairs , as well as in weakly supervised semantic segmentation [ 40 ] and unsupervised domain adaptation [ 1 ] . In contrast to this body of work , we learn to establish temporal correspondences via a suitable feature representations learned in a completely unsupervised way . Clustering . Contrastive learning methods are inherently expensive due to the size of pairwise feature comparisons they consider in training [ 8 , 14 ] . Clustering provides a more computationally efficient alternative [ 6 ] , since pairwise distances need to be computed only w. r. t. a ( moderately sized ) set of pre-defined clusters . Combining these ideas , Caron et al . [ 5 ] propose to cluster the embeddings and then enforce consistency between the cluster assignments by contrastive learning . The main challenge in this approach is to impose an expressive prior on the cluster assignments . For example , a uniform prior prevents trivial solutions where feature assignments collapse to a single cluster . Here , we neither explicitly define the clusters nor the prior . Instead , we sample attractor points , called anchors , directly from the available feature representations using a spatial grid sampling strategy . 3 Method . Learning feature representations from unlabelled data is an inherently ill-posed problem and requires specifying assumptions about what constitutes a useful feature embedding with a downstream task in mind . We begin with outlining the assumptions guiding our unsupervised learning method for video segmentation , before delving into the technical specifics of their implementation . We then look in more detail into the mechanisms that our method implements to comply with those assumptions . 3.1 Assumptions . Assumption 1 : Non-local feature diversity . We assume that the distinctness of visual artefacts in a natural scene exhibits spatial correlation : the more distant two visual phenomena are on the pixel grid , the more likely they are to belong to different semantic entities , and vice versa . Dense representations of the images should mirror this property , i. e. be spatially distinguishable . Note that this assumption is weaker than that of spatial smoothness , which is problematic at object boundaries , where it would encourage the same representation of the object and the background . In contrast , non-local feature diversity is agnostic to the spatial arrangement of the objects and the background in the scene . On a feature grid , where every cell corresponds to its own receptive field in the image , we can discriminate the representation on all levels of the semantic hierarchy ( provided sufficient grid resolution ) : between the objects and the background , as well as between object parts . Jabri et al . [ 17 ] implicitly relied on this assumption to spatially distinguish the nodes of the space-time graph . Assumption 2 : Temporal coherence . We posit that the feature representation extracted from one frame in a video should be closer in the embedding space ( e. g. , in terms of the cosine similarity ) to a feature vector from another temporally close frame of the same sequence , rather than to a feature embedding derived from another video . The premise for this assumption is the temporal coherence in videos : the changes in appearance within a video sequence are seldom as significant as between two independent video shots . Also leveraged by Wang et al . [ 38 ] , this is a desirable property for dense tracking , which is our focus in this work . Assumption 3 : Temporal persistence of semantic content . We further assume that the semantic content of video clips remains unchanged , at least for a short time span . Specifically , if we represent a given reference frame with a set of distinct features ( following Assumption 1 ) , the semantic content of temporally close frames can be faithfully represented with the same feature set . Note that the contrastive random walk ( CRW ) [ 17 ] makes a somewhat stronger assumption that every frame in a sequence must contain the same feature set as the reference frame . By contrast , under our assumption every frame may comprise only a ( proper ) subset of the reference feature set . Although the edge dropout technique improves model robustness to partial occlusions in [ 17 ] , following our assumption allows to handle full occlusions , provided the occluded object is visible in the reference frame . Assumption 4 : Equivariance to similarity transformations . Finally , we require the feature representation be equivariant to similarity transformations , i. e. scaling or flipping of an input video frame should produce a correspondingly scaled or flipped feature embedding . This assumption has not been yet explored in dense unsupervised learning from videos , and is in contrast to the common practice of unsupervised learning from static image sets , where two views produced by a random similarity transform need to yield the same global , hence invariant representation [ 8 ] . While Assumptions 1 and 2 have been studied independently in prior work [ 17 , 38 ] , we investigate their combination here , and further complement them with our occlusion-aware variant of Assumption 3 and the yet unexplored Assumption 4 on equivariance . 3.2 Overview . The core of our training procedure is a self-supervised loss imposing four assumed properties on the feature representation . First , we learn to densely represent every video frame in terms of spatially discriminative features ( Assumption 1 ) . Second , the set of discriminative features representing a video is learned to be distinguishable from the set of another video sequence ( Assumption 2 ) . Third , our method learns to represent every frame in a video sequence as a composite of discriminative features extracted from a single reference frame of the same sequence ( Assumption 3 ) . Last , we learn a feature representation satisfying these properties under the equivariance constraint ( Assumption 4 ) . Specifically , we minimise the distance between every pair of spatially corresponding features extracted from two views and related by a similarity transform w. r. t. non-corresponding pairs .

Summary: This paper proposes a method for self-supervised pretraining on video:  Given a batch of video clips, dense per-frame features are extracted from a backbone architecture like ResNet. The objective is to apply a standard contrastive penalty where a softmax loss picks out a positive pair of feature embeddings amidst a large set of negatives. Rather than compare every feature vector to every other vector in the batch (across samples, time, and space) a subset of representative vectors are chosen for comparison. Starting from BxTxHxW features, we choose from each sample a single frame and a subsampled grid of values to get BxNxN "anchors" to compare to. The full set of BxTxHxW features will each be compared to all of the sampled BxNxN anchors.  To apply the loss, for each feature we need one of the comparison anchors to serve as a positive target. This is done by finding the closest match to features from an augmented version of the same clip. That is, another inference pass of the backbone takes in a randomly resized/cropped/flipped version of the clip. Those features are compared to the anchors to find the argmax closest match. This creates a pseudo-ground truth label used to supervise the original features. Importantly, the gradient is stopped so nothing is backpropped through the pseudo-label generation.  Pretraining is done with the proposed set up and evaluated on the downstream setting of unsupervised mask propagation on DAVIS and YT-VOS.