Watching the World Go By: Representation Learning from Unlabeled Videos

1 INTRODUCTION . The world seen through our eyes is constantly changing . As we move through the world , we see much more than a single static image : objects rotate revealing occluded regions , deform , the surroundings change , and we ourselves move . Our internal visual systems are constantly seeing temporally coherent images . Yet many popular computer vision models learn representations which are limited to inference on single images , lacking temporal context . Representations learned from static images are inherently limited to an understanding of the world as many unrelated static snapshots . This is especially true of recent unsupervised learning techniques ( Bachman et al. , 2019 ; Chen et al. , 2020a ; He et al. , 2020 ; Hénaff et al. , 2019 ; Hjelm et al. , 2019 ; Misra & Maaten , 2020 ; Tian et al. , 2019 ; Wu et al. , 2018 ) , all of which train on a set of highly-curated , well-balanced data : ImageNet ( Deng et al. , 2009 ) . Scaling up these techniques to larger , less-curated datasets like Instagram1B ( Mahajan et al. , 2018 ) has not provided large improvements in performance ( He et al. , 2020 ) . Only so much can be learned from a single image : no amount of artificial augmentation can show a new view of an object or what might happen next in a scene . This dichotomy can be seen in Figure 1 . In order to move beyond this limitation , we argue that video supplies significantly more semantically meaningful content than a single image . With video , we can see how the world changes , find connections between images , and more directly observe the underlying scene . Prior work using temporal cues has shown success in learning from unlabeled videos ( Misra et al. , 2016 ; Wang et al. , 2019 ; Srivastava et al. , 2015 ) , but has not been able to surpass supervised pretraining . On the other hand , single image techniques ( Gutmann & Hyvärinen , 2010 ) have shown improvements over stateof-the-art by using Noise Contrastive Estimation ( NCE ) . In this work , we merge the two concepts with Video Noise Contrastive Estimation ( VINCE ) , a method for using unlabeled videos as a basis for learning visual representations . Instead of predicting whether two feature vectors come from the same underlying image , we task our network with predicting whether two images originate from the same video . Not only does this allow our method to learn how a single object might change , it also enables learning which things might be in a scene together , e.g . cats are more likely to be in videos with dogs than with sharks . Additionally , we generalize the NCE technique to operate on multiple positive pairs from a single source . To facilitate this learning , we construct Random Related Video 1Code and the Random Related Video Views dataset will be made available . Views ( R2V2 ) , a set 960,000 frames from 240,000 uncurated videos . Using our learning technique , we achieve across-the-board improvements over the recent Momentum Contrast method ( He et al. , 2020 ) as well as over a network pretrained on supervised ImageNet on diverse tasks such as scene classification , activity recognition , and object tracking . 2 RELATED WORK . 2.1 NOISE CONTRASTIVE ESTIMATION ( NCE ) . The NCE loss ( Gutmann & Hyvärinen , 2010 ) is at the center of many recent representation learning methods ( Bachman et al. , 2019 ; Chen et al. , 2020a ; He et al. , 2020 ; Hénaff et al. , 2019 ; Hjelm et al. , 2019 ; Misra & Maaten , 2020 ; Tian et al. , 2019 ; Wu et al. , 2018 ) . Similar to the triplet loss ( Chechik et al. , 2010 ) , the basic principle behind NCE is to maximize the similarity between an anchor data point and a positive data point while minimizing similarity to all other ( negative ) points . A challenge for using NCE in an unsupervised fashion is devising a way to construct positive pairs . Pairs should be different enough that a network learns a non-trivial representation , but structured enough that the learned representation is useful for downstream tasks . A standard approach used by ( Bachman et al. , 2019 ; Chen et al. , 2020a ; He et al. , 2020 ) is to generate the pairs via artificial data augmentation techniques such as color jitter , cropping , and flipping . Contrastive Multiview Coding ( Tian et al. , 2019 ) uses multiple “ views ” of a single source image such as intensity ( L ) , color ( ab ) , depth , or segmentation , training separate encoders for each view . PIRL ( Misra & Maaten , 2020 ) uses the jigsaw technique ( Noroozi & Favaro , 2016 ) to break the image into non-overlapping regions and learns a shared representation for the full image and the shuffled image patches . Similarly , Contrastive Predictive Coding ( CPC ) ( Hénaff et al. , 2019 ) uses crops of an image as “ context ” and predicts features for the unseen portions of the image . We provide a more natural data augmentation by using multiple frames from a single video . As a video progresses , the objects in the scene , the background , and the camera itself may move , providing new views . Whereas augmentations on an image are constrained by a single snapshot in time , using different frames from a single video gives entirely new information about the scene . Additionally , rather than restricting our method to only use two frames from a video , we generalize the NCE technique to use many images from a single video , resulting in more computational reuse and a better final representation ( Bachman et al . ( 2019 ) similarly makes multiple comparisons per pair , but each anchor has only one positive ) . 2.2 UNSUPERVISED LEARNING USING VIDEO CUES . In contrast with supervised learning which requires hand-labeling , self-supervised and unsupervised learning acquire their labels for free . These techniques create datasets which are orders of magnitude larger than comparable fully-supervised datasets . Whereas self-supervised learning requires extra setup during data generation ( Godard et al. , 2017 ; Pinto et al. , 2016 ; Schmidt et al. , 2016 ) , unsupervised learning can use existing data without any specific data generation constraints . Unsupervised single image methods such as auto-encoders ( Kramer , 1991 ) , colorization ( Zhang et al. , 2016 ) , GANs ( Radford et al. , 2015 ) , jigsaw ( Noroozi & Favaro , 2016 ) , and NCE ( Wu et al. , 2018 ) rely on properties of the images themselves and can be applied to arbitrary datasets . However these datasets can not represent temporal information , nor can they show novel object views or occlusions . Video data automatically provides temporal cohesion which can be used as additional supervisory signal to learn these phenomena . There is a long history of using videos for low level ( Li et al. , 2019 ; Meister et al. , 2018 ; Srivastava et al. , 2015 ) and high-level tasks ( Misra et al. , 2016 ; Wang et al. , 2019 ) . One of the most common unsupervised setups is using the present to predict the future . The Natural Language Processing community has embraced language modeling as an unsupervised task which has resulted in numerous breakthroughs ( Devlin et al. , 2019 ; Mikolov et al. , 2013 ; Radford et al. , 2019 ) . However , similar systems applied to unlabeled videos have not revolutionized computer vision . These representations still underperform supervised methods due to several issues . Primarily , neighboring video frames do not change nearly as much as neighboring words in a sentence , so a network which learns the identity function would perform well at next frame prediction . Additionally , words are reused and can thus be tokenized in an effective way whereas images never repeat , especially between two disparate video sources . To avoid these issues , many have opted for other methods . Anand et al . ( 2019 ) use the NCE loss to discriminate between temporally near frames and temporally far frames of ATARI gameplay but do not compare across games . Han et al . ( 2019 ) also use the NCE loss and the CPC technique on a 3D-ResNet to learn spatio-temporal features . Similar to our proposal , Tschannen et al . ( 2020 ) and Purushwalkam & Gupta ( 2020 ) advocates using videos to learn invariances to deformations , color changes , occlusions , and other difficult scenarios which preserve the semantic relationships of the video subjects but drastically change the appearance . These works provide additional evidence that videos offer a richer visual backing as compared with single image methods . Aside from the NCE approach , others have proposed alternative video training tasks . Misra et al . ( 2016 ) shuffle the frames of a video and train a network to predict whether they are correctly temporally ordered . Wang Wang et al . ( 2019 ) and Vondrick et al . ( 2018 ) use cycle consistency and color as a form of tracking from one frame onto another . Earlier work from Wang & Gupta ( 2015 ) uses hand-crafted features to track patches of a video and learn similarities between the patches . Others have explored learning representations from multi-modal inputs provided by video such as optical flow or audio Korbar et al . ( 2018 ) ; Owens & Efros ( 2018 ) ; Piergiovanni et al . ( 2020 ) ; Zhao et al . ( 2018 ) . Our approach is inspired by these works but focuses on learning a semantic representation of the entire scene based on a single frame from a video . Many of these prior works require multiple frames for their representation , and thus can not be used on single-image downstream tasks . However , if a network can consistently represent visually dissimilar images from the same video with similar vectors , then not only has it learned how to recognize what is in each image , but it can also represent what might happen in the past or future of that scene . 3 METHODS . In order to learn a semantically meaningful representation , we exploit the natural augmentations provided by unlabeled videos . In this section , we first outline the dataset generation process . We then describe the learning algorithm used to train our representation . 3.1 DATASET . Using ImageNet as a basis for representation learning has shown remarkable success both with supervised pretraining as well as unsupervised learning . However , even without labels , the images of ImageNet have been hand selected and are unnaturally balanced . To improve learned representations using existing techniques may require significantly larger datasets ( He et al. , 2020 ) , but obtaining data with similar properties automatically and at scale is not practical . Instead , we turn to unlabeled videos as a source of additional supervision . In order to train on a diverse set of realistic video frames , we collect a new dataset which we call Random Related Video Views ( R2V2 ) . We use the following fast and automated procedure to generate the images in our dataset . 1 . Use YouTube Search to find videos for a set of queries , and download the top K videos licensed under the Creative Commons . In practice we use the ImageNet 1K classes . 2 . Filter out videos with static images . 3 . Pick a random point in the video and extract T images with a gap of G frames between each image . In practice T = 4 and G = 150 . Using this procedure , we are able to construct R2V2 in under a day on a single machine . Because our dataset is constructed automatically , we can easily gather more data ( more frames per video , more videos overall ) . In this work we limit the scale to roughly that of comparable datasets ( see Table 1 ) . More details on data collection are provided in Appendix 1 .

Summary: The idea of learning representations from video rather than single images is an appealing one with many favorable properties to allow a system to get direct signal on appearance of objects under various natural transformations (occlusion, lighting, etc). Combining instance discrimination ideas of loss based on unlabelled images for which it is known whether they are similar or not, with the idea of curating the images from video is hypothesized to yield learned representations that capture properties enabling improved performance across a variety of single image tasks. The authors create a dataset based on video with positive pairs for noise contrastive estimation, conduct fairly comprehensive experiments and promise to make their newly constructed dataset available. The experiments showcase this type of learned representation outperform alternatives not based on videos on a variety of tasks.