Watching the World Go By: Representation Learning from Unlabeled Videos
Authors: Daniel Gordon
Kiana Ehsani
Dieter Fox
Ali Farhadi
Venue:  (2021)
Keywords: Representation Learning
Unsupervised Learning
Video Analytics

Abstract: Recent unsupervised representation learning techniques show remarkable success on many single image tasks by using instance discrimination:  learning to differentiate between two augmented versions of the same image and a large batch of unrelated images.  Prior work uses artificial data augmentation techniques such as cropping, and color jitter which can only affect the image in superficial waysand are not aligned with how objects actually change e.g. occlusion, deformation,viewpoint change.  We argue that videos offer this natural augmentation for free.Videos can provide entirely new views of objects,  show deformation,  and even connect semantically similar but visually distinct concepts.We propose Video Noise Contrastive Estimation,  a method for using unlabeled video to learnstrong, transferable, single image representations.We demonstrate improvements over recent unsupervised single image techniques,as well as over fully supervised ImageNet pretraining, across temporal and non-temporal tasks.

Under review as a conference paper at ICLR 2020
WATCHING THE WORLD Go BY: REPRESENTATION LEARNING FROM UNLABELED VIDEOS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent unsupervised representation learning techniques show remarkable success on many single image tasks by using instance discrimination: learning differ- entiate between two augmented versions of the same image and a large batch of unrelated images. Prior work uses artificial data augmentation techniques such cropping, and color jitter which can only affect the image in superficial ways and are not aligned with how objects actually change e.g. occlusion, deformation, viewpoint change. We argue that videos offer this natural augmentation for free. Videos can provide entirely new views of objects, show deformation, and even connect semantically similar but visually distinct concepts. We propose Video Noise Contrastive Estimation, a method for using unlabeled video to learn strong, transferable, single image representations. We demonstrate improve- ments over recent unsupervised single image techniques, as well as over fully supervised ImageNet pretraining, across temporal and non-temporal tasks
INTRODUCTION
The world seen through our eyes is constantly changing. As we move through the world, we see much more than single static image: objects rotate revealing occluded regions, deform, the sur- roundings change, and we ourselves move. Our internal visual systems are constantly seeing tem- porally coherent images. Yet many popular computer vision models learn representations which limited to inference on single images, lacking temporal context, Representations learned from static images are inherently limited to an understanding of the world as many unrelated static snapshots.
This is especially true of recent unsupervised learning techniques (Bachman et al. 2019 Chen al. 2020al He et al. 2020;
Hénaff al., 2019 Hjelm et al.. 2019 Misra & Maaten [2020] Tian
et al. 2019 Wu et al. 2018)) all of which train on set of highly-curated well wall-balanced data: Ima-
geNet (Deng et al. | 2009). Scaling up these techniques to larger, less-curated datasets like Instagram-
1B (Mahajan et al. 2018) has not provided large improvements in performance (He et al.) 2020
Only so much can be learned from single image: no amount of artificial augmentation can show a new view of an object or what might happen next in scene. This dichotomy can be seen in Figure] I
In order to move beyond this limitation, we argue that video supplies significantly more semanti- cally meaningful content than single image. With video, we can see how the world changes, find connections between images, and more directly observe the underlying scene. Prior work using temporal cues has shown success in learning from unlabeled videos (Misra et al. 2016 Wang al. 2019 Srivastava et al. 2015), but has not been able to surpass supervised pretraining. On the other hand, single image techniques (Gutmann & Hyvärinen 2010) have shown improvements over state- of-the-art by using Noise Contrastive Estimation (NCE). In this work, we merge the two concepts with Video Noise Contrastive Estimation (VINCE), a method for using unlabeled videos basis for learning visual representations. Instead of predicting whether two feature vectors come from the same underlying image, we task our network with predicting whether two images originate from the same video. Not only does this allow our method to learn how single object might change, it also enables learning which things might in a scene together, e.g. cats are more likely to be in videos with dogs than with sharks. Additionally, we generalize the NCE technique to operate on multiple positive pairs from a single source. To facilitate this learning, we construct Random Related Video
"ooee and the Random Related Video Views dataset will be made available.

---

Under review as a conference paper at ICLR 2020
Standard Contrastive Loss
Video Noise Contrastive Estimation
8
Image Augmentations
Culor, Crop,
Temporal Changes
Defommation, New Views, Related Cibjects
Figure I: NCE vs VINCE. The standard contrastive setup learns to separate artificial augmentations of the same image. Our method uses novel views and temporal consistency which single images cannot provide.
Views (R2V2), set 960,000 frames from 240,000 uncurated videos. Using our learning technique, we achieve
improvements over the recent Momentum Contrast method (He et al. 2020) well network pretrained supervised ImageNet diverse tasks such scene classification, activity recognition, and object tracking.
RELATED WORK
2.1
NOISE CONTRASTIVE ESTIMATION (NCE)
The NCE loss (Gummann & Hyvärinen 2010) the center of many recent representation learning
methods (Bachman et al.] 2019 Chen et al. [2020a] He et al.) 2020 Hénaff et al. [219 [Hjelm et al.
2019
Misra & Maaten [2020] laa et al. 2019 Wu et al.] 2018). Similar the triplet loss (Chechik)
al. 2010), the basic principle behind NCE to maximize the similarity between an anchor data
point and a positive data point while minimizing similarity to other (negative) points.
A challenge for using NCE in an unsupervised fashion is devising a way to construct positive pairs. Pairs should be different enough that a network learns a non-trivial representation, but structured enough that the learned representation is useful for downstream tasks. A standard approach used by (Bachman et al. 2019 Chen et al. 2020a He et al., 2020) is to generate the pairs via artificial data augmentation techniques such color jitter, cropping, and flipping. Contrastive Multiview Coding (Tian et al.
2019) uses multiple "views" of a single source image such as intensity (L), color (ab), depth, segmentation, training separate encoders for each view. PIRL Missa & Maaten 2020) uses the jigsaw technique (Noroozi & Favaro 2016) to break the image into non-overlapping regions and learns shared representation for the full image and the shuffled image patches. Simi- larly, Contrastive Predictive Coding (CPC) (Hénatf et al. 2019) uses crops of an image as "context" and predicts features for the unseen portions the image. We provide more natural data augmen- tation by using multiple frames from single video. As video progresses, the objects in the scene, the background, and the camera itself may move, providing new views. Whereas augmentations on an image constrained by single snapshot in time, using different frames from a single video gives entirely new information about the scene. Additionally, rather than restricting our method to many images from
only two frames from video, generalize the NCE technique single video, resulting in more computational reuse and a better final representation (Bachman et al. 2019) similarly makes multiple comparisons per pair, but each anchor has only positive).
2

---

Under review as a conference paper at ICLR 2020
2.2 UNSUPERVISED LEARNING USING VIDEO CUES
In contrast with supervised learning which requires hand-labeling, self-supervised and unsupervised learning acquire their labels for free. These techniques create datasets which are orders of magni- tude larger than comparable fully-supervised datasets. Whereas self-supervised learning requires extra setup during data generation (Godard al., 2017 Pinto et al.) 2016 Schmidt et al. 2016). unsupervised learning can use existing data without any specific data generation constraints. Un- supervised single image methods such as auto-encoders (Kramer 1991] I. colorization (Zhang al. 2016), GANs (Radford et al. 2013), jigsaw (Noroozi & Favaro 2016). and NCE (Wu et al. 2018) rely on properties of the images themselves and can be applied to arbitrary datasets. However these datasets cannot represent temporal information, nor can they show novel object views or occlusions.
Video data automatically provides temporal cohesion which can used as additional supervisory signal to learn these phenomena. There is a long history of using videos for low level (Li et al. 2019 Meister et al. 2018 Srivastava et al., 2015) and high-level tasks (Misra et al. [016 Wang al. 2019). One of the most common unsupervised setups is using the present predict the future. The Natural Language Processing community has embraced language modeling as an unsupervised task which has resulted in numerous breakthroughs (Devlin al.] [219 Mikolov et al., 2013) Rad- ford et al. 2019). However, similar systems applied to unlabeled videos have not revolutionized computer vision. These representations still underperform supervised methods due several issues. Primarily, neighboring video frames do not change nearly as much as neighboring words in sen- tence, so network which learns the identity function would perform well at next frame prediction. Additionally, words are reused and can thus be tokenized in an effective way whereas images never repeat, especially between two disparate video sources.
To avoid these issues, many have opted for other methods. [Anand et al. (2019) use the NCE loss to discriminate between temporally near frames and temporally far frames of ATARI gameplay but do not compare across games. Han et al. (2019) also use the NCE loss and the CPC technique 3D-ResNet to learn spatio-tempporal features. Similar to our proposal, Tschannen et al. (2020) and Purushwalkam & Gupta (2020) advocates using videos to learn invariances to deformations, color changes, occlusions, and other difficult scenarios which preserve the semantic relationships of the video subjects but drastically change the appearance. These works provide additional evidence that videos offer a richer visual backing as compared with single image methods.
Aside from the NCE approach, others have proposed alternative video training tasks. Misra et al. (2016) shuffle the frames of video and train a network to predict whether they are correctly tempo- rally ordered. Wang Wang et al. (2019) and Vondrick et al. (2018) use cycle consistency and color as a form of tracking from one frame onto another. Earlier work from Wang & Gupta (2015) uses hand-erafted features to track patches of video and learn similarities between the patches. Others have explored learning representations from modal inputs provided by video such as optical flow or audio Korbar et al.) (2018;; Owens & Efros (018): |Piergiovanni (018)). Our approach is inspired by these works but focuses on learning a semantic representation of the entire scene based on a single frame from a video. Many of these prior works require multiple frames for their representation, and thus cannot be used single-image downstream tasks. How- ever, a network can consistently represent visually dissimilar images from the same video with similar vectors, then not only has it learned how to recognize what is in each image, but it can also represent what might happen in the past or future of that scene.
al. 2020); Zhao et al.
METHODS
In order learn semantically meaningful representation. exploit the natural augmentations provided by unlabeled videos. In this section, we first outline the dataset generation process. We then describe the learning algorithm used to train representation.
3.1 DATASET
Using ImageNet as a basis for representation learning has shown remarkable success both with supervised pretraining as well unsupervised learning. However, even without labels, the images of ImageNet have been hand selected and are unnaturally balanced. To improve learned representations using existing techniques may require significantly larger datasets (He et al. 2020), but obtaining
3

---

Under review as a conference paper at ICLR 2020
Dataset
Number of Images (Train)
Number Vidcos (Train)
Number Categorics
Mean Image Size
ImageNet 1K (Deng et 2009
1.3 M
D
1000
(428, 406)
YouTube 8M
Hama 2016
3.7
Kinerics and
Kay er al
0
220 K
400
GOT-10k (Hling
2019
14M
9 K
563
(1600, 912) (467.
Relateo Vioco (Uur)
0.96
2
Table 1:
Comparison of various image and video datasets. While we have neither the most images nor the most videos, we provide good diversity between videos which is crucial for learning strong, generic image representation.
data with similar properties automatically and scale not practical. Instead, we turn to unlabeled videos as a source of additional supervision.
In order to train on a diverse set of realistic video frames, we collect new dataset which we call Random Related Video Views (R2V2). We use the following fast and automated procedure to gen- erate the images in our dataset.
1. Use YouTube Search to find videos for set of queries, and download the top K videos licensed under the Creative Commons. In practice we use the ImageNet IK classes.
2. Filter out vidcos with static images.
3. Pick random point in the video and extract T' images with a gap of G frames between each image. In practice T and G 150.
Using this procedure, we are able to construct R2V2 in under day on single machine. Because our dataset is constructed automatically, we can easily gather more data (more frames per video, more videos overall). In this work we limit the scale to roughly that of comparable datasets (see Table T. More details on data collection are provided in Appendix ]
3.2 NOISE CONTRASTIVE ESTIMATION (NCE) LEARNING
Given a dataset of diverse video frames, we learn representation which takes advantage of the structure of the data. We choose the Noise Contrastive Estimation technique (Gutmann & Hyväri-
nen 2010) which has been popular in many recent works (Bachman al. 2019 Chen et al. 2020a
He et al. 2020 Hénaff et al. 2019: Hjelm et al.] [2019] Misra & Maaten 2020 Tian et al. 2019 Wu al. 2018), augmented with temporal supervision.
The standard NCE implementation (used in (Chen et al. 2020a Bachman et al. 2019 Hjelm et al. ) uses the following procedure. First, batch of anchor images A is selected. Second, batch of positive images P is selected, one for each anchor. Positive matches for one example are reused as negatives for the other samples without the need recompute the features. The NCE loss for a single batch is shown in Equation [] where sim(X, Y) is any similarity metric between the two inputs. This similarity in a learned feature space rather than directly in image space, and this leamed feature encoding is what we use on downstream tasks. The NCE Loss produces strong representational encoding for inputs by maximizing the similarity of the encoding of positive pairs while minimizing similarity of negative ones, encouraging the encoder to be discriminative over many possible matches, but general in that it must work on any inputs. As in other works, we use the dot product of the feature embeddings of the data points (as seen in Equation 2) as the similarity metric due to its computational efficiency (Bachman et al. 2019 Chen et al., [2000a] He et al. 2020 HHénaff et al. 2019 Misra & Maaten 2020 Wu et al. 2018). The similarity is rescaled by
temperature scalar T to create peaked softmax distributions.
1 log LNEE
si.(AA.P.))
i=1 3 (aim(A..P.)
n
S(X)
g(Y)
sim(X,y) ))=
[[((X]]
[
3.3 MULTI-FRAMM NCE
All recent works perform some sort of transformation on a single image to create Anchor-Positive pairs for the NCE loss (Bachman al. 2019 Chen al. 2020a He et al. 2020 Hénaff al. 2019
4

---

Under review as a conference paper at ICLR 2020
Standard NCE
Multi-Frame NCF
Multi-Frime Multi-Pair NCE
Sume Finme Posiiives
Multi Frame Posiiives
Cumrnt Posilives
MoCo MemBank
N2
Ng
Ny.. N2
Andiers
Anchoms
Figure 2: Left: Standard NCE using "Same Frame" where all positive pairs come from the same image. Middle: NCE using "Multi-Frame" where positive pairs come from the same video. Right: Multi-Frame Multi-Pair NCE which uses more than one positive pair per video. The gray boxes indicate the true match pairs. The Memory Bank adds more negatives for each anchor.
Mísra & Maaten 2020 Wu al., 2018). We refer to this as "Same Frame." We differ from these works by using multiple images from single video to form our pairs. This allows our network to see truly different views, deformations, similar objects, and larger scene changes. Additionally, this encodes temporal consistency For example in a video of two cats playing, the camera may focus on one cat, or may even pan to previously unseen dog, but it is unlikely pan shark. Note that in practice, select frames with replacement, making our potential pairs a strict superset of those in prior works.
the semantic contents of video are unlikely to change suddenly.
3.4 MEMORY BANKS AND MOMENTUM CONTRAST
NCE-based methods benefit greatly from large pools of negatives because this increases the like- lihood of finding at least one hard negative for each positive example. In some works (Misra & Maaten 2020 Wu al. 2018), negatives sampled from large memory bank which filled with earlier outputs of the network. The NCE loss can be modified to use negatives from prior batches as shown in equation 3 for memory bank of negatives NI...m
n Σ
(i.
LCCE
log prim(A..P)) + E/= (sim((A,NN)
72
As the network trains, its output distribution will shift. A potential issue when using memory bank is the network learns simple classifier between the current distribution and an old one. Momentum Contrast (MoCo) (He al. (020) alleviates this issue by using a quickly updating primary network (f in equation 2 and a slowly updating secondary network (g). f is updated based on the NCE loss in equation and g is updated using momentum rule g t ag + (1 a. The memory bank is filled with previous outputs from the slowly changing network reducing the likelihood that f will be able to learn simple recent batch/old batch classifier. For more details, see (He al. 2020
3.5 MULTI-PAIR NCE
By using a memory bank, we increase the number of negatives without a large computational cost. Yet standard NCE only n positive pairs per batch of size We can further increase the number of positives per batch (while holding batch size constant) by selecting e videos and i: samples from each video where k P. By computing the pairwise similarity between each pair, we reuse each
positive sample h times, resulting in k2 .!" positives per batch. Using a simple biock-diagonal mask as shown in Figure 2 we can efficiently compute the similarities and NCE loss both between elements of the batch and across memory bank, achieving large number of positive comparisons per batch while retaining a large negative size. In practice, we notice no meaningful computational cost to this approach. We refer to this full method using Multi-Pair video data and the Multi- Frame learning procedure as Video Noise Contrastive Estimation (VINCE). See Appendix 2 for detailed pseudocode.
5

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
We evaluate our method (VINCE) on both single image and temporal tasks by freezing our learned representation and adding small network (in most cases single linear layer) for adaptation to new end-tasks. Our learned representation transfers well to variety of visual tasks, especially tasks which require temporal reasoning. To show this, we compare with multiple strong baselines:
MoCo-IN: Network pretrained on ImageNet (Deng al. 2009) using the MoCo algorithm.
MoCo-R2V2 Network pretrained on R2V2 using the MoCo algorithm. This uses exactly the same data as VINCE but the Same Frame technique described in Sec. [.3] We also prevent multiple images from the same video being in the Memory Bank at the same time,
Sup-IN: Network pretrained on fully supervised ImageNet.
To validate the benefits of learning from videos, we additionally compare against an unsupervised image dataset. Since ImageNet itself required time, effort, and money to create, construct new static image dataset analogous to our video dataset. Specifically, we search Google Images for the ImageNet synsets and download the top K results for each category. We refer to this as MoCo- G). Although other unsupervised video- based methods exist, they have generally underperformed pretrained supervised ImageNet weights are only applicable to video tasks.
4.1 TARGET TASKS
We compare each method on several diverse end-tasks using both ResNet18 (He et al. 2016) and ResNet50 backbones. More training implementation details can be found in Appendix 3) Results for these tasks shown in T We train all end-task models using Adam (Kngma & Ba 2014) and a shared learning rate schedule per task. For each dataset, we use standard data augmentation approaches (crop, flip, color jitter except for tracking). One overall trend note is the relative gain over MoCo-R2V2. If the single-frame algorithm performed as well as our multi-frame method on temporal tasks. it would indicate minimal temporal understanding. However, the relative gain of VINCE over MoCo-R2V2 on Kinetics (13.85% and tracking (13.33% and 15.38%) are higher than those on single-frame frame tasks, showing that our method can incorporate temporal cues.
ImageNet: For this task, we use our frozen learned representations, adding single linear layer after the global average pool. Although none of the methods match the fully supervised performance of ResNet on ImageNet, they do achieve reasonable performance given only single linear layer. It unsurprising that MoCo pretrained on ImageNet images (MoCo-IN) outperforms our method (0.447 vs 0.400) due to the domain shift between pretrain and end-task. However MoCo pretrained on R2V2 (MoCo-R2V2) suffers nearly a 2x drop in accuracy (8.9%) compared to our method (4.7%), indicating that pretraining multi-frame matching provides clear benefit over single frame pairs. Our method, which has never seen an image from ImageNet before, still learns a representation which generalizes well to this new type of data. Even drawing images from a similar class distribution (MoCo-G) does not outperform our method.
SUN Scenes: SUN Scenes is a classification dataset in which each image is categorized into one of 397 possible scene types such as airplane cabin, bedroom, and coffee shop. Again we train a single linear layer on top of each pretrained network. This data is quite similar ImageNet in that each image is well-curated and contains single, unambiguous subjects. As such, the ImageNet fully supervised baseline transfers quite well to SUN Scenes. However VINCE outperforms Sup-IN by small margin. Again we note a large improvement of VINCE over MoCo-R2V2 (0.495 vs. 0.450). This shows that our method learns to recognize not just the main subject of an image but also the surrounding scene, which requires a richer understanding of the world.
Kinetics 400: This dataset consists of 10 second clips from YouTube videos and action labels for each segment. We first download each video and subsample each clip to one frame per second (10 frames per clip). We train a single layer LSTM (Hochreiter & Schmidhuber, 1997) followed by a single linear layer to predict the action category for each segment. Kinetics acts as crucial test to evaluate whether our model learns temporal cues. VINCE greatly outperform all other methods, whereas traditional baselines such fine-tuning supervised ImageNet do not adapt well at all. This shows that contrary to popular belief, representations pretrained on ImageNet many not be a good fit for other visual domains, especially on temporal tasks.

---

Under review as a conference paper at ICLR 2020
Imagee Classification Nmege:Ned Deng al. al. 2009)
Scene Clussifiction
Action Recognition
Tracking
SUN Sccnes Niño et al, 2010
Kinetles 4000 Kay et 2017
OTB 2015 Wu et al. 2015
Trained Layeris)
Lineur
Linear
LSTM - Linear Accuracy (Top 1)
[xl Conv Success
Mesric
Accuncy (Top 1)
Accarccy (Top
Precision
Sup-IN
0.696
0.491
01.007
0.557
0.396
RosNet
MoCo-IN
0.447
0.487
0.336
0.583
0.429
MoCo-Ei
0.313
0.551
0.413
McCo C2Co-R2V2
0.318
VINCE(OWs))
0,400
0.495
0.362
0.629
0.465
Relative over McCo-R2V2
11.91%
9.93%
13,85%
13.33%
15.38%
RosNe50
0.762
0.593
0.303
0.458
0.320
MrCo-V2-IIN impl)
0.652
0.450
McCo B2Co
0.536
0.456
(Owc)(Ows)
0.544
0.491
0.401
0.300
McCo-R2V2
Relative over
1.36%
5.29%
7.72%
4.15%
0.33%
Table 2: Comparison of representation performance across a variety of end tasks. We show improvements over MoCo trained on the same data all tasks, and outperform MoCo trained on ImageNet as well as supervised pretraining on ImageNet on all tasks but ImageNet itself (and tracking for ResNet50). Each representation uses the same ResNet convolutional backbone, sharing weights across all tasks. Linear (for Kinetics LSTM - Linear) classifiers are the only learned weights for each end task.
Object Tracking Benchmark (OTB) 2015: In this dataset, given an initial bounding box around an arbitrary object in the first image, model must locate the object in the following frames. Difficult tracking datasets like OTB 2015 implicitly require robustness to occlusion and appearance change of the target object, so improved performance would indicate representation's 's relative robustness to these challenges. We use the SiamFC (Bertinetto al., 2016) tracking algorithm on top of our learned representation. SiamFC localizes an object of interest by convolving deep features from the initial bounding box with the current image, similar to template matching (Briechle & Hanebeck 2001) but in deep feature space. For a more complete explanation, see (Bertinetto et al.] 2016). OTB is evaluated using two metrics precision and success. Precision measures the percentage of frames where the (normalized) center error is less than certain threshold, averaging over many thresholds. Similarly, success measures the percentage of frames where the Intersection Over Union is more than range of thresholds.
representation which works well for SiamFC would have the property that the cross correla- tion of two images of the same object is high, but the cross correlation of two different objects, or a poorly cropped image of the same object, is low. Pretraining our representations on multiple frames from the same video coincides well with the first objective, however since we use cropped data augmentations, the representations tend to be somewhat invariant to poorly-cropped candidates. Still, the models perform quite well across a variety of difficult tracking instances. Our ResNet18 model transfers significantly better than all other methods indicating a clear benefit to using tem- poral cues during pretraining. We find, somewhat unexpectedly, that the ResNet18 network fares better than the ResNet50. A likely explanation for this is that the original SiamFC method uses AlexNet (Krizhevsky et al. 2012) with no padding in a fully-convolutional manner. When using ResNet, padding must be applied to keep the outputs the same dimensionality as the inputs. Thus, at training time the network may latch onto zero- aero-padding cues which will not be applicable at test time. This becomes more of an issue the larger the receptive field which is why ResNet50 struggles but R8 is somewhat less affected.
4.2 METHOD ABLATION
We validate the effectiveness of Multi-Frame (Sec. the number of images from each video used in a batch of comparisons shown in Table 3 (due to computational constraints, we only perform ablations on ResNet18). The first row is equivalent to the procedure done in MoCo (He et al. 2020) i.e. the anchor and positive pairs are two data augmentations of the same image. The second row uses the MoCo procedure as well, however the anchors and positives may be from different images from the same video. The third row uses our Multi-Pair NCE method taking positives and 4 anchors from each video, resulting in 16 positive pairs. A pictorial representation can be seen in Figure 2 Note that when selecting images for row and 3, we use sampling with replacement, making our method a strict super-set of MoCo.
3.3 and Multi-Pair (Sec. 3.5 learning by ablating
7

---

Under review as a conference paper at ICLR 2020
Test Task
Images Per Video
ImageNet
SUN Scene
Kinetics 400
OTB 2015 Precision
OTB 2015 Success
Same Frame
0.450
0.318
0.403
Multi-Frame
0.381
0.478
0.361
0,622
0.464
Multi-Frame Multi-Pair
0.400
0.495
0.362
0.629
0.465
Table 3: Method ablation for VINCE. We compare using one source image with two augmenta- tions (the standard approach), two different images, or set of different images. Using Multi-Frame results in large boost across the board. Multi-Frame Multi-Pair further increases the power of the representation. Note that all methods use the entire dataset, but only Multi-Frame methods use multiple images from video within one batch.
Test Task
Pretraining Data
ImageNet
SUN Scene
Kinetics 400
OTB 2015 Precision
OTB 2015 Success
R2V2 IN-Queries
0.400
0.495
0.362
0.629
0.465
R2V2 YT8M URLs
0.367
0.478
0,343
0.667
0.492
Kinetics 400 URLS
0.368
0.494
0.390
0.612
0.456
Table 4: Pretraining data ablation for VINCE. Each method uses the same training setup but different training data. Since R2V2 uses ImageNet search queries, it outperforms the others on ImageNet. Similarly, pretraining on Kinetics increases performance on the Kineties task.
We observe across-the- thoaodd improvements from both modifications the MoCo approach. The majority of the improvement comes from using two non-identical frames for matching, but we still gain additional improvement from using Multi-Pair NCE. Our intuition that using the Multi- Pair NCE creates gradients that pull each feature towards global video representation whereas the standard NCE remains more mnstance-based. only moving a representation in one direction at time. Thus, we would expect the Multi-Pair NCE features to be more holistically semantic whereas the standard NCE may retain more uniquely identifying features. In fact, we observe a larger perfor- mance gap on the more semantic ImageNet and SUN Scene tasks. In contrast, because the Kinetics model uses an LSTM to reason over all input images at once, instance-level features are equally useful global video features for overall accuracy.
4.3 PRETRAINING DATA ABLATION
In Table explore the effect of different pretraining datasets end-task performance. For each experiment, we use VINCE but use video data from three different sources: our method of searching ImageNet synset queries, using the URLs from YouTube 8M (Abu-El-Haija al. 2016), and using the URLs from Kinetics 400 (Kay et al. (017). Again, we only test ResNet18. Our YouTube8M(YT8M) pretraining data uses the same filtering procedure as R2V2 and contains 5.8 million images from 1.4 million videos. As noted in Table 2 MoCo-IN results, using the same dataset for pretraining as the end-task results in boost in performance on that specific task but does not indicate that the representation will be better on all tasks. We see this trend is true again when pretraining on Kinetics data. Similarly, since R2V2 uses ImageNet synset for search queries, pretraining on it performs better on ImageNet than the other less-aligned datasets.
In general, this would indicate that given a large enough set of diverse videos, pretraining directly on the unlabeled source data would result in the best performing representation possible, then pretraining on a large external source of data may still result in a useful representation. It also indicates that the VINCE method works well The increased performance on tracking when using YT8M data could be explained by it simply having access to larger number of video sources and frames. For generic object tracking, class diversity may be less important than number of samples because the class identity is ignored.
that data. If this is not
variety of different pretraining datasets.
5 CONCLUSIONS
In this work we introduced Video Noise Contrastive Estimation, a process for using unlabeled videos to learn an unsupervised image representation. By training on multiple images from the same video, learn from natural changes such deformation and viewpoint rather than artificial 2D augmen-
8

---

Under review as a conference paper at ICLR 2020
tations. To learn from large variety of diverse videos, collect Random Related Video Views in completely automated fashion. We show across-the-board improvements over the recently pro- a wide variety of tasks, and we believe Video Noise Contrastive Estimation will extend to other unsupervised methods such as SimCLR (Chen et al.) (020a) and PIRL (Misra & Maaten| 2020) well as other end-tasks. As representation learning techniques improve, we believe that videos rather than images will prove an invaluable resource for pushing the state-of-thearr forward.
posed MoCo (He al., 2020) technique
REFERENCES
Sami Abu-EI-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. XrXiv preprint 160iv: 08675, 2016.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Mare-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems, pp. 8766-8779, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp- 15509-15519, 2019.
Luca Bertinetto, Jack Valmadre, Joao Henriques, Andrea Vedaldi, and Philip HS Torr. Fully- convolutional siamese networks for object tracking. European conference on computer vision, pp. 850-865. Springer, 2016.
Kai Briechle and Uwe D. Hanebeck. Template matching using fast normalized cross correlation. In SPIE Defense Commercial Sensing, 2001.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11 IM 109-1135. 2010.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Clement Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised monocular depth estima- tion with left-right consistency. IEEE Conference on Computer Vision and Pattern Recognition, July 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Thirteenth International Conference on Artificial Intelli- gence and Statistics, pp. 297-304, 2010.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020. 2020.

---

Under review as a conference paper at ICLR 2020
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint 1XXv:1905.09272. 2019.
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estima- tion and maximization. In International Conference on Learning Representations, 2019. URL https://openreview net/ 2or
Devon Hjelm, Alex Fedorov, Samuel
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9:1735- 1780, 1997.
Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine In- telligence, pp. 1-1, 2019. ISSN 1939-3539. doi: 10.1 109110mm2220999295446500
Stthttp:
7/dx. www.:oi...gggg/. wtt../..........///./........... 1I0.TT/MMM... http://.....0EEE...22221..221550 201...97461999977488
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.069550 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 4490,698, 2014.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-super synchronization. In Advances in Neural Information Processing Systems, pp. 7763-7774, 2018.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233-243, 1991.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, 2012.
Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T Freeman. Learning the depths moving people by watching frozen people. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4521-4530, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):5579-2603 2008.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In European Conference on Computer Vision, pp. 181-196, 2018.
Simon Meister, Junhwa Hur, and Stefan Roth. Unflow: Unsupervised learning of optical flow with bidirectional census loss. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 31 1111-3119, 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision, pp. 527-544. Springer, 2016.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
10

---

Under review as a conference paper at ICLR 2020
AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unsupervised video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 133-142, 2020.
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, and Abhinav Gupta. The curious robot: Learning visual representations via physical interactions. In European conference on computer vision, 2016.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive sulf-super learning: In- variances, augmentations and dataset biases. arXiv preprint 1rXiv:2007.33116 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 0rXiv:1511.064:44 2015.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
models are unsupervised multitask learners. 2019.
Tanner Schmidt, Richard Newcombe, and Dieter Fox. Self-supervised visual descriptor learning for dense correspondence. IEEE Robotics and Automation Letters, 2(2):420-427, 2016.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 843-852, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.0530 05849, 2019.
Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced visual invariances. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13806 13815, 2020.
Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Track- ing emerges by colorizing videos. In European Conference on Computer Vision, pp. 391-408, 2018.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In IEEE International Conference on Computer Vision, pp. 2794-2802, 2015.
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. IEFE Conference on Computer Vision and Puttern Recognition, pp- 2566-2576, 2019.
Yi Wu, Jongwoo Lim, and King-Hsuan Yang. Object tracking benchmark. IEEE Transactions Pattern Analysis and Machine Intelligence, 37(9): 834-1848, 2015.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. IEEE Conference on Computer Vision and Pattern Recog- 2018
nition, pp.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485-3492. IEEE, 2010.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick. Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
11

---

Under review as a conference paper at ICLR 2020
DU PERARAAA
AD Tu UTTAAA DYY
VOUTH DE MOO
POURO DENOAE SOTSCOS
Figure A1: Sample from Random Related Video Views (train set).
APPENDIX 1 DATASET COMPARISON
Random Related Video Views provides nearly 250,000 video links as well as multiple images per video across broad variety of categories. We show more samples from R2V2 in Figure AT Videos each have four images 150 frames apart. Each separate video (outlined in blue) lists its correspond- ing YouTube link. Using ImageNet synsets for search queries provides reasonable visual diversity, but could be substituted with another set of queries. While we acknowledge that using YouTube's Search feature not truly random, this procedure resulted in significantly more diverse samples than using existing datasets like You (Abu-El-Haal
et al. 2016) which is heavily unbalanced with unnatural videos [ We do no additional data cleaning
ensure that the videos or extracted images actually contain the search term (many do not), nor do we search for "high interest" video segments as in Misra et al. (2016). We also discard the search term itself as form supervision. We find that a gap of 5 seconds between each saved image typically results in visually distinct but semantically related images. Too much shorter results in images which are less individually distinct, and too much longer may result in large and unpredictable changes. A sample from each dataset can be seen in the supplemental material.
The second most common category in YouTubeSM, comprising 540k of the 3.7 million training videos is "Video Game," and the fifth "Cartoon" with 240k. The category "Minecraft" itself has over 57,000 videos the dataset whereas the category "Pear" has only 138.
R2V2
YTBM
R2V2
Kindics 400
GDT 10k
Figure A2: Random sampling of pairs of images from videos in each dataset. In GOT- 10k, sometimes different video clips are segments from the same original video as seen in the first and second sample. Images are square cropped for visualization purposes only.
12

---

Under review as a conference paper at ICLR 2020
We show samples from various existing video datasets in Figure A2
Datasets such as
YouTube8M (Abu-El-Haij) et al. 2016) and Kinetics400 (Kay et al. 201
Kinetics contains only
videos of humans performing actions. Alternative datasets such GOT-10k (Huang et al., 2019) provide a comparatively small number of videos but with dense annotations (in GOT-10k's case for object tracking).
APPENDIX 2
ALGORITHM PSEUDOCODE
The pscudocode below efficiently computes the Multi-Pair NCE for batch of inputs. Shapes of each input are provided for clarity where:
The number of videos in batch.
k The number of frames video.
72 (v k)
The total number of images in the batch.
m The number of elements the Memory Bank.
d The dimensionality of the feature space.
Algorithm Python-style pseudo code for Multi-Pair NCE.
milti f_ppa_paa___ee
f_cutput,
[v, k, 3] outpet f encoder
g_output,
19, k, d] output of g encoder
moco_mem,
[m, d]
mask,
[Ry in m) block diagonal bool
temparature):
8 [1]
:_cutput s_output. reshape
g_output g_output reshape
compare consatenate put boco_reni, axis=D)
[(n m),
similarities matmul it_oupoo, compare.7:
[ny m)
similarities /- temperature
pes pes_similarities similarities
[ny k]
heg similarities stnliaritiea [masi)))]]
[ny IE +.m K)
exp_pore pos_sim exp wttp__inillrrtteee.. camallizng__oostarc reduce_ tneg_stnennnniila
[ity k]
brozdcast
[ay k]
axia-1),
shape (nuneretor)
exp_pos_stt lexp_pos_ainn hermaltzing_constantt loss (log (score)]
return loss
practice we use Log-Sum-Exp trick for numerical stability but omit here for clarity. broadcast repeats the input until it matches the provided dimensions. maak flips the booleans of each point in the mask.
APPENDIX 3 VINCE IMPLEMENTATION DETAILS
For training ResNet18 (He al. 2016) representations average pooling followed by a fully connected layer (512 x 512), Leaky-ReLU and final embedding layer (512 64). The features 1.2-normalized and multiplied by 0.07 in MoCo (He al. 2020). We use 8 GPUs, a training batch size of 256 and a MoCo Memory Bank of size 65536 and g-network momentum of 0.999. For multi-GPU training we employ the Shuffle-BN technique shuffling both the anchors and the positives to reduce the correlation between batches filled with multiple images from the same video. We use SGD with a learning rate of 0.03, momentum=0.9. and weight decay=0.0001. All of these hyperparameters are shared with MoCo (He et al. 2020). All methods are trained for approximately 450k iterations. When selecting frames from video, we pick with replacement. In addition to the natural augmentation, we perform standard data augmentation (color jitter, cropping, flipping) on the inputs. This prevents the network from relying too heavily on shared video statistics like mean frame color. After cropping, all images resized to 224 x 224. It worth noting that both VINCE and our data (R2V2) can be used with other network architectures or learning algorithms such AMDIM (Bachman et al. 2019) PIRL (Misra & Maaten 2020). or SimCLR (Chen et al.) (220a)). We choose ResNet18 and MoCo due to implementation simplicity and relatively low computational constraints.
the network up to the global
For ResNet50 training, we closely match the hyperparameters in MoCo v2 (Chen et al. 2020b). Specifically, we use the blur augmentation and stronger color augmentation and cosine annealing
13

---

success and failure cases Under review as a conference paper at ICLR 2020
R2V2
ImageNet
Qurery Image
To Nearest Neighbors
Query Image
Top Nearest Neighbors
VINCE
VINCE
MuCo
MoCa
R2V2
R2V2
MoCCo IN
McCo IN
MoCo R2V2
MoCo
R2V2
MoCo IN
McCo-IN
Figure A3: Nearest neighbor results for sampling of query images from R2V2 and ImageNet using various models. VINCE shows a clear understanding each image and finds highly relevant neighbors.
learning rate for 286,000 iterations (200 epochs of ImageNet equivalent R2V2 regardless of the number of unique positives per batch). The batch size we use is 896 with an initial learning rate of 005 and embedding dimensionality of 128. The temperature parameter used was n.2 We did not perform any hyperparameter searches for VINCE or R2V2, so these results may be suboptimal.
APPENDIX 4 QUALITATIVE RESULTS
We additionally provide two qualitative analyses to better understand the of VINCE: Nearest Neighbors, and t-SNE.
APPENDIX 4.1 NEAREST NEIGHBORS:
We query ImageNet Val and set of test videos for nearest neighbor matches, taking at most one neighbor per video. We visualize the top 5 neighbors for VINCE, MoCo-R2V2, and MoCo-IN in Figure A3 We observe that VINCE seems to understand the semantics of an image more than MoCo-R2V2 and MoCo-IN, For instance, although MoCo-R2V2 and MoCo-IN find other control panels and buttons in query they do not make the scene-level connection as VINCE does. Query 2 shows an interesting quirk case of our method. Rather than matching the semantics of the image, VINCE relies the news logo differentiating feature due to its discriminative nature. Each image in VINCE's query 2 results from a separate video, but from the same news source. For the ImageNet queries, despite never seeing ImageNet inputs during pretraining, VINCE is able to find good matches as well as MoCo-IN which was trained using only ImageNet inputs.
interiors well
14

---

Under review as a conference paper at ICLR 2020
Faces
Hands
Food
Fuzzy Animals
Under Water
Figure A4: 1-SNE embedding of images from R2V2 test set.
APPENDIX 4.2 T-SNE:
Using set held-out video frames, we project the 64-D embedding space from VINCE to 2D using t-SNE (Maaten & Hinton, 2008) and visualize the formed clusters in Figure Not only does this assist in verifying the quality of the embedding, it also serves as a visual method for evaluating the diversity of the dataset itself. The largest of the clusters seems to be the face cluster. YouTube is full of videos of people looking and talking directly to camera, and our random subset reflects this pattern. Other interesting, yet unexpected clusters emerge well such as cats (YouTube loves cats), hands (demo videos), and food (cooking videos).
APPENDIX 5 PRECISION AND SUCCESS PLOTS FOR OTB 2015
We provide full breakdowns of the Precision and Success of each method on OTB 2015 (Wu et al. 2015). The values in the legend correspond to the (mean) area under the curve for each method.
Pricblbe Ne: for CTB 2335
1.5
SuccessPtert ona
3.8
32
Over trrandd
(a)
(b)
Figure A5: Precision (a) and Success (b) plots for OTB 2015 for various backbones.
15


Reviews:
[{"review_id": "XWYGO_DNtx-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an idea to use NCE for videos where positive/negative training pairs are created by temporally sampling different frames in the video.\n\nThe idea is fairly straightforward, there is not anything particularly novel about the approach, other than how the training pairs are created. I think it would have been more interesting and insightful if more ways to generate these pairs were proposed and evaluated. For example, video has many modalities (e.g., RGB, audio, optical flow, etc) in addition to the spatial and temporal dimensions. Exploring ways of creating pairs using multi-modal data would have been more interesting.\n\nExperimentally, there aren't really any comparisons to previous self-supervised learning methods. This is a pretty major weakness as it makes it difficult to understand how well this task is doing. Methods like SimCLR provide >70% accuracy on ImageNet and others do well on video tasks (see missing related works below). Currently, I'm not convinced by the experiments.\n\nThe studies comparing different pretraining data and multi-frame vs. single frame are interesting and show the potential of the approach. \n\nThere are some missing related works, for example:\n- \"Cooperative learning of audio and video models from self-supervised synchronization\", NeurIPS'18\n- \"Audio-visual scene analysis with self-supervised multisensory features\", ECCV'18\n- \"Evolving Losses for Unsupervised Video Representation Learning\", CVPR'20 \n\nThese works all provide strong performance on unsupervised video representation learning, yet are not mentioned or compared to.\n\nOverall, I think the proposed idea is not especially novel and the experiments aren't strong enough to show that the simple idea is good. I think there is some potential in the paper, but needs more to be convincing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "L7Sjr9-lIb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of learning representations from video rather than single images is an appealing one with many favorable properties to allow a system to get direct signal on appearance of objects under various natural transformations (occlusion, lighting, etc). Combining instance discrimination ideas of loss based on unlabelled images for which it is known whether they are similar or not, with the idea of curating the images from video is hypothesized to yield learned representations that capture properties enabling improved performance across a variety of single image tasks. The authors create a dataset based on video with positive pairs for noise contrastive estimation, conduct fairly comprehensive experiments and promise to make their newly constructed dataset available. The experiments showcase this type of learned representation outperform alternatives not based on videos on a variety of tasks. \n\nQuality : this seems like a solid paper offering a good intuitive idea with well supported experimental section to show case its relevance. \n\nClarity : the paper is quite clearly written for the most part. The dataset section 3.1 seems to have an omitted paragraph, please see point 1 below. Section 3.2 is quite lean and does not stand on its own, but relies heavily on previous work omitting much of the essence. I would recommend spending a bit more time on ensuring it is more rigorously written. See for example my comment 2 below. \n\nOriginality : the paper is modestly original. It combines two existing ideas - that of using discrimination loss for unsupervised learning of image features, and that of using video based data to allow for rich example of the same data that takes into account real world type transformations. Thus, it is hard to claim more than moderate originality. However,  \n\nSignificance : the improvements over existing baselines are solid, though I would not categorize them as dramatic. Given the originality is also solid, the overall significance is moderate. \n\nComments:\n1. The dataset generation section is strange, did you omit too much? \"We use the following fast and automated procedure to generate the images in our dataset. Using this procedure,...\" it almost seems like a few sentences were dropped between the first and second sentences. While the information exists in the appendix, a sentence or two seem to be warranted in the main test. \n2. \"Gradients flow through the positive pairs\" - at this point in the text you have only introduced a loss. The sentence warrants the question of gradients with respect to what? for rigor, and clarity of exposition, this intuition related statement should come after you talk of what is the parameterized aspect of eq (1) wrt which gradients are taken (so after eq 2 is introduced and the idea that the feature representations are captured through learned method, and in particular some reference to the NN you are using. This makes it a bit more complete as a description of the method and presented in a more methodical order. \n3. Why would you not remove the option of choosing an anchor and positive as the same image? seems easy to avoid\n4. how do you know if the video doesn't contain a shift to a different scene, with different content, thus making the positive actually very different? \n5. Since the representation you provide is supposed to learn representations that are somehow more informed about natural transformations (occlusion, lighting) - is there an experiment you can conceive of to test this specific hypothesis? i think it would both be interesting, and also give insight on whether this is indeed what is being learned. This might also make this representation useful for other types of tasks that are not looked at in the paper that I would encourage the authors to explore. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "G2MJmqKXIbk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper incorporate the popular contrastive with unsupervised learning from video. Specifically, multiple frames from the same video is used as positive pairs and frames from different videos is viewed as negative pair.  The author also proposed a simple and effective ways to collect class-balanced and diverse video frame dataset from Youtube.  The author conducted extensive evaluation experiments on both video recognition and image recognition downstream tasks. Extensive ablation experiments demonstrated the effectiveness  of utilizing multiple frames and the balanced data collection algorithms. \n\nMy concerns:\n1.  Lack of novelty.  The idea of mapping different frames in one video closer,  predicting other frames in one videos, or maximizing mutual informations of embedding of different frames in one video is widely explored.  And adopting contrastive learning in video unsupervised learning scenario has been done before. (Mentioned in the related work section of your paper too)\n2. No comparison against other video based unsupervised learning algorithms.  From my viewpoint, improve over single framed based contrastive learning only proves that your algorithm successfully utilized temporal information encoded in the data and provide limited insights for exploiting more useful information from videos.  \n3. If you can demonstrate your way of incorporating contrastive learning into video based unsupervised learning offers a non-trivial improvement or have a significant difference with other video based unsupervised learning, the impact of your work will be larger. \n\nOverall, I think this paper is interesting but its contribution is limited. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was a difficult decision. Overall it seems to be a quality paper, well written and with many experiments, in particular evaluating learned representations across various tasks and datasets. The authors were also quite courteous in their replies which is appreciated. I really like the point the paper makes about video as a natural augmentation and I find that novel amid the recent NCE surge, where most papers rely critically on augmentation. R4 was also very positive about the paper overall concept.

In terms of paper weaknesses two of the reviewers voted for rejection because the paper ignores existing work on contrastive learning from videos. The authors rebuttal is that they are the first evaluating on images, not on videos. All reviewers also point out limited technical novelty, which the authors acknowledge. Finally, R1 is not very confident about the experiments.

Overall, and after calibration, the appropriate recommendation seems to be rejection.


Decision: Reject