Title: UNSUPERVISED LEARNING FROM VIDEO WITH DEEP NEURAL EMBEDDINGS

Abstract: Because of the rich dynamical structure of videos and their ubiquity in everyday life, it is a natural idea that video data could serve as a powerful unsupervised learning signal for visual representations. However, instantiating this idea, especially at large scale, has remained a significant artificial intelligence challenge. Here we present the Video Instance Embedding (VIE) framework, which trains deep nonlinear embeddings on video sequence inputs. By learning embedding dimensions that identify and group similar videos together, while pushing inherently different videos apart in the embedding space, VIE captures the strong statistical structure inherent in videos, without the need for external annotation labels. We find that, when trained on a large-scale video dataset, VIE yields powerful representations both for action recognition and single-frame object categorization, showing substantially improving on the state of the art wherever direct comparisons are possible. We show that a two-pathway model with both static and dynamic processing pathways is optimal, provide analyses indicating how the model works, and perform ablation studies showing the importance of key architecture and loss function choices. Our results suggest that deep neural embeddings are a promising approach to unsupervised video learning for a wide variety of task domains.

1. INTRODUCTION:

A video's temporal sequence often contains information about dynamics and events in the world that is richer than that in its unordered set of frames. For example, as objects and agents move and interact with each other, they give rise to characteristic patterns of visual change that strongly correlate with their visual and physical identities, including object category, geometric shape, texture, mass, deformability, motion tendencies, and many other properties. It is thus an attractive hypothesis that ubiquitiously available natural videos could serve as a powerful signal for unsupervised learning of visual representations for both static and dynamic visual tasks.

However, it has been challenging to embody this hypothesis in a concrete neural network that can consume unlabelled video data to learn useful feature representations, especially in the context of at-scale real-world applications. Even for the case of single static images, the gap in representational power between the features learned by unsupervised and supervised neural networks has been very substantial, to the point where the former were unsuitable for use in any at-scale visual task. However, recent advances in learning with deep visual embeddings have begun to produce unsupervised representations that rival the visual task transfer power of representations learned by their supervised counterparts (Wu et al., 2018b; a; Zhuang et al., 2019; Caron et al., 2018) . These methods leverage simple but apparently strong heuristics about data separation and clustering to iteratively bootstrap feature representations that increasingly better capture subtle natural image statistics. As a result, it is now possible to obtain unsupervised deep convolutional neural networks that outperform "early modern" deep networks (such as AlexNet in Hinton et al. (2012) ) on challenging recognition tasks such as ImageNet, even when the latter are trained in a supervised fashion.

In this work, we show how the idea of deep unsupervised embeddings can be used to learn features from video datastreams, introducing the Video Instance Embedding (VIE) method. In VIE, videos are projected into a compact latent space via a deep neural network, whose parameters are then tuned to optimally distribute embedded video instances so that similar videos aggregate while dissimilar videos separate. We find that VIE learns powerful representations for transfer learning to action recognition in the large-scale Kinetics dataset, as well as for single-frame object classification in the ImageNet dataset. Moreover, where direct comparison to previous methods is possible (UCF101 and HMDB51 datasets), we find that VIE substantially improves on the state-of-the-art. We evaluate several possibilities for the unsupervised VIE loss function, finding that the Local Aggregation metric, which has previously been shown to be state-of-the-art in single-frame unsupervised learning (Zhuang et al., 2019) , is also most effective in the video context. We also explore several neural network embedding and frame sampling architectures, finding that different temporal sampling statistics are better priors for different transfer tasks, and that a two-stream static-dynamic architecture is optimal. Finally, we present analyses of the learned feature representations giving some intuition as to how the models work, and a series of ablation studies illustrating the importance of key architectural choices.

2. RELATED WORK:

Unsupervised Learning of Deep Visual Embeddings. In this work, we employ a framework derived from ideas first introduced in the recent literature on unsupervised learning of embeddings for images (Wu et al., 2018b) . In the Instance Recognition (IR) task, a deep nonlinear image embedding is trained to maximize the distances between different images while minimizing distances between augmentations (e.g. crops) of a given image, thus maximizing the network's ability to recognize individual image instances. In the Local Aggregation (LA) task (Zhuang et al., 2019) , the embedding loss also allows selected groups of images to aggregate, dynamically determining the groupings based on a local clustering measure. Conceptually, the LA approach resembles a blending of IR and the also-recent DeepCluster method (Caron et al., 2018) , and is more powerful than either IR or DeepCluster alone, achieving state-of-the-art results on unsupervised learning with images. Another task Contrastive Multiview Coding (CMC) (Tian et al., 2019) achieves similar performance to LA within this embedding framework while aggregating different views of a given image. CMC have also been directly applied to videos, where single frame is clustered with its future frame and its corresponding optical flow image. The VIE framework allows the use of any of these embedding objectives, and we test several of them here. Other recently proposed methods such as Hénaff et al. (2019) and Bachman et al. (2019) have achieved comparable results to LA and CMC through optimizing the mutual information of different views of the images, though they use much deeper and more complex architectures.

Supervised Training of Video Networks. Neural networks have been used for a variety of supervised video tasks, including captioning (Krishna et al., 2017) and 3D shape extraction (Matsuyama et al., 2004; Akbarzadeh et al., 2006) , but the architectures deployed in those works are quite different from those used here. The structures we employ are more directly inspired by work on supervised action recognition. A core architectural choice explored in this literature is how and where to use 2D single-frame vs 3D multi-frame convolution. A purely 2D approach is the Temporal Relational Network (TRN) (Zhou et al., 2018) , which processes aggregates of 2D convolutional features using MLP readouts. Methods such as I3D (Carreira & Zisserman, 2017) have shown that combinations of both 2D and 3D can be useful, deploying 2D processing on RGB videos and 3D convolution on an optical flow component. A current high-performing architecture for action recognition is the SlowFast network (Feichtenhofer et al., 2018) , which computes mixed 2D-3D convolutional features from sequences of images at multiple time scales, including a slow branch for low frequency events and a fast branch for higher-frequency events. The dynamic branch of our two-stream architecture is chosen to mimic the most successful SlowFast network parameters. However, we find it useful to include in our architecture a static pathway that is not equivalent to either of the SlowFast branches.

Unsupervised Learning on Videos. The literature on unsupervised video learning is too extensive to review comprehensively here, so we focus our discussion on several of the most relevant recent approaches. Temporal autoencoders such as PredNet (Lotter et al., 2016) , PredRNN (Wang et al., 2017) , and PredRNN++ (Wang et al., 2018) are intriguing but have not yet evidenced substantial transfer learning performance at scale. Transfer learning results have been generated from a wide variety of approaches including the Geometry-Guided CNN (Gan et al., 2018) , motion masks (Pathak et al., 2017) , VideoGAN (Vondrick et al., 2016) , a pairwise-frame siamese triplet network (Wang & Gupta, 2015) , the Shuffle and Learn approach (Misra et al., 2016) , and the Order Prediction Network (OPN) (Lee et al., 2017) . More recent works, including the Video Rotations Prediction task (3DRotNet) (Jing et al., 2018) , the Video Motion and Appearance task (MoAp) (Wang et al., 2019) , and the Space-Time Puzzle task (ST-puzzle) (Kim et al., 2018) , have reported improved performance, with the help of pretraining on large-scale datasets and using spatiotemporal network architectures. All these works only operate on relationships defined within a single video, differentiating them from VIE, which exploits the relationships both within and between different videos through a loss function defined on the distribution of video embeddings. These differences contribute to VIE's greatly improved performance.

3. METHODS:

VIE Embedding framework. The general problem of unsupervised learning from videos can be formulated as learning a parameterized function φ θ (•) from input videos V = {v i |i = 1, 2, .., N }, where each v i consists of a sequence of frames {f i,1 , f i,2 , ..., f i,mi }. Our overall approach seeks to embed the videos {v i } as feature vectors E = {e i } in the D-dimension unit sphere S D = {x ∈ R D+1 with ||x|| 2 2 = 1}. This embedding is realized by a neural network φ θ : v i → S D with weight parameters θ, that receives a sequence of frames f = {f 1 , f 2 , ..., f L } and outputs e = φ θ (f ) ∈ S D . Although the number of frames in one video can be arbitrary and potentially large, L must usually be fixed for most deep neural networks. Therefore, the input f on any single inference pass is restricted to a subset of frames in v chosen according to a frame sampling strategy ρ -that is, a random variable function such that v ⊂ v for all samples v drawn from ρ(v). Given ρ, we then define the associated Video Instance Embedding (VIE) e for video v as the normed (vector-valued) expectation of e under ρ, i.e.

EQUATION

In addition to choosing φ θ and ρ, we must choose a loss function L : E → R such optimizing L with respect to θ will cause statistically related videos to be grouped together and unrelated videos to be separated. Note that in theory this loss function depends on the whole group of embedded vectors, although in practice it is only ever evaluated on a stochastically-chosen batch at any one time, with dataset-wide effects captured via a memory bank.

In the following subsections, we describe natural options for these three main components (architecture φ θ , sampling strategy ρ, and loss function L). As shown in Section 4, such choices can not only significantly influence the quality of learned representations, but also change the focus of the learned representations along the spectrum between static and dynamic information extraction.

Architecture φ and sampling strategy ρ. Recent exploration in supervised action recognition has provided a variety of choices for φ θ . Although very complex network options are possible (Carreira & Zisserman, 2017) , since this work is an initial exploration of the interplay between video processing architecture and unsupervised loss functions, we have chosen to concentrate on five simple but distinct model families. They are differentiated mainly by how their frame sampling assumptions represent different types of temporal information about the inputs:

1. Single-frame 2D-CNNs. Although deep 2D convolutional neural networks (CNNs) taking one frame as input ignore temporal information in the videos, they can still leverage context information and have achieved nontrivial performance on action recognition datasets (Carreira & Zisserman, 2017) . They are also useful baselines for measuring the effect of including temporal information.

2. 3D-CNNs with dense equal sampling. 3D-CNNs, with spatiotemporal filters, can be applied to dense evenly-sampled frames to capture fine-grained temporal information. This architecture has proven useful in the R3D networks of Tran et al. (2018) and the 3DResNets of Hara et al. (2018) .

3. Shared 2D-CNNs with sparse unequal sampling. The Temporal Relation Network bins videos into half-second clips, chooses L consecutive bins, and randomly samples one frame from each bin. A shared 2D-CNN is then applied to each frame and its outputs are concatenated in temporal order and fed into an MLP that will create the final embedding vector. Unlike the dense sampling approach, this method can capture long-range temporal information through sparse sampling, but as the intervals between frames are uneven, the temporal signal can be very noisy.

4. 2D-3D-CNNs with sparse equal sampling. Addressing the issue of noisy temporal information in the third family, the Slow branch of the SlowFast (Feichtenhofer et al., 2018) architecture samples frames equally but sparsely from the input video. These frames are then passed through 2D-CNNs with spatial pooling, applying 3D convolutions downstream once spatial redundancy is reduced.

5. Multi-stream architectures. Combination architectures allow the exploitation of the multiple temporal scales present in natural videos, including the SlowFast approach (Feichtenhofer et al., 2018) (combining 2 and 4), and true two-stream networks with both static and dynamic pathways (combining 1, 2, and 4).

In our experiments ( §4), we implement these models with CNN backbones that, while accommodating small unavoidable differences due to the input structure, are otherwise as similar as possible, so that the qualities of learned representations can be fairly compared.

Loss function L.:

Recent work in unsupervised learning with single images has found useful generic metrics for measuring the quality of deep visual embeddings (Wu et al., 2018b; Zhuang et al., 2019) , including the Instance Recognition (IR) and Local Aggegation (LA) loss functions. These methods seek to group similar inputs together in the embedding space, while separating dissimilar inputs. They are based on a simple probabilistic perspective for interpreting compact embedding spaces (Wu et al., 2018b; Zhuang et al., 2019) . Specifically, the probability that an arbitrary feature e is recognized as a sample of v i is defined to be:

EQUATION

where temperature τ ∈ (0, 1] is a fixed scale hyperparameter. Both {e i } and e are projected onto the unit sphere S D . With this definition in mind, we can define the IR and LA loss functions, adapted to the video context via eq. 1.

IR algorithm. The VIE-version of the loss used in IR is:

L IR (v i , E) = -log P (i|e, E) + λ θ 2 2 (3)

where λ is a regularization hyperparameter, and where for computational efficiency the denominator in P (i|e, E) is estimated through randomly choosing a subset of Q out of all N terms (see Wu et al. (2018b) for further details). Intuitively, optimizing this loss will group embeddings of frame groups sampled from the same video, which then implicitly gathers other similar videos.

LA algorithm. Local Aggregation augments the IR concept by allowing for a more flexible dynamic detection of which datapoints should be grouped together. Define the probability that a feature e is recognized as being in a set of videos A as P (A|e, E) = i∈A P (i|e, E). For a video v i and its embedding e i , the LA algorithm identifies two sets of neighbors, the close neighbors C i and background neighbors B i . C i is computed via dynamic online k-means clustering, and identifies datapoints expected to be "especially similar" to v i ; B i is computed via k-nearest neighbors method, and sets the scale of distance in terms of which closeness judgements are measured. Given these two neighbor sets, the local aggregation loss function measures the negative log-likelihood of a point being recognized as a close neighbor given that is already a background neighbor:

EQUATION

Intuitively, the LA loss function encourages the emergence of soft clusters of datapoints at multiple scales. See Zhuang et al. (2019) for more details on the LA procedure.

Memory Bank. Both the IR and LA loss functions implicitly require access to all the embedded vectors E to calculate their denominators. However, recomputing E rapidly becomes intractable as dataset size increases. This issue is addressed, as in Wu et al. (2018b; a) ; Zhuang et al. (2019) , by approximating E with a memory bank Ē keeping a running average of the embeddings.

4. EXPERIMENTS AND RESULTS:

Experimental settings. To train our models, we use the Kinetics-400 dataset (Kay et al., 2017) , which contains approximately 240K training and 20K validation videos, each around 10 seconds in length and labeled in one of 400 action categories. After downloading, we standardize videos to a framerate of 25fps and reshape all frames so that the shortest edge is 320px. After sampling according to the frame sampling strategy for each model architecture, we then apply the spatial random cropping and resizing method used in Feichtenhofer et al. (2018) . Following Zhuang et al. (2019) we also apply color noise and random horizontal flip, using the same spatial window and color noise parameters for all the frames within one video. At test time, we sample five equally-spaced sequences of frames, resize them so that their shortest side is 256px, and take center 224 × 224 crop. Softmax logits for the five samples are then averaged to generate the final output prediction.

We use ResNet-18v2 (He et al., 2016) as the convolutional backbone for all our model families, to achieve a balance between model performance and computation efficiency. Implementations for the different model families are denoted VIE-Single ("Family 1"), VIE-3DResNet1 ("Family 2"), VIE-TRN ("Family 3"), VIE-Slow ("Family 4"), and VIE-SlowFast ("Family 5"). Two-stream models are created by concatenating single-and multi-frame network outputs, yielding VIE-TwoStream-S (combining VIE-Single and VIE-Slow) and VIE-TwoStream-SF (combining VIE-Single and VIE-Slowfast). We follow Zhuang et al. (2019) for general network training hyperparameters including initial learning rate, optimizer setting, learning rate decay schedule, batch size, and weight decay coefficient. Further details of model architecture and training can be found in Appendices A and B.

Transfer to action recognition on Kinetics. After training all models on the unlabelled videos from Kinetics, we evaluate the learned representations by assessing transfer learning performance to the Kinetics action recognition task. Though previous work on unsupervised video learning has not been evaluated on large-scale datasets such as Kinetics, these results are useful both for understanding the effect of architecture on representation quality and providing a strong unsupervised benchmark for future works. Transfer learning is assessed through the standard transfer procedure of fixing the learned weights and then training linear-softmax readouts from different layers of the fixed model. We implemented this procedure following the choices in Zhuang et al. (2019) , but using the Kinetics-specific data augmentation procedure described above. Since some of the models generate outputs with a temporal dimension, directly adding a fully-connected readout layer would lead to more trainable readout parameters as compared to single frame models. To ensure fair comparisons, we thus average the features of such models along the temporal dimension before readout.

Results are shown in Table 1 . Multi-frame models substantially outperform single-frame versions, an improvement that cannot be explained by the mere presence of additional frames (see Appendix C). The two-stream models achieve the highest performance, with a maximum accuracy of approximately 48.5%. The rank order of unsupervised performances across VIE variants are aligned with that of supervised counterparts, indicating that the unsupervised training procedure takes advantage of (Zhuang et al., 2019) .

Transfer to action recognition on UCF101 and HMDB51. To compare VIE to previous methods, we evaluate results on the more commonly-used UCF101 (Soomro et al., 2012) and HMDB51 (Kuehne et al., 2011) action recognition benchmarks. We initialize networks by pretrained weights on Kinetics and then finetune them on the smaller datasets, following the procedures used for the most recent work in unsupervised video learning (Jing et al., 2018; Wang et al., 2019; Kim et al., 2018) . Because details of the data augmentation pipelines used during finetuning can influence final results, we test all models with the pipeline used by ST-Puzzle, which has the highest reported performance. ST-Puzzle's multi-scale augmentation pipeline differs from the color-noise augmentation pipeline used in our original Kinetics training, so we report fine-tuning performance for VIE models with both pipelines. Details of the finetuning procedure can be found in Appendix E.

Table 2 shows that VIE substantially outperforms other methods. Making these comparisons requires some care, because results reported in previous works are often confounded by the variation of multiple factors at once, making it hard to determine if improvements are really due to a better algorithm, rather than a larger training dataset, a more powerful network architecture, or a difference in input data types. First, holding network architecture and training dataset fixed, VIE-3DResNet surpasses the previous state-of-the-art (ST-Puzzle) by 6.5% on UCF101 and 11.1% on HMDB51, approaching the supervised upper-bound. With the better augmentation pipeline, the improvement on UCF101 is 9.7%. 3DRotNets are trained on Kinetics-600, which contains more than twice as much training data and 50% more categories than that used for ST-Puzzle and VIE models, and are trained with larger inputs (64-frame RGB vs VIE's 16-frame inputs). Nonetheless, VIE still shows improvement of more than 6.2% when compared with comparable input type. 3DRotNet also reports results for a larger fused model trained with both 64-frame RGB and frame-difference inputs. VIE-Slowfast nonetheless substantially outperforms this model, using fewer trainable parameters (21M vs 32M) and much less training data (Kinetics-400 vs Kinetics-600). Again, VIE makes good use of more complex architectures, as can be seen in the SlowFast vs ResNet18 comparison. Moreover, Two-Stream networks achieve even better performance than SlowFast.

Transfer to static object categorization. To determine the extent to which the VIE procedure learns general visual representations, we also evaluate the learned representations for transfer to image categorization in ImageNet. For models requiring multi-frame inputs, we generate a "static video" by tiling still images across multiple frames. Results are shown in Table 1 . As they for the action recognition transfer, the two-stream models are highest performing for this task as well. Interestingly, however, unlike for the case of action recognition, the multi-frame dynamic models are substantially worse than the single frame models on the ImageNet transfer task, and show a performance drop at the highest convolutional layers. In fact, the transfer performance of the single-frame unsupervised model trained on Kinetics is actually substantially better than that of any model supervised on Kinetics (see Appendix Table 5 ). Taken together, these results strongly motivate the two-stream architecture, as features that contribute to high performance on action recognition -e.g. processing of dynamical patterns -are not optimal for static-image performance. However, the relatively high performance Table 2 : Top-1 finetuning results on UCF101 and HMDB51 datasets using models pretrained on Kinetics. We report results under two data augmentation pipelines, where "Multi-scale Aug." represents the pipeline used in Hara et al. (2018) and "Color-noise Aug." is ours used during Kinetics training. We also provide performance of training from scratch ("Scratch") and from supervisedly trained models on Kinetics ("Supervised"). For 3DRotNet, we compare to its model trained with RGB inputs, where 64-RGB means 64 RGB frames. † : AlexNet results are all pretrained on UCF101. of the static and two-stream models shows that VIE can achieve useful generalization, even when train and test datasets are as widely divergent as Kinetics and ImageNet.

Networks:



5. ANALYSIS:

Benefit from long-range temporal structure. A key idea in VIE is to embed entire videos into the latent space, which is intended to leverage contextual information contained in the video. This may even have utility for videos with multiple scenes containing widely-diverging content (a common occurrence in Kinetics), as the high-dimensional embedding might learn to situate such videos in the latent space so as to retain this structure. As a preliminary test of the validity of this approach, we generated new training datasets by dividing each video into equal-length bins and then use these temporally-clipped datasets to train VIE models. Transfer learning results from these models (Table 3 ) show that the full-video model outperforms both 2-bin and 5-bin models, especially on ImageNet transfer learning performance, supporting the choice of embedding entire videos and also indicating that even better performance may be obtained using longer, more contextually complex videos.

Benefit from more data. Although VIE achieves state-of-the-art unsupervised transfer performance on action recognition, and creates representations that transfer to static object categorization better than supervision on action recognition, its learned representation is (unsurprisingly) worse on Im-ageNet categorization than its counterpart directly trained on the (much larger) ImageNet training set (Zhuang et al., 2019) . To test whether VIE would benefit from more videos, we subsample the Kinetics training set and retrain VIE-Single (see Table 3 ). Performance on ImageNet increases consistently and substantially without obvious saturation, indicating VIE's representation generalizability would benefit substantially if trained on a video dataset of the scale of ImageNet.

Video retrieval. We conduct a video retrieval experiment using distance in the video embedding space. Representative examples are shown in Figure 2 . Oftentimes, qualitatively similar videos were retrieved, although some failure cases were also observed. To further investigate the idea that multiframe models develop representations focusing on the dynamic features, while single-frame models better extract static information. VIE-Slowfast appears to extract context-free dynamic information, 

6. CONCLUSION:

We have described the VIE method, an approach that combines multi-streamed video processing architectures with unsupervised deep embedding learning, and shown initial evidence that VIE is promising for large-scale unsupervised video learning. Together with recent results in static image categorization (Zhuang et al., 2019; Wu et al., 2018b) , our results suggest that the deep embedding approach is an increasingly viable framework for general unsupervised learning.

However, there are a number of critical limitations in the current method that will need to be overcome in future work. A natural direction for improvement of the architecture is to investigate the use of recurrent neural network motifs (Hochreiter & Schmidhuber, 1997; Nayebi et al., 2018) and attentional mechanisms (Vaswani et al., 2017) . Exploring the use of other unsupervised learning losses such CMC (Tian et al., 2019) is also of great interest. Further, our current results are likely impacted by limitations in the Kinetics dataset, especially for harnessing the importance of dynamic processing, since even in the supervised case, single-frame performance is comparatively high.

Seeking out and evaluating VIE on additional datasets will be critical -perhaps most importantly, for applications involving large and previously uncurated video data where the potential impact of unsupervised learning is especially high. It will also be critical to test VIE in video task domains other than classification, including object tracking, dynamic 3D shape reconstruction and many others.

A LA SPECIFIC PARAMETERS:

For the LA-specific parameters, we use cluster size m = 8000 for constructing close neighbors C i and nearest neighbor size k = 512 for constructing background neighbors B i . These parameters depart somewhat from the optimal parameters found in Zhuang et al. (2019) , due to the substantial difference in size, and thus density in the embedding space, between the Kinetics training set (240K points) and the ImageNet dataset used in Zhuang et al. (2019) (1.2M points).

B NETWORK IMPLEMENTATION DETAILS:

For VIE-Single, we directly apply the ResNet-18 architecture and follow exactly the same preprocessing pipeline as described in the main text.

For VIE-3DResNet, in order to be comparable to other works (Kim et al., 2018; Jing et al., 2018) which use a smaller input resolution for their networks, we correspondingly scale down our input image size. More specifically, during training, we first resize the chosen frames so that their shortest edges are between 128 and 160px and then get 112 × 112 images through random crops. We then apply the same color noise and random horizontal flip to get the final inputs to the networks. During testing, the frames are resized so that their shortest side is 128px, and then the center 112 × 112 crops are chosen as inputs. Same as in Kim et al. (2018) ; Jing et al. (2018) , the input clip contains 16 consecutive frames.

For VIE-TRN, we sample four consecutive half-second bins, and then one frame from each bin, using ResNet-18 as the shared 2D-CNN across multiple frames, with the outputs of the Conv5 concatenated channel-wise and input into a fully-connected layer to generate the final embedding. This is a simplified version of the TRN, which runs faster and achieves only slightly lower supervised action-recognition performance than the full 8-frame TRN introduced in Zhou et al. (2018) .

For VIE-Slow and VIE-SlowFast, we follow Feichtenhofer et al. (2018) but modify it to use ResNet-18 rather than ResNet-50. The Slow model/pathway evenly samples one frame from every 16 to assemble a 4-frame input sequence, while the Fast pathway samples one frame from every 4 to assemble a 16-frame input sequence.

C SINGLE-FRAME MODELS WITH MULTI-FRAME INPUTS:

To control for the fact that multi-frame models received more total inputs than single-frame models, we also built models which, for any given multi-frame model, takes VIE-Single model, applies it to multiple frames using the same sampling strategy as for the multi-frame model, and then averages across the per-frame outputs before training the softmax classifier. These models are denoted with Input-Single. And their performance is shown in Table 4 . 

:

To be comparable to previous work, our 3DResNet uses a lower input resolution. See Appendix B.

Figures and Tables:

Figure 1: Schematic of the Video Instance Embedding (VIE) Framework. a. Frames from individual videos (v 1 , v 2 , v 3 ) are b. sampled into sequences of varying lengths and temporal densities, and input into c. deep neural network pathways that are either static (single image) or dynamic (multi-image). d. Outputs of frame samples from either pathway are vectors in the D-dimensional unit sphere S D ⊂ R D+1 . The running mean value of embedding vectors are calculated over online samples for each video, e. stored in a memory bank, and f. at each time step compared via unsupervised loss functions in the video embedding space. The loss functions require the computation of distribution properties of embedding vectors. For example, the Local Aggregation (LA) loss function involves the identification of Close Neighbors C i (light brown points) and Background Neighbors B i (dark brown points), which are used to determine how to move target point (green) relative to other points (red/blue).

Figure 2: Video retrieval results for VIE-Single and VIE-Slowfast models from Kinetics validation set. GT=ground truth action label, Pred=model prediction. For each query video, top three nearest training neighbors are shown. Red font indicates a kNN-classifier prediction error.

Top-1 transfer learning accuracy (%) on the Kinetics and ImageNet validation sets. "Random" means a randomly initialized ResNet-18 without any training.

Table content: <table><tr><td>Datasets</td><td/><td colspan="2">Kinetics</td><td/><td/><td>ImageNet</td><td/></tr><tr><td>Metric</td><td colspan="7">Super. Conv3 Conv4 Conv5 Conv3 Conv4 Conv5</td></tr><tr><td>Random</td><td>-</td><td>9.40</td><td>8.43</td><td>6.84</td><td>7.98</td><td>7.78</td><td>6.23</td></tr><tr><td>VIE-Single (IR)</td><td>57.59</td><td>23.50</td><td>38.72</td><td>43.85</td><td>22.85</td><td>40.49</td><td>40.43</td></tr><tr><td>VIE-Single</td><td>57.59</td><td>23.84</td><td>38.25</td><td>44.41</td><td>25.02</td><td>40.49</td><td>42.33</td></tr><tr><td>VIE-TRN</td><td>59.43</td><td>25.72</td><td>39.38</td><td>44.91</td><td>27.24</td><td>40.28</td><td>37.44</td></tr><tr><td>VIE-3DResNet</td><td>53.22</td><td>33.01</td><td>41.34</td><td>43.40</td><td>30.18</td><td>35.37</td><td>32.62</td></tr><tr><td>VIE-Slow</td><td>60.84</td><td>24.80</td><td>40.48</td><td>46.36</td><td>20.10</td><td>37.02</td><td>37.45</td></tr><tr><td>VIE-Slowfast</td><td>62.36</td><td>28.68</td><td>42.07</td><td>47.37</td><td>22.61</td><td>36.84</td><td>36.60</td></tr><tr><td>VIE-TwoStream-S</td><td>-</td><td>26.38</td><td>41.80</td><td>47.13</td><td>23.98</td><td>40.52</td><td>44.02</td></tr><tr><td>VIE-TwoStream-SF</td><td>-</td><td>29.89</td><td>43.50</td><td>48.53</td><td>23.23</td><td>40.73</td><td>43.69</td></tr><tr><td colspan="8">increased architectural power when available. The LA-based VIE-Single model performs better than</td></tr><tr><td colspan="6">the IR-based model, consistent with the gap on static object recognition</td><td/><td/></tr></table>

Top-1 accuracy (in %) of transfer learning to Kinetics and ImageNet from VIE-Single models trained using different amount of videos or with videos cut into different number of bins.

Table content: <table><tr><td>Dataset</td><td/><td>Kinetics</td><td/><td/><td>ImageNet</td><td/></tr><tr><td>Layer</td><td colspan="6">Conv3 Conv4 Conv5 Conv3 Conv4 Conv5</td></tr><tr><td>VIE-Single</td><td>23.84</td><td>38.25</td><td>44.41</td><td>25.02</td><td>40.49</td><td>42.33</td></tr><tr><td colspan="2">70%-VIE-Single 26.18</td><td>38.87</td><td>43.59</td><td>23.05</td><td>39.63</td><td>39.85</td></tr><tr><td colspan="2">30%-VIE-Single 25.54</td><td>37.49</td><td>40.72</td><td>23.33</td><td>38.49</td><td>36.23</td></tr><tr><td colspan="2">2bin-VIE-Single 24.54</td><td>39.16</td><td>44.24</td><td>25.55</td><td>41.43</td><td>39.36</td></tr><tr><td colspan="2">5bin-VIE-Single 25.17</td><td>38.73</td><td>43.33</td><td>23.90</td><td>40.46</td><td>37.83</td></tr></table>

Top-1 transfer learning accuracy (%) on Kinetics for Input-Single models.

Table content: <table><tr><td>Models</td><td colspan="3">Conv3 Conv4 Conv5</td></tr><tr><td colspan="2">TRN-Input-Single 25.52</td><td>39.25</td><td>44.27</td></tr><tr><td colspan="2">Slow-Input-Single 26.17</td><td>39.24</td><td>44.62</td></tr><tr><td>Sf-Input-Single</td><td>25.72</td><td>39.38</td><td>44.29</td></tr><tr><td colspan="4">D TRANSFER LEARNING PERFORMANCE ON IMAGENET FOR SUPERVISED</td></tr><tr><td>KINETICS MODELS</td><td/><td/><td/></tr><tr><td colspan="3">E FINE-TUNING IMPLEMENTATION DETAILS</td><td/></tr></table>

Peer review (iclr2020):
Summary The paper applies an existing self-supervised method (Local Aggregation) on videos that was initially developed with images. The paper mainly studies how effective the video representations are for 2 downstream tasks (action recognition and image recognition) with different video architectures and frame sampling strategies. Strengths 1) Their method achieves a significant improvement in performance over baseline self-supervised video representation learning methods on 2 datasets: UCF101 and HMDB51 2) The paper presents though experiments with different architectures and training settings in order to be fair to the baselines. Weakness 1) Comparison to other self-supervised methods trained on Kinetics data but tested on ImageNet are missing. The problem is that the authors state that the features learnt with VIE are competitive for ImageNet classification but there are no other baselines to compare their approach with. Most self-supervised methods on video do not provide results on ImageNet classification. So it is understandable why they don*t have these metrics but without them it is difficult to place the importance of their method for the ImageNet classification task. 2) There are some claims in the paper that remain unsubstantiated. The only form of validation for these claims is the final performance of the downstream task which does not necessarily validate the claims in the paper. For example: i) *Benefit from long-range temporal structure* In this section the authors train the model with temporally clipped videos by dividing the videos into N bins. They compare it with models trained on full videos and conclude that because performance with entire video on downstream tasks is higher, their training mechanism is capturing long range temporal structure present in videos. It is unfair to compare these two models as they both are trained with different amounts of data. The simplest explanation seems that the model that sees full videos sees more data. It is not clear from this experiment if the long-range temporal structure is benefiting the task or more data is benefiting it. It is difficult to isolate the source of the performance improvement in this manner. But it might be more easier to test performance on a downstream task that requires some temporal reasoning[3, 4] and check if the embeddings are indeed capturing the structure. ii) *However, the relatively high performance of the static and two-stream models shows that VIE can achieve useful generalization, even when train and test datasets are as widely divergent as Kinetics and ImageNet.* This claim is based by comparing the supervised learning and self-supervised learning on Kinetics and then testing on ImageNet. The last layers are usually learning more task-specific features which is why transfer learning performance is worse for Conv5 (Table 5 in Appendix). But Conv4 features are still generalizing well (41.13 is best for VIE-TRN) for the supervised learning models. These are still better than all the ImageNet numbers of VIE-TRN. I see that VIE-TwoStream numbers are much better but he comparison with supervised learning counterparts is not fair as VIE-TwoStream is 2 different models clubbed together and all the Supervised Learning baselines are single model. Decision Their method showcases the power of the Local Aggregation training algorithm for unsupervised representation learning. Their approach scales well to videos and their paper has interesting experiments. But in the present version there are some unsubstantiated claims in the paper. Minor comments: 1) *has remained a significant artificial intelligence challenge* - Too informal a description for an academic paper. 2) *, showing substantially improving on the state of the art * - improvement over 3) *early modern” - please rephrase. 4) Use of the term TwoStream is confusing as most literature in action recognition[1,2] refers to using RGB and flow as two streams. Two-stream models in this paper refer to using 2 base networks for extracting embeddings. As the two-stream terminology is already used in many papers in action recognition, please rephrase this to something less ambiguous. References [1] *Two-stream convolutional networks for action recognition in videos* Karen Simonyan, Andrew Zisserman. [2] *Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset* Joao Carreira, Andrew Zisserman. [3] *The *Something Something* Video Database for Learning and Evaluating Visual Common Sense.*1 Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, Roland Memisevic. [4] *Scaling Egocentric Vision: The EPIC-KITCHENS Dataset* Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, Michael Wray.