Self-supervised video pretraining yields strong image representations
Authors: Nikhil Parthasarathy
S. M. Ali Eslami
Joao Carreira
Olivier J Henaff
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised
contrastive
video
representation learning
image segmentation
object detection

Abstract: Videos contain far more information than still images, and hold the potential for learning rich representations of the visual world. Yet, pretraining on image datasets has remained the dominant paradigm for learning representations that capture spatial information and previous attempts at video pretraining have fallen short on image understanding tasks. In this work we revisit self-supervised learning of image representations from the dynamic evolution of video frames. To that end, we propose a dataset curation procedure that addresses the domain mismatch between video and image datasets, and develop a contrastive learning framework which handles the complex transformations present in natural videos. This simple paradigm for distilling knowledge from videos to image representations, called VITO, performs surprisingly well on a variety of image-based transfer learning tasks. For the first time, our video-pretrained model closes the gap with ImageNet pretraining on semantic segmentation on PASCAL and ADE20k and object detection on COCO and LVIS, raising the possibility of video-pretraining becoming the new default for learning image representations. 

Under review as a conference paper at ICLR 2023
SELF-SUPERVISED VIDEO PRETRAINING YIELDS GENERAL IMAGE REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Videos contain far more information than still images and hold the potential for learning rich representations of the visual world. Yet pretraining on image datasets has remained the dominant paradigm for learning representations that capture spa- tial information. Prior attempts at video pretraining made progress towards solv- ing video-based tasks, but did so the cost of their image understanding capa- bilities. In this work we revisit self-supervised learning of image representations from the dynamic evolution of video frames. To that end, we propose procedure for data curation that addresses the domain mismatch between video and image datasets, and develop a contrastive learning framework which handles the com- plex transformations present in natural videos. This simple paradigm for distilling knowledge from videos to image representations, called VITO, far outperforms all prior video pretraining methods on object detection and semantic segmenta- tion tasks, and for the first time, closes the gap with ImageNet pretraining. Fur- thermore, VITO remains effective when transferring video understanding tasks such as DAVIS segmentation and UCF-101 action recognition. Together, these results suggest that peetranntt now strictly more general than image- pretraining and could become the new default for learning visual representations.
INTRODUCTION
Pretraining on large image datasets has been the dominant paradigm for learning representations that understand the visual world (Krizhevsky al., 2012; He et al., 2016). In particular, self-supervised methods which learn represen- tations that are invariant to specific image trans- formations have proven very powerful, surpass- ing supervised pretraining on a variety of down- stream tasks (He et al., 2020; Hénaff al.,
Prior Video SSL
mean loU, PASCAL fine fine-tuning
76
Our Video SSL
ImageNet
72
70
66
64
2019; Chen et al., 2020; Caron et al., 2021).
Although the synthetic augmentations used in these transformations capture important image priors such as scale-, color-, and translation- invariance, they pale in comparison to the com- plex changes in pose and viewpoint that arise in natural videos.
62
VPS
ViVI
VINCE
CycleContrast
Videotat: McCLR
Videottet: VITO
ImageMet: MoCLR
Therefore, one would expect that learning from videos, as opposed to images, should pro- duce strictly more general visual representa- tions. However, while self-supervised video representation learning has seen a variety re- cent successful applications when evaluating video-basee based tasks (Qian et al., 2021; Feichten-
Figure Closing the gap between video- and poage-pretraining. Prior work learning image rep- resentations from video have lagged behind Ima- geNet pretraining. We propose method for data curation, VideoNet, and self-supervised learning objective, VITO, which together close this gap.
hofer et al., 2021; Toering et al., 2022; Dave et al., 2022; Feichtenhofer al., 2022: Ni al., 2022), it has typically done so by sacrificing performance on image classification (Gordon et al., 2020; Wu & Wang, 2021) relative to image pretraining. Furthermore, the specifics of video--rpresennttiion

---

Under review as a conference paper at ICLR 2023
chitectures make their comparison with image- based architectures difficult, obfuscating the role of the underlying data and learning paradigm in the quality of the resulting representations.
In this work we perform systematic comparison of image- and video-based learning of image representations. Starting from a strong, self-supervised contrastive baseline, we find the spatial content of standard video datasets have detrimental effect on the quality of the resulting repre- sentations, as measured by their performance on canonical scene understanding tasks. We therefore introduce a straightforward video curation Soncedurr-ViieoNeeeee tion with that of ImageNet, and which partially redresses the imbalance between image and video learning. Additionally, we propose three simple modifications to the standard contrastive paradigm to account for the particularities of video data: less aggressive crop augmentation, multi-scale at- tention pooling, and enriching view generation with natural temporal deformations. Together, these improvements yield large gains over prior video pretraining efforts on semantic segmentation on PASCAL and ADE20K and object detection on COCO and LVIS, closing the gap between image- and video-based representation learning for the first time. Notably, we maintain the expected ben- efits of transfer to video-based tasks (DAVIS segmentation and UCF-101 action recognition). This gives new life to the promise of video pretraining serving general purpose means of learning visual representations.
aligns their class distribu-
RELATED WORK
Vidco-based pretraining, Many prior works have considered the problem of self-supervised repre- sentation learning for capturing spatio-tempont invariances. These span a wide range of approaches, beginning with traditional methods that leveraged temporal coherence, optical flow, and object track- ing (Wiskott & Sejnowski, 2002; Hurri & Hyvärinen, 2003; Agrawal al., 2015; Wang Gupta, 2015; Pathak et al., 2017; Goroshin et al., 2015; Misra et al., 2016; 6; Srivastava et al., 2015; Kulkarni et al., 2019). More recently, there have been many successful examples of approaches that leverage contrastive learning, masked autoencoding, and other self- supervied pretext tasks to learn strong video representations (Sermanet et al., 2018; Recasens et al., 2021; Qian et al., 2021; Dave et al., 2022; Dorkenwald et al., 2022; Feichtenhofer al., 2021; 2022). However, most of these methods employ specialized video architectures and transfer to video-based tasks (action recognition, motion segmentation, object tracking, etc.) to measure the quality of the learned representations.
Natural motion-induced deformations are powerful learning signals that should allow for learning better image representations well, and recent works (Gordon et al., 2020; Alayrac et al., 2020; Wu & Wang, 2021; Tschannen et al., 2020; Xu & Waan, 2021; Jabri et al., 2020; Bian et al., 2022; Xiong al., 2021) have made attempts in this direction. One family of successful, recent methods, use pyele-consistencybeee -ased objectives that encourage learning correspondences between tempo- rally ordered image patches via graph random walks (Jabri al., 2020; Bian et al., 2022). However, this is generally computationally expensive to train, and has had noted issues scaling to larger model architectures Xu Wang (2021). Alternate approaches have used optical flow to supervise cor- respondence learning Sharma et al. (2022); Xiong et al. (2021), which can be quite powerful, but also limited to capturing dynamics over short time-scales. The most similar works to ours are Gor- don et al. (2020), Xu & Wang (2021), and Wu & Wang (2021), which utilize simple variants of contrastive learning to learn global frame-level representations from videos. Our approach differs in its curation of video datasets and its ability to handle temporal deformations in the contrastive learning framework via learned attention. Although these works report gains on video-centric tasks such as object tracking and video segmentation, in the context of canonical scene understanding tasks used to evaluate image representations, we find these methods to underperform state-of-the-art -ateeNeet-retrainne models.
Contrastive learning for fine-grained scene understanding. In this work, we specifically focus on evaluations that assess real-world scene understanding, namely semantic segmentation and ob- ject detection (Van Gansbeke et al., 2021; Hénaff et al., 2021; Xie et al., 2021a). Self-supervised learning has greatly benefited fine-grained scene understanding tasks, and there has been significant progress using dense contrastive losses that chose positive pairs for local features by spatial prox- imity and/or feature affinity across two views (Xie et al., 2021b; Bai et al., 2022; O Pinheiro et al., 2020; Wang et al., 2021b). However, as described in Sharma et al. (2022), dense contrastive losses fail when ground truth correspondences cannot be easily obtained across views, as is the case when
2

---

Under review as a conference paper at ICLR 2023
including natural temporal augmentations. Sharma et al. (2022) propose to resolve this by tracking local features via optical flow, but these computations can be brittle and we therefore ask whether correspondences can be established using semantic similarity instead. In this work, we revert to the standard, global contrastive loss formulation, and find that it discovers semantic correspondences when equipped with lightweight attentional module.
METHODS
In experiments we pretrain image representations using image video datasets, then transfer them to a range of downstream image tasks that test their spatial understanding. We adopt the ResNet-50 architecture used throughout the self self-vuue learning literature.
3.1 PRETRAINING VISUAL REPRESENTATION
We start by describing our self-supervised baseline for learning representations from images or individual video frames, MoCLR, before adding the simple modifications which together make our method for distilling videos into image representations, which we call VITO.
A strong contrastive baseline. MoCLR (Tian et al., 2021) is simple but powerful way of learning image representations from image data. Given an image x (or a frame from a video), we generate small number of "views" via random cropping, resizing, and color jittering. Each view of is encoded with a feature extractor Se into a spatial map of hidden vectors h2 = fo(v') where the parameters of the online network being optimized. Following Chen al. (2020), these hidden vectors are average pooled into single vector h then transformed with a two-layer MLP go. yielding non-linear projections z/p = go (h()) which we rescale such that their Euclidean norm is equal 1/ VT, where 0.1.
We wish to enforce invariance of these features across views. In theory, one could regress one projection zj onto its target z/i- however helpful to stabilize these targets by encoding them instead with specific target networks fe and g5. whose parameters & vary more slowly, as shown by Grill et al. (2020). We enforce this invariance using the standard contrastive loss of van den Oord al. (2018)
exp(z} z
(B: = llog
(1)
exp(z}
z + Σn. exp(zx) )
where {>] in are negative features from other images in the batch. We generalize this loss to mul- tiple views by evaluating it for all pairs L(0;E) Eits ()) We update the online network
with gradients from the contrastive loss, and the target network exponential moving average of the online network
0 - optimizezr,( VoL(O, Ao) + (1 Xe) + X=O,
where À and Aę are learning rates for the online and target networks respectively. By combining contrastive formulation of SimCLR (Chen et al., 2020) and the momentum architecture of MoCo (He et al., 2020) and BYOL (Grill et al., 2020), MoCLR (similarly to MoCo v3 (Chen et al., 202 I), which it is akin to) benefits from the best of each approach and has been shown to yield state-of-the- art performance
variety of downstream tasks (Tian et al., 2021).
Adapting synthetic augmentations to video frames. The synthetic augmentation pipeline that has become ubiquitous in the self-supervised learning literature is tailored to pretraining models on ImageNet (Chen et al., 2020; Grill al., 2020). However, video frames (or even uncurated image data) typically differ from the statistics of ImageNet images. In particular, uncurated video frames generally have more variable viewpoints, and a larger field-of-view that can cover multiple (not necessarily centered) objects in complex scenes. The standard random resized crop (RRC) operation, which has been found to be essential in self-super methods like SimCLR and B BOL (Chen et al., 2020; Grill et al., 2020), is aggressive scale transformation where the smallest crops can cover only 8% of the original image. While this enables learning strong invariances across views when the image is relatively homogeneous in content (e.g. featuring single object), for
3

---

Under review as a conference paper at ICLR 2023
video frames this result in views that have very different semantic content (e.g. entirely different objects), dampening the selectivity of the representation for different object classes. As a result, we suggest and will empirically validate that larger crop sizes (e.g. increasing the minimum crop size to 40%) are beneficial when applying contrastive learning to video frames.
Learning from natural temporal transformations. When learning from still images, we apply the random pre-processing pipeline of B YOL (Grill et al., 2020), which includes random cropping, flip- ping, blurring, and point-wise color transformations v' ~ A.()), see appendix A. for the the detail of their distributions. When learning from videos, we sample frames according to distribution T and transform each frame using the same pipeline above:
(4) Recent works have suggested similar methodologies for learning from natural augmentations. In Gordon et al. (2020). 7 samples pairs with a fixed time delay. Xu & Wang (2021) choose a distri- bution that involves uniform sampling over independent chunks of a given video clip. In this work, propose a simpler approach where sample from uniform distribution over the entire video segment length T' = 2.56s. In Figure A.2, we show that these sampling schemes induce very different distributions of absolute time differences between pairs of frames. Our marginal sampling scheme is arguably the most natural as the mode of the distribution is at O, meaning that it is not biased over-represent any specific time difference (similarly the random-resized crop operation in space). While not a huge effect, we find that when evaluated on multiple downstream tasks, this temporal sampling method outperforms other methods (Figure A.2).
v ~ (z11)
v² A2(22
21.22 ~ T((x))==1
frame 1
view 1
hiddens
attention mask
augment
fo
a
mask pooling
frame 2
view 2
EMA
Contrastive loss
augment
fe
ag
Figure 2: Learning to attend to related video content. Each augmented frame is encoded by the network f as a spatial array of hidden vectors. The attention module a takes as input features from one view and produces mask that isolates features that are likely to be predictive of the other, cumporally-displaee view. The attention feature extractor and attention module are trained to satisfy the contrastive objective. Subscripts and refer to online and target (EMA) networks respectively.
features are pooled accordingly, and both the
Multi-scale contrastive attention pooling. Typical contrastive frameworks use a simple global average pooling of hidden vectors to obtain a single feature for each input that can then be passed through the projector network go- This global average pooling (GAP) aggregates content across the whole image, which can enable higher-level semantic tasks like image classification. However, global pooling makes it hard to localize features in space. As result, for fine-gra fine-grained tasks like semantic segmentation, it has been shown that using dense contrastive losses can lead to significant improvements (Wang et al., b021b; Xie et al., 2021b; Bai et al., 2022; Hénaff et al., 2021).
Most local contrastive methods require establishing local correspondences across the two views of the input image such that the contrastive loss can be applied to features that represent the same image content. While these correspondences easily obtained when learning from static images, when tem- poral deformations introduced they require some form of object or point tracking (Sharma al.,
4

---

Under review as a conference paper at ICLR 2023
2022). Yet these methods can be quite cumbersome and involve tuning many datuset-dependent hy- perparameters, so in this work, we propose a more general, adaptive method for learning (at multiple scales) what features should be attended to in order to solve the contrastive learning problem across temporally displaced views.
As shown in Figure 2, given view of the feature extractor outputs spatial map of feature vectors given scale 8, where different scales correspond the outputs of different
h Rhxuxe blocks of ResNet for example, At each scale, we introduce a 2-layer attention MLP ag which outputs mask m 1 softmaz(as) (hog.)))) that we to spatially weight and pool hidden vectors:
1 Eig ml [
Em Em/"i.. [1,J] h/ [e.
Given the attention-pooled features from multiple scales, we concatenate them before transforming them with the two-layer MLP projector: z/ go (o())) where h'g = [h1"" 8 1...S).
This framework allows the projector to utilize a multi-scale adaptively localized representation to solve the contrastive learning problem. This is especially important given the much larger dynamic range of scales which objects can appear in videos as opposed single-ohject ImageNet images. This method is related to that of Jetley et al. (2018), which applied variant of spatial attention pool- ing in the context of supervised image classification, and more loosely to attention- based backbones which have shown great success in self-supervised learning (Caron et al., 2021). Note however that, rather than requiring specialized network operations, our multi-scale attention pooling module can be lightweight addition to standard convolutional architectures. In experiments, find that for the canonical ResNet-50 architecture, attending over the outputs of the last two ResNet blocks (i.e. S 2) is optimal given our evaluations.
3.2 EVALUATING VISUAL REPRESENTATIONS
Classification has traditionally been the default means of evaluating image representations. Clas- sifying single objects however does not require many of the defining features of real-world scene understanding. Semantic segmentation and object detection provide more relevant tests as they require that representation understand fine and coarse object boundaries, shapes, sizes, and view- points. We therefore evaluate on two semantic segmentation datasets, PASCAL VOC (Everingham et al., 2015) and ADE20K (Zhou et al., 2017), which respectively test object-level understanding and complex scene understanding. We also evaluate on the well-knoww COCO object detection dataset (Lin et al., 2014) and the more challenging long-tailed LVIS dataset (Gupta al., 2019). To test generalization beyond image benchmarks, we also evaluate on two video-based tasks, DAVIS 2017 Segmentation and UCF-105 101 action recognition. For details on the specific training and evaluation protocols see Sec A.2.
3.3 ADDRESSING DATASET DOMAIN MISMATCH
We begin investigating the potential for video learning with standard datasets including Kinetics, AudioSet, and YouTube-SM. Yet prior work has shown that even self-supervised methods are sen- sitive to the pretraining distribution. We therefore hypothesized that video pretraining might benefit from a data distribution that is more aligned with the statistics of standard image datasets.
As a test of this hypothesis, we developed simple data curation pipeline (which we refer to as VideoNet) filter online videos such that our training data more closely matches the distribution of ImageNet categories. For each of the 1,000 ImageNet categories, we retrieved 5,000 video clips whose title included the category's 's name or a synonym. We then filtered these videos by applying an image classifier to verify that the videos contained the intended object category. For this ran pretrained ResNet-50 ImageNet classifier on the first 100 frames of each video and discarded videos for which the query category
not equal to the ResNet's top-1 prediction for any of the frames. We additionally discarded videos of less than 10s in length. This procedure resulted a dataset of 1,180,042 videos in total.
We note that while the VideoNet procedure is close in conceptualization to the method used to create the R2V2 dataset proposed by Gordon et al. (2020), it differs in a few ways. First, we utilize full
5

---

Under review as a conference paper at ICLR 2023
Kinetics700
Audioset
VideoNet
ImageNet
PASCAL semantic segmentate megmentation (mloU)
pre training Image pre-training
78
76
74
THE LOCCERR PAPER OWEE
72
70
66
Kinetics700
YouTubeBM
HowTcloOM
AudioSet
VideoNaat
JFT
ImageNet
Figure 3: Impact of pretraining data's s spatial content on representation quality. Left: transfer perfor- mance of models pretrained on single frames from image datasets (grey bars) or individual videos (blue bars). All models ResNet-50 backbone and 2-eiew MoCLR pretraining for 300 epochs. Right: example frames from different video and image dataests.
video clips that allow us to uniformly sample frames at any time point rather than the fixed sampling of frames that are apart in R2V2. This coarse temporal sampling reduces the total number of frames model can learn from, but also limits the resolution of temporal deformations used in the contrastive framework: displacements of 5s are more likely to have changes in semantic content than continuously sampled frames from small interval. Second, by using the ImageNet classifier to filter videos, we can reduce mismatch with the ImageNet distribution that can arise from incorrect tagging and noisy labeling of online videos. This is somewhat verified by the fact that only 24% of the retrieved videos met our filtering criteria.
4 RESULTS
4.1 EFFECT OF PRETRAINING DATA
To demonstrate the effect of the pretraining data distribution on transfer performance, we first pre- train the baseline MoCLR model (using views) on variety of image and video datasets, where initially treat video datasets as collections of individual frames. We train each model for 300 ImageNet-equivalenn epochs, referred to hereafter as "epochs" (i.e. epoch learning from 1.28M examples, irrespective of the dataset), such that each model benefits from the same amount of com- putation. Figure 3 (left) shows their transfer performance on PASCAL semantic segmentation.
Training on standard datasets. As expected, ImageNet pretraining works very well, but pretraining on standard video datasets results in a substantial drop performance (e.g. -6.8% or -5% mloU from pretraining on Kinetics700 or AudioSet). This performance gap between video and image pretraining can be attributed frames and domain mismatch between the dataset categories (Figure 3, right). Consistent with this, training on JFT (Sun et al., 2017), an uncurated dataset with a heavy-tailed class distribution, also results in loss in performance. Notably, this is despite the much larger size of JFT (300M images), which indicates that having more training data does not necessarily lead to increased performance.
combination of increased complexity and field-of-view of video
Training with VideoNet curation. We find that applying the same bascline pretraining to frames from curated video dataset performs better than existing large-scale video datasets like Audioset (+1.6% mloU), but still underperforms image pretraining on JFT and ImageNet (Figure 3).
This result demonstrates the importance of aligning the image frame distributions between video and image datasets. As a result, we choose this filtered video dataset as our primary pretraining dataset for the rest of this work, with the goal of closing the gap with ImageNet pretraining performance.

---

Under review as a conference paper at ICLR 2023
4.2 LEVERAGING VIDEO DATA WITH VITO
We now turn to our proposed method for distilling videos into image representations, VITO. We investigate the choices in our pretraining paradigm by varying each of the three main components in isolation: adaptation of spatial crop size, natural augmentations, and multi-scale attention pooling. For the following ablations, we train VITO and its variants for 300 epochs on VideoNet, using views, then transfer to PASCAL semantic segmentation. Similarly, we pretrain MoCLR baselines on ImageNet and VideoNet for 300 epochs using 3 views.
Adapting spatial augmentations. We validate in Figure 4 (left) our hypothesis that increasing the minimum crop-scale in the random-resized crop operation during training leads to models that gen- cralize better to fine-grained tasks like semantic segmentation. Specifically, we find that a minimum crop scale of 0.4 (as opposed to the traditional 0.08) results in the best transfer performance (+1.7% mloU). Note that this conclusion differs slightly from that of Feichtenhofer et al. (2021) who find more aggressive cropping be beneficial for action recognition.
Natural augmentations. As described in section 3.1, for each training example, we sample views using marginal sampling of each frame from the video clip of length 7 2.56 seconds. This length determines the distribution (and accordingly the mean) time difference between any pair of frames. As result, the total length impacts the time-scale over which the contrastive model learns invariances. We verify our choice by varying the total length of clips. While going to longer time- scales T = 3.2s does not hurt performance much, we find significant improvement over using shorter clips (e.g. 7 = 1.28s, + +1.0% mloU; Figure 4, center). This suggests that invariance to the rich temporal deformations present in video clips is indeed a beneficial criterion for learning fine- grained spatial representations. Note however that the optimal temporal displacement is relatively short (median- 0.76s when T = 2.56s, Figure A.2) and that sampling video datasets too coarsely (e.g. every 58 as in Gordon et al. (2020)) may therefore limit their utility.
Multi-scale attention pooling. We decompose the proposed multi-scale contrastive attention pool- ing isolate the effects of multi-scale learning from those of attention pooling. While we find only modest gains from adding attention pooling to a single-scale version of the model (+0.2% mIoU), we find that the 2-scale model (without attention pooling) improves over the single scale model more robustly (+0.6% mloU). Interestingly, we find that the combination of the 2-scale model with atten- tion pooling has synergistic effect (+1%) 1+1% mloU over the single-scale attention model), highlighting the importance of handling the variability in scales present in natural vidcos. Furthermore, through visualization of the attention masks, we also discover an interesting property of semantic-binding that we believe underlies these perfomrance gains (see Sec A.3 for more discussion).
Combined, the three modifications VITO makes the contrastive framework result in a 2.8% mloU improvement over MoCLR pretrained on VideoNet, closing the gap with ImageNet pretraining when transferring to PASCAL semantic segmentation. In the next sections, we seek to understand the mechanism underlying this improvement, and validate it on other downstream tasks.
4.3 EVALUATION: IMAGE UNDERSTANDING TASKS
Having shown that VITO is potentially learning novel visual representations that transfer well to PASCAL segmentation, we present in Table the transfer performance of VITO against recent image and video pretraining methods on all of the semantic segmentation and object detection tasks.
Comparison to video-pretraining. Given a similar computational budget as prior works (200 epochs and 3 views) VITO delivers large gains over all prior methods. For example, VITO im- proves over VIVI (Tschannen et al., 2020) by 10%/5%/2%/%% highlighting the importance of data curation and our contrastive formulation. VITO improves over VINCE (Gordon et al., 2020) by 7%/4/1 %/1%, highlighting the importance of fine-grained temporal deformations. Finally, VITO improves even over MMV (Alayrac et al., 2020) by 5%//7%/29/2%% despite their use of large- scale text supervision, highlighting the relevance of video-only learning. Finally, we disentangle the power of our method and dataset by confirming that each independently have strong effects: MoCLR trained on VideoNet still outperforms all prior work. Similarly VITO trained on standard datasets (Audioset or YTTM) also outperform all prior work.
7

---

Under review as a conference paper at ICLR 2023
Ablation:
scale
Ablation:
multi-srale attention pooling
PASCCAL Semantic Segmenttation (milat)
76.5
25.3
75.3
75.5
75.0
75.0
75.0
24.0
MoGLR
74.0
MACUR
NoCLR
73.5
73.5
735
ImageNet
ImageNet
Imagahat
73.0
73.0
75.
crsp Cr3p
1.00
2.50
(6lo afr)
Figure 4: Effects of crop scale, natural augmentations, and multi multi-scale attention on representation quality. All ablations are performed relative to VITO's configuration (denoted green asterisk) which uses 2-scale attention pooling, a less aggressive crop scale of 40%, and natural augmentations uniformly sampled window of length 2.56s. We also compare to our baseline MoCLR model trained on single frames, either from ImageNet (dotted gray line) or VidcoNet (dashed blue line). All models evaluated by transferring PASCAL semantic segmentation.
Semantic segmentation
Object detection
Pretraining
Dataset
Epochs
PASCAL
ADE20K
COCO
LVIS
Random Init
53.0
27.9
39.0
21.1
Comparison to video pretraining
VES (Xu Wang, 2021)
K400
100
63.9
31.4
41.6
23.2
VIVI (Tschannen et al., 2020)
YouTubeSM
192
65.8
34.2
41.3
23.2
VINCE (Gordon et al., 2020)
R2V2
200
69.0
35.7
42.4
24.4
CycleContrast (Wu & Wang, 2021)
R2V2
200
69.2
35,6
42.8
24.5
MMV (Alayrac al., 2020)
AS HT
1600
24.2
MoCLR
VideoNet
200
72.8
37.5
42.6
24.6
VTTO
YT8M
200
71.8
37.8
42.7
24.6
VTTO
AudioSet
200
73.6
38.5
43.2
25.0
VITO
VideoNet
200
75.5
39.2
43.6
25,6
Comparison to ImageNet pretraiiing
Supervised
ImageNet
200
71.3
33.5
44.2
25.2
BYOL (Grill et a 2020)
ImageNet
300
76.1
38.8
43.7
25.5
MoCLR (Tian al., 1)
ImageNet
300
76.4
39.2
43.9
25.8
DINO (Caron al., 2021)
ImageNet
300
76.1
39.0
44.3
26.4
VITO
VidcoNet
300
76.3
39.4
44.0
25.7
Table I: VITO outperforms prior video pretraining and closes the gap with ImageNet-pretraininn of ResNet-50 models. For external models, we finetune publicly available checkpoints.
Comparison to ImageNet pretraining. Finally, we compare our VITO- perer-rete VideoNet model to host of state-of- the-art ImageNet--retrained methods. Surprisingly, find VITO to be compet- itive with the best of such methods, outperforming B BYOL and DINO on PASCAL, matching BYOL and MoCLR COCO and LVIS, and surpassing all methods ADE20K. VITO largely surpasses supervised ImageNet pretraining on downstream evaluations. This is, to the best of our knowledge, the first example of video pretraining achievingImageNet-- demonstrate that our results are not specific to the ResNet-50 architecture and scale well when using Swin transformers (see Sec. A.4).
performance on such tasks. We also
4.4 EVALUATION: VIDEO UNDERSTANDING TASKS
To further emphasize the value of video pretraining, we evaluated VITO on tasks that have been proven difficult when using image pretraining, specifically video segmentation and action recogni- tion. We find these tasks to be a good combination as they test both I-shot fine-grained capabilities (DAVIS segmentation) and coarser temporal understanding (UCF101 full video classification).
8

---

Under review as a conference paper at ICLR 2023
Pretraining
Dataset
Im
Fm
DetCong (Hénaff et al., 2021)
ImageNet
63.1
66.4
MoCLR (Tian et al., 2021)
ImageNet
63.1
67.8
BYOL (Grill et al., 2020)
ImageNet
63.8
69.4
VITO
VideoNet
65.5
70.8
Table 2: VITO outperforms all image-pretraining baselines on DAVIS 2017 video segmentation
Pretraining
Dataset
Backbone
Top-1
Video architectures
Supervised (Wang et al., 102
ImageNet
B3D
67.1
VidcoMoCo (Pan al., 2021)
K400
R(2+1)D
78.7
Temporal-ssl (Jenni et al., 2020)
K400
R(2+1)D
81.6
VTHCL (Yang et al., 2020)
K400
3D-R50
82.1
CVRL (Qian et al., 2021)
K400
3D-R50
92.9
p-BYOL (Feichtenhofer et al., 2021)
K400
3D-R50
95.5
Image architectures
Shuffle and Learn (Misra et al., 2016)
UCFI01/HMDBS1
AlexNet
50.6
OPN (L.ce et al., 2017)
UCF101
VGG-M
59.8
TCE (Knights al., 2021)
K400
R50
71.2
CycleContrast (Wu Wang. 2021)
R2V2
RI8
76.8
CycleContrast (Wu Wang, 2021)
R2V2
R50
82.1
VITO
VideoNet
R50
85.3
Table 3: VITO outperforms pretained backbones finetuning on UCF101 w/ just pooling of frame representations, Performance is even above all video architectures except gray numbers.
DAVIS segmentation. Here we demonstrate the value of video pretraining compared directly with ImageNet pretraining. In addition displaying competitive scene understanding capabilities, that VITO specifically learns features relevant to temporal dynamics, and conclusively outperforms strong ImageNet pretraining methods, on both the region jaccard (T) and boundary F measure (F) (Table 2).
UCF101 action recognition. In Table are either pretrained on video data using vidco-specific architectures (3D convolutions and variants) or standard image architectures that produce Trame-based representations which are simply averaged over clips. Traditionally, the latter have not been able to come close to the performance of the former, but we find that VITO has the best performance of all image backbones and even outperforms many recent video-specific SSL methods.
present finetuned top-1 accuracy of many methods that
DISCUSSION
We propose VITO, a simple method for distilling videos into image representations. The key fea- tures of our method include improved dataset curation, adapting standard synthetic augmentations video frames, and using attention-guided contrastive learning. With these components, VITO out- performs all prior video pretraining methods on object detection and semantic segmentation tasks, and for the first time, closes the gap with ImageNet pretraining. Furthermore, unlike image pretrain- ing, VITO VUTO-pretraiiing generalizes to tasks that require temporal understanding, achieving surpris- ingly strong performance on video segmentation and action recognition.
We believe this work can be a foundation for future video pretraining efforts, our approach is powerful, yet simple and extensible almost every aspect. For example, because we base our learning paradigm on standard architectures and contrastive learning methods, it is easy to extend or adapt our approach leverage continuing advancements image-based contrastive, and more generally, self-supervised learning objectives. Additionally, while we have shown the benefits of a simple attention module for learning from video data, there are great opportunities to extend our approach take advantage of more powerful attentional architectures. In sum, despite the many successes in video representation learning. our results suggest that there a great untapped potential in video pretraining as paradigm for learning general visual representations.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
We will release our pretrained models along with the code needed to implement the VITO model architecture. Pretraining and evaluation details about architectures, optimization, and hyperparame- ters have all been detailed in the appendix. The VideoNet procedure for curating video datasets can be reproduced with standard ImageNet classifiers and publicly available online videos.
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015.
Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised mul- timodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg. Point-level gion contrast for object detection pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16061-16070, 2022.
Zhangxing Bian, Allan Jabri, Alexei Efros, and Andrew Owens. Learning pixel trajectories with multiscale contrastive random walks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4508-6519 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 964009964,.000 2021.
Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve, and Mubarak Shah. Telr: Temporal contrastive learning for video representation. Computer Vision and Image Understanding, 219:1033006, 2022.
Michael Dorkenwald, Fanyi Xiao, Biagio Brattoli, Joseph Tighe, and Davide Modolo. Sevrl: Shuf- fled contrastive video representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4132-4141, 2022.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-1 -1)6 2015.
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3299-3309, 2021.
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spa- tiotemporal learners. arXiv preprint arXiv:2205. 09100,091,,,...1 2022.
Chengjian Feng, Yujie Zhong, Yu Gao, Matthew Scott, and Weilin Huang. TOOD: Task-aligned one-stage object detection. In Int. Conf. Comput. Vis., 2021.
Daniel Gordon, Kiana Ehsani, Dieter Fox. and Ali Farhadi. Watching the world go by: Representa- tion learning from unlabeled videos. arXiv preprint 000e:20033.79900 2020.
Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised leam- ing of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pp. 4086-4099 2015.
10

---

Under review as a conference paper at ICLR 2023
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraaro Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self- learning, Advances in Neural Information Processing Systems, 33, 2020.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 5356-5364, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint ar 110.::9900000200 2019.
Olivier J Hénaff, Skanda Koppula, Jean-Baptie Baptiste Alayrac, Aaron den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. In ICCV, 2021.
Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisser- man, João Carreira, and Relja Arandjelovic. Object discovery and representation networks. arXiv preprint arXiv:2203.0877
2022.
Jarmo Hurri and Aapo Hyvärinen. Simple-cell-ike receptive fields maximize temporal coherence in natural video. Neural Computation, 15(3):663 691, 2003.
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. Advances in neural information processing systems, 33, 2020.
Simon Jenni, Givi Meishvili, and Paolo Favaro. Video representation learning by recognizing tem- poral transformations. In European Conference on Computer Vision, pp. 425 442. Springer, 2020.
Saumya Jetley, Nicholas A Lord, Namhoon Lee, and Philip HS Torr. Learn to pay attention. arXiv preprint arXiv:1804.022: 02391, 2018.
Joshua Knights Ben Harwood, Daniel Ward, Anthony Vanderkop, Olivia Mackenzie-Roo Ross, and Peyman Moghadam. Temporally coherent embeddings for self-supervised video representation learning. In 2020 25th International Conference Pattern Recognition (ICPR), pp. 8914 8921. IEEE, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32, 2019.
Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Wing-Hsuan Yang. Unsupervised represen- tation learning by sorting sequences. In Proceedings of the IEEE international conference on computer vision, pp. 667-676. 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. Eur. Conf. Comput, Vis., pp. 740-755. Springer, 2014.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 3131-333440 2015.
11

---

Under review as a conference paper at ICLR 2023
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Int. Conf. Learn. Represent., 2019.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European conference on computer vision, pp. 527-544. Springer, 2016.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12475-12486, 2021.
Jingcheng Ni, Nan Zhou, Jie Qin, Qian Wu, Junqi Liu, Boxun Li, and Di Huang, Motion sensitive contrastive learning for self-supervised video representation. arXiv preprint arXiv:2208.06110 06105, 2022.
Pedro OO Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. Advances in Neural Information Process- ing Systems, 33:4489-444 45000 2020.
Tian Pan, Yibing Song, Tianyu Yang. Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11205-11214, 2021.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning fea- tures by watching objects move. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2701-2710, 2017.
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6964-6974, 2021.
Adrià Recasens, Pauline Luc, Jean- Captistt Alayrae Luyu Wang, Ross Hemsley, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altché, Michal Valko, Jean- Bastien Grill, Aäron van den Oord, and Andrew Zisserman. Broaden your views for self- supervised vidco learning. In Int. Conf. Comput. Vis., 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018.
Yash Sharma, Yi Zhu, Chris Russell, and Thomas Brox. Pixel-level correspondence for self- supervised learning from video. arXiv preprint arXiv:2207.03866. 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 12I2v:122..4400 2012.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video rep- resentations using Istms. In International conference on machine learning, pp. 843-852, PMLR, 2015.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Yonglong Tian, Olivier Henaff, and Aäron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 10063-10074, 2021.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully convolutional one-stage object detection. In Int. Conf. Comput. Vis., 2019.
12

---

Under review as a conference paper at ICLR 2023
Martine Toering, Ioannis Gatopoulos, Maarten Stol, and Vincent Tao Hu. Self-supervised video rep- resentation learning with stream prototypical contrasting. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 108-118, 2022.
Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced visual invariances. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13806 13815, 2020.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0372 2018.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsuper- vised semantic segmentation by contrasting object mask proposals. ICCV, 2021.
Jinpeng Wang, Yuting Gao, Ke Li, Yiqi Lin, Andy J Ma, Hao Cheng, Pai Peng, Feiyue Huang, Rongrong Ji, and Xing Sun. Removing the background by adding the background: Towards background robust self- supervvid video representation learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11804-11813 1813, 2021a.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, pp. 2794-2802, 2015.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3024-3033, 2021b.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 14(4):715-770, 2002.
Haiping Wu and Xiaolong Wang. Contrastive learning of image representations with cross-video cycle-consistency. In Proceedings of the IEEE/CVF International Conference Computer Vi- sion, pp. 10149-10159, 2021.
Shengkai Wu, Xiaoping Li, and Xinggang Wang. IoU-aware single-stage object detector for accurate localization. Image and Vision Computing, 2020.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8392-8401, 2021a.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16684-16693,
2021b.
Yuwen Xiong, Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. Self-supervised representation learning from flow equivariance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10191-10200, 2021.
Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame- level similarity perspective. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision pp. 10075-10085 2021.
Ceyuan Yang, Yinghao Xu, Bo Dai, and Bolei Zhou Video representation learning with visual tempo consistency. arXiv preprint arXiv:2006. 15489, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708 170.0088, 2017.
Shifeng Zhang, Cheng Chi, Yongqiang Yao. Zhen Lei, and Stan Z. Li. Bridging the gap between anchor- based and anchor-free detection via adaptive training sample selection. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
13

---

Under review as a conference paper at ICLR 2023
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 933-641,2011
14

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 IMPLEMENTATION: DATA PRE-PROCESSING
Self-supervised pretraining. Each frame is randomly augmented by composing the following op- erations, each applied with given probability:
1. random cropping: a random patch of the image is selected, whose is uniformly sam- pled in [s Al, where A is the area of the original image, and whose aspect ratio is logarithmically sampled [3/4, 4/3). s a scale hyper-parameter set 0.08 when learn- ing from ImageNet, and 0.4 when learning from videos. Regardless, the patch is then resized to 224 x224 pixels using bicubic interpolation;
2. horizontal flipping:
3. color jittering: the brightness, contrast, saturation and hue are shifted by a uniformly dis- tributed offset;
4. color dropping: the RGB image is replaced by its grey-scale values;
5. gaussian blurring with x33 square kernel and a standard deviation uniformly sampled from 2.00;
6. solarization: a point-wise color transformation x -> x Laç0.5 + (1 - x)) 1220. with pixels in [0, 1].
The augmented frames v and v² result from augmentations sampled from distributions A1 and A2 respectively. These distributions apply the primitives described above with different probabilities, and different magnitudes. The following table specifies these parameters for the BYO framework (Grill et al., 2020). which we adopt without modification. When learning from three views, we use the distribution A1 to generate the third view.
Parameter
A
A
Random crop probability
1.0
Flip probability
0.5
Color jittering probability
0.8
Color dropping probability
0.2
Brightness adjustment max
0.4
Contrast adjustment max
0.4
Saturation adjustment max Hue adjustment max
0.2
0.1
Gaussian blurring probability
1.0
0.1
Solarization probability
0.0
0.2
Transfer to PASCAL and ADE20K. During training, images are randomly flipped and scaled by a factor in (0.5, 2.0). Training and testing performed with 512x512-resolution images. When fine- tuning on ADE20K, we aditionally use photometric transformations from the mmseg' codebase.
Transfer to COCO and LVIS. The target resolution is 800x 1024. During testing, image is resized by a factor s while preserving the aspect ratio, such that is tightly contained inside the target resolution, and then padded. When fine-tuning, the image is rescaled by factor of where u is uniformly sampled [0.8, 1.25], and is then cropped or padded to the target resolution.
A.2 IMPLEMENTATION OPTIMIZATION
Self-supervised pretraining. We pretrain ResNet-50 using the LARS optimizer (You et al., 2017) with batch size of 4096 split across 128 Cloud TPU v3 workers. We adopt the optimization details of BYOL, scaling the learning rate linearly with the batch size and decaying it according to a cosine schedule. The base learning rate is 0.3 and the weight decay is 10-6
hhttps://gi thub. www.como//////// open-mm http//mmemmeetttiiini
15

---

Under review as a conference paper at ICLR 2023
Semantic segmentation on PASCAL and ADE20K. We evaluate ResNet models by attaching fully-convolutions network (FCN, Long et al. (2015)) and fine-tuning end-to-end, following He et al. (2020). We fine-tune for 45 and 60 epochs PASCAL and ADE20K respectively, and report the mean intersection over union (mloU) averaged across 5 runs.
We fine-tune for 45 epochs the PASCAL trainnaug200 12 set 60 epochs the ADE20K t train set. We use stochastic gradient descent with a batch size of 16 and weight decay of 0.005. The learning rate is initially set to 0.04 and decayed exponentially with factor of 0.9" where is the iteration number. When fine-tuning external models, we sweep over the base learning rate and weight decay and report their performance given the optimal configuration. In all cases we report mloU on the val set averaged across 5 runs.
Transfer to COCO and LVIS with FCOS*. The network is fine-tuned for 30 epochs on the COCO crain2017 7 set or the LVIS vi_train n set. We use Adam (Loshchilov & Hutter, 2019) with weight decay 10-4, base learning rate of 10-3, and batch size 128 split across 16 workers. The learning rate rises linearly for 1 of an epoch, and is dropped twice by factor of 10, after 3 and of the total training time. We report mAP on the COCO va12017 set and the LVIS vl-val set, averaged across runs.
2
We evaluate pretrained ResNet's using the FCOS* architecture, following Hénaff et al. (2022). FCOS* is the implementation of a single-stage detector based on FCOS (Tian et al., 2019), and improved with the collection of techniques from Wu et al. (2020), Zhang al. (2020), and Feng et al. (2021), full details can be found in Hénaff et al. (2022). The pretrained network is used to initialize the backbone of the FCOS* model, which is then fine-tuned for 30 epochs. We report bounding-box mean average precision (mAP) averaged across 5 runs.
Video segmentation on DAVIS
As a further test of scene understanding, we assess whether learned representations can continue to recognize parts of an object as they evolve over time. Video object segmentation, specifically in its semi-supervised setting, captures this ability, which we evaluate on the DAVIS' 17 benchmark. Having evaluated learned representation on video independently across frames, we segment these features with nearest neighbor matching from frame to frame, given a segmentation of the first frame. In this way, the segmentation is propagated according to the similarity of the representation across space and time. We reuse the segmentation procedure from Xu & Wang (2021) without modification.
Action recognition UCF101 We evaluate action recognition classification on the UCF101 dataset (Soomro et al., 2012). We follow the procedure for finetuning used in Wu & Wang (2021) which is based on Morgado al. (2021). Clip representations are obtained by averaging the frame representations for the video, and one fully connected layer is used for predicting the action class. 10 clips are sampled from each video and the predictions of the clips are averaged for the final results.
A.3 SEMANTIC BINDING WITH CONTRASTIVE ATTENTION POOLING
The ablation study demonstrated that multi-scale attention improves the performance of VITO in semantic segmentation. To probe why this may be, we visualize and interpret the learned attention masks (Figure A. For simplicity, we only visualize the masks from the coarsest scale (output feature map). but the interpretation naturally extends to the multi-scale version as these masks are learned with independent attention modules.
Because the attention masks are not computed jointly across each view, for a given video frame, the attention module must marginalize over the training data to make a statistical prediction what should be attended to in the first view in order to minimize the contrastive loss across possible second views? Specifically, the attention must focus on content that is most likely to be stable across time while still being discriminative (or unique) relative to other frames from other videos. Different examples appear to trade-off these criteria differently, yet systematically. For example, in the third column of Figure A.1 even though the animated characters on the right side of both frames may be discriminative content, the attention module has learned to focus on the static picture on the left as it is the content that is most likely be stable across time. For this pair of frames the prediction is
16

---

Under review as a conference paper at ICLR 2023
Figure A.1: Example augmented frames with overlaid (resized) learned attention masks. Attention is computed from the output of the final block of the VITO trained ResNet-50. Crucially, the attention masks are computed independently, such that the attention module can only use spatial cues.
Pascal Semantic Segmentation (mlou)
ADE 20k Semantk: Segmentation (micU) Distribution of Time Delays Between Sampled Views
marg nal
76.0
[medd inn 3.58g)
36.0
datta
75.5
75.0
36.3
745
16.0
MoCLK
Vi0cofic:
37.5
MotLE
McCLR
unforr
marginal
02k3
anformm
margral
0.0
0.5
1.0
1.5
2.0
2.5
Figure A.2: Ablating different temporal sampling schemes. Delta refers fixed time sampling beteween frames as in Gordon et al. (2020). Uniform refers to chunking time into non-overlapping blocks and uniformly sampling within each chunk as in Xu & Wang (2021). Marginal sampling (ours) refers to simple uniform sampling from the full video clip of length T 2.56s. First two panels show that marginal sampling best overall across transfer to PASCAL and ADE20K. Third panel shows the distribution of absolute time- diffeeiffeeesees between any two pairs of frames under each sampling scheme (assuming 3 views are sampled per clip).
therect-the attention disregards content that is changing too abruptly-deppit despite not having access to motion cues. On the other hand, the example in the fourth column demonstrates scenario where the model has attended to stable, but primarily discriminative content (the bird) rather than the background, which is also very stable but most likely less unique relative to other videos.
Even beyond the ability to localize stable, yet discriminative content, it seems that our method also enables "semantic binding" of visually different, but semantically related features. This can be seen in the first pair of frames, the model has learned to associate with the dumbbell (in the second frame), demonstrating an understanding that these two semantically related concepts co-occur and thus are predictive of one another given the right embedding.
elbow (in the first frame)
Binding co-occuring features appears as an intuitive explanation for why these representations would perform well on semantic segmentation. It is particularly interesting that training end-to-end with standard contrastive loss can produce complex behavior reminiscent of the DINO approach (Caron al., 2021) even though use a single, two-layer MLP attention module opposed large-scale transformer architectures which use attention throughout the network.
17

---

Under review as a conference paper at ICLR 2023
A.4
MODEL SCALING RESULTS
Semantic segmentation
Object detection
Pretraining
Dataset
Backbone
PASCAL
ADE20K
COCO
LVIS
VITO
ImageNet
R50
76.3
39.4
44.0
25.7
MoCLR (Tian al., 2021)
ImageNet
Swin-S
78.6
43.7
48.4
32.7
VITO
VideoNet
Swin-S
81.3
46.1
49.8
33.5
Detcon (Hénaff et al., 2021)
ImageNet
Swin-S
81.4
46.1
50.4
33.1
Table A.1: VITO scales larger model architectures (Swin-S), improving performance compared to the ResNet-50 baseline and remaining competitive with ImageNet pretraining.
18


Reviews:
[{"review_id": "iaOf-a3Z0ez", "reviewer": "Reviewer_iRvv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a video self-supervised pretraining pipeline called VITO. It made several modifications over existing contrastive learning frameworks including larger crop size, improved temporal sampling scheme, and multi-scale attention feature pooling for the projector. The authors also investigated the data domain mismatch and propose a video curation procedure to further improve the model pretraining performance. Experimental results on semantic segmentation and object detection show that the proposed framework can achieve better results among methods pretrained on video datasets, and be competitive with the models pretrained on ImageNet.", "summary_of_the_review": "Overall I believe this paper makes novel technical contributions (using learnable attention gates to align features from different frames, pointing out that the video data quality actually matters a lot for existing models on several benchmarks, etc.). However, the evaluation part is my major concern, would be great if the authors could better motivate why video is important (although this might be shown in other works) and show either video+image could perform better than image only, or video pretraining learn better representations on some tasks compared with image-based and other video-based baselines. ", "strength_and_weaknesses": "Pros:\n\nThis paper is well-organized and easy to follow. The technical contribution and the way to do data curation make sense to me. The ablation is comprehensive and the final results compared with other video-based model looks good, which shows the effectiveness of the proposed framework.\n\nCons:\n\nThe motivation of this paper is not very clear, as the results using video pretraining do not show superior performance compared with image pretraining based on the benchmark the authors chose. If the authors believe video is beneficial maybe choose another benchmark (like video segmentation or tracking) or at least combine ImageNet with the proposed VideoNet data to showcase better performance compared with the ImageNet baseline.\nFor other misc comments see the section below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RK-ZnXyW6ie", "reviewer": "Reviewer_ACJo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors adapt existing contrastive learning approaches for self-supervised image representation learning to learning from video frames instead of static images. In particular, they first adapt the data augmentation steps, which are key to contrastive learning, to the video domain. Two main adaptations are proposed: first, the crop size is increased to account for the fact that, unlike ImageNet, in-the-wild videos are not centered on a single object, hence very aggressive cropping can make the task unnecessary hard. Second, they utilize a multi-scale attention pooling module to account for the fact that the frame content can shift significantly over time. Finally, they collect a new dataset which is automatically curated (using an ImageNet classifier) to match the ImageNet category distribution. They then experimentally demonstrate that, by combining these model and data improvements, self-supervised pre-training on video frames can come close to pre-training on ImageNet in terms of downstream performance on the tasks of semantic segmentation and object detection. ", "summary_of_the_review": "This is a (for the most part) solid empirical study on adapting image-level contrastive learning methods to video frames. That said, a crucial factor is missing from the ablation analysis (control for the amount of training data) and the overall value of using video frames in this framework is not well justified. Moreover, related work overview has significant omissions. I hope the authors can address these concerns in the rebuttal.", "strength_and_weaknesses": "Strengths:\n\nThe paper is well written and is easy to follow.\n\nThe proposed approach is sound and its effectiveness is convincingly confirmed by an ablation analysis.\n\nThe final model trained on curated video frames archives similar performance to models trained on ImageNet in terms of downstream semantic segmentation and object detection accuracy.\n\n\nWeaknesses:\n\nThis is a purely empirical study, there is no technical contribution (all that techniques that are used have been introduced in the past).\n\nRelated work overview is incomplete. In particular, the authors ignore the whole line of work on space-time walks for self-supervised image representation learning from videos  which is highly relevant (e.g. Jabri et al., NIPS'21 but also prior works and follow ups).\n\nIn the experiments the authors do not control for the amount of training data. In particular, they have collected 1,180,042 videos each of which is at least 10 seconds long. How many frames is that? There is indeed a lot of redundancy between frames from the same video, but the redundancy is by far not 100%, especially in relatively long videos. For a fair comparison, the authors need to report results when only one pair of frames is sampled from each vides in a pre-processing step, so that the overall amount of data is comparable to ImageNet.\n\nMost importantly, the value of using videos as a source of training data for image-level contrastive learning is not convincingly demonstrated. The authors have to go to great lengths to simply match the ImageNet performance. The downsides of this approach are clear (e.g. much higher storage requirements) but the benefits are not obvious from the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OqJRpqsYdtQ", "reviewer": "Reviewer_kndf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to learn image-based representations from video. The paper makes two contributions: (1) on the network side, the authors introduce simple modifications to the way augmentations are applied (e.g. change the scale and uniform temporal sampling ) and a simple attention head for attention pooling to replace average pooling before the features are passed to the MLP. (2) on the data side, a new video dataset is proposed using a data curation process which tries to address the domain gap between imagenet and other video datasets. They evaluate on semantic segmentation and object detection after fine-tuning.", "summary_of_the_review": "The paper reports some results that might be useful for the community, but I believe more work is required to increase the impact of the proposed method.", "strength_and_weaknesses": "On the positive side:\n+ The paper reports some results that might be useful for the community. \nOn the negative side:\n- The methodological contributions are quite thin, mainly comprising minimal changes to existing strategies for augmentation and attention pooling (which is also very standard).\n- The method is shown to work on the proposed curated dataset. Although the point of using a video dataset close to imagenet for a fair comparison is valid, in my opinion it also weakens the paper's contribution in a sense that learning from uncurated video data is a much more challenging and impactful goal. \n- Experiments: from the experiments (table 1) it is not clear how VITO performs when trained on other video datasets (like Kinetics or Youtube 8M). Similarly, it is not obvious how other previously proposed methods would perform when trained on the newly proposed curated dataset. I suppose a strong baseline would be also to train DINO on the new dataset.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7UQJjiuh6LJ", "reviewer": "Reviewer_zSuF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a series of hyper-parameter changes (i.e. scale of the crop, frame sampling) and data filtering for learning representation from videos. The changes are implemented on top MoCLR. On the dataset and tasks tested the paper results show that video pretraining can match imagenet pretraining. However, the technical novelty is limited and the experimental results insufficient.", "summary_of_the_review": "While the result itself is interesting, the evaluation is very limited, with insufficient ablation and comparisons. Moreover, on the technical side, the novelty is very low as the approach is a direct implementation of an existing technique with appropriate hyper parameter tuning for video data.", "strength_and_weaknesses": "Strengths:\n\n- Overall, the paper is well written\n- The results confirm that with careful considerations (by filtering the data and tuning the augmentations) video based pretraining can match in certain conditions imagenet pretraining.\n\nWeaknesses and questions:\n\nLow technical novelty:\n\n- the method itself is a direct adaptation/implementation of MoCLR.\n\n- The study of adapting the augmentation to videos, while reasonable, is straight-forward and its more of a hyper-parameter tuning than an actual contribution (e.g. setting the augm scale and frame sampling).\n\n- The data filtering pipeline addresses the distribution mismatch between ImageNet and the video dataset. It's not surprising that matching the data distribution will improve the performance and such pipelines, are common, albeit perhaps not for this very specific use case.\n\nInsufficient evaluation, ablations and comparisons:\n\n- As multiple frames are present in a video, counting all the frames, the filtered dataset is still 1 order of magnitude higher than imagenet. What is the performance when the same number o frames as that of the imagenet images are used?\n\n- Expanding on the prior question, what is the performance of a model trained with a single frame (most representative according to the imagenet classifier used for filtering)? This should hopefully emphasize how much the additional dynamic present in video helps.\n\n- No results and ablation with other backbones: how does the performance vary when the backbone is larger (e.g: ResNet-50-2x, ViT-B/L) and respectively smaller?\n\n- Current state-of-the-art for video recognition is achieved by initializing from pre-training on static images. How does the performance vary?\n\n- Insufficient evaluations: only object detection and segmentation considered. What is the performance on classification for example? What is the few-shot performance of such models?\n\n- How does this compare with imagenet-22k pretraining? How would the filtering perform based on these categories?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject