Self-supervised video pretraining yields robust and more human-aligned visual representations
Authors: Nikhil Parthasarathy
S. M. Ali Eslami
Joao Carreira
Olivier J Henaff
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised learning
contrastive
video pretraining
representation learning
visual representation
human alignment
robustness
shape-bias
saliency

Abstract: Humans learn powerful representations of objects and scenes by observing how they evolve over time. Yet, outside of specific tasks that require explicit temporal understanding, static image pretraining remains the dominant paradigm for learning visual foundation models. We question this mismatch, and ask whether video pretraining can yield visual representations that bear the hallmarks of human perception: generalisation across tasks, robustness to perturbations, and consistency with human judgements. To that end we propose a novel procedure for curating videos, and develop a contrastive framework which learns from the complex transformations therein. This simple paradigm for distilling knowledge from videos, called VITO, yields general representations that far outperform prior video pretraining methods on image understanding tasks, and image pretraining methods on video understanding tasks. Moreover, VITO representations are significantly more robust to natural and synthetic deformations than image-, video-, and adversarially-trained
ones. Finally, VITO’s predictions are strongly aligned with human judgements, surpassing models that were specifically trained for that purpose. Together, these results suggest that video pretraining could be a simple way of learning unified, robust, and human-aligned representations of the visual world.

Self-supervised video pretraining yields robust and more human-aligned visual representations
Nikhil Parthasarathy'
S. M. Ali Eslami João Carreira
Olivier Hénaff
Google DeepMind
Abstract
Humans learn powerful representations of objects and scenes by observing how they evolve over time. Yet, outside of specific tasks that require explicit temporal understanding, static image pretraining remains the dominant paradigm for Iearn- ing visual foundation models. We question this mismatch, and ask whether video pretraining can yield visual representations that bear the hallmarks of human percep- tion: generalisation across tasks, robustness to perturbations, and consistency with human judgements. To that end we propose a novel procedure for curating videos, and develop contrastive framework which learns from the complex transforma- tions therein. This simple paradigm for distilling knowledge from videos, called VITO, yields general representations that far outperform prior video pretraining methods on image understanding tasks, and image pretraining methods on video understanding tasks. Moreover, VITO representations to natural and synthetic deformations than image-, video-, and adversarially-train ones. Finally, VITO's predictions are strongly aligned with human judgements, surpassing models that were specifically trained for that purpose. Together, these results suggest that video pretraining could be simple way of learning unified, robust, and numan-aligne representations of the visual world.
significantly more robust
Introduction
With the explosion recent AI breakthroughs, humans now interact with and depend on the outputs of these models an unprecedented rate. It is therefore increasingly important that these models be aligned with human abilities, judgements, and preferences. In the context of computer vision systems, human alignment can be quantified with accurate generalization across wide range of tasks [1-3], robustness to various input deformations [4], and consistency with human perceptual judgements [5]. While each of these challenges has been tackled separately, progress along axis has often come at the expense of the others. For example, gains in robustness [6] or temporal understanding [7-9] have thus far come at the cost of spatial understanding. and scaling the model and dataset size, while improving fask-generality and robustness [10, I], can be detrimental for their consistency with human perception [11. 12].
In this work we question this trend, and ask whether improvements to all aspects of human alignment can be made with the appropriate pretraining methodology. Specifically, humans and animals have long been thought to learn from the dynamic evolution of natural scenes [13-15] and we hypothesize that artificial visual systems will be more aligned by appropriately leveraging natural video pretraining. In particular, while many current self-supervised methods [16-19] learn representations that are invariant to synthetic augmentations that capture important image priors such as scale-, color-, and consslatio-invarinnne these represent small part of the complex (and signal-rich) changes in pose, viewpoint, and motion that captured from natural videos. Predicting the evolution of videos is also a natural means of learning intuitive physics and model based reasoning [20-22].
Current affiliaaion: NYU Center for Neural Science, work done while interning at DeepMind.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Practically, we develop a self-supervised contrastive framework which learns to locate the most stable and distinctive elements in temporally displaced video frames, and maximizes their invariance. Secondly, we find the statistics of standard video datasets to have a detrimental effect on the quality of the resulting representations, as measured by their performance on canonical scene understanding tasks. We therefore introduce a simple, yet powerful video curation procedure-Vident -VideoNet-wiie aligns their class distribution with that of ImageNet, and which redresses the imbalance between image and video learning. In concert, this paradigm constitues new methodology for distilling the knowledge of videos into visual representations: VITO.
VITO yields musk-geneaal representations that perform well across both spatial and temporal under- standing tasks. Particularly, VITO shows large gains over prior video pretraining efforts in scene understanding tasks, while achieving similarly large performance gains over image pretraining on video understanding tasks. Furthermore, VITO significantly outperforms the default ImageNet pre- training well as adversarial pretraining on image classification tasks subject to natural distribution shifts. Finally, we find that even without a significant expansion in model size, VITO not only task-general and robust in performance, but also quantitatively captures multiple aspects of human perceptual judgements, surpassing models specifically trained for that purpose.
2
Related work
Learning general visual representations from videos. Many prior works have considered self- supervised representation learning for capturing spatio-temporal invariances, beginning with methods that leveraged temporal coherence, optical flow, and object tracking [23-31]. More recently, many successful approaches have leveraged contrastive learning, masked autoencoding, and other self- supervised pretext tasks to learn strong video representations [32-38]. However, most of these methods employ specialized video architectures and only transfer to video-based tasks such as action recognition and motion segmentation.
Yet natural motion-induced deformations are powerful learning signals that should allow for learning better image representations as well. Indeed, human infants can form complex understanding of objects and shape within months, specifically driven by their observations of how they move [39, 40]. Given this inspiration, some works have demonstrated that self-supe contrastive learning in videos can lead to aspects of efficient human learning and robust recognition [41-43]. In computer vision, spele-consistencyy [44, 45] and optical flow [46, 47] have been used to learn correspondences between temporally ordered image patches. The most similar works to ours utilize video-based contrastive learning [7-9] to improve performance on temporal understanding tasks, however they do so at the cost of spatial scene understanding.
Robustness to distribution shifts. As standard benchmarks have been progressively saturated [48], the community has turned to measuring robustness to adversarial attacks [49], corruptions [50], and out-t--distribution datasets [4, (1-53J. We focus on a subset of these benchmarks that are as "natural" as possible, evaluate generalization with respect shifts that are most likely to appear in the real world. While there have been many efforts to specifically encourage regularize models for these kinds of robustness [54-57], we instead investigate the complementary question of whether image and video pretraining differ in this respect.
Human-aligned representations. Most recent progress in achieving more behaviorally-matcher representations has been by scaling existing approaches. Indeed, recent examples [10, 18) show that as data and model sizes grow by orders of magnitude, generality and robustness of representations tend to emerge. Moreover some aspects of human perception such as an increased thape-bias and consistency with human perceptual behavior [11, 59] can be captured reasonably well by certain large models. However this scaling property tends be brittle, with some large-scale models displaying significantly worse consistency with human perception [11, 12]. Additionally, more recent work on alignment has found that scaling and architecture are not as important for alignment on specific benchmarks, in comparison to the training dataset and objective function [60]. Therefore, while scaling may continue to lead to task-performance gains, it is unclear whether only scaling image-based pretraining will close the gap with general human behavior. We therefore explore the complementary and potentially synergistic question of whether video pretraining can improve the task-generality, robustness, and behavioral similarity of learned visual representations.

---

frame
view
hiddens
ottention mask
sugment
fo
a
mask pooling
frame 2
view
EMA
Contrastive loss
augment
fe
ag
Figure 1: Learning to attend related vidco content. Each augmented frame is encoded by the network as spatial array of hidden vectors. The attention module takes imput features from one view and produces mask that isolates features that are likely to predictive of the other, temporally-displee features are pooled accordingly, and both the feature extractor and attention module are trained to satisfy the contrastive objective. Subscripts 0 and refer to online and target (EMA) networks respectively.
view. The attention attention-gated
Method
We pretrain image representations using video datasets, then transfer them to a range of downstream tasks that test image, video, and robust understanding. We adopt the ResNet-50 architecture for our initial exploration, then validate our results with Swin transformers (see Sec. B.4).
3.1 Self-supervised pretraining
Our method for distilling videos into image representations, VITO, builds robust visual representa- tions by learning to track stable and distinctive content in videos while they evolve over time.
Natural video pipeline. The key to our method to distill the natural transformations present in videos into image-based based representations. Given video-clip, sample frames according to distribution T and further transform each frame with image-based augmentations:
(1) where the distribution T samples frames uniformly from video clip of length T' = 2.56s and the image transformations At include random cropping, flipping, blurring, and point-wise color transformations [61], see appendices A. and B.2, and Figure B.3 for an ablation.
v! A1 (211
v² 2~~ A2(x2
11, 22 ~ T(
We note that video frames (or even uncurated image data) typically differ from the statistics of (object centered) ImageNet images, with more variable viewpoints and larger field-of-view that can cover multiple objects in complex scenes. As a result, the aggressive random cropping from [61] (whose smallest crops cover only 8% of the original image) can result in "positive" pairs with very different semantic content (e.g. entirely different objects). We therefore suggest and empirically validate that larger crop sizes (e.g. increasing the minimum crop size to 40%) are beneficial when learning from real-world video frames (see Figure B.2).
Multi-scale contrastive attention pooling, Standard contrastive frameworks use global average pooling of hidden vectors to obtain single representation of each view. It has been shown that using dense contrastive losses can lead to significant improvements [62-65], but these methods require establishing correspondences across views. Whereas correspondences can easily be obtained from statie images, when temporal deformations are introduced they require some form of object or point tracking [46]. Furthermore, with the larger field-of-view of video frames, correspondence learning becomes an increasingly difficult task. this work, we propose a more general, adaptive method for learning correspondences at multiple scales. Our method learns what features should be attended to in order to solve the contrastive learning problem across temporally displaced views.
As shown in Figure 1, given view v' the feature extractor outputs spatial map of feature vectors L.s R xxxxxe at given scale where different scales correspond to the outputs of different blocks h/

---

of a ResNet for example. At each scale, we introduce a 2-layer attention MLP aj which outputs a mask m softmax (ha (hl))) that we use spatially weight and pool hidden vectors:
L. g
Σ m mi, [i,
which we we concatenate and transform with the two-layer MLP projector: z/1 go())) where h2 [hg s E I...S]. In our experiments, we find that for the canonical ResNet-50 architecture. attending over the outputs of the last two ResNet blocks (i.e. S = 2) is optimal given our evaluations. These hidden vectors are then transformed with a standard two-layer MLP go. yielding projections z/} - go (h)). We enforce invariance across views using the standard InfoNCE loss [66], encoding targets with slowly-varying larget networks Se and g that are exponential moving averages of the online network [61]
exp(z) x
C (0: =- -log
exp(z)) x + En exp(z z
{="),, are negative features computed from frames from other videos in the batch. The final, multi-view loss is evaluated for all pairs [(0:E) = Σitj 2 (0;
3.2 Addressing dataset domain mismatch
We began investigating the potential for learning general representations from videos, using standard datasets including Kinetics, AudioSet, and YouTube-8M.. However, Kinetics is quite small and is limited in scope human actions. On the other-hand, AudioSet and YouTube--M are noisy and have very imbalanced class distributions. Additionally, prior work has shown that even self-supervised methods are quite sensitive the pretraining distribution [67]. Yet over the last decade, it has been shown that ImageNet can be used for learning image representations that transfer well to many downstream tasks. As a result, hypothesized that collecting minimally-curatee video dataset matched to the rough properties of ImageNet would be beneficial for learning more general visual
model from videos.
To test of this hypothesis, we developed a data curation pipeline- pipeline-VideoNeer VideoNet to filter online videos such that our training data more closely matches the distribution of ImageNet categories. For each of the 1 1,000 ImageNet categories, we retrieved 5,000 video clips whose title included the category's 's name synonym. We then filtered these videos by applying 50 on ImageNet) to verify that the videos contained the intended object category. We classified the first 100 frames of each video and discarded videos for which the query category was not equal the ResNet's top-1 prediction for any of the frames. We also discarded videos of less than 10s in length.
image classifier (pretrained ResNet-
While the VideoNet procedure is close in conceptualization to the method used to create the R2V2 dataset proposed by Gordon et al. [7], it differs in a few ways. First, we utilize full video clips that allow us to uniformly sample frames at any time point rather than the fixed sampling of frames that are 58 apart in R2V2. Second, by using the ImageNet classifier to filter videos, we can reduce mismatch with the ImageNet distribution that can arise from incorrect tagging and noisy labeling of online videos. This is verified by the fact that only 1.18M of the 5M retrieved videos met our filtering criteria. We also note that the of classification-basse While we demonstrate in Sec. 4.3, that this curation does provide large benefits in the context of video pre-training compared with existing datasets, there is still great potential to make improvements by utilizing larger target datasets (such as ImageNet-22K] and utilizing alternative curation strategies such as the nearest- neighbor retrieval proposed by [10] in creating the LVD-142M, image dataset.
filtering is just method of curation.
4 Results
Humans are able to solve a range of visual tasks that require complex spatial and temporal reasoning, including generalizing to noisy or out-of-distribution (OOD) scenarios. Therefore, we first benchmark VITO against image and video pretrained models on variety of tasks to demonstrate sufficient generality and robustness task performance. We then assess whether VITO not only captures these bask- properties, but also displays strong quantitative alignment with human behavior.

---

4.1 VITO generalizes across diverse visual tasks
We present in Table the transfer performance of VITO compared to strong supervised and self- supervised baselines on dense scene understanding (semantic segmentation and object detection), video understanding (video segmentation and action recognition), and out-of-distribution (OOD) object recognition. On every benchmark, VITO either outperforms or competitive with the best baselines for that specific task.
Scene Understanding
Video Understanding
OOD Recognition
Pretraining
Dataset
ADE20K (mloU)
COCO (mAP)
DAVIS (J&F mean)
UCF101 (top-1)
IN-A (top-1) I)
IN-Vid (pm0/
pm10)
Random
27.9
39.0
Standard image pretraining
Supervised
ImageNet
33.5
44.2
66.1
83.4
2.2
67.7/52.4
BYOL [61]
ImageNet
38.8
43.7
66.6
85.6
MoCLR [68]
ImageNet
39.2
43.9
65.5
85.5
3.7
64.7/50.0
DINO [19]
ImageNet
39.0
44.3
65.3
85,4
5.0
65.2/52.0
Robust image pretraining
Stylized- IN [56]
SIN+IN
83.3
2.0
68.4/51.7
L2-Robust [54]
ImageNet
83.7
2.1
65.2/51.6
Video pretraining
VIVI [69]
YT8M
34.2
41.3
0.5
57.9/36.5
MMV-VA [70]
AS + HT
32.5
41.3
VINCE [7]
R2V2
35.7
42.4
66.1
VFS [8]
K400
31.4
41.6
67.8
CycleCon [9]
R2V2
35.6
42.8
82.8
0.4
50.4/30.1
VITO
VideoNet
39.4
44.0
68.2
87.4
5.4
70.6/57.2
Table 1: VITO representations generalize variety of tasks in both image and video modalities, surpassing models specialized for each task. For external models, we finetune publicly available checkpoints.
Scene understanding. We first note that VITO provides large gains over all prior video pretraining methods on scene understanding and robust object recognition. We further validate these comparisons on three additional benchmarks and find that VITO strongly outperforms the prior work across all datasets (PASCAL/ADE2O//COOOOO VIVI [69] by 2- 2-10%, highlighting the importance of data curation and our contrastive formulation. VITO improves over VINCE [7] by 1-12%, highlighting the importance of fine- eraiggr temporal deformations. Finally, VITO improves over MMV [70] by 2-15%, despite their use of large-scale scale text supervision, highlighting the relevance of video-only learning.
see Table B.3). For example, VITO improves over
Compared with the best supervised and self-supervised maatr-iretraining methods VITO achieves competitive performance on these same benchmarks (Table and Table B.3). To our knowledge, VITO is the first video pretrained method to close the gap with ImageNet pretraining on large-scale scene understanding benchmarks such as these.
Video understanding. We next ask whether this increased spatial understanding come at the cost of traditional benefits of video pretraining video tasks. We find that this is the case, evaluating on DAVIS segmentation and UCF-101 action recognition. On DAVIS, which tests the ability segment an object over its dynamic temporal evolution, VITO features capture fine-grained temporal deformations of objects far better than ImageNet pretraining methods, as well as the best video pretraining methods (See Table B.4 for additional comparisons). On 101, which tests the ability to classify global apatio-temporal features, we find that simple average pooling of VITO frame representations again outperforms all image pretraining and prior frame-based based video pretraining significantly. VITO even outperforms number of recent methods that use specialized video architectures (See Table B.5). While VITO under-performs relative to the best video models, we note that these methods either cannot be tested or under-perform on spatial understanding, Additionally, as shown in Table B.5 and Sec. A.5, simple learned temporal pooling strategies on top of VITO representations further close the gap with the best video architectures.
5

---

ImageNet-3DCC Validation
ImageNet-3DCC Validation
-20
Corrupt- Cloan Accuracy)
-20
-25
-55
Clean
-30
-30
-35
35
WiD-hatin
-00
VITO Locaeekakktn
MoCLA VidcoNet
VIO (T=0)
MatzR Videottel
-05
MaCLR
MoCLR magetett
Corruption Severity
Corruption Severity
Figure ImageNet-3DCC validation accuracy for different levels corruption severity. (Left): Comparisons with prior work including methods specifically designed to enhance robustness (SIN+INIK and L2-Robust). (Right): comparisons with ablations of the VITO method/model.
Object recognition under distribution shifts. A key feature of human perception is being able to generalize under distribution shifts away from the training data. The standard ImageNet benchmark does not test this, as the validation set is drawn from a similar distribution as the train set. We hypoth- esize that while ImageNet pretraining can lead strong performance in-distribution, pretraining on videos can endow models with better generalization capabilities.
We thus evaluate nition under natural shifts we evaluate on the ImageNet- (Table I). imageNet-Vid-Robue Vid-Robust tests generalization of image classifiers to natural deformations over time. The anchor frame is identified as the cleanest frame capturing the object, and as time evolves, recognition becomes more difficult. We see that VITO surpasses all models on the anchor frame accuracy (+3% relative to supervised ImageNet training for pmi),, but more importantly, the accuracy gap grows for the largest temporal displacement (+5% for pm10). ImageNet-A on the other hand contains ImageNet-like images that systematically fool ImageNet classifiers (i.e. 'natural adversarial examples"), ) On this dataset, while performance is very low across all models, VITO again shows more robustness. For additional comparison, we also evaluate two models (SIN-IN and L2-Robust (c= 1)) which are models trained specifically for robustness (to shape-bias and adversarial attacks respectively). While SIN-IN yields modest improvements on ImageNet- Vid-Robust, neither method approaches the gains in robustness afforded by VITO.
suite of benchmarks designed to test distributional robustness [4]. To test recog-
and ImageNet-A benchmarks
Finally, we evaluate robustness on the ImageNet-3DCCC dataset, which contains naturalistic and synthetic corruptions applied to clean images from the ImageNet validation set [53]. To test robustness to conditions of real-world deployment, we choose the subset of corruptions designed with 3D models be consistent with scene geometry. These include things like fog, near/far focus, motion blur, etc. and have 5 different severity levels per image. In Fig. 4.1 (Left), we plot the difference in accuracy between clean (ImageNet val) and corrupted accuracy across severity levels. This "A -cccuurry provides measure of how robust a model is as distortion levels increase. We see that across all corruption strengths, VITO shows increased robustness compared to supervised and supervupee (MoCLR, DINO) ImageNet pre-trained models. The robustness gap grows significantly at the highest corruption levels, demonstrating the generality of this effect (+ 10% relative to supervised ImageNet training). While the robust training methods (SIN+ INKK and 1,2-Robust) outperform supervised and MoCLR models, VITO remains significantly more robust, demonstrating that learning from video deformations may endow more general form of robustness than that provided by either style-transfer or adversarial images.
To quantify further the specific impact of individual components of VITO on robust recognition, we show the same plot (Fig. 4.1 (Right)), now with the ablations described in Sec 4.3. We find that all components of our method and architecture are necessary for best robustness, but in particular there a striking split between models trained with only spatial deformations (VITO (T=0), MoCLR ImageNet, McCLR VideoNet) and those trained with video deformations. We find that the models that learn only from image-level spatial deformations suffer significantly robustness against all of the models that learn from video deformations.

---

Human
Harmonized
Trained for Human alignment
Method
Alignment
CLIP
MoCLR [68]
x
21.4
Supervised
X
34.4
Supervised
CLIP [58]
X
41.8
Harmonized [72]
V
45.5
VITO
X
47.7
Figure 3: Example human saliency maps from the ClickMe dataset [71] and ResNet-50 models. Gradient- based saliency shown for Supervised and Harmonized [72]. Attention maps are shown for CLIP and VITO model. We use multi-heaa attention pool weights for CLIP and average of weights from last attention pool- ing scales VITO.
Table 2: Quantitative comparison between gradient-based saliency maps (from Super- vised. MoCLR, CLIP-RN50 (attention-map)) and Harmonized networks), VITO attention weights, with human saliency maps using correlation based alignment score from [71]
4.2 Measuring explicit human-alignment
Given that VITO representations display strong generalization across many tasks and robustness to distribution shifts, two signatures of human perceptual intelligence, we now directly ask whether they align with human perceptual representations.
Visual saliency via contrastive attention. We start by comparing VITO's learned attention masks human saliency data from the ClickMe dataset [71], as well saliency maps obtained from collection of ResNet-50 models. For the supervised and MoCLR ResNets we use standard gradient- based saliency as in [72]. Since our model contains two attention maps at two scales of the ResNet, we upsample both maps to the image size and simply average them to obtain single map. We compare our attention maps additionally to those obtained from the modified CLIP ResNet [58], which also utilizes attention-pooling in the final layer but is trained for image-language alignment (the canonical approach for training state-of-the-art visual language models). Because the CLIP pooling uses multi-head attention, we upsample these maps and average them across heads. Finally, we also compare to the gradient-baat based saliency maps from a "harmonized" model explicitly trained to align with human saliency ([72)).
Qualitatively, VITO saliency maps appear significantly more aligned with human perception maps than the supervised and CLIP ResNets (Figure 3). Surprisingly, VITO appears more aligned than the Harmonized saliency maps across the examples. Quantitatively (using Spearman rank correlation) VITO outperforms the supervised, MoCLR, and CLIP models by a large margin, and even surpasses Harmonized model which has been specifically trained for this purpose (Table 2).
This result suggests that as opposed to image-base based objectives or image-language alignment, human perception of feature importance across the visual scene can be better explained as a consequence of learning what to attend to in the context of self-supervised video-based learning. We hypothesize that these attention masks could underlie the formation of high-level concepts via "semantic binding". which we investigate in Figure B. and Section B.I.
Human error consistency in shape-biased tasks. Based this result relating object saliency, we hypothesize that VITO may be capturing global object shape features better than traditional deep networks which have been shown to heavily rely textural cues for classification [56].
To evaluate this quantitatively, we used a subset of the dataset proposed in [59] to test both the accuracy and consistency with human judgments of model classifications of stimuli that require
7

---

Method
accuracy diff. 4
obs. consistency
ceiled error consistency 1
Image pretraining
DINO [73]
0.236
0,504
0.291
Supervised
0.215
0,511
0.329
SIN+INIK [56]
0,203
0,527
0.330
MoCLR [68]
0,190
0,536
0.335
1.2-Robust [54]
0.178
0,544
0,389
CLIP [58]
0.108
0,612
0.482
Video pretraining
R3M [74]
0.392
0.359
0.054
CycleCon [9]
0.237
0,484
0.258
VINCE [7]
0,210
0,501
0.269
VITO
0.157
0.564
0.422
Table 3: Accuracy difference and consistency with human judgments on stimuli that are biased to requiring global-shape understanding (instead of texture) for recogntion/discriinnttoo VITO surpasses all comparable trained models (both image and video pretraining) in all benchmarks, including those that are trained specifically he robust (SIN+INI,, and L2-rrobstt.. We underperform the CLIP model; however, note that CLIP trained with an order magnitude more images (400M) and explicit human-language supervision.
shape-cues for effective discrimination (Table 3). Specifically, these stimuli are categorized into groups: edge drawings, sue-conflict/ stylized (mixing of shapes with contradictory textures through style-transfer, variable low-pass pass filtering (to remove high-froee figh-frequency local content), uniform noise (corrupts local texture features). Based on the original methodology proposed in [5], we report the accuracy difference (from human accuracy), the raw consistency with human judgments, and ceiled error consistency (method from [5]).
We compare to supervised and MoCLR ResNets, the robust training methods cited earlier, well as CLIP [58]. We also compare to various video pre-training methods cited earlier and another (R3M [74]), which has specifically shown to have human- and neurally-aligned representations of dynamic, object-centric scenes [75]. For all networks, we train linear classifiers on the ImageNet validation set and evaluate on the modified shape- biased stimuli. Compared with all other comparable image pretrained models, VITO achieves stronger robustness to shape-biasing transformations (lower accuracy difference relative to original images). Furthermore, VITO makes predictions more consistent with human judgements in terms of per-trial classification behavior. This is particularly surprising as VITO even outperforms the adversarially-trainee robust model without requiring any explicit robust training procedure. Moreover, this improvement is not captured by prior video pretraining efforts (which are in fact far worse than the image pretraining methods). The R3M model, in particular, performs surprisingly poorly. Because the images used collect the human judgmentss are modified versions of those from the ImageNet validation set, we hypothesize that this performance be attributed to the poor transfer of the Ego4D datasets the diverse classes present in ImageNet (contrarily to VideoNet). Indeed, the R3M model only achieves 13% accuracy on the clean ImageNet validation set (see Table B.3). Finally, we note that VITO does underperform CLIP on this benchmark; however, this comparison is not truly fair as CLIP is trained with explicit human supervision via large-scale image-language mappings. In fact, we believe that our method can be augmented with similar language supervision to improve human alignment even further.
In summary, VITO captures aspects of how humans process shape- informmttoontoion that cannot be captured by other strong visual models. Understanding more about this effect and what aspects of learning from videos lead to this remain interesting opportunities for future work.
4.3 Ablations
To understand more about how the components of VITO training contribute its performance, we vary the different aspects of our paradigm in isolation: our method for data curation (VideoNet). multi-scale attention pooling, and details the input data (spatial crop size and the temporal sampling scheme). We explore some ablations in detail on an example benchmark (PASCAL segmentation), but also evaluate ablations across many of the benchmarks used in this work. Finally, we provide brief exploration demonstrating that our method scales well to larger architectures.
8

---

PASCAL semartic sagmentation (mlol))
Video pre-training Image pre-training
Kinetics700
Audioset
VideoNet
ImageNet
78-
76-
74-
DELAN TOVEL
66
Kinetics700
You' nuTubuuM
How hnwToloos
AudloSet
VideoNet
JPy
Imagelet
Figure 4: Impact of pretraining data's spatial content on representation quality. Left: transfer performance of models pretrained on single frames from image datasets (grey bars) or individual videos (blue hars). Right: example frames from different video and image datasets.
Effect of pretraining data. To demonstrate the effect of the pretraining data distribution on transfer performance, we pretrain a baseline MoCLR model (using 2 views) on a variety of image and video datasets, where initially treat video datasets collections of individual frames. We train each model for 300 ImageNet-cquivalent epochs, referred to hereafter as "epochs" (i.c. epoch = learning from 1. 1.28M examples, irrespective of the dataset), such that each model benefits from the same amount of computation. Figure 4 (left) shows their transfer performance on PASCAL semantic segmentation. As expected, ImageNet pretraining works very well, but pretraining standard video datasets results in a substantial drop in performance (e.g. -6.8% or -5% mloU from pretraining on Kinetics700 or AudioSet). This performance gap between video and image pretraining can be attributed to a combination of increased complexity and field-of-view of video frames and domain mismatch between the dataset categories (Figure 4, right). Consistent with this, training on JFT [76]. an uncurated dataset with a heavy-tailed class distribution, also results in loss in performance. Notably, this is despite the much larger size of JFT (300M images). We find that applying the same baseline pretraining to frames from our curated video dataset performs better than existing large-scale video datasets like Audioset 1+1.6% mloU), but still underperforms image pretraining on JFT and ImageNet (Figure 4). This demonstrates the importance of aligning the distribution of video frames with that of common image datasets. We therefore use VideoNet as our primary pretraining dataset for the rest of the study. In Sec B.3 we disentangle the power of our method and dataset by confirming that each independently have strong effects: MoCLR trained on VideoNet, and VITO trained on standard datasets (Audioset or YT8M) also outperform all prior work (including models trained on much larger image datasets like JFT-300M).
Multi-scale attention pooling. We decompose the proposed multi-scale contrastive attention pooling to isolate the effects of multi-scale learning from those of attention pooling (Figure B.2, right). While we find only modest gains from adding attention pooling to a single-scale version of the model (+0.2% mloU), we find that the 2-scale model (without attention pooling) improves over the single scale model more robustly (+0.6% mloU). Interestingly, we find that the combination of the 2-scale model with attention pooling has a synergistic effect (+1% mloU over the single-scale attention model), highlighting the importance of handling the variability in scales present in natural videos.
Spatial and temporal augmentation parameters. We first validate in Figure B.2 (left) our hypothe- sis that increasing the minimum crop-scale in the random-resized crop operation during training leads to models that generalize better to fine-gre -uu-ggr tasks like semantic segmentation. Specifically, we find that minimum crop scale of 0.4 (as opposed to the traditional 0.08) results in the best transfer performance (+1.7% mloU). Note that this conclusion differs slightly from that of [37] who find more aggressive cropping to be beneficial for action recognition.
Next, to study the effect of different temporal sampling schemes, for each training example, we sample 3 views using marginal sampling of each frame from the video clip of length T = 2.56 seconds. This length determines the distribution of time differences between any pair of frames, and thus the time-scan scale over which the contrastive model learns invariances. We verify our choice by varying the total length of clips. While going to longer time-scales performance much, we find significant improvement over using shorter clips (c.g. T = 1.28s,
3.2s does not hurt

---

Pretraining
Dataset
PASCAL (mloU)
UCF101 (top-1)
IN-A
IN-Vid (pm0/pm10)
Human
(top-1)
sistency
MoCLR
VideoNet
72.8
83.0
2.3
53.5/40.5
0.224
VTTO
Iscale (w/o attn)
VideoNet
75.2
85.5
3.9
67.3/55.5
0.359
VTTO
Iscale (attn)
VideoNet
75.4
85.7
3.5
65.6/52.9
0.368
VTTO
2scalc (w/o attn)
VideoNet
75.8
86.2
4.2
67.4/54.9
0.390
VITO
(T=0)
VideoNet
74.8
83.2
3.9
63.9/49.5
0.323
VITO
AudioSet 73.8
84.8
3,4
55.7/42.4
0.401
VITO
VideoNet 76.3
87.4
70.6/57.2
0.422
Table 4: Summary ablation models on key evaluations covering image understanding, video understanding. and human alignment on ood object recognition. In summary, is clear that components (pretraining data, temporal deformations, and the multi-scale attention pooling) are required for best performance across all tasks.
+ +1.0% mloU; Figure B.2, center). This suggests that invariance to the rich temporal deformations present in video clips is indeed beneficial criterion for learning fine-grained spatial representations.
Comprehensive ablation summary. In Table 4, we extend the above ablation studies to more comprehensive benchmark set. In addition to the PASCAL segmentation task, we evaluate the key ablated models on video understanding (UCF101), OOD recognition (IN-A/IN-Vid Vid) and human alignment on the shape-bias tasks specified in Sec 4.2. We confirm that all of the major methodological components (VideoNet dataset, multi-scale attention pooling, and using temporal deformations) work in concert, and are required for best performance across all tasks. Notably, we see a particularly striking dichotomy between models trained with and without temporal deformations on human error-consistency. Specifically, models trained without temporal deformations (MoCLR and VITO (T=0)) have significant drop in human error-consistency relative all other models trained with temporal deformations, highlighting the importance of learning these kinds of invariances.
Scaling model architectures. We briefly demonstrate that VITO scales to more recent larger architectures. Specifically, show preliminary results that VTTO achieves highly competitive performance on four scene understanding benchmarks using the Swin-S transformer architecture [77]. In Sec. B.4, we show that performance improves dramatically over the ResNet-50 architecture and is competitive with a strong, specialized ImageNet pretrained baseline for fine-grained scene understanding (DetCon [65]].
5 Discussion
Summary. We propose VITO, simple method for distilling videos into visual representations. The key features of our method include improved dataset curation, adapting augmentation pipelines to appropriately handle video frames, and using srrention-guided contrastive learning. With these com- ponents, VITO surpasses both prior video pretraining in spatial understanding, and image pretraining on temporal understanding and robustness. addition to these hallmarks of human perception, VITO explicitly aligns with aspects of human saliency and image recognition behavior that are not captured by other high-performance representation learning techniques. In sum, despite the many successes in video representation learning, our results suggest that there is great untapped potential in video pretraining as paradigm for learning general, human- human-aligned visual representations.
We believe this work can be foundation for future video pretraining efforts, as our approach is powerful, yet simple and extensible. However, we recognize that this demonstration is mostly limited to single contrastive learning framework and ResNet-50 architecture. We leave for future work, the validation and exploration of similar analyses with larger models and other self- supervueee training objectives (such as MAEs and self-distillations methods like DINO). Additionally, while we have shown the benefits of a surprisingly simple attention module learning correspondences in video data, there are more powerful attentional architectures we can leverage along with scaling dataset size as in [10]. We have started these experiments with our exploration of Swin transformer architectures.
Limitations and Future Work.
10

---

References
[1] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: retrospective. International journal of computer vision, 111(1):98-136, 2015.
[2] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torraba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 633-641, 2017.
[3] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint 1rXiv:1212.0402, 2012.
[4] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. Advances Neural Information Processing Systems, 1588339918599 2020.
[5] Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of enns and humans by mcasuring error consistency. Advances Neural Information Processing Systems, 33:1 1138903919000 2020.
[6] lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
[7] Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled videos. arXiv preprint a7X1v:2003.07990, 2020.
[8] Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A vidco frame-level similarity perspective. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10075-10085, 2021.
[9] Haiping Wu and Xiaolong Wang. Contrastive learning of image representations with cross-video cycle- consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10149-10159) 2021.
[10] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0711 2023.
[11] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Thrahim Alahdulmohsin, al. Scaling vision transformers 22 billion parameters. arXiv preprint arXiv:2302.05442. 2023.
[12] Manoj Kumar, Neil Houlsby, Nal Kalchbrenner, and Ekin Dogus Cubuk. Do better imagenet classifiers assess perceptual similarity better? Transactions of Machine Learning Research, 2022.
[13] Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217-233, 1961.
[14] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical ficeptie-field effects. Nature neuroscience, 2(1):79, 1999.
[15] Stephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in a sensory population. Proceedings of the National Academy of Sciences, 112(22):6908 6913,,911111 2015.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[17] Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: / 190500922.... 2019.
[18] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[19] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2021.
11

---

[20] Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience, 22(6):984-99) 2019.
[21] Peter W Battaglia. Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110445):8327 18332, 2013.
[22] Olivier Hénaff, Yoon Bai, Julie A Charlton, Ian Nauhaus, Eero P Simoncelli, and Robbe LT Goris. Primary visual cortex straightens natural video trajectories. Nature communications, 12(1))5982, 2021.
[23] Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-7700 2002.
[24] Jarmo Hurri and Aapo Hyvärinen. Simple-cell-he receptive fields maximize temporal coherence in natural video. Neural Computation, 15(3):663-691, 2003.
[25] Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015.
[26] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using vidcos. In Proceedings of the IEEE internationaa conference computer vision, pages 2799-2880, 2015.
[27] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2701-2710, 2017.
[28] Ross Goroshin, Joan Bruma, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pages 4086-4093 2015.
[29] Ishan Misra, C Lawrence Zätnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European conference computer vision, pages 527-544. Springer, 2016.
[30] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representa- tions using Istms. In Internatioonal conference on machine learning, pages 843-852. PMLR, 2015.
[31] Tejas D Kulkarni, Ankush Gupta, Catalin lonescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32, 2019.
[32] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang. Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 11 1134-1141. IEEE, 2018.
[33] Adrià Recasens, Pauline Luc, Jean- Baptiste Alayrac, Luyu Wang, Ross Hemsley, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altché. Michal Valko. Jean- -astten Grill, Aäron van den Oord, and Andrew Zisserman. Broaden your views for sule-super video learning. Int. Conf. Compur. Vis., 2021.
[34] Rui Qian, Tianjian Meng, Boqing Gong, Ming- Hsuan Yang. Huisheng Wang. Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964-6974, 2021.
[35] Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve. and Mubarak Shah. Telr: Temporal contrastive learning for video representation. Computer Vision and Image Understanding, 219:103406, 2022.
[36] Michael Dorkenwald, Fanyi Xiao, Biagio Brattoli, Joseph Tighe, and Davide Modolo. Sevrl: Shuffied contrastive video representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4132-4141, 2022.
[37] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He, A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3299-3309, 2021.
[38] Christoph Feichtenhoofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal leamnrss arXiv preprint 20Xiv:2205.09113, 2022.
2022.
[39] Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89 96, 2007.
[40] Elizabeth S Spelke, Principles of object perception. Cognitive science, 14(1):29-56, 1990.
12

---

[41] Emin Orhan, Vaibhav Gupta, and Brenden M Lake. Self-supervised learning through the eyes of a child. Advances in Neural Information Processing Systems, 33:9960-99711 2020.
[42] Chengxu Zhuang. Ziyu Xiang. Yoon Bai, Xiaoxuan Jia, Nicholas Turk- Browne, Kenneth Norman, James J DiCarlo, and Dan Yamins. How well do unsupervised learning algorithms model human real-time and life-long learnin?? Advances Neural Information Processing Systems, 35:22628-22642. 2022.
[43] Nathan Kong and Anthony Norcia, Are models trained adversarially robust? In SVRHM 2021 Workshop@ NeurIPS, 2021.
conttiolyy data streams more
[44] Allan Jabri, Andrew Owens, and Alexii Efros. Space-time correspondence contrastive random walk. Advances in neural information processing systems, 33, 2020.
[45] Zhangxing Bian, Allan Jabri, Alexei A Efros, and Andrew Owens. Learning pixel trajectories with multiscale contrastive random walks. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition. pages 6508-6519, 2022.
[46] Yash Sharma, Yi Zhu, Chris Russell, and Thomas Brox. Pixel-level correspondence for self- supervied learning from video. arXiv preprint arXiv:2207.03860 2022.
[47] Yuwen Xiong, Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. Self-supervised representation learning from flow equivariance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10191-10200, 2021.
[48] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07133 071529 2020.
[49] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019.
[50] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: /2261, 2019.
[51] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[52] Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9661-966 166619969,220 2021.
[53] Oguzhan Fatih Kar, Teresa Yeo, and Amir Zamir. 3d common corruptions for object recognition. In ICML 2022 Shift Happens Workshop.
[54] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep lcarning models resistant to adversarial attacks. arXiv preprint /706::/0006068833 2017.
[55] Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks rohust against diverse image corruptions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16. pages 53-69. Springer, 2020.
[56] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet- imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint 1r1iv:1811.122: 2018
[57] Qizhe Xie, Kinh-Thhng Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PMLR, 2021.
[59] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885 22893,,23999... 2021.
13

---

[60] Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A Vandermeulen, and Simon Kornbiith. Human alignment of neural network representations. arXiv preprint arXiv:2211.012011 2022.
[61] Lean-Basties Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre Richemond, Elena Buchatskaya, Carl Doersch. Bernardo Avila Pires. Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in Neural Information Processing Systems, 33, 2020.
[62] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[63] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16684-16600 16693, 2021.
[64] Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg. Point-level region contrast for object detection pre-training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16061-16070, 2022.
[65] Olivier J Hénaff, Skanda Koppula, Jean- Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. In ICCV, 2021.
[66] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint urXiv:/807 Net::0307035 03748, 2018.
[67] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer FisionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XI 16. pages 776-794. Springer, 2020.
[68] Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. Divide and contrast: Self supervised learning from uncurated data. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10063-10074, 2021.
[69] Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby. Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced visual invariances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13806-13815, 2020.
[70] Lean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider. Relja Arandjelović, Jason Ramapuram, Jeffrey Fauw, Lueas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-33 33,, 2020.
[71] Drew Linsley, Dan Shiebler. Sven Eberhardt, and Thomas Serre. Learning what and where to attend. arXiv preprint or / 808.08819, 2018.
[72] Thomas Fel, Ivan Felipe, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. arXiv preprint arXiv:2211.0455 2022.
[73] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski. and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
[74] Suraj Nair. Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
[75] Aran Nayebi, Rishi Rajalingham, Mehrdad Jazayeri, and Guangyu Robert Yang. Neural foundations of mental simulation: Future prediction of latent representations on dynamic scenes. arXiv preprint arXiv:2305. / 2023.
[76] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017.
[77] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022. 2021.
[78] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint 17X0v 2017.
14

---

[79] Jonathan Long, Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and patiern recognition, pages 3431-3440, 2015.
[80] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unitied perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018.
[81] Olivier Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. arXiv preprint 3XXiv:223.087777 2022.
[82] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully convolutional one-stage object detection. In Int. Conf. Comput. Vis., 2019.
[83] Shengkai Wu, Xiaoping Li. and Xinggang Wang. IoU-aware single-stage object detector for accurate localization. Image and Vision Computing, 2020.
[84] Shifeng Zhang, Cheng Chi, Yonggiang Yao, Zhen Lei, and Stan Z. Li. Bridging the gap between anchor- based and anchor free detection via adaptive training sample selection. In IEEE Conf. Comput. Vis. Puttern Recog,, 2020.
[85] Chengjian Feng, Yujie Zhong, Yu Gao, Matthew R Scott, and Weilin Huang. TOOD: Task-aligned one-stage object detection. In Int. Conf. Comput. Vis., 2021.
[86] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Int. Conf. Learn. Represent.. 2019.
[87] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross- modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recogni- tion. pages 12475-12486, 2021.
[88] Xiaolong Wang, Allan Jabri, and Alexci A Efros. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2566-2576, 2019.
[89] Xucting Li, Sifci Liu, Shalimi De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint-task self-supervised learning for temporal correspondence. Advances in Neural Information Processing Systems, 32, 2019.
[90] Jinpeng Wang, Yuting Gao, Ke Li. Yiqi Lin, Andy J Ma, Hao Cheng. Pai Peng, Feiyue Huang, Rongrong Ji. and Xing Sun. Removing the background by adding the background: Towards background robust self- superviedd video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1804-11813, 2021.
[91] Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11205-11214, 2021.
[92] Simon Jenni, Givi Meishvili, and Paolo Favaro. Video representation learning by recognizing temporal transformations. In European Conference on Computer Vision, pages 425-442. Springer, 2020.
[93] Ceyuan Yang, Yinghao Xu, Bo Dai, and Bolei Zhou. Video representation learning with visual tempo consistency. arXiv preprint arXiv:2006. 2020.
[94] Tengda Han, Weidi Xie, and Andrew Zisserman. self-super co-training for video representation learning. Advances in Neural Information Processing Systems, 33:5679 5690, 2020.
[95] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics datasct. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages -308, 2017.
[96] Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Hsuan Vang. Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE international conference on computer vision, pages 667-676, 2017.
[97] Joshua Knights, Ben Harwood, Daniel Ward, Authony Vanderkop, Olivia Mackenzie-Ron Ross, and Peyman Moghadam. Temporally coherent embeddings for self suffssupee video representation learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 891 8914-89211 IEEE, 2021.
15

---

Appendix: Implementation details
A.1 Self-supervised learning
Data pre-processing. Each frame is randomly augmented by composing the following operations, each applied with given probability:
random cropping: random patch image selected, whose area is uniformly sampled in [s. where A is the area of the original image, and whose aspect ratio is logarithmically sampled in [3/4, 4/3]. is scale haper-parameter set to 0.08 when learning from ImageNet, and when learning from videos. Regardless, the patch is then resized to 224 x224 pixels using bicubic interpolation;
2. horizontal flipping:
3. color jittering: the brightness, contrast, saturation and hue shifted by uniformly distributed offset;
4. color dropping: the RGB image is replaced by its grey-scale values;
5. gaussian blurring with a 23 x 23 square kernel and standard deviation uniformly sampled from [0.1, 2.0];
6. solarization: point- wise color transformation
10.00. + (1-x) 1220.. with pixels æ in [0, 1].
The augmented frames and o² result from augmentations sampled from distributions A/ and As respectively. These distributions apply the primitives descrihed ahove with different prohabilities, and different magnitudes. The following table specifies these parameters for the B BYOL framework [61], which we adopt without modification. When learning from three views, we the distribution Ai generate the third view.
Parameter
A
Aa
Random crop probability
1.0
Flip probability
0.5
Color jittering probability
0.8
Color dropping probability
0.2
Brightness adjustment max
0.4
Contrast adjustment max
0.4
Saturation adjustment max
0.2
Hue adjustment max
0.1
Gaussian blurring probability
1.0
0.1
Solarization probability
0.0
0.2
Optimization. We pretrain ResNet-50 using the LARS optimizer [78] with a batch size of 4096 split across 128 Cloud TPU v3 workers. We adopt the optimization details of YOL, scaling the learning linearly with the batch size and decaying it according to cosine schedule. The base learning rate is 0.3 and the weight decay is 10-6,
A.2 Transfer to PASCAL and ADE20K semantic segmentation
We evaluate ResNet models by attaching fully-convolutional network (FCN, Long et al. [79]) and fine- tuning end-to-end, following He al. [16]. When evaluating Swin transformers we instead use UperNet segmentation architecture [80].
Architecture.
Data pre-proceee rreppoeesssig, During training, images are randomly flipped and scaled by a factor in [0.5, 2.0]. Training and testing are performed with 512 images. When fine-tuning on ADE20K, aditionally use photometric transformations from the mmseg' codebase.
Inttpa://githul com.uu..oomm//eeoo com/ http.///....p...mmmm//////meeeee
16

---

tune for epochs on PASCAL train_ aug2012 set or 60 epochs on the ADE20K train set. We use stochastic gradient descent with a batch size of 16 and weight decay of 0.005. The learning rate is initially set to 0.04 and decayed exponentially with a factor of 0.9" where n the iteration number. When fine-tuning external models, we sweep over the base learning rate and weight decay and report their performance given the optimal configuration. In all cases we report mloU on the val set averaged across runs.
Optimization. We fine-tune
A.3 Transfer to COCO and LVIS object detection
Architecture. We evaluate both ResNet and Swin transformers using the FCOS' architecture, following Hénaff et al. [81]. FCOS" is the implementation of a single-stage detector based on FCOS [82], and improved with the collection techniques from Wu et al. [83], Zhang et al. [84], and Feng et al. [85], full details can be found in Hénaff et al. [81].
The target resolution is 800x 1024. During testing, an image is resized by a factor while preserving the aspect ratio, such that it is tightly contained inside the target resolution, and then padded. When fine-tuning, the image is rescaled by a factor of u s where u is uniformly sampled in [0.8, 1.25]. and is then cropped or padded to the target resolution.
Data pre- pre-processigg
Optimization The network is fine-tuned for 30 epochs the COCO train2017 set or the LVIS v1 traiin set. We use AdamW [86] with weight decay 10 10-*, +, base learning rate 10 and batch size 128 split across 16 workers. The learning rate rises linearly for & of an epoch, and is dropped twice by a factor of 10, after 5 and 3 of the total training time. We report mAP on the COCO va12017 set and the LVIS vi_val set, averaged across 5 runs.
A.4 Transfer to DAVIS video segmentation
As further test of scene understanding, we assess whether learned representations can continuc to recognize parts of object as they evolve over time. Video object segmentation, specifically in its semi-supervised setting, captures this ability, which we evaluate on the 17 benchmark. Having evaluated learned representation across frames, we segment these features with nearest neighbor matching from frame to frame, given a segmentation the first frame. In this way, the segmentation is propagated according to the similarity of the representation across space and time. We reuse the segmentation procedure from Xu and Wang [8] without modification, and report region (J) and boundary quality (F).
video independently
A.5 Transfer to UCF-101 action recognition
We evaluate action recognition classification on the UCF101 dataset [3]. We follow the procedure for finetuning used [9] which is based on [87]. We utilize clips of 2 seconds in length at 12fps. Each frame is processed by the ResNet-50 backbone. Clip representations are obtained by one of three methods for temporal integration:
1. Average pooling the standard baseline, producing a 2048-d vector output for clip which then fed and one fully connected (2048x x 101) layer for predicting the action class.
2. MS avg-pool: we pool the block3 representations (1024-d) over the two subclips of Is each that make up the larger clip. This is done because the features at this scale have smaller receptive fields and are selective for less complex content. Then we concatenate the two (1024-d) vectors with the average pooled feature from the block4 output to get a single 4096-d vector for each clip that again is fed through a fully-connected layer to predict the action class. By concatenating the two subelip representations, the fully-connected layer can in fact compute complex temporal relationships such as differences etc. along with the final layer's 'sses invariant representation that is pooled for the full clip.
3. MS temp-attn:
We perform the same methodology above for integrating multiple scales, but replace the average pooling over time with an attention pooling layer. Given representations for an L-frame clip E BBCCXL at given scale, compute temporal attention weights w, E R where w f(z). We choose
17

---

f be 2))) where Rxx1 a linear weighting of channels. Finally the pooled representation
V Σ, Wz
We show results using method in the main text and demonstrate the improvements from methods and 3 Appendix Table B.5. 10 clips are sampled from each video and the predictions of the clips are averaged for the final results. We fine-tune for 16 epochs using the ADAM optimizer with a multi-step LR decay schedule at epochs 6, 10, and 14. The initial leamning rate is set 0.0001. The implementation is adopted from https: /ttp://gggttt/tt/eeeeeee com.ffacebooreeeeaee.e..ooo AVID-CMA.
A.6 Transfer to ImageNet classification
For all models we freeze the ResNet-50 encoder (which outputs 2048-d embeddings). We then train a linear head to classify the 1000 categories in the ImageNet training using the standard split. To train the classifier, use the SGD optimizer with nesterov momentum and momentum parameter equal to 0.9. We use weight-decay of and sweep the learning rate for each model in the range [0.4, 0.3, 0.2, 0.1, 0.05] and pick the best classifier based ImageNet validation accuracy.
A.7 Transfer to -ustrr-distribution evaluations
For all OOD evaluations, we evaluate on datasets that utilize all (or subsets) of the ImageNet validation set. Therefore, for these evaluations use the pre-trained encoder and linear classifier (trained as in Sec A.6). We freeze the encoder and linear classification head and evaluate task performance on images from either the ImageNet-A, ImageNet-vid-robust, and Imagenet- CDCC datasets. For ImageNet-A and ImageNet-vin rid-rouss,, we use the evaluation code and method from [4].
For ImageNet-3DD 3DCCC, we do not use the entire corruption set because we wanted to specifically test models under the more natural 3-d corruptions. As is deseribed in [53], the dataset can be broken down into sets of corruptions: 3-d informed corruptions (using a depth model to generate natural corruptions informed by 3-d information) and standard 2-d noise and artifacts (like in ImageNet-C). For our experiments, we chose to evaluate specifically the 3-d comruptions, which were found to induce larger robustness effects for evaluating standard networks [53]. Nevertheless, we found similar results when evaluating robustness to 2-d noise and artifacts. All images from the following classes of corruptions were used for evaluation: far focus, near focus, fog, flash, xy motion blur, z motion blur, view jitter.
A.8
Alignment with human saliency
Human saliency measurements are obtained from the ClickMc dataset. Aligmment is measured as the Spearman rank correlation between model and human saliency averaged over the datasct, normalized by inter-rater alignment of humans.
A.9 Human error consistency evaluation
We evaluate accuracy and human error consistency on shape-hias datasets using the code from https: / sttpp://github.. com/ bethgelab/ hatigeelabmooell-aee modellvva vs-i--v-aaa human/ / rrre/mmater. We choose the subset of images that remove textural cues in different ways (forcing humans and models utilize global shape during discrimination): edge drawings, cue-conflict stimuli, graded low-pass filtering, and uniform gaussian noise. We report three metrics from [59]:
1. Accuracy difference: measure of human vs model classification accuracy on each OOD dataset and then averaged.
2. Observed consistency: measures the fraction of samples for which humans and model get the same sample either both right both wrong.
3. Error consistency: Score that measures whether there ahove-chance consistency. This is important because e.g. two decision makers with 95% accuracy each will have at least 90% observed consistency, even if their 5% errors non-overlapping subsets of the test data (intuitively, they both get most images correct and thus
18

---

Figure B.1: Example augmented frames with overlaid (resized) learned attention masks. Attention is computed from the output of the final block of the VITO trained ResNet-50. Crucially, the attention masks computed independently, such that the attention module can only use spatial cues.
observed overlap is high). Error consistency indicates whether the observed consistency is larger than what could have been expected given two independent binomial decision makers with matched accuracy [5].
The mathematical details on each of these metrics are provided in [5].
B Appendix: Additional results
B.1 Semantic binding with contrastive attention pooling
The ablation study demonstrated that multi-scale attention improves the performance of VITO in semantic segmentation. To probe why this may be, we visualize and interpret the learned attention masks (Figure B. For simplicity, we only visualize the masks from the coarsest scale (output feature map), but the interpretation naturally extends to the multi-scale version as these masks are learned with independent attention modules.
Because the attention masks are not computed jointly across each view, for a given video frame, the attention module must marginalize over the training data to make statistical prediction what should be attended the first view order to minimize the contrastive loss across possible second views? Specifically, the attention must focus on content that is most likely be stable across time while still being discriminative (or unique) relative other frames from other videos. Different examples appear to trade-off these criteria differently, yet systematically. For example, in the third column Figure B.I even though the animated characters on the right side of both frames may be discriminative content, the attention module has learned to focus on the static picture on the left as it is the content that is most likely to be stable across time. this pair frames the prediction correct- the attention disregards content that is changing too Chruptly-despite not having access motion cues. On the other hand, the example in the fourth column demonstrates a scenario where the model has attended to stable, but primarily discriminative content (the bird) rather than background, which is also very stable but most likely less unique relative to other videos.
Even beyond the ability to localize stable, yet discriminative content, it seems that our method also enables "semantic binding" of visually different, but semantically related features. This can be seen in the first pair of frames, as the model has Iearned associate an arm elbow (in the first frame) with the dumbbell (in the second frame), demonstrating an understanding that these two semantically related concepts co-occur and thus are predictive of one another given the right embedding.
Binding co-occuring features appears as an intuitive explanation for why these representations would perform well on semantic segmentation. It is particularly interesting that training end-to-end with standard contrastive loss can produce complex behavior reminiscent of the DINO approach [19] even though we use single, two-layer MLP attention module opposed to large-scale transformer architectures which attention throughout the network.
19

---

B.2 Ablating the components of VITO
In Figure B.2 we demonstrate on an example scene understanding task (PASCAL) how VITO is impacted by crop-scale, clip length, and type of attention pooling used (or used). In Figure B.3 additionally deeper analysis the temporal sampling scheme and demonstrate that our choice performs best across tasks and is arguably the most natural.
Ablation: RRC min scale
Ablation: video diip length
Ablation:
multi-scaee attention pooling
PASCAL Semande Segmentation (mlou))
76.0
76.0
70.0
75.0
75.0
74.5
74.5
74.5
McCUR
MOCLR
MOCLR RcCLR
73.5
McCUR
73.5
MOCLR
73.5
73.0
73.0
73.0
T-1.855
T-258s
(W/b attn) [W/ aton!
twic attn
(Al atin)
Figure B.2: Effects of crop scale, natural augmentations, and multi-scale attention on representation quality. All ablations are performed relative VITO's configuration (denoted by a green asterisk) which uses 2-scale attention pooling. a less aggressive crop scale of 40%, and natural augmentations uniformly sampled in window of length T = 2.56s. We also compare our baseline MoCLR model trained on single frames, either from ImageNet (dotted gray line) VidcoNet (dashed blue linc). All models are evaluated by transferring to PASCAL semantic segmentation.
Pascal Semantic Segmentation (mloU)
ADE-20K Semantic Segmentation (mloU)
Distribution Time Delays Belween Sampled Views
1.0
(mediann 0.7 0.75k
unifaam (madiann 1.056
76.0
detta CO. c0.85st
75.5
2400
25.0
74.5
74.0
MoCLA
MoCLR
0.2
73.5
McCLR
MoELR
73.0
data
uniform
marginal
00/ta uniform
marignal
0.0
0.5
1.0
1.5
2.0
2.5
Figure B.3: Ablating different temporal sampling schemes. Delta refers fixed time sampling beteween frames Gordon et al. [7]. Uniform refers to chunking time into non-overlapping blocks and uniformly sampling within each chunk as in Xu and Wang [8]. Marginal sampling (ours) refers to simple uniform sampling from the full vidco clip of length T 2.56s. First two panels show that marginal sampling is best overall across transfer to PASCAL and ADE20K. Third panel shows the distribution of absolute time-differences between any pairs frames under each sampling scheme (assuming views are sampled per clip). Our marginal sampling scheme is arguably the most natural as the mode the distribution is at O, meaning that it not biased to over weereeepeseen any specific time difference (similarly to the random undom-resized crop operation in space).
B.3 Dataset and method ablations
In Table B.1 we show that both our learning objective VITO and choice of dataset, VideoNet, are important for achieving top performance. However, these results also show that outpeform exisitng video pretraining when using
20

---

standard datasets Audioset YT8M. In addition, by comparing MoCLR trained on JFT-300M, we demonstrate the benefits of our method are not the result of simply having more frames of training data.
Semantic segmentation
Object detection
Pretraining
Dataset
Epochs
PASCAL
ADE20K
COCO
LVIS
MoCLR
VideoNet
200
72.8
37.5
24.6
VITO
YT8M
200
71.8
37.8
42.7
24.6
VITO
AudioSet
200
73.6
38.5
43.2
25,0
VTTO
VideoNet
200
75.5
39.2
43.6
25.6
McCLR
JFT-300M
200
74.3
38.7
43.2
25.4
Table B.1: VITO dataset and method ablations. We compare the baseline method MoCLR trained on VideoNet to
demonstrate the impact of our methodology. VTTO on VideoNet performs significantly hetter due to the methodological improvements (attention pooling, adaptation of spatial and temporal augmentations). We also evaluate VITO on traiditonal video datasets such YT8M and AudioSet. We that these numbers still greatly outperform prior vidco pretraining VideoNet dataset is clear
(See Table B.3. However the imapet Finally. we show that VideoNet does not simply provide benefits due to increased number of total frames vs. ImageNet. In fact, MoCLR trained on JFT-300M has order of magnitude more frames and yet still underforms.
best model VITO trained on VideoNet.
B.4 Scaling architectures
Here we demonstrate that VITO scales effectively to more powerful Swin transformer architectures. Results on scene understanding benchmarks improve greatly over ResNet-50 models and are competitive with specialized fine-grained scene understanding models from recent literature [65]. See Table. B.2
Semantic segmentation
Object detection
Pretraining
Dataset
Backbone
PASCAL
ADE20K
COCO
LVIS
VTTO
VideoNet
R50
76.3
39.4
44.0
25.7
MoCLR
VideoNet
Swin-S
78.6
43.7
48.4
32.7
VITO
VidcoNet
Swin-S
81.3
46.1
49.8
33.5
Delcon
ImageNet
Swin-S
81.4
46.1
50.4
33.1
Table B.2: VITO scales to larger model architcctures (Swin-S), improving performance compared the ResNet-50
baseline and remaining competitive with strong ImageNet pretrained baseline (Detcon) from Hénaff et al. [65].
21

---

B.5 Comparisons on additional scene understanding tasks
VITO outperforms all prior video pretraining (of image representations) understanding tasks. In addition the evaluations in the main text, we add PASCAL segmentation, LVIS object detection, ImageNet-18 classification. VITO remains highly competitive with the best ImageNet pretraining these tasks. (See Table B.3).
Semantic segmentation
Object detection
Classification
Video Pretraining
Dataset
PASCAL
ADE20K
COCO
LVIS
IN-IK
Random Init
53.0
27.9
39.0
21.1
Methods pretraining on video datasets
R3M [74]
13.3
VFS [8]
K400
63.9
31.4
41.6
23.2
VIVI [69]
YT8M
65.8
34.2
41.3
23.2
62.6
VINCE [7]
R2V2
69.0
35.7
42.4
24.4
54.4
CycleContrast [9]
R2V2
69.2
42.8
24.5
55.6
MMV TSM [70]
AS HT
70.6
32.5
41.3
24.2
51.4
VITO
VidcoNet
76.3
39.4
44.0
25.7
66.2
Methods pretraining on ImageNet
Supervised
ImageNet
71.3
33.5
44.2
25.2
76.1
BYOL [61]
ImageNet
76.1
38.8
43.7
25.5
MoCLR [68]
ImageNet 76.4
39.2
43.9
25.8
71.4
DINO [19]
ImageNet 76.1
39.0
44.3
26,4
75.3
Table B.3: Image and pretraining evaluated on object-detection semantic segmentation, and ImageNet-1K classification.
B.6
Comparison to image pretraining on video-based tasks
Here we demonstrate more thoroughly that compared with image pretraining methods (image backbones), we perform significantly better video-level tasks. On both DAVIS segmentation (Table B.4) and UCF-101 action recognition (Table B.5), VITO outperforms strong ImageNet trained baselines and methods pretrained on video datasets.
Pretraining
Dataset
Fm
Fm
ImageNet pretraining
Supervised
ImageNet
63.7
68.4
MoCo [16]
ImageNet
63.2
67.6
DetCong [65]
ImageNet
63.1
66.4
MoCLR [68]
ImageNet
63.1
67.8
BYOL [61]
ImageNet
63.8
69.4
Video pretruining
VINCE [7]
Kinctics
63.4
67.8
TimeCycle [88]
VLOG
41.9
39.4
UVC [89]
Kinetics
54.5
58.1
CRW [44]
K400
64.8
70.2
VFS [8]
K400
65.3
70.2
VITO
VideoNet
65.5
70.8
Table B.4: VITO significantly outperforms all image-pretts peeteepretaaining baselines DAVIS 2017 video segmentation. VITO also oulperforms many recent successful video pretraining methods.
22

---

Pretraining
Dataset
Backbone
Top-1
Video architectures
Supervised [90]
ImageNet
67.1
VideoMoCo 191]
K400
R(++))D
78.7
Temporal-ssl [92]
K400
R(2+1)D
81.6
VTHCL [93]
K400
3D-R50
82.1
CoCLR [94]
K400
S3D
87.9
CVRL [34]
K400
3D-R50
92.9
p-BYOL [37]
K400
3D-R50
95.5
Supervised [95]
K400
I3D
95.1
Image architectures
OPN [96]
UCF101
VGG-M
59.8
TCE [97]
K400
R50
71.2
CycleContrast [9]
R2V2
R50
82.1
MoCLR [68]
ImageNet
R50
85.5
BYOL [61]
ImageNet
R50
85.6
VITO (avgpool)
VideoNet
R50
87.4
VITO (MS-avgpool)
VidcoNet
R50
88.5
VITO (MS-attppool)
VidcoNet
R50
89.4
Table B.5: VITO outperforms all image representations when finetuning for UCF101 action recognition, using temporally- pooled level-level representations. VITO's performance even competitive with many video architectures.
23


Reviews:
[{"review_id": "2blUUD5ezK", "reviewer": "Reviewer_i1Wo", "summary": "The paper proposes VITO, a new method for self-supervised video pretraining that learns general and human-aligned visual representations. It made several modifications over existing contrastive learning frameworks, including larger crop sizes, improved temporal sampling scheme, and multi-scale attention feature pooling for the projector. The paper also creates a video dataset (VideoNet) that aligns the class distribution with ImageNet, and partially redresses the imbalance between image and video learning. This improves spatial understanding compared to other video datasets.\nExperiments are conducted on various tasks, including semantic segmentation / object detection on image, video object segmentation and classification, as well as OOD object recognition. ", "questions": "Please see the weakness part.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-organized and easy to follow. \n\nThe technical contribution and the way to do data curation make sense to me. \n\nThe results shows that VITO matches or exceeds image pretraining on spatial tasks like object detection and segmentation while \noutperforming other video pretraining methods. This shows it learns a general representation.\n\nThe ablation is comprehensive.\n\n", "weaknesses": "The human alignment evaluations are somewhat weak. Human alignment is a big area and whether saliency and shape bias tasks can serve as representative tasks for the evaluation concerns me. More comprehensive alignment benchmarks could be explored.\n\nIt would be great if the authors could also show baselines with the masked image modeling objective.\n\nOverall I think this paper shows good results and presents things clearly. Thus my rating is weak accept. I encourage the authors to perform the above things I suggest to improve the quality of this paper further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. Regarding the human alignment evaluations, we agree that more evaluations should be done; however, we focused on saliency and shape-bias tasks as these have been recent and prevalent tasks where many deep image models fail to capture important aspects of human behavior. In fact, the saliency benchmark that we evaluate on is an extremely large-scale study and we believe that this result is quite strong, given the fact that VITO provides greater saliency alignment than even the “harmonized” model (trained with an objective to explicitly match the human saliency maps). Additionally, shape vs texture bias has been a prevalent issue in the vision community given how strongly many models differ from humans. While the gap has closed for very large models trained on billions of images, our results demonstrate that even in a smaller scale setting, video pre-training may be a more natural solution. Nevertheless, more evaluations are always beneficial so if the reviewer has any suggestions for a preferable quantitative alignment benchmark, we are happy to evaluate and include this in the final paper. \n\nRegarding masked image modeling, we understand that this is a dominant recent SSL paradigm. However, much of the masked autoencoding (MAE) literature is focused on using transformer architectures, due to the need for operating on patchified inputs. Because we seek architecture-matched comparisons, this makes comparing to any masked image modeling objectives difficult. We have provided a brief scaling evaluation using Swin transformers in Supp Table B.4, but leave it to future work to do a thorough comparison across transformer architectures.\n"}, {"review_id": "SZM4FIzP8t", "reviewer": "Reviewer_3k9B", "summary": "The paper introduces a SSL method using video pre-training to produce general visual representations for both image and video tasks. The proposed VITO pipeline includes a data curation process and a video SSL technique based on MoCLR. The authors conduct a comprehensive evaluation of VITO's performance on diverse benchmarks, spanning detection, segmentation, video segmentation, classification, and out-of-distribution object recognition. Additionally, the paper includes a comparison of human-alignment on two benchmarks.", "questions": "Overall I think the video pre-training direction and results are interesting. \nHowever, my main concerns revolve around the limited evaluation and results presentation, which lack sufficient ablation studies and comparisons, as mentioned in the weaknesses.", "limitations": "See weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "•\tThe paper is well-organized and easy to follow. The proposed components for the method, including data curation and various modifications on MoCLR, all make sense to me. \n\n•\tAblation studies are conducted by the authors to validate the effectiveness of their proposed method.", "weaknesses": "1. The main method is based on the image SSL method MoCLR. While the authors demonstrate the effectiveness of multi-scale contrastive attention pooling, this component appears applicable to image SSL as well. Therefore, the technical novelty of applying an image SSL method to video datasets seems limited, especially considering the existence of other established video pre-training works.\n\n2. The motivation behind choosing MoCLR as the starting baseline should be clarified. Given the availability of better contrastive learning methods and the superior performance of Masked-based pre-training (e.g., MAE), it would be beneficial to provide comparisons or discussions to justify this selection.\n\n3. The data curation process plays a critical role in the overall pipeline. To gain more insights, additional ablation or analysis would be helpful.\n \n3a.  For example, what are the the effect of the number of curated videos and different filtering strategies.\n\n3b. For the ablation (Figure 4), considering that performance can be significantly influenced by pre-training data and downstream dataset distribution, it would be better to verify on more downstream tasks instead of only one dataset.\n\n3c. It would be interesting to Investigate the scaling behavior on the training schedule, as 300 ImageNet epochs might be insufficient for large-scale video datasets.\n\n\n4. In Table 1, incorporating more standard benchmarks would enhance the evaluation's comprehensiveness. For instance, Kinetics is a more widely adopted benchmark for video understanding compared to the small-scale UCF-101.\n\n5. In B.5, VITO doesn’t show superior performance on image-based tasks, especially it has significant drop on ImageNet compared to ImageNet pre-trained methods. It might be worth to have some discussions around that as this is also important observations. \n\n6. The paper lacks comparisons with some video-specific SSL methods, including contrastive learning-based and Masked-based methods (e.g., VideoMoCo, VideoMAE, MAE_ST, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and direct them to the global response for our summary. We will address each cited weakness point by point.\n\n**Weakness 1: technical novelty**\n\nWe agree with the reviewer that the basic idea of applying an image-based SSL framework to a video dataset is not novel. In fact this has been attempted previously in the many cited works we compare to in Table 1 (CycleCon, VINCE, VFS, VIVI etc.). The novelty of our work lies in the careful methodological choices (for both the learning paradigm and dataset curation), which lead to significant empirical gains over all prior image-based video pretraining work. We concede that there are many successfully works which specifically train video-based architectures on video datasets, but these methods are severely limited in their application to general scene understanding, robust recognition, and comparisons to human behavior due to the fact that they cannot be adequately modified to maintain high performance on image-based tasks. We emphasize that a key aspect of our work is demonstrating how image-based representations trained on video data can be far more general: maintaining strong performance on spatiotemporal tasks while far outperforming on static image tasks. \n\n**Weakness 2: why MoCLR?**\n\nWe will clarify this in the text, but we choose MoCLR based on it’s strong performance in [1] showing that it outperforms both contrastive (SimCLR) and non-contrastive (BYOL) formulations. We disagree that it is not a strong baseline. However, we have additionally compared to DINO in the robustness and alignment evaluations and find similar and sometimes worse behavior compared with MoCLR. We do not compare to all recent SoTA SSL methods such as MAE primarily because they use significantly larger architectures (generally transformers) that would not be architecture-matched and would obfuscate the current set of results (see general response for longer discussion). \n\n**Weaknesses 3a/3b: impact of curation is unclear**\n\nRegarding VideoNet data curation and ablations, we refer the reviewer to our general response. Briefly, we present ablations on the 4 scene understanding tasks using two other prominent video datasets (AudioSet and YT8M) (Supp. Table B1). While the performance drops slightly using alternative video datasets, we see that our method performs far better than prior image-based video pretraining efforts. VideoNet curation helps to close the gap with ImageNet pretraining but it is not necessary to outperform prior work. Nevertheless, we agree with the reviewer that more work should be done to understand how our results are impacted by different curation strategies. In particular, we believe a purely unsupervised curation approach may be possible in which we simply measure the similarity of our dataset and ImageNet images (without labels) in an unsupervised embedding space to better align the two, rather than using a classifier (similarly to DINO v2). We leave this for future work.\n\n**Weakness 3c: scaling epochs**\n\nWe agree that investigating scaling behavior to longer epoch schedules is valuable and interesting. We did not have time to run further experiments on this, but will add text to cite this as future work. \n\n**Weakness 4: insufficiency of UCF101**\n\nUCF-101 is to our knowledge quite standard and widely accepted. While it may be easier than some benchmarks, many prior works do evaluate on it (See Supp Table B5). Additionally, there are very few prior works in image-based video pretraining which evaluate on Kinetics, which is the subset of prior work that we feel is the most important and direct comparison. Finally, many video-based pre-training in fact trains on the Kinetics dataset, so for those models, Kinetics is an in-distribution evaluation. Therefore, given these constraints, UCF presents a reasonable example of an OOD video benchmark. We also note that outside of the UCF benchmark, we additionally evaluate on video segmentation via DAVIS, which is a standard benchmark in the field. \n\n**Weakness 5: ImageNet numbers**\n\nThe reviewer notes that VITO does not show superior performance on image-based tasks citing the ImageNet classification number. This is not true as VITO does outperform many ImageNet pretraining methods on tasks which use OOD datasets (segmentation, detection, and OOD recognition). The reason VITO does not outperform other methods on ImageNet validation accuracy is that this is an in-distribution evaluation for models trained on the clean ImageNet dataset. While we align the class distribution of VideoNet to that of ImageNet, the frames themselves are significantly more diverse and noisy meaning that ImageNet recognition is not strictly an “in-distribution” task for VideoNet pretraining, making the comparison unfair. \n\n**Weakness 6: comparing to video models**\n\nWe provide some video-based comparisons in Supp Table B5 for comparing VITO on the UCF evaluation. However, on all image-based evaluations it is unclear how we can compare to these video-specific SSL methods as they all utilize video-specific architectures. It is well-known that these architectures cannot easily be adapted to handle image-based evaluations so we do not see an easy way to compare with these methods. This issue in fact exemplifies a benefit of VITO as it generalizes across image and video tasks seamlessly while maintaining strong performance in both domains. \n"}, {"review_id": "4jM22WUG3o", "reviewer": "Reviewer_Pto4", "summary": "This paper studies how to take advantage of natural temporal distortions in video to learn image spatial representations. The paper first proposes a VideoNet dataset that filters the video data from common video datasets by an ImageNet classifier. The paper further proposes to multi-scale attention pooling to improve the baseline algorithm. The algorithm trained on the proposed the dataset is evaluated on both image and video datasets. The paper also studies the human alignment of the learned attention.", "questions": "I have a few questions on the experiment settings:\n\n1. In Table 1, why are some results different from what are reported in their original papers? For example, VFS [8] on DAVIS is reported to be 68.9 in its paper, but 67.8 in this manuscript. Is this 67.8 number produced by the authors with a different experimental setting? Are most of numbers in Table 1 produced by the authors by fine-tuning on the publicly available checkpoints?\n\n2. Why are the UCF-101 results in Table 1 significantly lower than other self-supervised learning algorithms on video? For example, in Feichtenhofer et al. [34] a ResNet-50 pre-trained on K400 can easily obtain an over 90% accuracy on UCF101?\n\n3. The experiments on ImageNet-3DCC in Figure 2 are interesting. I am wondering why only evaluate MoCLR as the representative self-supervised algorithm, while not others like DINO?\n\n4. I am wondering the training cost on VideoNet. For example, how many ImageNet-equivalent epochs have been used and how many crops are there in each step?\n", "limitations": "The authors adequately addressed the limitations potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Pre-training image representations with video datasets is a natural and valuable idea to explore.\n\n2. The VideoNet dataset could be a valuable contribution to the community provided the authors have a plan to release it publicly.\n\n3. The paper provided abundant experiments to demonstrate the effectiveness of the proposed algorithm. ", "weaknesses": "1. Although using an ImageNet classifier to filter the video data is a practical idea to reduce the domain gap between video and image datasets, this also introduces unfairness to the comparison to other self-supervised methods. This is because the filtering process with an ImageNet classifier implicitly takes advantage of ImageNet labels, and ImageNet labels are very effective on downstream image tasks as validated by the authors in Figure 4.\n\n2. It is unsure how much the temporal deformations provided by a video dataset help with the pre-training VITO, which I think is one of the central questions for video contrastive learning for image downstream tasks. Because if temporal deformations are not useful, then why bother filtering a video dataset which costs much more than a image dataset. I am not sure if T=0 in Figure B.2 is a baseline that generate two views from the same frame in VideoNet. If so, this would be a helpful ablation that can be highlighted in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and address the concerns below.\n\n**Weakness 1: VideoNet curation**\n\nWe realize that the VideoNet curation procedure does involve utilizing the class distribution or implicit labels. However, most of the high-performance SSL methods we compare to were trained on ImageNet (which itself was manually curated) so we do not agree that there is anything unfair in our comparisons. Even the VINCE/CycleContrast video models leverage a similar curation strategy in construction of their R2V2 dataset.  Particularly, we outperform ImageNet pre-trained methods on video tasks, OOD robustness, and human alignment, demonstrating that we can leverage the benefits of ImageNet’s class diversity along with complex spatiotemporal deformations of objects to learn more general visual representations. However, to additionally alleviate concerns we have provided ablations of VITO using VideoNet and YT8M in Supp Table. B.1. We find that using these uncurated datasets, VITO outperforms all prior video-pretraining (including prior video pretraining leveraging the same datasets) and even outperforms an SSL baseline trained on the significantly larger, but uncurated JFT-300M image dataset.\n\n**Weakness 2: temporal ablations**\n\nWe agree with the reviewer that assessing the impact of the temporal deformations is a central question. Indeed in our ablations in Fig B.2, T=0 is the spatial version of training where augmentations are only generated from individual frames. Therefore, we see a significant increase in performance including temporal deformations. We have now additionally shown the impact of temporal deformations through many more evaluations specifically emphasizing the impact of the temporal deformations on increased robustness (ImageNet 3dcc recognition) and human alignment on shape-biased tasks. For more information, see the global response and rebuttal PDF figures. \n\n**Question 1: DAVIS inconsistency** \n\nFor fair comparison, we re-ran all the models we compare to by fine-tuning or evaluating linear classifiers (for ImageNet-based evals) on the publicly available checkpoints. Therefore, there are slight discrepancies in the DAVIS video segmentation numbers, but we believe all these comparisons are fair as they are all fine-tuned under the same conditions. \n\n**Question 2: UCF low numbers**\n\nOur UCF numbers are to the best of our knowledge, the best performance based on a pure image-based backbone. In fact, in Supp Table B.5 it can be seen that with simple temporal pooling strategies we surpass many prior video-based backbones and can get close to the Feichtenhofer numbers (under the same pre-training epochs). All prior work that show accuracies in the 90% range use training procedures that train with 3D ResNets, (2+1)D ResNets, or other architectures that allow for temporal integration in the model during training. We think it is significant that we can achieve numbers that are not too far from this just with spatial feature extraction and pooling of the output representations. In our new ablations (see global response), we see that this performance does not appear without the inclusion of temporal deformations during the training, indicating that there is implicit knowledge of the features relevant for defining action categories that can be captured in a purely image-based representation. \n\n**Question 3: ImageNet-3DCC**\n\nSee our global response. We have now added DINO and many of the prior video pretraining works (cycle contrast, vince, vfs) and our ablations. We find that the strong robustness of VITO only appears with the combination of its methodological components, and is not replicated by other powerful SSL methods such as DINO or other video pretraining efforts. \n\n**Question 4: Computational budget**\n\nWe designed the VITO training procedure such that it matched the computational budget of all of the models we compare to that are trained on ImageNet. In each step, for all models we train with 3 views and the 300 epochs refer to ImageNet-equivalent epochs (i.e. the same number of iterations needed for 300 ImageNet epochs). . \n"}, {"review_id": "RHnSosaXgq", "reviewer": "Reviewer_Xjw1", "summary": "The paper proposes a self-supervised method for learning image representations from videos.\nTowards this end, a procedure to curate video datasets most suitable for such pre-training is proposed by selecting videos that best match the distribution of visual classes found in ImageNet. \nSecondly, this dataset is leveraged through a contrastive training approach where the common global average pooling of video frame features is replaced with an attentional module that learns to select spatial regions of several network layers to construct the image embedding. \nThe learned representations are evaluated on a large set of downstream tasks, including spatial scene understanding (segmentation + detection), video understanding (segmentation + action recognition), and robust image classification.", "questions": "I would appreciate it if the authors could address the weaknesses outlined above. I'm particularly interested in how the technical contributions affect downstream results across multiple benchmarks (i.e., how do the ablations look when evaluated across the multiple benchmarks in Table 1)?", "limitations": "Limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method is very well presented, and the paper is well written\n- The downstream evaluation of the learned image representation is extensive, and the model achieves good performance across a variety of tasks\n- The finding that a video dataset with similar category distribution to ImageNet appears to perform considerably better than many existing video datasets is interesting (see Fig 4)", "weaknesses": "- I would have preferred to see more extensive ablation experiments to support the technical contributions of the paper (e.g., the attentional pooling module for contrastive learning). These ablations are currently only reported on a single task and dataset. It would be more convincing to see the influence on all the downstream tasks in Table 1 and to summarize these ablation results in a Table. As it is, it is unclear if the benefits of the proposed training are consistent across tasks or if most of the benefit is due to the data curation process.\n- The data curation process relies fundamentally on reproducing the object category distributions of ImageNet in a video dataset. While it is interesting that this appears to lead to a benefit across many downstream tasks, it is unclear what explains these benefits. Is it greater visual diversity, lesser class imbalance, or maybe some other property?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and address the concerns below.\n\n**Weakness 1: More extensive ablations**: \n\nSee our global response for a detailed discussion of more extensive ablations we have now performed. Notably, we have verified that all components (dataset, contrastive attention pooling, and temporal augmentations) all are necessary for the strongest performance. In addition, we find a particularly strong relationship between robustness/good human alignment with models that have been trained with temporal augmentations. We believe that strengthening this link is important and will update our main text accordingly. \n\n**Weakness 2: unclear what explains VideoNet performance**\n\nWe agree with the reviewer that there is still much more to unpack about the VideoNet curation and what aspects are necessary or not. We have provided ablations comparing to YT8M and training on the far more diverse JFT-300M image dataset in Supp Table B.1. In both cases, VideoNet outperforms by a large margin across the scene understanding benchmarks. This suggests that the key feature has to do with the shape of the class distribution rather than just greater visual diversity. Specifically, we hypothesize that having densely sampled classes and lesser class imbalance (removing long tails) greatly improves performance because it provides more difficult yet useful negative examples for the contrastive loss. It remains to be seen whether we can test this more empirically by controlling for different properties of the distribution and evaluating, but we believe our work will be a solid basis for this investigation.\n"}, {"review_id": "mIgcWGLOhw", "reviewer": "Reviewer_3hVk", "summary": "This paper introduces a new video dataset (VideoNet), and architectual adjustments (VITO) that enables allows pretraining on video datasets to improve performance on downstream transfer tasks, such as segmentation, object detection, and generalization. The VITO network uses a ResNet backbone, and architectural changes include extracting spatial feature maps at the two penultimate blocks, applying a learned softmax function to obtain a weighting over features, and then projecting this into a final feature map. This feature extractor is trained using the InfoNCE loss on two sampled video frames, with additional image-space augmentations applied. Additional experiments in the supplementary material investigate the use of transformer backbones. The VideoNet dataset retrieves video clips matching the ImageNet categories, and applies an image classifier on these videos to filter for the correct category. Additional experiments compare the attention maps from the VITO model and other baselines to human ground truth from the ClickMe dataset.", "questions": "- L124: what is the softmax normalized over?\n- L130: is $a_\\xi$ also a target network? It may be helpful to also illustrate the network $g$ in Fig. 1\n- Fig 3: Why was the CLIP Resnet chosen for the attention map, as opposed to the attention maps extracted from CLIP or DINO transformer models?", "limitations": "Limitation are adequately addressed. The current setup explores primarily the ResNet backbone with the InfoNCE loss, but there are promising initial results with transformer models. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The proposed method and training dataset presents strong empirical results. It outperforms, or is competitive with, image pretraining on ImageNet, video pretraining, or robust image pretraining on a variety of transfer tasks. Notably, it demonstrates improved robustness on datasets with image corruptions even when compared to models trained for robust classification.\n- Ablations evaluate the independent contributions of both the dataset on the model design. With the same VITO model, but different datasets, VideoNet can outperform AudioSet as a prior video dataset on segmentation and object detection tasks. On the same dataset (VideoNet) but different models (MoCLR vs VITO), VITO also outperforms the prior method. (Table B.1)\n- On additional experiments on Swin transformers show promising improvements. I think additional details on the transformer setup, and  which spatial feature maps are extracted in the transformer would be helpful. \n- This paper is well written and clear to follow. \n", "weaknesses": "- The benefit of VITO seems to be primarily on transfer tasks. On standard classification, it falls short of ImageNet pretraining by ~10%, but it is still outperforming other video tasks. It is also slightly lower on object detection compared to the DINO representation. However, I think there are still valuable insights to be gained from the methodology and dataset presented here. \n- There are some additional parts that could benefit from clarification. Please see the below questions section.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and now address the weaknesses/questions.\n\n**Weakness 1: lower ImageNet validation numbers**\n\nYes VITO does underperform supervised ImageNet pretraining on ImageNet classification by ~10% and other SSL ImageNet pretraining methods by 5-7%. This however can be attributed to the fact that ImageNet classification is an “in-distribution” task for models pretrained on ImageNet. Even though a separate validation set is used, the training images are sampled from the same distribution, as they share properties such as single-object field-of-view, centrally cropped, high resolution etc. Even though VideoNet matches the overall class distribution to ImageNet, we cannot expect our models to transfer as well to ImageNet as it is still out-of-distribution in many respects. Indeed VideoNet frames contain multiple objects and more global scenes, strong motion deformations, and wider variability in resolution and quality. Even still, VITO outperforms all ImageNet trained models on benchmarks based on ImageNet distribution shifts (ImageNet categories under distribution shifts). For example, VITO strongly outperforms all ImageNet pretraining on ImageNet-Vid and ImageNet-3DCC. These recognition benchmarks are arguably better tests of generalization under real-world conditions.\n\nAdditionally, while VITO is competitive on all other scene understanding benchmarks and strongly surpasses ImageNet pre-training on transfer to video-based tasks (DAVIS, UCF). Therefore, on average, VITO generalizes across all tasks significantly better than the comparable ImageNet pretrained models. \n\n**Question 1: L124: what is the softmax normalized over?**\n\nThe softmax is normalized over space such that the attention-weights across space to sum to 1. This encourages competition across spatial locations, forcing the attention to be more localized.\n\n**Question 2: L130: is a_xi also a target network? It may be helpful to also illustrate the network g in Fig. 1**\n\nA_xi is also a target network as the EMA is applied to all parameters for the target.  We understand that missing g in Fig 1 can be confusing and will update the figure and caption to emphasize it’s role. \n\n**Question 3: Fig 3: Why was the CLIP Resnet chosen for the attention map, as opposed to the attention maps extracted from CLIP or DINO transformer models?**\n\nWe choose the CLIP ResNet for attention map comparison as it provides an architecturally matched comparison to ours. We fully agree that scale in data and architecture can produce improvements, but our goal was to understand the impact of video pretraining on the image representations for this we require comparisons that are matched in model capacity to isolate these effects. \n"}, {"review_id": "pEVZVnfFKT", "reviewer": "Reviewer_jN1F", "summary": "The research question addressed by the paper is whether self-supervised training on\nvideos leads to image features that are better aligned with human perception. To that\nend, the authors propose a contrastive learning method that leverages natural\nchanges over time as different views, beyond adapted image augmentations. Moreover, an\nattention mechanism is introduced that allows the model to compare image pairs based\non features from selected regions only. The model is trained on videos that have been\ncurated to better match the ImageNet category distribution than existing video datasets.\nTrained this way, the model performs better than previous methods when transferred\nto several standard benchmarks and shows a better alignment with human vision.", "questions": "As described above, the paper has a much stronger focus on performance in terms of\ncomputer vision (where the proposed method is very competitive) than alignment with\nhuman vision. I would recommend to change the title and introduction to better align\nwith this focus and I am happy to increase my score if this issue is addressed.\n\nL186ff needs clarification for me: When evaluating video understanding, the paper claims\nthat \"VITO learns features that capture finegrained temporal deformations of objects\".\nHowever, VITO is based on image networks so that spatio-temporal features cannot be\nlearned.", "limitations": "The authors discuss limitations of the paper due to the focus on a single architecture\n(ResNet-50). Potential societal impacts are not discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Training computer vision models to better align with human perception is an active\n  area of research with the potential to improve, e.g., the robustness of current\n  approaches. Self-supervised training on natural videos is closer to how humans learn\n  than standard training methods. Exploring this direction therefore is well motivated.\n- The authors compare to a broad range of previous models on several computer viion\n  benchmarks. The newly proposed method improves over previous approaches in most cases.\n- Several ablation studies are performed and reported in the supplement to disect the\n  influences of the contributions on the improved performance.\n", "weaknesses": "The paper aims at better aligning neural network features with human perception.\nHowever, the range of models tested for alignment with human perception is much smaller\nthan the range of models tested for performance on computer vision bechmarks. In\nparticular, none of the other video based self-supervised methods is tested and none\nof the ablation studies considers alignment with human vision.\n\nThe title of the paper, \"Self-supervised video pretraining yields human-aligned visual\nrepresentations\", and the introduction therefore raised wrong expectations for me. Due to the\nmissing ablations, one cannot judge whether the improved human alignment is due to the\nvideo pretraining or other contributions, such as the new training set or the\nmulti-scale contrastive attention pooling. Moreover, I find the title too bold given\nthat other methods perform much better in terms of human error consistency according to\nthe official benchmark repository (e.g., CLIP).\n\n**Update:** Both points raised above have been addressed during the rebuttal. Therefore I improved my rating of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and address the concerns below.\n\n**“none of the other video based self-supervised methods is tested and none of the ablation studies considers alignment with human vision.”**\n\nWe agree this was a limitation of our current results and have addressed this by greatly expanding the models evaluated on our benchmarks and human alignment. (See global response). The results stand that VITO outperforms all prior video pretraining on both the visual tasks and human alignment, and we have verified that these effects are strongest only with the combination of the contrastive attention pooling and the use of temporal deformations during training. \n\n**“Due to the missing ablations, one cannot judge whether the improved human alignment is due to the video pretraining or other contributions, such as the new training set or the multi-scale contrastive attention pooling.”**\n\nWe refer the reviewer to our general response and ablation figures in the rebuttal PDF for a complete analysis of the impact of the ablations on the improved human alignment and robustness. In summary, we have verified that all components (dataset, contrastive attention pooling, and temporal augmentations) are necessary for the strongest performance. In addition, we find a particularly strong relationship between robustness/good human alignment with models that have been trained with the temporal augmentations. \n\n**“Moreover, I find the title too bold given that other methods perform much better in terms of human error consistency according to the official benchmark repository (e.g., CLIP).would recommend to change the title and introduction to better align with this focus and I am happy to increase my score if this issue is addressed.”**\n\nWe refer the reviewer to the global response clarifying the intentions behind our claims and title (which we will incorporate in the text), and are happy to discuss/change based on what the reviewer thinks would make the most appropriate title. Regarding comparisons with CLIP, we first would like to note that CLIP in fact does not outperform our method on the saliency benchmarks which are a strong test of one aspect of perceptual alignment. Additionally regarding the shape-bias alignment, we have now included CLIP in our evaluations and while we find that it performs better than VITO, it is not by a very large margin. Moreover, CLIP is also trained with large-scale language supervision (400M image-text pairs), which is far larger than VideoNet, so we find it significant that we can obtain similar gains over standard ImageNet pretraining with far less data and supervision than CLIP training. More relevant are comparisons to image- and video-only self-supervised learning methods, where VITO outperforms prior work. Nevertheless, it would be very interesting to combine our proposed self-supervised video pretraining with multimodal language supervision. \n\n**“When evaluating video understanding, the paper claims that \"VITO learns features that capture finegrained temporal deformations of objects\". However, VITO is based on image networks so that spatio-temporal features cannot be learned.”**\n\nWe fully agree that VITO cannot capture spatio-temporal features explicitly. However, as seen in the ablations (see main response), VITO does in fact perform significantly better when trained with temporal deformations. We believe this is due to the fact that VITO learns to attend to the features in an image which are more likely to be predictable under the fine-grained temporal deformations of objects. These features can be quite different than those that are predictable under standard image-based augmentations. Therefore, even though VITO does not capture the deformation explicitly, it is learning to become robust to the fine-grained temporal deformations of objects. We will clarify this in the main text.\n"}]

Author rebuttal:
We thank all reviewers for the feedback and acknowledging our work's clear presentation of a novel step towards learning more general human-aligned visual models. We will address global concerns here and individual comments separately. Please see the rebuttal PDF for new results. 

**Ablations:** Reviewers emphasized the need for more evaluations on ablated models to justify the critical components of our final model. To answer these questions, we present new results evaluating multiple ablations of VITO along different dimensions: UCF101 action recognition, OOD image recognition (ImageNet-A/ImageNet-vid), recognition under natural corruption (ImageNet 3dcc) and human alignment (shape bias tasks). We evaluated the following models on these tasks, seen in Table 1, Table 2,  and Figure 3 (right panel):
* MoCLR VideoNet: MoCLR applied to VideoNet frames, no temp. augmentations.
* VITO spatial: 2-scale attention pooling, no temp. augmentations.
* VITO (1scale w/o attn): All augmentations, 1-scale embedding, no attention pooling.
* VITO (2scale w/o attn): 2-scale embeddings, all augmentations, no attention pooling.
* VITO (1scale): Full VITO, 1-scale embedding.
* VITO (AudioSet): Full VITO, pretraining data: uncurated AudioSet.

These models exemplify the key method elements (spatial vs spatio-temporal deformations, multi-scale contrastive attention, and VideoNet dataset). 

_Rebuttal Table 1_ data is consistent with the more limited ablations in App Fig. B2 and App Table B.1. First, incorporating temporal deformations contributes greatly to the best model performance (temporal VideoNet models outperform the VITO-spatial and MoCLR models on all benchmarks). Second, architecturally, the multi-scale aspect is the most significant component on its own, but when combined with attention pooling there is a synergistic effect that leads to significant improvements, especially on OOD recognition benchmarks. Moreover, VideoNet models outperform those trained on other datasets such as Audioset. Along with the ablations in App. Table B.1, this suggests that the importance of VideoNet is in shaping the class distribution rather than just having large visual diversity.

_Rebuttal Table 2_ shows that the full VITO method produces the best human alignment in terms of ceiled error consistency. Notably, other high-performing models are _all VITO ablations that were trained with temporal deformations_. This surprisingly includes the AudioSet pretrained model, even though it has poorer performance on the computer vision tasks. This result is quite striking, as previous image-based video pretraining methods perform quite poorly (VINCE [7], VFS [8], and CycleCon [9]). This suggests that our specific methodology better leverages the spatio-temporal data to produce learned representations that are more human-aligned. As suggested, we now include a comparison with CLIP, which does have better error-consistency than our best model. However, CLIP is trained with large-scale language supervision (400M image-text pairs), and is thus not directly comparable. More relevant are comparisons to image- and video-only SSL methods, where VITO significantly outperforms prior work.
 
_Rebuttal Fig 3_ (right panel): In recognition under natural corruptions (ImageNet-3DCC), VITO outperforms all ablated versions. Again, there is a striking gap between models trained with spatio-temporal deformations, which are far more robust than those trained only with spatial augmentations. Coupled with the alignment results, this strongly highlights the importance of video vs. image pretraining.

**Comparisons to MAEs and transformer-based architectures:** While we agree with the reviewers that transformer architectures would greatly benefit the work, this would require a significant replication effort that we leave to future work. It would be unfair to just compare ResNet-50 models with transformers due to the vast differences in expressive power. To reiterate, the goal of this work is to _compare image representations learned from videos with those learned from images, assuming matched model capacities_. In addition, some of our most important comparisons are with prior work that also attempted to learn image representations from video data. These also mostly performed experiments with ResNet-50s, informing our initial architecture choice to have apples-to-apples comparisons with prior work. 

**Comparisons to SoTA video models**: Most SoTA video-based methods utilize specialized video architectures, and thus cannot be evaluated on general image-based tasks such as scene understanding, robust recognition, and human alignment. As a result, while these methods perform well on video tasks such as action recognition, they remain less general than ours and are less appealing candidates for robust, human-aligned visual representations.

Lastly, we wish to clarify potential confusion over our title and introduction. We do not mean to claim we have learned the most human-aligned, general visual representation. Yet, because most of the current literature in this space is solely focused on achieving more general representations via scale (larger image datasets and models), we wanted to probe alternative ways to achieve this goal. Our work shows that video pretraining can yield more general, robust, and human-aligned visual representations by leveraging natural spatio-temporal deformations, even at a relatively small scale. This contribution can be complementary with scaling to larger architectures, datasets, and sources of supervision (e.g. language alignment). As an example, recent work has shown the benefits of combining image-level self-supervision and multimodal image-text alignment [Mu]. Naturally, combining our work in self-supervised video pretraining with multimodal image-text alignment appears as a potentially promising direction for future work. 
[Mu] Mu, Norman, et al. "Slip: Self-supervision meets language-image pretraining."

Meta-review: This paper proposes a new method for self-supervised video pretraining that learns general and human-aligned visual representations. The results are quite compelling and the reviewers seem to generally agree there is a solid methodological contribution to the field. Reviewers raise points about limited ablation, but the additional points brought up in the rebuttal help in this regard.  What is most striking is that the method seems to improve robustness and be competitive with models specifically trained to combat corruption robustness, leading some credence to the main motivation of the paper.

Thus the decision will be to accept.
Decision: Accept (poster)