SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To quantify further the specific impact of individual components of VITO on robust recognition, we show the same plot (Fig. 4.1 (Right)), now with the ablations described in Sec 4.3. We find that all components of our method and architecture are necessary for best robustness, but in particular there is a striking split between models trained with only spatial deformations (VITO  $(\mathrm{T} = 0)$ , MoCLR ImageNet, MoCLR VideoNet) and those trained with video deformations. We find that the models that learn only from image-level spatial deformations suffer significantly in robustness against all of the models that learn from video deformations.
Figure 3: Example human saliency maps from the ClickMe dataset [71] and ResNet-50 models. Gradient-based saliency is shown for Supervised and Harmonized [72]. Attention maps are shown for CLIP and VITO model. We use multi-head attention pool weights for CLIP and average of weights from last 2 attention pooling scales in VITO.
Method	Trained for alignment	Human Alignment
MoCLR [68]	X	21.4
Supervised CLIP [58]	X	34.4
X	41.8
Harmonized [72]	✓	45.5
VITO	X	47.7
Table 2: Quantitative comparison between gradient-based saliency maps (from Supervised, MoCLR, CLIP-RN50 (attention-map), and Harmonized networks), VITO attention weights, with human saliency maps using a correlation based alignment score from [71]
4.2 Measuring explicit human-alignment
Given that VITO representations display strong generalization across many tasks and robustness to distribution shifts, two signatures of human perceptual intelligence, we now directly ask whether they align with human perceptual representations.
Visual saliency via contrastive attention. We start by comparing VITO's learned attention masks to human saliency data from the ClickMe dataset [71], as well as saliency maps obtained from a collection of ResNet-50 models. For the supervised and MoCLR ResNets we use standard gradient-based saliency as in [72]. Since our model contains two attention maps at two scales of the ResNet, we upsample both maps to the image size and simply average them to obtain a single map. We compare our attention maps additionally to those obtained from the modified CLIP ResNet [58], which also utilizes attention-pooling in the final layer but is trained for image-language alignment (the canonical approach for training state-of-the-art visual language models). Because the CLIP pooling uses multi-head attention, we upsample these maps and average them across heads. Finally, we also compare to the gradient-based saliency maps from a "harmonized" model explicitly trained to align with human saliency ([72]).
Qualitatively, VITO saliency maps appear significantly more aligned with human perception maps than the supervised and CLIP ResNets (Figure 3). Surprisingly, VITO appears more aligned than the Harmonized saliency maps across the 4 examples. Quantitatively (using Spearman rank correlation) VITO outperforms the supervised, MoCLR, and CLIP models by a large margin, and even surpasses the Harmonized model which has been specifically trained for this purpose (Table 2).
This result suggests that as opposed to image-based objectives or image-language alignment, human perception of feature importance across the visual scene can be better explained as a consequence of learning what to attend to in the context of self-supervised video-based learning. We hypothesize that these attention masks could underlie the formation of high-level concepts via "semantic binding", which we investigate in Figure B.1 and Section B.1.
Human error consistency in shape-biased tasks. Based on this result relating to object saliency, we hypothesize that VITO may be capturing global object shape features better than traditional deep networks which have been shown to heavily rely on textural cues for classification [56].
To evaluate this quantitatively, we used a subset of the dataset proposed in [59] to test both the accuracy and consistency with human judgments of model classifications of stimuli that require
Method	accuracy diff. ↓	obs. consistency ↑	ceiled error consistency ↑
Image pretraining
DINO [73]	0.236	0.504	0.291
Supervised	0.215	0.511	0.329
SIN+IN1K [56]	0.203	0.527	0.330
MoCLR [68]	0.190	0.536	0.335
L2-Robust [54]	0.178	0.544	0.389
CLIP [58]	0.108	0.612	0.482
Video pretraining
R3M [74]	0.392	0.359	0.054
CycleCon [9]	0.237	0.484	0.258
VINCE [7]	0.210	0.501	0.269
VITO	0.157	0.564	0.422
Table 3: Accuracy difference and consistency with human judgments on stimuli that are biased to requiring global-shape understanding (instead of texture) for recognition/discrimination. VITO surpasses all comparable trained models (both image and video pretraining) in all benchmarks, including those that are trained specifically to be robust (SIN+IN1K, and L2-robust). We underperform the CLIP model; however, we note that CLIP is trained with an order of magnitude more images (400M) and explicit human-language supervision.
shape-cues for effective discrimination (Table 3). Specifically, these stimuli are categorized into 4 groups: edge drawings, cue-conflict / stylized (mixing of shapes with contradictory textures through style-transfer), variable low-pass filtering (to remove high-frequency local content), uniform noise (corrupts local texture features). Based on the original methodology proposed in [5], we report the accuracy difference (from human accuracy), the raw consistency with human judgments, andCeiled error consistency (method from [5]).
We compare to supervised and MoCLR ResNets, the robust training methods cited earlier, as well as CLIP [58]. We also compare to various video pre-training methods cited earlier and another (R3M [74]), which has specifically shown to have human- and neurally-aligned representations of dynamic, object-centric scenes [75]. For all networks, we train linear classifiers on the ImageNet validation set and evaluate on the modified shape-biased stimuli. Compared with all other comparable image pretrained models, VITO achieves stronger robustness to shape-biasing transformations (lower accuracy difference relative to original images). Furthermore, VITO makes predictions more consistent with human judgements in terms of per-trial classification behavior. This is particularly surprising as VITO even outperforms the adversarially-trained robust model without requiring any explicit robust training procedure. Moreover, this improvement is not captured by prior video pretraining efforts (which are in fact far worse than the image pretraining methods). The R3M model, in particular, performs surprisingly poorly. Because the images used to collect the human judgments are modified versions of those from the ImageNet validation set, we hypothesize that this performance can be attributed to the poor transfer of the Ego4D datasets to the diverse classes present in ImageNet (contrarily to VideoNet). Indeed, the R3M model only achieves  $13\%$  accuracy on the clean ImageNet validation set (see Table B.3). Finally, we note that VITO does underperform CLIP on this benchmark; however, this comparison is not truly fair as CLIP is trained with explicit human supervision via large-scale image-language mappings. In fact, we believe that our method can be augmented with similar language supervision to improve human alignment even further.
In summary, VITO captures aspects of how humans process shape-information that cannot be captured by other strong visual models. Understanding more about this effect and what aspects of learning from videos lead to this remain interesting opportunities for future work.
To understand more about how the components of VITO training contribute to its performance, we vary the different aspects of our paradigm in isolation: our method for data curation (VideoNet), multi-scale attention pooling, and details of the input data (spatial crop size and the temporal sampling scheme). We explore some ablations in detail on an example benchmark (PASCAL segmentation), but also evaluate ablations across many of the benchmarks used in this work. Finally, we provide a brief exploration demonstrating that our method scales well to larger architectures.
Figure 4: Impact of pretraining data's spatial content on representation quality. Left: transfer performance of models pretrained on single frames from image datasets (grey bars) or individual videos (blue bars). Right: example frames from different video and image datasets.
Effect of pretraining data. To demonstrate the effect of the pretraining data distribution on transfer performance, we pretrain a baseline MoCLR model (using 2 views) on a variety of image and video datasets, where we initially treat video datasets as collections of individual frames. We train each model for 300 ImageNet-equivalent epochs, referred to hereafter as "epochs" (i.e. 1 epoch = learning from 1.28M examples, irrespective of the dataset), such that each model benefits from the same amount of computation. Figure 4 (left) shows their transfer performance on PASCAL semantic segmentation. As expected, ImageNet pretraining works very well, but pretraining on standard video datasets results in a substantial drop in performance (e.g.  $-6.8\%$  or  $-5\%$  mIoU from pretraining on Kinetics700 or AudioSet). This performance gap between video and image pretraining can be attributed to a combination of increased complexity and field-of-view of video frames and domain mismatch between the dataset categories (Figure 4, right). Consistent with this, training on JFT [76], an uncurated dataset with a heavy-tailed class distribution, also results in a loss in performance. Notably, this is despite the much larger size of JFT (300M images). We find that applying the same baseline pretraining to frames from our curated video dataset performs better than existing large-scale video datasets like Audioset (+1.6% mIoU), but still underperforms image pretraining on JFT and ImageNet (Figure 4). This demonstrates the importance of aligning the distribution of video frames with that of common image datasets. We therefore use VideoNet as our primary pretraining dataset for the rest of the study. In Sec B.3 we disentangle the power of our method and dataset by confirming that each independently have strong effects: MoCLR trained on VideoNet, and VITO trained on standard datasets (Audioset or YT8M) also outperform all prior work (including models trained on much larger image datasets like JFT-300M).
Multi-scale attention pooling. We decompose the proposed multi-scale contrastive attention pooling to isolate the effects of multi-scale learning from those of attention pooling (Figure B.2, right). While we find only modest gains from adding attention pooling to a single-scale version of the model  $(+0.2\% \mathrm{mIoU})$ , we find that the 2-scale model (without attention pooling) improves over the single scale model more robustly  $(+0.6\% \mathrm{mIoU})$ . Interestingly, we find that the combination of the 2-scale model with attention pooling has a synergistic effect  $(+1\% \mathrm{mIoU}$  over the single-scale attention model), highlighting the importance of handling the variability in scales present in natural videos.
Spatial and temporal augmentation parameters. We first validate in Figure B.2 (left) our hypothesis that increasing the minimum crop-scale in the random-reshed crop operation during training leads to models that generalize better to fine-grained tasks like semantic segmentation. Specifically, we find that a minimum crop scale of 0.4 (as opposed to the traditional 0.08) results in the best transfer performance  $(+1.7\% \mathrm{mIoU})$ . Note that this conclusion differs slightly from that of [37] who find more aggressive cropping to be beneficial for action recognition.
Next, to study the effect of different temporal sampling schemes, for each training example, we sample 3 views using marginal sampling of each frame from the video clip of length  $T = 2.56$  seconds. This length determines the distribution of time differences between any pair of frames, and thus the time-scale over which the contrastive model learns invariances. We verify our choice by varying the total length of clips. While going to longer time-scales  $T = 3.2s$  does not hurt performance much, we find a significant improvement over using shorter clips (e.g.  $T = 1.28s$ ,
Pretraining	Dataset	PASCAL (mIoU)	UCF101 (top-1)	IN-A (top-1)	IN-Vid (pm0/pm10)	Human error consistency
MoCLR	VideoNet	72.8	83.0	2.3	55.5/40.5	0.224
VITO 1scale (w/o attn)	VideoNet	75.2	85.5	3.9	67.3/55.5	0.359
VITO 1scale (attn)	VideoNet	75.4	85.7	3.5	65.6/52.9	0.368
VITO 2scale (w/o attn)	VideoNet	75.8	86.2	4.2	67.4/54.9	0.390
VITO (T=0)	VideoNet	74.8	83.2	3.9	63.9/49.5	0.323
VITO	AudioSet	73.8	84.8	3.4	55.7/42.4	0.401
VITO	VideoNet	76.3	87.4	5.4	70.6/57.2	0.422
Table 4: Summary of ablation models on key evaluations covering image understanding, video understanding, and human alignment on ood object recognition. In summary, it is clear that all components (pretraining data, temporal deformations, and the multi-scale attention pooling) are required for best performance across all tasks.
+1.0% mIoU; Figure B.2, center). This suggests that invariance to the rich temporal deformations present in video clips is indeed a beneficial criterion for learning fine-grained spatial representations.
Comprehensive ablation summary. In Table 4, we extend the above ablation studies to a more comprehensive benchmark set. In addition to the PASCAL segmentation task, we evaluate the key ablated models on video understanding (UCF101), OOD recognition (IN-A/IN-Vid) and human alignment on the shape-bias tasks specified in Sec 4.2. We confirm that all of the major methodological components (VideoNet dataset, multi-scale attention pooling, and using temporal deformations) work in concert, and are required for best performance across all tasks. Notably, we see a particularly striking dichotomy between models trained with and without temporal deformations on human error-consistency. Specifically, models trained without temporal deformations (MoCLR and VITO  $(\mathrm{T} = 0)$ ) have a significant drop in human error-consistency relative to all other models trained with temporal deformations, highlighting the importance of learning these kinds of invariances.
Scaling model architectures. We briefly demonstrate that VITO scales to more recent larger architectures. Specifically, we show preliminary results that VITO achieves highly competitive performance on four scene understanding benchmarks using the Swin-S transformer architecture [77]. In Sec. B.4, we show that performance improves dramatically over the ResNet-50 architecture and is competitive with a strong, specialized ImageNet pretrained baseline for fine-grained scene understanding (DetCon [65]).
Summary. We propose VITO, a simple method for distilling videos into visual representations. The key features of our method include improved dataset curation, adapting augmentation pipelines to appropriately handle video frames, and using attention-guided contrastive learning. With these components, VITO surpasses both prior video pretraining in spatial understanding, and image pretraining on temporal understanding and robustness. In addition to these hallmarks of human perception, VITO explicitly aligns with aspects of human saliency and image recognition behavior that are not captured by other high-performance representation learning techniques. In sum, despite the many successes in video representation learning, our results suggest that there is a great untapped potential in video pretraining as a paradigm for learning general, human-aligned visual representations.
Limitations and Future Work. We believe this work can be a foundation for future video pretraining efforts, as our approach is powerful, yet simple and extensible. However, we recognize that this demonstration is mostly limited to a single contrastive learning framework and ResNet-50 architecture. We leave for future work, the validation and exploration of similar analyses with larger models and other self-supervised training objectives (such as MAEs and self-distillations methods like DINO). Additionally, while we have shown the benefits of a surprisingly simple attention module for learning correspondences in video data, there are more powerful attentional architectures we can leverage along with scaling dataset size as in [10]. We have started these experiments with our exploration of Swin transformer architectures.
A.1 Self-supervised learning
Data pre-processing. Each frame is randomly augmented by composing the following operations, each applied with a given probability:
1. random cropping: a random patch of the image is selected, whose area is uniformly sampled in  $[s\cdot \mathcal{A},\mathcal{A}]$ , where  $\mathcal{A}$  is the area of the original image, and whose aspect ratio is logarithmically sampled in  $[3 / 4,4 / 3]$ .  $s$  is a scale hyper-parameter set to 0.08 when learning from ImageNet, and 0.4 when learning from videos. Regardless, the patch is then resized to  $224\times 224$  pixels using bicubic interpolation;
2. horizontal flipping;
3. color jittering: the brightness, contrast, saturation and hue are shifted by a uniformly distributed offset;
4. color dropping: the RGB image is replaced by its grey-scale values;
5. gaussian blurring with a  $23 \times 23$  square kernel and a standard deviation uniformly sampled from [0.1, 2.0];
6. solarization: a point-wise color transformation  $x \mapsto x \cdot \mathbb{1}_{x < 0.5} + (1 - x) \cdot \mathbb{1}_{x \geq 0.5}$  with pixels  $x$  in [0, 1].
The augmented frames  $\pmb{v}^1$  and  $\pmb{v}^2$  result from augmentations sampled from distributions  $\mathcal{A}_1$  and  $\mathcal{A}_2$  respectively. These distributions apply the primitives described above with different probabilities, and different magnitudes. The following table specifies these parameters for the BYOL framework [61], which we adopt without modification. When learning from three views, we use the distribution  $\mathcal{A}_1$  to generate the third view.
Parameter	A1	A2
Random crop probability	1.0
Flip probability	0.5
Color jittering probability	0.8
Color dropping probability	0.2
Brightness adjustment max	0.4
Contrast adjustment max	0.4
Saturation adjustment max	0.2
Hue adjustment max	0.1
Gaussian blurring probability	1.0	0.1
Solarization probability	0.0	0.2
Optimization. We pretrain ResNet-50 using the LARS optimizer [78] with a batch size of 4096 split across 128 Cloud TPU v3 workers. We adopt the optimization details of BYOL, scaling the learning rate linearly with the batch size and decaying it according to a cosine schedule. The base learning rate is 0.3 and the weight decay is  $10^{-6}$ .
A.2 Transfer to PASCAL and ADE20K semantic segmentation
Architecture. We evaluate ResNet models by attaching a fully-convolutional network (FCN, Long et al. [79]) and fine-tuning end-to-end, following He et al. [16]. When evaluating Swin transformers we instead use the UperNet segmentation architecture [80].
Data pre-processing. During training, images are randomly flipped and scaled by a factor in [0.5, 2.0]. Training and testing are performed with  $512 \times 512$ -resolution images. When fine-tuning on ADE20K, we additionally use photometric transformations from the mmseg† codebase.
$\dagger$ https://github.com/open-mmlab/mmsegmentation
Optimization. We fine-tune for 45 epochs on the PASCAL train_aug2012 set or 60 epochs on the ADE20K train set. We use stochastic gradient descent with a batch size of 16 and weight decay of 0.005. The learning rate is initially set to 0.04 and decayed exponentially with a factor of  $0.9^n$  where  $n$  is the iteration number. When fine-tuning external models, we sweep over the base learning rate and weight decay and report their performance given the optimal configuration. In all cases we report mIoU on the val set averaged across 5 runs.
A.3 Transfer to COCO and LVIS object detection
Architecture. We evaluate both ResNet and Swin transformers using the FCOS\* architecture, following Henaff et al. [81].  $\mathrm{FCOS}^{\star}$  is the implementation of a single-stage detector based on FCOS [82], and improved with the collection of techniques from Wu et al. [83], Zhang et al. [84], and Feng et al. [85], full details can be found in Henaff et al. [81].
Data pre-processing. The target resolution is  $800 \times 1024$ . During testing, an image is resized by a factor  $s$  while preserving the aspect ratio, such that it is tightly contained inside the target resolution, and then padded. When fine-tuning, the image is rescaled by a factor of  $u \cdot s$  where  $u$  is uniformly sampled in [0.8, 1.25], and is then cropped or padded to the target resolution.
Optimization The network is fine-tuned for 30 epochs on the COCO train2017 set or the LVIS v1_train set. We use AdamW [86] with weight decay  $10^{-4}$ , base learning rate of  $10^{-3}$ , and batch size 128 split across 16 workers. The learning rate rises linearly for  $\frac{1}{4}$  of an epoch, and is dropped twice by a factor of 10, after  $\frac{2}{3}$  and  $\frac{8}{9}$  of the total training time. We report mAP on the COCO va12017 set and the LVIS v1_val set, averaged across 5 runs.
A.4 Transfer to DAVIS video segmentation
As a further test of scene understanding, we assess whether learned representations can continue to recognize parts of an object as they evolve over time. Video object segmentation, specifically in its semi-supervised setting, captures this ability, which we evaluate on the DAVIS'17 benchmark. Having evaluated a learned representation on a video independently across frames, we segment these features with nearest neighbor matching from frame to frame, given a segmentation of the first frame. In this way, the segmentation is propagated according to the similarity of the representation across space and time. We reuse the segmentation procedure from Xu and Wang [8] without modification, and report region  $(\mathcal{I})$  and boundary quality  $(\mathcal{F})$ .
A.5 Transfer to UCF-101 action recognition
We evaluate action recognition classification on the UCF101 dataset [3]. We follow the procedure for finetuning used in [9] which is based on [87]. We utilize clips of 2 seconds in length at 12fps. Each frame is processed by the ResNet-50 backbone. Clip representations are obtained by one of three methods for temporal integration:
1. Average pooling is the standard baseline, producing a 2048-d vector output for a clip which is then fed to and one fully connected  $(2048\times 101)$  layer for predicting the action class.
2. MS avg-pool: we pool the block3 representations (1024-d) over the two subclips of 1s each that make up the larger clip. This is done because the features at this scale have smaller receptive fields and are selective for less complex content. Then we concatenate the two (1024-d) vectors with the average pooled feature from the block4 output to get a single 4096-d vector for each clip that again is fed through a fully-connected layer to predict the action class. By concatenating the two subclip representations, the fully-connected layer can in fact compute complex temporal relationships such as differences etc. along with the final layer's invariant representation that is pooled for the full clip.
3. MS temp-attn: We perform the same methodology as above for integrating multiple scales, but replace the average pooling over time with an attention pooling layer. Given representations for an L-frame clip  $z \in \mathbb{R}^{B \times C \times L}$  at a given scale, we compute temporal attention weights  $w_{t} \in \mathbb{R}^{L}$  where  $w = f(z)$ . We choose
$f$  to be  $\tanh (Wz)$  where  $W\in \mathbb{R}^{C\times 1}$  is a linear weighting of channels. Finally the pooled representation  $v = \sum_{L}w_{t}\cdot z$
We show results using method 1 in the main text and demonstrate the improvements from methods 2 and 3 in Appendix Table B.5. 10 clips are sampled from each video and the predictions of the clips are averaged for the final results. We fine-tune for 16 epochs using the ADAM optimizer with a multi-step LR decay schedule at epochs 6, 10, and 14. The initial learning rate is set to 0.0001. The implementation is adopted from https://github.com/facebookresearch/AVID-CMA.
A.6 Transfer to ImageNet classification