SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For all models we freeze the ResNet-50 encoder (which outputs 2048-d embeddings). We then train a linear head to classify the 1000 categories in the ImageNet training set using the standard split. To train the classifier, we use the SGD optimizer with nesterov momentum and momentum parameter equal to 0.9. We use weight-decay of 0 and sweep the learning rate for each model in the range [0.4, 0.3, 0.2, 0.1, 0.05] and pick the best classifier based on ImageNet validation accuracy.
A.7 Transfer to out-of-distribution evaluations
For all OOD evaluations, we evaluate on datasets that utilize all (or subsets) of the ImageNet validation set. Therefore, for these evaluations we use the pre-trained encoder and linear classifier (trained as in Sec A.6). We freeze the encoder and linear classification head and evaluate task performance on images from either the ImageNet-A, ImageNet-vid-robust, and Imagenet-3DCC datasets. For ImageNet-A and ImageNet-vid-robust, we use the evaluation code and method from [4].
For ImageNet-3DCC, we do not use the entire corruption set because we wanted to specifically test models under the more natural 3-d corruptions. As is described in [53], the dataset can be broken down into two sets of corruptions: 3-d informed corruptions (using a depth model to generate natural corruptions informed by 3-d information) and standard 2-d noise and artifacts (like in ImageNet-C). For our experiments, we chose to evaluate specifically on the 3-d corruptions, which were found to induce larger robustness effects for evaluating standard networks [53]. Nevertheless, we found similar results when evaluating robustness to 2-d noise and artifacts. All images from the following classes of corruptions were used for evaluation: far focus, near focus, fog, flash, xy motion blur, z motion blur, view jitter.
A.8 Alignment with human saliency
Human saliency measurements are obtained from the ClickMe dataset. Alignment is measured as the Spearman rank correlation between model and human saliency averaged over the dataset, normalized by inter-rater alignment of humans.
A.9 Human error consistency evaluation
We evaluate accuracy and human error consistency on shape-bias datasets using the code from https://github.com/bethgelab/model-vs-human/tree/master. We choose the subset of images that remove textural cues in different ways (forcing humans and models to utilize global shape during discrimination): edge drawings, cue-conflict stimuli, graded low-pass filtering, and uniform gaussian noise. We report three metrics from [59]:
1. Accuracy difference: measure of human vs model classification accuracy on each OOD dataset and then averaged.
2. Observed consistency: measures the fraction of samples for which humans and a model get the same sample either both right or both wrong.
3. Error consistency: Score that measures whether there is above-chance consistency. This is important because e.g. two decision makers with  $95\%$  accuracy each will have at least  $90\%$  observed consistency, even if their  $5\%$  errors occur on non-overlapping subsets of the test data (intuitively, they both get most images correct and thus
Figure B.1: Example augmented frames with overlaid (resized) learned attention masks. Attention is computed from the output of the final block of the VITO trained ResNet-50. Crucially, the attention masks are computed independently, such that the attention module can only use spatial cues.
observed overlap is high). Error consistency indicates whether the observed consistency is larger than what could have been expected given two independent binomial decision makers with matched accuracy [5].
The mathematical details on each of these metrics are provided in [5].
B Appendix: Additional results
B.1 Semantic binding with contrastive attention pooling
The ablation study demonstrated that multi-scale attention improves the performance of VITO in semantic segmentation. To probe why this may be, we visualize and interpret the learned attention masks (Figure B.1). For simplicity, we only visualize the masks from the coarsest scale (output feature map), but the interpretation naturally extends to the multi-scale version as these masks are learned with independent attention modules.
Because the attention masks are not computed jointly across each view, for a given video frame, the attention module must marginalize over the training data to make a statistical prediction—what should be attended to in the first view in order to minimize the contrastive loss across possible second views? Specifically, the attention must focus on content that is most likely to be stable across time while still being discriminative (or unique) relative to other frames from other videos. Different examples appear to trade-off these criteria differently, yet systematically. For example, in the third column of Figure B.1 even though the animated characters on the right side of both frames may be discriminative content, the attention module has learned to focus on the static picture on the left as it is the content that is most likely to be stable across time. For this pair of frames the prediction is correct—the attention disregards content that is changing too abruptly—despite not having access to motion cues. On the other hand, the example in the fourth column demonstrates a scenario where the model has attended to stable, but primarily discriminative content (the bird) rather than the background, which is also very stable but most likely less unique relative to other videos.
Even beyond the ability to localize stable, yet discriminative content, it seems that our method also enables "semantic binding" of visually different, but semantically related features. This can be seen in the first pair of frames, as the model has learned to associate an arm or elbow (in the first frame) with the dumbbell (in the second frame), demonstrating an understanding that these two semantically related concepts co-occur and thus are predictive of one another given the right embedding.
Binding co-occurring features appears as an intuitive explanation for why these representations would perform well on semantic segmentation. It is particularly interesting that training end-to-end with a standard contrastive loss can produce complex behavior reminiscent of the DINO approach [19] even though we use a single, two-layer MLP attention module as opposed to large-scale transformer architectures which use attention throughout the network.
B.2 Ablating the components of VITO
In Figure B.2 we demonstrate on an example scene understanding task (PASCAL) how VITO is impacted by crop-scale, clip length, and the type of attention pooling used (or not used). In Figure B.3 we additionally do a deeper analysis of the temporal sampling scheme and demonstrate that our choice performs best across tasks and is arguably the most natural.
Figure B.2: Effects of crop scale, natural augmentations, and multi-scale attention on representation quality. All ablations are performed relative to VITO's configuration (denoted by a green asterisk) which uses 2-scale attention pooling, a less aggressive crop scale of  $40\%$ , and natural augmentations uniformly sampled in a window of length  $\mathrm{T} = 2.56\mathrm{s}$ . We also compare to our baseline MoCLR model trained on single frames, either from ImageNet (dotted gray line) or VideoNet (dashed blue line). All models are evaluated by transferring to PASCAL semantic segmentation.
Figure B.3: Ablating different temporal sampling schemes. Delta refers to fixed time sampling between frames as in Gordon et al. [7]. Uniform refers to chunking time into non-overlapping blocks and uniformly sampling within each chunk as in Xu and Wang [8]. Marginal sampling (ours) refers to simple uniform sampling from the full video clip of length  $T = 2.56s$ . First two panels show that marginal sampling is best overall across transfer to PASCAL and ADE20K. Third panel shows the distribution of absolute time-differences between any two pairs of frames under each sampling scheme (assuming 3 views are sampled per clip). Our marginal sampling scheme is arguably the most natural as the mode of the distribution is at 0, meaning that it is not biased to over-represent any specific time difference (similarly to the random-resized crop operation in space).
B.3 Dataset and method ablations
In Table B.1 we show that both our learning objective VITO and choice of dataset, VideoNet, are important for achieving top performance. However, these results also show that we can outperform existing video pretraining even when using
standard datasets like Audioset and YT8M. In addition, by comparing to MoCLR trained on JFT-300M, we demonstrate the benefits of our method are not the result of simply having more frames of training data.
Pretraining	Dataset	Epochs	Semantic segmentation	Object detection
PASCAL	ADE20K	COCO	LVIS
MoCLR	VideoNet	200	72.8	37.5	42.6	24.6
VITO	YT8M	200	71.8	37.8	42.7	24.6
VITO	AudioSet	200	73.6	38.5	43.2	25.0
VITO	VideoNet	200	75.5	39.2	43.6	25.6
MoCLR	JFT-300M	200	74.3	38.7	43.2	25.4
Table B.1: VITO dataset and method ablations. We compare the baseline method MoCLR trained on VideoNet to demonstrate the impact of our methodology. VITO on VideoNet performs significantly better due to the methodological improvements (attention pooling, adaptation of spatial and temporal augmentations). We also evaluate VITO on troidonal video datasets such as YT8M and AudioSet. We note that these numbers still greatly outperform prior video pretraining (See Table B.3. However the imapct of the VideoNet dataset is clear as the best model is VITO trained on VideoNet. Finally, we show that VideoNet does not simply provide benefits due to increased number of total frames vs. ImageNet. In fact, MoCLR trained on JFT-300M has an order of magnitude more frames and yet still underforms.
B.4 Scaling architectures
Here we demonstrate that VITO scales effectively to more powerful Swin transformer architectures. Results on scene understanding benchmarks improve greatly over ResNet-50 models and are competitive with specialized fine-grained scene understanding models from recent literature [65]. See Table. B.2
Pretraining	Dataset	Backbone	Semantic segmentation	Object detection
PASCAL	ADE20K	COCO	LVIS
VITO	VideoNet	R50	76.3	39.4	44.0	25.7
MoCLR	VideoNet	Swin-S	78.6	43.7	48.4	32.7
VITO	VideoNet	Swin-S	81.3	46.1	49.8	33.5
DetconB	ImageNet	Swin-S	81.4	46.1	50.4	33.1
Table B.2: VITO scales to larger model architectures (Swin-S), improving performance compared to the ResNet-50 baseline and remaining competitive with a strong ImageNet pretrained baseline (Detcon) from Henaff et al. [65].
B.5 Comparisons on additional scene understanding tasks
VITO outperforms all prior video pretraining (of image representations) on scene understanding tasks. In addition to the evaluations in the main text, we add PASCAL segmentation, LVIS object detection, ImageNet-1K classification. VITO remains highly competitive with the best ImageNet pretraining on these tasks. (See Table B.3).
Video Pretraining	Dataset	Semantic segmentation	Object detection	Classification
PASCAL	ADE20K	COCO	LVIS	IN-1K
Random Init	53.0	27.9	39.0	21.1	-
Methods pretraining on video datasets
R3M [74]	-	-	-	-	-	13.3
VFS [8]	K400	63.9	31.4	41.6	23.2	-
VIVI [69]	YT8M	65.8	34.2	41.3	23.2	62.6
VINCE [7]	R2V2	69.0	35.7	42.4	24.4	54.4
CycleContrast [9]	R2V2	69.2	35.6	42.8	24.5	55.6
MMV TSM [70]	AS + HT	70.6	32.5	41.3	24.2	51.4
VITO	VideoNet	76.3	39.4	44.0	25.7	66.2
Methods pretraining on ImageNet
Supervised	ImageNet	71.3	33.5	44.2	25.2	76.1
BYOL [61]	ImageNet	76.1	38.8	43.7	25.5	-
MoCLR [68]	ImageNet	76.4	39.2	43.9	25.8	71.4
DINO [19]	ImageNet	76.1	39.0	44.3	26.4	75.3
Table B.3: Image and pretraining evaluated on object-detection, semantic segmentation, and ImageNet-1K classification.
B.6 Comparison to image pretraining on video-based tasks
Here we demonstrate more thoroughly that compared with image pretraining methods (image backbones), we perform significantly better on video-level tasks. On both DAVIS segmentation (Table B.4) and UCF-101 action recognition (Table B.5), VITO outperforms strong ImageNet trained baselines and methods pretrained on video datasets.
Pretraining	Dataset	Jm	Fm
ImageNet pretraining
Supervised	ImageNet	63.7	68.4
MoCo [16]	ImageNet	63.2	67.6
DetConB [65]	ImageNet	63.1	66.4
MoCLR [68]	ImageNet	63.1	67.8
BYOL [61]	ImageNet	63.8	69.4
Video pretraining
VINCE [7]	Kinetics	63.4	67.8
TimeCycle [88]	VLOG	41.9	39.4
UVC [89]	Kinetics	54.5	58.1
CRW [44]	K400	64.8	70.2
VFS [8]	K400	65.3	70.2
VITO	VideoNet	65.5	70.8
Table B.4: VITO significantly outperforms all image-pretraining baselines on DAVIS 2017 video segmentation. VITO also outperforms many recent successful video pretraining methods.
Pretraining	Dataset	Backbone	Top-1
Video architectures
Supervised [90]	ImageNet	I3D	67.1
VideoMoCo [91]	K400	R(2+1)D	78.7
Temporal-ssl [92]	K400	R(2+1)D	81.6
VTHCL [93]	K400	3D-R50	82.1
CoCLR [94]	K400	S3D	87.9
CVRL [34]	K400	3D-R50	92.9
ρ-BYOL [37]	K400	3D-R50	95.5
Supervised [95]	K400	I3D	95.1
Image architectures
OPN [96]	UCF101	VGG-M	59.8
TCE [97]	K400	R50	71.2
CycleContrast [9]	R2V2	R50	82.1
MoCLR [68]	ImageNet	R50	85.5
BYOL [61]	ImageNet	R50	85.6
VITO (avgpool)	VideoNet	R50	87.4
VITO (MS-avgpool)	VideoNet	R50	88.5
VITO (MS-attnpool)	VideoNet	R50	89.4
Table B.5: VITO outperforms all image representations when finetuning for UCF101 action recognition, using temporally-pooled frame-level representations. VITO's performance is even competitive with many video architectures.