Revisiting Feature Prediction for Learning Visual Representations from Video
Authors: Adrien Bardes
Quentin Garrido
Jean Ponce
Xinlei Chen
Michael Rabbat
Yann LeCun
Mido Assran
Nicolas Ballas
Venue: Accepted by TMLR ()

Abstract: This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision. The models are trained on 2 million videos collected from public datasets and are evaluated on downstream image and video tasks. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks, without adaption of the model’s parameters; e.g., using a frozen backbone. Our largest model, a ViT-H/16 trained only on videos, obtains 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet1K.

Under review as submission to TMLR
Revisiting Feature Prediction for Learning Visual Represen- tations from Video
Anonymous authors Paper under double-blind review
Abstract
This paper explores feature prediction a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision. The models are trained on 2 million videos collected from public datascts and are evaluated on downstream image and vidco tasks. Our results show that learning by predicting video features leads to versatile visual represenlations that perform well on both motion and appearance-basee tasks, without adaption of the model's parameters; c-g., using frozen backbone. Our largest model, VIT-H/16 trained only on videos, obtains 81.9% on Kinetics-400, 72.2% on Sompehhinn--Smmething--r- and 77.9% on ImageNet
1 Introduction
Humans possess the remarkable ability to map low-level signals originating from the retina into semantic spatio-temporal understanding of the world; synthesizing notions such as objects and global motion (Spelke
et al. (995). A long-standing goal of the machine learning community is to identify the principles or objectives
that may guide such unsupervised learning in humans (Field 1994 Berkes & Wiskott 2005 Hinton 1989).
One related hypothesis is based on the predictive feature principle (Rao Ballard 1999), which posits that
representations of temporally adjacent sensory stimuli should be predictive of each other.
In this work, we revisit feature prediction a stand-alone objective for unsupervised learning of visual representations from video. Numerous advances in the field such as the standard use of transformer architectures in vision Dosovitskiy et al. 2020), the maturing of masked autocncoding frameworks {Xic et al.
2021 Bao et al. 2021: He et al. (0221) query-based feature pooling (Chen et al. 2022 joint-embedding predictive architectures (JEPA) LeCun 2022 Assran et al. 2023 Baevski et al., 2022b). and larger datasets form a unique arsenal of tools, which we integrate in a modern and conceptually simple method, the video Seint-embedding predictive architecture or V-JEPA, which is based solely on feature prediction, without using pretrained image encoders, text, negative examples, human annotations, pixel-level reconstruction.
We seek to answer the simple question:
How effective is feature prediction as a sland-alone objective for unsupervised learning from video with modern tools?
To that end, we pretrain a family of V-JEPA models on a dataset of 2 million videos collected from publicly available datasets by combining a masked modeling prediction task with a joint-embedding predictive architecture (sce Figure p We measure performance on several downstream image and video tasks, using both frozen evaluation and end-to-end fine-tuning. Our findings suggest that feature prediction can indeed serve effective stand-alone objective for unsupervised learning from video, while using significantly shorter training schedules than pixel prediction methods. Specifically:
Feature prediction leads versatile visual representations that perform well across downstream image and video tasks without adaption of the model's weights; i.e,, using a frozen backbone. V-JEPA achieves

---

Under review as submission to TMLR
Frozen Evaluation
SOTA line tuned lask-speclii
model 5802 (MVD)
V-JPAA VIT-Hiv
Something Something-Someiie Sometiintt
70
WIFL/6
OmsiMAE VIT-
VideoMAE
SOTA dine-tuned
60
Hicradi
VIT-H/66
Imk-spucific model
K402
VideoMAEsz
50
LJEPA
ViTg/4
40
Video Feature Prod.
Video Proel Pred.
OpenCTIP VIT-Gi
Image Models
70
72
74
76
78
80
82
84
86
88
90
92
Kinetics 400
V-JEPA models pretrained video learn versatile visual representations. It performs well motion-based tasks Something-v2) and appearance-bain bassed tasks (Kinetics 400) without adaptation of the model's parameters, ie., using the same frozen backbone for both tasks.
Figure 1:
the best performance among methods we consider (+6% accuracy) on the SomethingSoettingg22 task, which requires fine-grained temporal understanding, V-JEPA also competitive on tasks like Kinetics400, where appearance- based features are sufficient and hence state-of-ther image models such as DINOv2 excel (Figure and Table 6
Models trained with feature prediction are superior pixel prediction approaches under frozen evaluation protocol (attentive probing) and are competitive with pixel prodiction under full fine-tuning, while using significantly shorter training schedules (Tables 5 and
Models trained with feature prediction more label-efficient than pixel prediction approaches. Decreasing the available number of labeled examples results in an increase in the performance gap between V-JEPA and pixel-reconstruucioo models (Table S
2
Related Works
Slow Features. One way to encourage temporally adjacent representations to be predictive of each other is to ensure that they vary slowly over time. Early works targeting predictive features encouraged representations of individual video frames to be locally temporally invariant, while preventing representation collapse by using spectral methods, as in SFA (Wiskott & Sejnowski] 2002). SSA (Kayser et al. 2001 ) and Simulated Fixations (Zou et al, 2012). More recently, Goroshin et al. (2013); Wang et al. (2010) train a siamese convolutional network to map the representations of two subsequent frames to the same point, while encouraging distant frames have diverse representations via a pair- wise margin loss and a triplet loss, respectively. Other works (Oord al. 2018 Surís et al. 2021 Feichtenhofer et al. 2021 implement temporal invariance using noise-contrastive estimation (Gutmann & Hyvärinen] (0022). Our exploration in this paper goes beyond temporal invariance and explores feature prediction using masked modeling.
Predictive Features. Going beyond local invariance, a family of works trains a predictor network to map the representation of' frame or clip al one time-step to a distinct representation at another time-step. Srivastava et al. (2015): Vondrick et al. (2016); Wang et al. (2023b) train such a video feature predictor network on top of a frozen pretrained image or vidco encoder. Unfreezing the target feature extractor, several methods train the video encoder and the predictor network simultancously, while preventing collapse by using supervised action forecasting loss Girdhar & Grauman 2021 , or by using the representations of distant.
2

---

Under review as submission to TMLR
clips negative samples in contrastive loss (Han et al. 2019 2020
Tan et al 2023 often focusing small
convolutional encoders (Han et al. 2019 2020j. The idea of learning representation by predicting missing information in feature space is also core to the joint-embedding predictive architecture (JEPA) (LeCun) 2022),
which combines a siamese encoder with a predictor network. JEPAs have been successfully instantiated in
several modalities, such as with audio data (Baevski al. 2022b) and image data (Zhou el. al., 2021: Oquab
et al. 2023 Assran al., 2023). In this work, we extend this paradigm to video data by leveraging recent. advances in self-supervised learning.
Advances in Self-Supervised Learning. The use of vision transformers (Dosovitskiy et al. 2020 et al,
2022 has become standard practice in self-supervised learning with joint-embedding architectures (Chen ct al. 2021 Caron et al.) 2021 Oquab et al., 2023 Zhou et al., 2021 Assran et al. 2022). and unlocked
masked image modeling in pixel space by parameterizing the pixel decoder as transformer with learnable
mask tokens (Dosovitskiy et al.
2020 Xie et al 2021 He et al.
2021 Bao et al. 2021). demonstrating
step-change in the representation quality of autoencoding methods Vincent et al.
2010 This line of
generative methods was subsequently extended to video data using -emporl masking TTong et al.,
2023). It was also 2022: Feichtenhofer et al., 2022 Wang et al.) 2023a Kalluri et al.) 2023 Gupta et al.
recently shown that the representations of masked image autoencoders could be significantly improved by
using learnable pooling mechanisms based on cross-attention (Chen et al., 2022) Finally, through careful selection of design choices, the non-contrastive collapse prevention strategy in BYOL (Grill et al.] 2020
recently made to work with image feature prediction methods (Baevski ct al. 2022b Assran et al. 2023
which demonstrated the ability to learn representations that can be leveraged for various downstream lasks without relying on invariance to hand-crafted image transformations.
Feature Prediction versus Pixel Reconstruction. Approaches that predict in pixel space must dedicate significant model capacity and compute to capture all the low-level detail in the visual input. By contrast, approaches that predict in latent space have the ficxibility to climinate irrelevant or umpredictable pixel- level details from the target representation (Vondrick et al. 2016). Predicting in representation space has been shown to lead to versatile representations that perform well across many downstream tasks through lincar probing or low-shot adaptation dAssran et al. 2023 Oquab et al.) 2023 Assran al. 2022), while demonstrating an efficiency gain during pretraining compared pixel level reconstruction (Assran al.) 2023 Baevski et al., 2022b a). The works of Bacvski et al. 2202a b) additionally show that predicting in representation space results in competitive end-to-end fine-tuning performance in the image, audio and text
domains. In this work, we extend these findings to the video modality.
3 Methodology: Video-JEPA
Z
predictor
D(sy, sy)
Sy
Sy
x-encoder
y-encoder
x
Figure 2: Hoint-Embedding Predictive Architectures trained to predict the representation of input y from the representation of another input æ. The additional variable z provides the predictor with information about the transformation that computes y from
Our goal to explore the effectiveness of feature prediction as a stand-alone objective for learning visual representations from video. To that end, we use joint-embedding predictive architecture (JEPA) (LeCun 2022); see Figure 2 The main idea behind JEPA is to learn by predicting the representation of an input y
3

---

Under review as submission to TMLR
from the representation of another imput æ. The basic architecture is made up of an encoder, Eo(-), which computes the representation of the inpuls, and a predictor, Pa(-), which predicts the representation of from the representation of z, conditioned on a variable 2 indicating the transformation (or corruption) between æ and y. Conditioning on & enables the generation distinct predictions for various transformations x.
3.1 Training Objective
We train our visual encoder (.).) to satisfy the constraint that representations computed from one part of the video, y, should be predictable from representations computed from another part of the video, æ. The predictor network Po(-), which maps the representation of æ to the representation y, is trained simultancously with the encoder, and is provided specification of the spatio-temporal positions of y through the conditioning variable Ay
Naively implementing the objective using the regression
minimizco.6
[P - E
would admit trivial solution, where the encoder outputs a constant representation, regardless of its input. In practice, use the following modified objective to prevent representation collapse,
minimizeo. ()((()))), sg -gggsgg(())))) (Eo()))
(1)
where sg(-) denotes a stop-gradient operation, which does not backpropagate through its argument, and E-(-) an exponential moving average of the network E.(-). The use of an exponential-move -ovingg average feature extractor along with stop-gradient and predictor has been used as collapse prevention strategy for image pretraining (Grill et al. 2020), and studied empirically (Xie et al. 2021 and theoretically (Tian
et al. 2021 In fact, the objective in equalion equation is similar to the loss of Assran et. al. (2023 used for image pretraining, but we modify it to use an l regression, which we found to be more stable.
Theoretical motivation. was proposed in Grill et al. (2020) for the BYOI. method. We provide simple adaptation of their analysis for our loss. For ease of exposition, we will disregard the effect of the conditioning variable z and consider one dimensional representations. Denote the representation Ey(y) by random variable Y. The optimal predictor under equation equation is thus given by the following functional expression,
theoretical motivation for the effectiveness this collapse prevention strategy
P* (Fo(z)) = PegminnllPP(aaaaa) Y| [edian(Y([ogr))
Substituting this expression for the optimal predictor into the loss function and evaluating the expected gradient of the encoder gives
- Y = VoMAD(Y|EE(()),
where MAD(- [z)) is the median absolute deviation of a random variable conditioned on Eş(z). Thus, in the case where the predictor is optimal, the encoder must learn to capture much information about the video possible to minimize the deviation of the larget. The hypothesis is that incorporating exponential moving average to compute the representation of ensures that the predictor evolves faster than the encoder and remains close to optimal, thereby preventing collapse.
3.2 Prediction Task: Predicting from
The feature prediction task is based on a masked modeling formulation (He et al. 2021 Tong et al. 2022 i.e., regions & and from the video are sampled using masking. To sample y from a video, we sample several (possibly overlapping) spatially continuous blocks with various aspect ratios and repeat the spatial blocks across the entire temporal dimension of the video; x is taken to be the complement. Masking large continuous block that covers the full temporal dimension limits information leakage due to the spatial and temporal redundancy of videos, and results in harder prediction task (Tong et al.) 2022
4

---

Under review as submission to TMLR
Mask [TxH*W]
x encoder
y encoder
predictor
Eg
P
Ep
Concatenate mask tokens
Remove
Remove
masked
unmasked tokens
tokens
[Lxd]
[Nxd]
[Nxd]
[Lxd]
[Mxd]
[Mxd]
[Lxd]
[Lxd]
Figure 3: V-JEPA. Training operates video clip of T' frames with spatial resolution H x W, flattened into a sequence L tokens. (Left to right): We first obtain the input of the z-encoder by dropping tokens from the video clip. The x-encoder then processes the masked video sequence, and outputs embedding vector for each imput token. Next, the oulputs of the x-encoder are concatenated with a set learnable mask lokens containing positional embeddings of the masked spatio-temporal patches. The predictor network processes the combined token sequence, and outputs an embedding vector for each mask token. The outputs of the predictor are then regressed to the prediction targets using an loss. The prediction targets correspond to the output of the y-encoder.
We leverage two types of masks: short- masks, where we take the union of 8 randomly sampled target blocks covering 15% of each frame, and long-range masks, where we take the union of 2 randomly sampled target blocks covering 70% of each frame. In both cases, the aspect ratio for all sampled blocks is randomly chosen in the range (0.75, 1.5). Given that both short-range and long-range masks are produced by sampling many blocks and taking their union, the result is average masking ratio of ~ 90%. We refer to masking strategy multi-block and compare it to other possible masking strategics in Section 4 Refer to Appendix for more details on the multiblock implementation.
3.3 Network Parameterization
We Vision Transformer (ViT) (Dosovisskiy et al., 2020 Arnab al., 2021 our video backbone. To process video with transformer network, split. the video clip into 3D grid of L spatio-temporal patches, where patch consists of 16 x 16 pixel block spanning 2 consecutive frames; we refer to these spatio-temporal patches as tokens. This sequence of tokens is then directly processed by the stack of transformer blocks. Inputs x and correspond to masked regions of a video, we apply the video masks by simply dropping a subset of the tokens. We apply masking at the imput of the x-encoder, and at the output of the y-encoder to construct contextualized targets (Baevski et al. 2022b). The encoder is parameterized using standard VIT networks, while the predictor narrow transtormer implemented using 12 blocks with an embedding dimension of 384. Taking inspiration from masked autoencoders (He et al. 2021 our predictor takes input. the sequence of embeddings produced by the z-encoder as well sequence of learnable mask tokens with positional ernbeddings indicating the spalio-temppral positions of the y tokens. The sequence of mask tokens implement the conditioning variable z (see Figure 2 which allows for outputting different representations for different spatio-temporal positions. The output of the predictor is an embedding vector for cach mask token; see Figure 3 and refer to Appendix B for more details.
3.4 Pretraining Data and Evaluation Setup
Pretraining. We combine several public datascts to construct an unsupervised vidco pretraining datasct, which we refer to as VideoMix2M. Specifically, we combine the videos from HowTo110M (HT) (Miech et al., 2019), Kinetics-4066 100040000/000000 (K710) (Kay et al.) 2017), and Something SeweethinnSSSmeeinneg 2017), and remove any overlap with the validation sets of Sinctics-400/6000 /0007700 and Something-Somene Something-2,, resulting in approximately million videos. We train a ViT-L/16, a ViT-H/16, and a ViT-H/ H/1668s transformer model on VideoMix2M. We batch size of 3072 for the VIT-I./1 and VYT-H/16 models, and batch
(SSv2) (Goyal et al,,
5

---

Under review as submission to TMLR
Table 1: Pizels us. Featurized Targets. We ablate the effect of computing the prediction loss in feature space vs pixel space. All models trained VideoMix2M for 90K iterations with batch size of 3072 using the multi-block prediction task. We examine downstream performance using frozen backbone with attentive probing, and report top-1 accuracy using single center view. We also examine eud-lo-end line-tuning performance of the models K400. Predicting in feature space provides a consistent improvement over pixel space prediction.
Frozen Evaluation
Fine- Teeong
K400
SSv2 (6xx1x))
IN1K
K400-ft (16x5x3)
Target
Arch.
(6xx1x1)
Pixels
ViT-1/16
68.6
66.0
73.3
85.4
Features
VIT-L/16
73.7
66.2
74.8
85.6
Table 2: Pretraining Data Distribution. We pretrain all models for 90K iterations using a batch size of 3072, and evaluate downstream performance of the frozen backbones with an attentive probe using single center view. Average performance across tasks increases with the pretraining dataset size.
Prozen Evaluation
K400
SSv2
IN1K
Avg.
Arch.
Data
# #Samples
06x110
K710
700K
75.8
63.2
73.7
70.9
K710+SSv2
900K
72.9
67.4
72.8
71.0
ViT-L/16
K710+HT
1900K
74.5
64.2
74.8
71.1
25% VideoMix2M
500K
72.7
73.9
70.9
VideoMix2M
2000K
73.7
66.2
74.8
71.5
K710+SSv2
900K
75.7
66.8
73.7
72.0
ViT-H/16
25% of VidcoMix2M
500K
73.5
67.6
75.2
72.1
VideoMix2M
2000K
74.0
68.5
75.9
72.8
size of 2400 for the ViT-H/16gsı model. Each model takes as input vidco clip of 16 frames sampled with frame-skip of 4, corresponding to roughly second clips on average. The ViT-L,/16 and ViT-H/16 process the video at a spatial resolution 224, while the ViT-H/163s1 uses an input resolution of 384; cf. Appendix C
Evaluations. Pretrained models are evaluated on downstream video and image tasks. On video tasks, we use a subset of the VideoGLUE benchmark (Yuan et al.] 2023) to test for various capabilities; specifically, we investigate action recognition on Kinetics-400 (K400) (Kay al. 2017). motion classification on Something- Something-v2 (SSv2) (Goyal et al., 2017). and action localization on AVA (Gu et al.] 2018). Action classification on Kinetics evaluates the appearance- based understanding of the model, as many action classes in the dataset be inferred from the presence of specific objects in the video (Sevilla-Late et al. 2021 evaluates the temporal understanding of the model,
Motion classification on action classes in the dataset are decoupled from the appearance appearance/preeeence preeence of specific objects in the video (Goyal iet al. 2017). Finally, action localization on AVA evaluates the ability of the model to understand and localize motions in the video. We follow standard practice and report accuracy on K400 and SSv2 by sampling several spatial and temporal views. For static image tasks, we explore object recognition on ImageNet (Russakovsky et al. 2015), scene classification on Places205 (Zhou et al. 2014), and fine-grained recognition iNaturalist.
2021
Van Horn al. 2018).
4 What Matters for Learning Representations from Video?
this section, we isolate the contributions of several design choices, including: a) the use a feature prediction versus pixel prediction objective, b) the construction of the pretraining data distribution, e) the feature pooling strategy for leveraging the model's representations in downstream tasks, and d) the masking strategy, towards identifying: what to predict from what?

---

Under review as submission to TMLR
Table 3: Average Pooling us, Adoptive Pooling. We pool the feature map output by the frozen V-JEPA encoder using attentive probe, which then fed into linear classifier for downstream supervised tasks (K400 and SSv2). We evaluate two pooling strategies: 1) average pooling (Avg.), and attentive pooling (Att.). Results are reported using a single center view. Using adaptive pooling with cross-altention layer leads to improvements of +17.3 points on K400 and +16.1 points on SSv2.
Frozen Evaluation
K400
SSv2
06xtxb
Method
Arch.
Avg.
Att.
Avg.
Alt.
V-JEPA
VIT-L/16
56.7
73.7
50.1
66.2
4.1 Predicting Representations versus Pixels
We first ablate the effect computing the prediction loss in representation space. We train a pair of ViT-L/16 models using either a V-JEPA feature prediction loss, or a mean-squared error loss with the normalized pixel values, in masked autoencoders (He et al. 2021] and VideoMAE Tong al. (2022). We perform a sweep over the learning rate and weight decay schedules for both approaches. All models are pretrained on VideoMix2M for 90K iterations with batch size of 3072 using multi-block masking. We examine performance on Kinetics-400 (K400), with an allentive probe, and report top-1 accuracy using a single center view. We also examine end-to-end fine-tuning performance of the models on Kinetics-400,
(SSv2), and ImageNet-IIK (INIK), using a frozen backbone
Results of this comparison are reported in Table [ and indicate that predicting in feature space provides consistent performance improvement over pixel space prediction in both frozen evaluation of the video backbone, well as end-to-end fine-tuning.
4.2 Pretraining Data Distribution
Next we study the impact of the pretraining data distribution in Table Leveraging large scale datasets has been critical for enabling the surge of advancements other modalities, such as text and images (Kaplan et al. 2020: Cherti et al., 2023). We investigate whether similar trend holds for video data. To control for the possible confounding variable of compute budget, we pretrain all models in Table for 90K iterations using a batch-size of 3072. We report downstream results on K400, SSv2, and INIK using frozen backbone with an attentive probe, and report top-1 accuracy using a single center view.
Table _ shows that average performance across tasks monotonically increases as we increase the size of the pretraining dataset, but the best. lask-specific performance is oblained by independently selecting the pretraining data for each specific downstream task. For instance, the L/16 obtains its best SSv2 performance when pretrained on K710+SSv2, its best K400 performance when pretrained only K710, and its bost INIK performance when pretrained only on K710+HT. The best average performance across all tasks is achieved by pretraining VideoMix2M, which combines all the data sources. Similarly, the H/16 pretrained on K710+SSv2 achieves a greater K400 score than the H/16 pretrained on VideoMix2M, however, the top performing H/16 on average is pretrained on VideoMix2M.
4.3 Evaluation: Attentive Probing
Next explore the feature pooling strategy for applying the model's representations in downstream tasks. Specifically, when evaluating the frozen pretrained backbone on downstream tasks, we learn a cross-attention layer with a learnable query token. The output of the cross-allention layer is then added back to the query token (residual connection), and then fed into two-layer MLP with a single GeLU activation, followed by a LayerNorm, and finally a lincar classifier. The attentive pooling strategy similar in spirit to the typical use of [cls] token in previous works to extract a single embedding from transformer representations in both vision (Oquab et al. 2023 Dosovitskiy et al. 2020 and language (Devlin et al. 2018
7

---

Under review as submission to TMLR
no masking
random tube
causal multi-block
multi-block
Figure 4: Mustration of various video masking strategies for V-JEPA pretraining. Random Tube masking corresponds to masking random wamber tubclots in the video clip. Causal Multi-Bloot masking corresponds masking out the last few fromes the video clip, well random of spotio-tent temporal blocks form the first few fromes. Multi corvesponds masking random
Hulti-Block Block masking
matio-tempooy blocks form the entire video clip.
Table 4: Ablating Prediction Task. Models VIT-L/16 networks pretrained on K710 and SSv2 and evaluated with an attentive probe using a single center view. 'he region æ is sampled by masking spatio-temporal regions in the video; y the mask complement. 1) random-tube(r): extended across the entire temporal duration) from the video, 2) causal multi-block(pl: x is restricted to the first frames the 16-frame video, which are then masked with random set of temporal blocks, 3) multi-block: r is obtained by masking random set of spatio- temporal blocks from the entire video. Best performance obtained by using multiblock masking.
obtained by masking fraction of tubes (spatial patches
Frozen Analuation
K400
SSv2
IN1K
Masking
(16xix1)
(lxxix1)
random-tube [0.9]
51.5
46.4
55.6
causal multi-blosn [6]
61.3
49.8
66.9
causal multi-block [12]
71.9
63.6
72,2
multi-block
72.9
67.4
72.8
In Table 3 that using adaptive pooling with learnable cross-attention layer leads to significant improvement of +17 points on K400 and +16.1 points SSv2. Using attentive-probe is also beneficial for other baseline models as reported in Appendix
4.4 Prediction Task: Predicting from
We conduct ablation on the masking strategy used in V-JEPA pretraining. We examine the following masking strategies: random-tube [x] in which is obtained by removing random fraction r of tubes (spatial patches extended across the entire temporal duration) from the video, causal multi-block [p] in which v is restricted to the first frames of the 16-frame video, which are then masked with a random set of spatio-temporal blocks, and multi-block in which x obtained by masking random set of spatio-temporal blocks from the entire video. Spatio-temporal blocks are sampled using the parameters described Section 4.4 an ablation on the size and quantity of masked spatio-temporal blocks is provided Appendix EX3 All ablations are constructed by sampling two sets of masks for each input video clip. Figure illustrates the conceptual differences between the various masking strategies.
Table indicates that the best results are obtained by sampling æ using multi-bbock stratogy, wherein the network is forced lo make predictions after removing large continuous blocks in the video. When æ is only sampled from the first few frames of the video, as in the causal multi-bloo strategy, we observe a decrease in downstream performances. Finally, the random-tube stratogy, wherein 90% of the tubes in the video are randomly masked, leads features of low-semantic quality when combined with our feature prediction objective.
8

---

Under review as submission to TMLR
Table 5: Comparison with Pizel Prediction Methods. We compare V-JEPA with OmniMAE (Grdhaar et al, 2023) VideoMAE (Tong al. 2022]. and Hiera (Ryali et al. 2023) which leverage pixestrrconntttcttinn loss. All models trained using a ViT-L architecture or a comparable Hiera-L. We evaluate the approaches on downstream image tasks (INIK, Places205, iNa1201) and video lasks (K400, SSv2, AVA) in both frozen evaluation (with frozen backbone), and end-to-end fine-tuning. All models are evaluated at resolution 224. On K400 and SSv2 we follow the standard practice of reporting accuracy from several spatial and temporal views from the video. In frozen evaluation, V-JEPA outperforms the baselines on all downstream tasks, except ImageNet, where the model achieves 74.8% compared to 75.1% of an OmniMAE model trained directly on ImageNet. V-JEPA also achieves the best fine- aun-tuning performance amongst all ViT-L models and matches the Hiera-L on SSv2. The V-JEPA results are achieved while processing significantly fewer examples during pretraining.
Prozen Evaluation w/ Att. Pooling
Fine- Tunkng
# Sammless Seen
K400 (16x8x3)
SSv2 (16x2x3)
AVA
IN1K
Places205
iNat21
K400-ft (16x5x3)
SSv2-ft
Method
Arch.
Iter.
(16x2x3)
Methods pretrained using pizel prediction
OmaiMAE
WT-L/16
2400M
1170K
65.6
60.6
14.4
75.1
59.8
66.1
84.0
74.2
VideoMAF
WT-//16
410M
400K
77.8
65.5
21.6
71.1
59.3
64.6
85.4
74.3
Hiera
Hicr-L
770M
1500K
75.5
64.2
15.8
68.9
56.9
87.3
75.1
WT-L/16
270M
75.1
Table 6: Comparison with State-of-the-Ar Models. We compare V-JEPA with state-of-the-art baselines in frozen evaluation with an attentive probe on downstream image tasks (INIK, Place205, iNat21) and video tasks (K400, SSv2, AVA). All models are evaluated at resolution 224, except I-JEPAs12 and V-JEPAas4 which are evaluated respectively at resolution and 384. On K400 and SSv2 we follow standard practice of reporting accuracy from several spatial and temporal views from the video. Compared to other video baselines, V-JEPA exhibits consistent improvement across downstream tasks. Compared to image-models that excel under the frozen evaluation, V-JEPA shows a significant performance improvement tasks requiring motion understanding (+21 points on SSv2), and reduces the gap between video and image models on tasks requiring static appearance- based features.
Video Tasks
Image Tasks
K400
SSv2
AVA
TNIK
Places208
INat21
Method Arch.
Params.
Data
Methoda pretrained Images
66.5
OpenCLIP
VIEG//4
1800M
LAION
81.8
34,8
23.2
70.2
83.6
DINOv2
VIT-g/14
1100M
LVD-142M
83.4
30.6
24.3
86.2
68.4
88.8
Methoda pretrained on Videos
MVD
VIT-L/16
200M
INIK-K40X K400
79.4
66.5
19.7
73.3
59.4
65.7
OmniMAE
VIT-H/16
630M
INIK
71.4
65.4
76.3
60.6
72.4
VIT-
20.7
59.1
VidcoMABv2
VIT-g/14
1100M
Un Hybrid
71.2
61.2
12.9
71.4
60.6
68.3
Hiem
Hiern-H
670M
K400
77.0
64.7
17.5
71.4
59,5
61.7
VIT-L/16
200M
80.8
69.5
25.6
74.8
60.3
67.8
VJEPA
1V16
630M
MideeMMx.2M
82.0
71.4
25.8
75.9
61.7
67.9
Vil-H/162s4
62.8
5 Comparison with Prior Work
Section 5.1 we investigate the impact feature prediction by comparing V-JEPA with video approaches that rely on pixel prediction, while using a similar architecture for all baselines. Subsequently, in Section 5.2 remove the architectural constraint and report the best performance across architectures for self-supervised vidco and image pretraining approaches. Finally, explore the label-officinccy of V-JEPA relative to other self-supervised video pretraining approaches in Section 5.3
We further detail the evaluation setup in
Appendix D
5.1 Comparison with Pixel Prediction
investigate the effectiveness of feature prediction pretraining, we first compare V-JEPA vidco masked modeling models relying on pixel prediction loss. We control for the possible confounding factor of model architecture by evaluating all models using either ViT-I./16 encoder, Hiera-L encoder, which has
9

---

Under review as submission to TMLR
Something Soaeeiin---eeiniggg
End-to-Ead Pine-Tuning
Something-Somee Saaiing-SSmmehing--
Evaluation
V-TEPA
Hieta Illaa
V-JEPA
25
Yideo Festurs Frod.
Video Festur Phod.
Valoo
70
1ideo Phod.
VideoMAE
74.5
VidroMAE
OmaiMAL VIrDIs
VideoMAEvz
74
60
1024
10
10
1012
1024
50
100
150
200
250
300
350
Samples Seen (M)
Pretraining Time (Hrs.)
(a) SSu2 fine-tuning us. Samples Seen.
(b) SSu2 frozen- evaluaaion us. Pretraining Time.
Figure 5: Efficiency of pretraining. (a) We report SSv2 fine-tuning for V-JEPA and rixestreeonstruction baselines using VIT-L/16 or Hiera-L architecture. V-JEPA outperforms all pixelrreconttruution methods using ViT-L/16 and matches the Hiera-L performance while seeing significantly less samples during pretraining. for all methods are measured and VideoMAEv2, and linearly extrapolated assuming a global batch size of 2400 samples. However, note that the SSv2 accuracies of video pixel prediction methods actually obtained with small batch sizes and significantly longer training schedules. V-JEPA outperforms pixel-reconstruction methods while training significantly faster.
(b) Wallclock times single GPU with batch size of 10 clips, using the official codebases for VideoMAE
similar number of parameters. For the pixel prodiction basclines we consider VidcoMAE (Tong et al. 2022 Wang et al. 2023a), which trains vision transformer autoencoders exchusively on video, Hiera (Ryali et al, 2023). which trains a hierarchical transformer autoencoder on video, and OmniMAE (Girdhar et al. 2023 which trains a vision transformer autoencoder on static images and video simultaneously.
Table 5 examiness both frozen evaluation with an attentive probe on downstream video and image tasks, well as end-to-end fine-tuning. In frozen evaluation, V-JEPA outperforms the baselines on all downstream tasks, except ImageNet, where we achieve 74.8% compared to 75.1% of an OmniMAE model trained directly on TmageNet;; hence, V-JEPA achieves comparable ImageNet performance despite only pretraining on video.
Under the fine-tuning protocol, V-JEPA also achieves the best performance of any model trained with a VIT-L/16. and matches the performance the Hicra-L on SSv2, which benefits from hierachical prior (Ryali) et al. 2023). The V-JEPA models achieve this result while processing significantly fewer samples during pretraining (Figure 5a),, demonstrating the efficiency of feature prediction as learning principle.
5.2 Comparison with State-of-the-Art
Next, in Table we inspect how the V-JEPA models pretrained vidco stack up next to the largest statc-of- the-art sell-supervised image and video models when freezing the backbone encoder and training an allentive probe on top. Our image pretrained baselines include OpenCLIP (Cherti et al.) 2023), DINOv2 (Oquab) et al. 2023), and I-JEPA (Assran et al. 2023). The OpenCLIP model is trained with a contrastive image- text alignment objective, DINOv2 and I-JEPA are trained with self-supervision. These models are known to excel their frozen-evaluation performance (Oquab al. (023);; i.e., their ability to produce visual features that can be applied to many downstream tasks simultaneoussl, without end-to-end fine-tuning, and thus provide highly competitive baselines. Our video pretrained baselines include VideoMAE (Tong et al. 2022), OmniMAE (Girdhar et al. 2023), Hiera (Ryali et al. 2023 VideoMAEv2 (Wang et al. 2023a and MVD (Wang et al. 2023b). The OpenCLIP, DINOv2 and VidcoMAEv2 models are parameterized as Giant/ Giant/Gigantia /Gianntic vision transformer architectures containing over 1B parameters trained on large-scale image or video datasets.
Comparison with vidco models. Compared to large-scale vidco basclines, the V-JEPA models outperform all previous models on every downstream video and image tasks by a notable margin (see Table G). Our H/16 model outperforms the largest publicly available VideoMAE, Video.AAVv, OmniMAE, MVD, and Hiera models by at least +5 points in motion understanding
+2 points action recognition (Kinetics-400), +5 points on action detection (AVA), +1 point on object recognition (IeageNet-1K), +2 points in scene recognition (Places205), and +0.2 points on fine-grained recognition
Soanething-Somehhnngg
10

---

Under review as submission to TMLR
Table 7: Finetuning results. We evaluate a V-JEPA model with the finetuning protocol on the K400 and SSv2 datasets using 16 frames per clip and mulli-view fusion (5x3 or 2x3) for inference. The #Samples Seen entry corresponds to the number of video clips processed during pretraining, which is larger than the size of the pretraining dataset for mulli-epoch training. We compare V-JEPA with different video self-supervised learming approaches. We report the VideoMAEv2 results without instruction-turiing for consistency with the other approaches. V-JEPA obtains competitive performance using the finetuning protocol.
K400
SSv2
Method
Arch.
Pretraining Data
#Samples Seen
(16x5x3)
(16x2x3)
VidcoMAEv1
ViT-L/16
K400|SSv2
380M 410M
85.4
74.3
VIT-H/11
K400]SSv2
380M 410M
86.6
74.8
VideoMAEv2
VIT-H/16
Un.Hybrid
1600M
86.9
76.8
VIT-L/I6
K400+INIK
2400M
86.4
76.7
MVD
VIT-H/16
K400+INIK
2400M
87.2
77.3
VIT-L/16
VideoMix2M
270M
85.6
75.1
V-JEPA
VIT-H/16
VideoMix2M
270M
86.6
77.0
(iNaturalist). Morcover, when comparing pretraining wallelock time in Figure 5b we see that V-JEPA achieves this performance with roughly 2x speedup compared to the large pixel prediction models.
Comparison with image models. On tasks that require fine-grained understanding of motion Soonething-Something--v the V-JEPA models provide major improvement (over +21 points) compared to large-scale image basclines, such DINOv2, OpenCLIP, and I-JEPA. Self-supervised pretraining from videos allows to model dynamic concepts that are not easily learned from static image datasets. Similarly, we observe that the V-JEPA models outperform image- based pretraining on action localization.
On Kinotics-400, we find image modcls to perform well; c.g., while DINOv2 (Oquab et al. 2023 proviously reported 78.4% on K400 with a linear probe, we improve the frozen evaluation of the g/14 model to 83.4% by using an attentive probe. In this case, our H/16 model achieves 82.0% top-1 accuracy. It is worth noting that the label for many Kinetics vidcos can be inferred using appearanco-basee basodd cues, without requiring an understanding of motion (Sevilla-Lara et al. 2021
The V-JEPA models narrow the gap with image models on image classification tasks. In particular, V-JEPA achieves a score of 77.4% on ImageNet using a one-layer attentive probe, which can be further improved to 77.9% using a two-layer attentive probe. More generally, we hypothesize that the datasets used to train V-JEPA and other video models are too constrained and lack the visual diversity of the internet-scale pretraining data used by the images models; as such, there is value in focusing future work on building diverse publicly available video datasets.
Finetuning In Table we evaluate V-JEPA by finetuning (separately) on K400 and SSv2. We compare V-JEPA with VideoMARe Av22 (Wang et. al. 2023a VideoMAE (Tong et al. 2022) and MVD (Wang el al. 2023b) using a VIT-L/16 and VIT-H/I6 16 architecture. V-JEPA obtains competitive performance using finetuning protocol. With a VIT-H/16 architecture, V-JEPA outperforms VidcoMAE by +1.2% and VidcoMAEV2 by +0.3% on the SSv2 dataset, while obtaining comparable performance on K400. V-JEPA also obtains performance similar to MVD on the SSv2 dataset. The MVD model achieves the best performance across models on the K400 dataset, and is trained using an additional image dataset (ImageNet1IK) in contrast to the other methods in the table, which only use video data. Additionally MVD requires the processing of significantly more samples during pretraining due to the cost of training the teacher encoder networks in pre-pre-training stage.
5.3 Label-efficiency
We examine the label-efficiency of V-JEPA compared to other self-supervised video models by measuring the ability of the pretrained backbones to adapt to downstream tasks with few labels. Specifically, we investigate the performance of the frozen models on Kinctics-400 and Something-Somethingg of labeled examples from each dataset available for training the attentive probe. We train the probes in several low-shot settings: using either 5% of the train set, 10,, 50%, and take 3 random splits in each
as we vary the percentage
11

---

Under review as submission to TMLR
Table Low-Shot Frozen Evaluation. Comparing V-JEPA to other video models frozen evaluation on Kinetics-400 vary the percentage of labeled examples from each dataset available for training the attentive probe. We train the probes in several low-shot seltings: using either 5% of the train set, 10%, or 50%, and lake random splits in each setting obtain more robust metrics, resulting different evaluation experiments for each model. We report the mean performances and standard deviation using the K400 and SSv2 validation sets. V-JEPA more label-efficient than other models; specifically, decreasing the available number of labeled examples from each class increases the performance between V-JEPA and the baselines.
and Sonnelhing-Something--
Frozen Evaluction
K400 (16x8x3)
SSv2 (1xx1x3)
50% orpples dari
10%
Method
Arch.
C-279 comples dasl
(-58 onglle per dase)
(- sopgle per dasi
Co-pli darq
(-190) dark
MVD
VIT-L/16
0.2
77.2
42.9
49.5
61.0 + 0.2
VideoMAE
VIT-H/16
62.3 0.3
68.5 - 0.2
78.2 + 0.1
41 0.8
48.1
0.2
60.5 + 0.4
VideoMAEv2
VIT-g/14
37.0 0.3
48.8 = 0.4
67.8 0.1
28.0 1:001. 1.0
37.3 0.3
54.0 + 0.3
V-JEPA
ViT-H/165s4
VIT-11/16
68.2 + 0.2
67.0 1 0.2
72.8
72.1
+ 0.2
1 0.1
80.6 + 0.2
80.2 0.2
54.0
$1.9 1 0.3
0.2
59.3 + 0.5
$7.5 1 0.4
67.9 + 0.2
67.3 1 0.2
setting to obtain more robust metrics, resulting in 9 different evaluation experiments for each model. Table reports the mean performances and standard deviation using the K400 and SSv2 validation sets.
g
We find V-JEPA to be more label-efficient than other self-supervised video models: decreasing the available number of labeled examples for training the attentive probe results an increase in the performance gap between V-JEPA and the other models. In particular, the performance of the largest V-JEPA model on K400 drops by 12% to 68.2% top-1 when we reduce the number of labeled examples by factor of 10x (from roughly 287 examples per class to 29 examples per class). By contrast, VideoMA Ev2 drops by 30% to 37.0% top-1, VideoMAE drops by 15.9% to 62.3% top-1, and MVD drops by 14.6% to 62.6% top-1.
Similar observations hold on SSv2. The performance of the largest V-JEPA model on SSv2 drops by 13.9% to 54.0% top-1 when we reduce the number of labeled examples by a factor of 10x (from roughly 440 examples per class 48 examples per class). By contrast, VidcoMAEv2 drops by 26% 28.0% top-1, VidcoMAE drops by 19.1% to 41.4% lop-1, and MVD drops by 18.1% to 42.9% top-1.
6 Evaluating the Predictor
Next, we seek qualitatively inspect the V-JEPA models. Recall that the predictor network V-JEPA predicts the representations of masked spatio-temporal region y from visible region x, given the positional information of the masked regions (see Section 3/. To qualitatively investigate the grounding of the feature- space predictions, we freeze the pretrained encoder and predictor networks and train conditional diffusion decoder to map the V-JEPA predictions to interpretable pixels (Bordes et al. 2021 Notably, the decoder is only fed the representations predicted for the missing regions of the video, and does not have access to the unmasked regions of the video (see Figure Gal.
Given a masked video, we use the V-JEPA pretrained models to predict the representations of the missing regions, and then the decoder to project the representations to pixel space. Figure 6b shows decoder outputs for various random seeds. Qualities that are common across samples represent information that is contained in the predictor representation.
Figure 6b shows that the V-JEPA feature predictions are indeed grounded, and exhibit. spatio-temporal consistency with the unmasked regions of the video. Specifically, the samples in Figure (bb show that the V-JEPA predictor correctly captures positional uncertainty and produces a variety of visual objects at various locations with consistent motion. Some of the samples also demonstrate an understanding of object-permanence,, the visual objects remain consistent after partial occlusion.
12

---

Under review as submission to TMLR
Frozen
x-encoder
predictor
decoder
(a) Visualization Methodology. We train conditional diffusion model to decode the V-JEPA feature-space predictions to interpretable pixels; the pretrained V-JEPA encoder and predictor networks are kept frozen in this process. The decoder is only fed representations predicted the missing regions of the video, and does not have access to the unmasked regions of the video.
minm
mimmn
(b) Visualizations. First Row: Masked videos used imput to the V-JEPA models (a pretrained VIT-H/16 encoder and corresponding predictor network). Other rows: Bounding boxes contain various samples from the decoder overlayed on the original video. V-JEPA is not generative model and the decoder does not have access to the context (first row), so we not expect samples to exactly match input. This experiment qualitatively illustrates what information is encoded and predicted by V-JEPA. In particular, characteristies that are common across samples represent information that encoded in the V-JEPA predictions. V-JEPA generates predictions that are spatially and temporally coherent with unmask region of the video. The predictions also capture consistent motion through time.
Figure 6: Qualitative Analysis. Offine visualizations of the V-JEPA feature-space predictions.
7 Limitations
While the focus of this work on exploring the effectiveness of feature prediction as a stand-alone objective for unsupervised learning for video, in this section, highlight current limitations of our instantiation of this learning objective, V-JEPA. Firstly, V-JEPA performance is sensitive to the masking strategy, as illustrated in Section 4.4 In future work, plan to investigate whether scaling the dataset size and pretraining computation make the approach more robust to this design choice. Secondly, the V-JEPA instantiation relies on vision transformers as video encoders, which limits the ability to process long or high-resolution video, because the computational complexity scales quadratically with the input size. We plan to investigate hicrarchical architectures to enable the processing of larger videos. Finally, the V-JEPA instantiation relics on bootstrapping prevent representation collapse (Grill al. 22200). Despite initial results (Tian al. 2021 understanding the learning mechanism behind this principle remains open question.
13

---

Under review as submission to TMLR
8 Conclusion
In this work, we explored the effectiveness of feature prodiction a stand-alone objective for unsupervised learning from video and introduced V-JEPA, collection of vision models trained solely using self-supervised feature prediction objective. The V-JEPA models demonstrate the ability to solve various downstream image and video tasks without adaption of the model parameters, and outperform previous video representation learning approaches in frozen evaluation on action recognition, spatio-temporal action detection, and image classification tasks. Additionally, we show that pretraining V-JEPA on videos is particularly effective for solving downstream tasks requiring fine-grained motion understanding, while large-scale image models trained on internet scale datasets fall short on such lasks. Finally, we empirically observed that V-JEPA models are label-efficient learners, and exhibit good performance on downstream tasks, even when only few labeled examples are available.
References
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34:24206-24221, 2021.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE international conference on computer vision, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas, Masked siamese networks for label-efficient learning. arXiv preprint ar arXfe:224..0744, 2022.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CV Conference Computer Vision and Pallern Recográtion, pp. 15619-15629 2023.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. arXie preprint orXiv:2212.07525, 2022a.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint or orXic:220..0355) 2022b.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08255 2021.
Pietro Berkes and Laurenz Wiskott, Slow feature analysis yields a rich repertoire of complex cell properties. Journal of vision, 5(6):9-9, 9, 2005.
Florian Bordes, Randall Balestriero, and Pascal Vincent.. High fidelity visualization of what your self-supervised representation knows about. arXiv preprint arXiv:2112.09161, 09164, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervisod learning of visual features by contrasting cluster assignments. arXiv preprinl Xrx:::0006. 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint drXi::21441122/, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. preprint ar Sfl:2002.007000 2020.
14

---

Under review as submission to TMLR
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervieed representation learning. arXiv preprint ArXi::2202. 03026, 2022.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint Xix:21104 0420070, 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ihaarco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pollern Recognition, pp. 1818-28201 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint or arXiv:1810.00 04805, 2018.
Ekin Dogus Cubuk, Barret Zoph, Vijay Mane, Dandelion and aadudddvan,, and Quoc V. Le. Autoaugment: Learning augmentation policies from data, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition al scale. Xiv preprinl ar Xrxiv:2000. 1 11929, 2020.
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neurol information processing systems, 35:35946 35958, 2022.
David J Field. What the goal of sensory coding? Neural computation, 6(4):559 601, 1994.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. Tn Proceedings of the IEFE/CVT Conference on Computer Vision and Pottern Recognition, pp. 6928 6938, 2020.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 13505 13515, 2021.
Rohit Girdhar, Alaaeddin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimac: Single model masked pretraining on images and videos. In Proceedings of the IEEE/ IEEE/CVF Conference on Compuler Vision and Pullern Recognition, pp. 10406 10417, 2023.
Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pp. 4086-4093, 2015.
Raghav Goyal, Samira Ebraaiii Kahou, Vincent Michalski, Joanna Materzynska, Susamne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXie preprint w=Xi::2006.0772 2020.
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vi- jayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE conference on computer vision and pattern vecognition, pp. 6047-6056, 2018.
15

---

Under review as submission to TMLR
Agrim Gupta, Jiajum Wu, Jia Deng, and Li Fei-Fei. Siamese masked autoencoders. arXiv preprint arXiv:2305.142 14344, 2023.
Michael U Gulmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2), 2012.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0, 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-angnnented dense predictive coding for video representation learning. In European conference on computer vision, pp. 312-329. Springer, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXw:2211..06377, 2021.
Geoffrey E. Hinton. Connectionist learning procedures. In Machine learning, pp. 355-610. Elsevier, 1989.
Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran. Flavr: Flow-agnostic video representa- tions for fast frame interpolation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2071-2082, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar (rXii:2001.08300 2020.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint Xrx: 1705. 00950, 2017.
Christoph Kayser, Wolfgang Einhäuser, Olaf Dümmer, Peter König, and Konrad Körding. Extracting slow subspaces from natural vidcos leads to complex cells. In Artificial Neural Networks-ICA. 2001: International Conference Vienna, Austria, August 21 25, 2001 Proceedings 11, pp. 1075 1080. Springer, 2001.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. 2016.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. 2017.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. 2022.
Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. In Proceedings of the TEEE international conference on computer vision, pp. 667-676, 2017.
Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint ar orXiv:2201.000 1.0466, 2022.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXio preprint ar 1 711 011001 2017.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrae, Makarand Tapaswi, Tvan Laptev, and Josef Sivie. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2630-2640, 2019.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual represontations by solving jigsaw puzzles.
In European conference computer vision, pp. 69 84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. aix preprint 111118070777 007.03748 2018.
16

---

Under review as submission to TMLR
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Nikhil Parthasarathy, SM Fslami, João Carreira, and Olivier J Hénaff. Self-supervised video pretraining yields strong image representations. arXiv preprint wrXiv:2210.06430 2022.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2336,2544, 2016.
Silvia Pintea, Jan van Gemert, and Arnold WM Smeulders. Déja vu: Motion prediction static images. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part III 19, pp. 172-187. Springer, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional intorpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79 87, 1999.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211122 12211-2222--1 2015.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hiera: A hierarchical vision transformer without the bells-and-whistlls.. arxiv preprinl arXic:2306. 00989, 2023.
Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Ouly time can tell: Discovering temporal data for temporal modeling. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp.
2021.
Elizabeth S Spelke, Peter Vishton, and Claes Von Hofsten. Object perception, object-directed action, and physical knowledge in infancy. 1995.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 843 852. PMLR, 2015.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7464-7473, 2019.
Didac Surís, Ruoshi Liu, and Carl Vondrick. Learning the predictability of the future. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12607-12617, 2021.
Reuben Tan, Matthias De Lange, Michael Iuzzolino, Bryan A Plummer, Kate Sacnko, Karl Ridgeway, and Lorenzo Torresani. Multiscale video pretraining for long-term activity forecasting, arXie preprint 0ll:2307.12844 2023.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weightt- consistency targets improve semi-supervised deep learning results. arXiv preprint arXiw:1703.00 01 780, 2017.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference Machine Learning, pp. 10268-10278. PMLR, 2021.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Vidcomac: Masked autoencoders are data-cfficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 2022.
17

---

Under review as submission to TMLR
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769 8778, 2018.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Antoine Manzagol. Extracting and composing robust features with denoising autocncoders. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pp. 1096 1103, 2008.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research, 11(1)), 2010.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In Proceedings of the IEEE conference on computer vision and pattern recognation, pp. 98-106, 2016.
Fei Wang, Ping Li, and Arnd Christian Konig. Learning a bi-stochastic data similarity matrix. In 2010 IEEE International Conference on Data Mining, pp. 551-5661 560. IEEE, 2010.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Vidcomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pottern Recognition, pp. 14549 14560, 2023a.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu- Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6312-6322, 6322, 2023b.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212. 09191, 2022.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- non-parametric instance discrimination. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 3733 3742, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXin:2111. 11.2211111888, 2021.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10334-10343, 2019.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot vidco-text understanding. Xiv preprinl arXiv:2109. 14044, 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXie preprint MXXi:2205500111 2022.
Liangzhe Yuan, Nitesh Bharadwaj Gundavarapu, Long Zhao, Hao Zhou, Yin Cui, Lu Jiang, Xuan Yang, Menglin Jia, Tobias Weyand, Luke Friedman, et al. Videoglue: Video general understanding evaluation of foundation models. arXiv preprint arXic:2307.0311 003166 2023.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16375-16387, 16387, 2022.
18

---

Under review as submission to TMLR
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, vol- ume 27. Curran Associates, Inc., 2014. URL https://proceeddngs wwwriss..errpp.... httpriisscc/ppppe/211//////
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Ibot: Image bert pre-training with online tokenizer. arXiv preprinl orXiv:2111.0000 2021.
Will Zou, Shenghuo Zhu, Kai Yu, and Andrew Ng. Deep learning of invariant features via simulated fixations in video. Advances in neural in information processing systems, 25, 2012.
A Extended Related Works
We first review approaches for learning visual perception from static images before discussing strategies for learning from video.
Weakly-Supervisse Learning from Static Images
One family of approaches for learning visual perception from static images trains visual encoder to predict the representations of text captions often found accompanying images from the Web, as in CLIP (Radford et al. 2021) or CoCa Yu et al. 2022). The largest open source CLIP model to date, numbering 2B parameters and trained on over 2B web-scraped images (Cherti et al., 2023), demonstrates impressive performance on a wide range of downstream image and video tasks. Notably, this is achieved using only the light- adaptation of task-specific heads, also referred to as frozen-evaluation, and does not require expensive end-to-end fine-tuning of the pretrained model.
Supervised Learning from Static Images
Other approaches for learning from static images leverage unsupervisod objectives. Initial works self- supervised approaches are based on sparse coding or hand-crafted pretext lasks, such as colorization [Larsson] et al.
2016 2017), rotation prediction (Gidaris et al. 2020). and jigsaws (Noroozi & Favaro 2016). More recent
approaches leverage invariance- objectives by training a visual encoder to be invariant to hand-crafted
image transformations
[W et al. 2018 Chen et al. 2020
Another family of mothods learn representations using denoising autoencoders (Vincent et al. 2008); image inpainting is one popular instantiation of this idea (Pathak et al.] 2016). More recently, masked autoen- coders (He et al. 2021) train an encoder-decoder transformer to predict missing pixels of masked image. Follow-up work addresses the indeterminism of pixel reconstruction by exploring instantiations of masked image modeling in latent space (Baevski et al.] 2022b Assran et al. 2023 Baevski et al.) 2022a). These approaches can be seen as applications of the predictive feature principle in the image modality.
There are also various methods that combine both masked image modeling and invariance criteria. learn visual representations from static images, such as iBOT (Zhou et al., 2021) and DINOv2 (Zhou et al. 2021 Oquab al. 2023), the latter is eurrently the most competitive instantiation of self-supervised learning with static images, scaled to model with over 1.1B parameters trained on curated dataset of 142M images.
Teakly-Supperiise Learning from Videos
One family of approaches for learning visual perception from videos relies weakly-supervised guidance from closed captioning, often computed from an ASR transcription of audio data accompanying internet videos. For instance, VideoBERT (Sun et al.) 2019 Xu et al. 2021) trains a video encoder to predict masked spans in the textual closed captions. Similarly, VideoCLIP (Xu et al. 2021) trains a video encoder to predict the representation of video captions computed by text encoder. Follow-up work such as MERLOT (Zellers) ct al.] 2022), VATT (Akbari al., [2021), and InternVideo (Wang et al. 2022) extended VidcoCLIP by incorporating additional unsupervised objectives.
19

---

Under review as submission to TMLR
Self- Sulf-vuee Learning from Videos
Similar to unsupervised learning from images, & lamily of unsupervised video representation learning approaches enforces spatio-temporal representation of video clip to be invariant to hand-crafted spatio- temporal data augmentations (Parthasarathy et al., 2022). However, onc obvious insight is that the temporal ordering of visual information in video can provide implicit supervision. Indeed, this insight is the key insight leveraged by many works on unsupervised video learning. Towards leveraging temporal information as supervision, some approaches train a visual encoder by predicting the temporal ordering of frames (Xu et al, Lee et al. 2017). Other approaches seek to predict low-level motion vectors computed from optical flow Pintea et al. 2014). to predict mixing pixels in video frames, using either frame-interpolation
2019
objective (Kalluri ct al. 2023 or a denoising autocncoder (Tong et al. 2022 Feichtenhofer et al. 2022
Wang
et al. 2023a).
B Extended Description of V-JEPA
In this section, we provide an in-depth description of our approach V-JEPA that is illustrated in Figure B
Input. Unless stated otherwise, during during pretraining, we always randomly sample a clip of 16 frames from each imput video with a temporal stride of 4 between sampled frames. An input video clip therefore covers 64 frames total, or roughly seconds of a given video running at frames per second. We then resize the video's spatial dimensions 224 224, resulting in an overall shape of 16 224 224 x 3 for the entire clip. Since ViT networks process 1D sequence of tokens, we must convert an input video clip into 1D token sequence. To do so, we apply a 3D convolution comprising d filters of size 2 x 16 16 with a temporal stride of 2 and spatial stride of 16, resulting in tensor of shape 8 x 14 x 14 x d. Next we add absolute 3D sin-cos positional embeddings to the spalio-tempora feature map and flatten it, resulting in a ID token sequence of shape 1568 x d. This process demonstrated in Figure
[8 14 14
3D sin-cos absolute position embeddings
16 video frames resolution 224 x 224
3D Conv [2 16 d6
flatten
[66 x 224 224 x
[8 14 14
[1568 x d]
Figure 7: V-JEPA training operates on video clip flattened into sequence of tokens. To convert video clip of size 16 224 224 3 into 1D token sequence, we apply a 3D convolution comprising filters size 2 16 with temporal stride of 2 and spatial stride of 16, resulting in tensor of shape 8 x 14 x 14 d. Next we add absolute 3D sin-cos positional embeddings to the spatio-temporal feature map and flatten it, resulting in a 1D token sequence of shape 1568 d.
V-JEPA. We sample both a video clip, and video mask in each iteration. We denote video clip represented as ID token sequence of length L 1568 by æz (II,..., Similarly, given mask of M < L patches, leaving N = L - M patches unmasked, we denote the indices of masked patches by (i1,..., and its complement (the indices of unmasked patches) by (ii...,,iN)
Computing the z-representations. To compute the V-JEPA loss, we first produce the r-reresentttions by masking the vidco clip and feoding into the æ-encoder; we denote the masked vidco by xN (x) ".,,," Applying the z-encoder Eo(-) the masked clip gives a sequence of patch representations, denoted by Fo(xN) = (hjr.-..,hy h.x).
20

---

Under review as submission to TMLR
Predicting the target. Next, the V-JEPA predictor network Po(:.-) takes as imput the tokens produced by the x-encoder and predicts the missing regions in the video clip, which are specified by a set of learnable mask tokens. Specifically, the mask tokens are parameterized as the sum of a shared learnable vector and an absolute 3D sin-cos positional embedding, denoted by mM = (m......, mss.). The output of the predictor is thus given by, 8M = Po (2,,,ma)) = (8ig:...., Bins); corresponding to a d-dimensional output for each of the M masked patches.
Computing the g-representations. Finally to compute the prediction targets, the entire unmasked video clip is processed by the y-encoder to obtain set of target representations, denoted by aL Es(æ1) ($1,...,8L))
The V-JEPA loss is now computed as
Loss
Σ
||6x - 8kl11,
M
which is simply the average L1 distance between the output of the predictor and the y-encoder. We then compute a gradient update with respect to the parameters of the x-encoder, 8, and the predictor, Φ, and subsequently update the parameters of the y-enodeer as an exponential moving average of the context encoder weights (Polyak average).
Multi-Mask Prediction. To increase the efficiency of V-JEPA, we multi-masien strategy (Caron et al. 2020: Baevski et al. 2022a), which enables us to amortize the cost of the target computation. As mentioned in Section 3 for a given video clip, we sample 2 different masks: short-range and long-range. While we need to forward propagate the x-encoder and predictor separately for each mask, we only need to compute the y-representation once. Specifically; we forward propagate the unmasked video through the y-encoder once to compute the target representations, and forward propagate through the æ-encoder and predictor twice, one time using long-range masks, and second time using short-range masks; since roughly 90% of the video clip is masked, the two forward passes through the r-encoder are reasonably efficient and amortize the cost of target computation.
C Pretraining details
In section, we report V-JEPA pretraining details. Table summarizes the main hyperparameters used during pretraining.
Architectures. We use Vision Transformer (Dosovitskiy et al. 2020 (ViT) architectures for the z-encoder and y-encoder. We train three V-JEPA encoders: a VIT-L/16221 ViT-H/16224 and a ViT-H/1660 All three encoders take input short video clip of 16 frames with temporal stride of 4 between consecutive frames. The subscripts, 224 and 384, indicate the spatial resolution of the video clip. V-JEPA flattens the video clip into a sequence of non-overlapping spatio-temporal patches of size 16 x 16 x 2 (see Figure 7 For all three models, the predictor is designed as a narrow ViT architecture, consisting of 12 transformer blocks with an embedding dimension of 384. For simplicity, we keep the number of self-attention heads in the predictor equal to that of the backbone used for the context-encoder haateee-eeoodde//tgeetenooeeeed ttagget-nnoooer V-JEPA pretrained withou! using a [cls] token.
Optimization. We use AdamW (Loshchilov Hutter 2017) to optimize the x-encoder and predictor weights. The ViT-L/16p 1/-24/162244/ and ViT-H/I6224 models use a batch size of 3072 while the ViT-H//623. uses a batch size of 2400. Models are trained for total of 90,000 iterations. The learning rate is linearly increased from 2 x 1 to 6.25 x 10-4 during the first 12, 000 iterations of pretraining, and decayed to 10-6 following cosine schedule. Weight-docay is also lincarly increased from 0.04 to 0.4 throughout pretraining. The y-encoder weights are initialized icalli to the z-encoder, and subsequently updated as an exponential moving average (EMA) [Tarvainen & Valpola 2017) of the r-encoder weights using a momentum value which starts at 0.998 and lincarly increased to 1.0 during training (Caron et al. 2021 Assran et al. 2022). We scale all hyper-parameter schedules 25% beyond the actual training schedule. Specifically, the learning rate schedule, weight-decay schedule, and EMA schedule are computed assuming a training length of 1 112,500
21

---

Under review as submission to TMLR
Table 9: pretraining hyper-parameters for V-JEPA.
Hyper-parameter
VIT-L/16221 VIT-H/1622 ViT-H/16631
data
datasets
VideoMix2M VideoMix2M VideoMix2M
resolution
224
224
384
num frames
16
16
16
temporal_stt stride horizontal_ flip
4
4
4
true
true
true
random Bestoo__rssiee_____ resize__scaa scale
(0.3, 1.0) (0.75, 1.35)
(0.3, 1.0)
(0.3, 1.0)
random resize _ssize__sppett_____ee_eee ratio
(0.75, 1.35)
(0.75, 1.35)
masking
block
block_aspect aspect ratio
(0.75, 1.5)
(0.75, 1.5)
(0.75, 1.5)
shortrange mask mum blocks
8
8
8
shortrange mask spatial scale
0.15
0.15
0.15
longrange mask num blocks
2
2
2
longrange mask spatial scale
0.7
0.7
0.7
oplimiiaation batch size total number of iterations 90000
3072
3072
2400
90000
90000
warmup iterations i
12000
12000
12000
Ir start Ir final Ir start momentum final momemtumn start start_weighht decay final final_weight weight decay scheduler scale factor
6.25e-4
6.25x10-4
6.25x10
2x 2x10
2x10-4
2x10-4
1x10
1x10-1
1x10-1
0.998
0.998
0.998
1.0
1.0
1.0
0.04
0.04
0.04
0.4
0.4
0.4
1.25
1.25
1.25
architecture
patch_ size
16
16
16
tubelet size
2
2
2
pred depth
12
12
12
pred embed dim
384
384
381
hardware
dtype
bfloat16
bfloat16
bfloat16
accelerator
A100 80G
A100 80G
A100 80G
22

---

Under review as submission to TMLR
Table 10: Frozen Evaluation hyper-parameters.
Hyper-parameter
K400
SSv2
INIK
Place205
iNat21
data num clips
N.A.
N.A.
N.A.
num frames
16
16
N.A.
N.A.
N.A.
temporal stride
4
4
N.A.
N.A.
N.A.
horizontal_ flip
true (0.08, 1.0)
true (0.08, 1.0)
true (0.08, 1.0)
true
true
random. Baddom_resize___e scale
(0.08, 1.0)
(0.08, 1.0)
random resize _esize__spectt.. ratio
(0.75, 1.33)
(0.75, 1.33)
(0.75, 1.33)
(0.75, 1.33)
(0.75, 1.33)
auto aoo_uuetentttt
false
false
true
true
true
optimization batch size epochs
256
256
1024
1024
1024
20
20
20
20
20
le-3
1e-3
1e-3
1e-3
1e-3
final Ir weight decay
0
0
0
0
0
0.01
0.01
0.01
0.01
0.01
iterations, although we only train our model for 90,000 iterations. We found the last 25% of the default schoduler period to update parrapppramotors too aggressively, and simply truncating the schodulers improved performance. To find the optimal values for the learning rale (scheduled from x 10 to v), weight decay (scheduled from v 0.4), and momentum (scheduled from to 1.0), we performed grid search with the following values, learning rate: 3e le 3, 3, 7.5e 4, 6.25e 4, 5e 4, 3.75e 4, 2e 4, weight decay: 0.04, 0.004, 0.0004, 0.00004, momentum: 0.99, 0.995, 0.996, 0.997, 0.998, 0.999.
Masking. As described in Section 3 we propose 3D Multi-Block masking strategy. We use two types of masks: short-range masks, where take the union of 8 randomly sampled target blocks with spatial scale of 0.15, and long-range masks, where take the union of 2 randomly sampled target blocks with spatial scale of 0.7. In both cases, the aspect ratio for all sampled blocks is randomly chosen in the range (0.75, 1.5).
D Evaluation details
D.1 Frozen classification
Attentive Probing. Given an input video, FL, the V-JEPA target encoder E.(.) outputs a sequence of L Lokens, (x(x1.) = ( 81), where 8; E Rd. To pool this sequence of tokens into a single feat ure vector, we apply a lightweight non-linear cross-attention block which replaces the self-attention operation of transformer block with cross attention. Specifically, cross-attention performs the following computation:
exp(g ( Wys;,
Σ E, exp(g WkS))
where Wk. W. Rdxd are the key and value matrices, and q E Rd learnable query token. The output of the cross-attention is then added back to the query token (residual connection), and then fed into two-layer MLP with a single GeLU activation, followed by LayerNorm, and finally linear classifier. The parameters of the cross-attention block are jointly learned with that of the linear classifier for the rowm task, while the encoder parameters are kept frozen. Note that in practice, we actually use an attentive probe with 12 heads, each of dimension 12. In Appendix we show that basclines benofit from the attentive probing protocol.
For all the lasks, we use AdamW optimizer with a cosine scheduler (no warmup) that decays the learning rate from 0.001 to O. We use a fixed weight-decay 0.01 and apply simple data augmentations (random resized crops and horizontal flips) during training of the attentive probe, except on image tasks, where we apply AutoAngment (Dogus Cubuk et al., 2019). Table 10 reports the hyperparameters for each downstream evaluation.
Optimization.
23

---

Under review as submission to TMLR
Table 11: Frozen Detection hyper-parameters.
Hyper-parameter
VIT-I,116
VIT-H/16
out layers
[18, 20, 22, 24]
[26, 28, 30, 32]
batch size
64
64
epochs
30
30
opt
AdamW
AdamW
opt opt_eps epe
0,00000001
0.00000001
momentum
0.9
0.9
weight Beiggh__decayy Ir warmup Ir min Ir warmup_eepop
0.05
0.05
0.0001
0.0001
0.000001
0.000001
0.000001 2
0.000001
2
warmup_sten
1
Extension to multiple clips. Unless stated otherwise, our attentive probe takes clips of 16 frames as input on Kinetics, and 2 clips of 16 frames on Something-Sometiinge to increase the temporal coverage of the video. Specifically, we first divide a video in 8 (or 2) equal-length temporal segments, and sample clip at random per segment. The video encoder Fe processes each clip separately and produces clip-level feature map. The feature maps for each clip then concatenated together and fed to the attentive probe. At lest time, average the prediction of 3 spatial views following standard practice in video classification.
Application of video models to images. To evaluate the video models on image tasks, we simply duplicate input images to generate still video clips of 16 frames. We perform this duplication operation simply for convenience in evaluation of the video models, however we find this step to be unnecessary in general. Given a video tokenizer implemented simply duplicate the image into 2 frame video clip. This would result in the same number of input tokens as that produced by a static image model with 2D-conv tokenizer.
3D-conv with temporal stride of 2, it is sufficient to
Application of image models to videos. To evaluate image models such as DINOv2 and OpenCLIP on video tasks, we simply process each frame independently with the image encoder to produce a frame-level feature map. The feature maps for each frame are then concatenated and fed the attentive probe, just do with the clip-level feature maps when evaluating video models.
D.2 Frozen detection
We evaluate our model on the AVA (Gu al.) 2018) spatio-temporal localization of human actions datasct, containing 211k training and 57k validation video segments. We follow the experimental protocol of (Feicht enhofer al. 2021), J, and precomputed masks from pretrained Faster-RCNN adapted to videos, which backbone and is pretrained on ImageNet and COCO. We train linear classifier
uses NestNett- on top of the frozen V-JEPA features to classify the extracted regions of interest and report mean Average Precision (mAP) on the 60 most. common classes. Hyper-parameters are provided in Table 11 Our frozen features are obtained by concatenating the last layer of the transformer encoder with three intermediate layers. We use a batch size of and pretrain for 30 epochs with AdamW using a learning rate of 0.0001 with 2 epochs of warmup and weight decay of 0.05.
D.3 Finetuning
Following Tong et al. (2022), , we finetune linear layer on top of our model, using layer decay schema and mixup as the data augmentation pipeline. We provide all hyper-parameters for both K400 and SSv2 in
Table 12
24

---

Under review as submission to TMLR
Table 12: Finetuning Evaluation hyper-parameters.
Hyper-paramcter
K400
SSv2
data
num segments
frames
16
sampling rate
resolution
224
model
model name
VIT-L/10 16 ViT-H/16
VTT-L/16
ViT-H/10 16
drop_path
0.1
0.2
0.2
0.2
head drop rale
0.
0.
0.5
0.5
optimization
batch size epochs
256 35
1024
256
256
25
15
15
opt
adamm
opt eps momentum
0.00000001
0.9
deeeyyy__ccyy_ Ir layer decay warmup ir min Ir warmup epochs
0.05
0.002
0.0005
0.0005
0.0005
0.75
0.75
0.75
0.75
1e-6
1e-8
1e-6
1e-6
1e-6
1e-5
1.5e-4
1.5e-3
augmentations color jitter horizontal flip num sample
0.4
True
True
False
False
nand-m77nttmstdd05---
smoothing train_ interpolation
0.1
bicubic
mum segment
5
5
2
2
test mum crop
3
3
3
3
erase
prob
0.25 pixel
mode
count
split
False
mirup
mixup
0.8
cutmix
1.0
mixup prob
1.0
mixup switch prob
0.5
mixup mode
batch
25

---

Under review as submission to TMLR
E Extra Results
E.1 Frozen Evaluation.
Table 13: Linear vs. Attentive Probe Evaluation for V-JEPA and VideoMAE. We evaluate the effect of
linear (Lin.) and attentive (Att.) probing when adapting V-JEPA to the K400 (16 5 x 3) and SSv2 (16 x 2) tasks. V-JEPA and VideoMAE benefit from using non-linear attentive probe.
K400
SSv2
Method
Arch.
Lin.
Att.
Lin.
Att.
VideoMAE
VIT-L/16
52.5
77.8
41.3
61.2
V-JEPA
VIT-L/16
56.7
80.8
50.1
69.5
Table 14: Linear vs. Attentive Probe Evaluation for DINOv2 and OpenCLIP. We evaluate the effect of linear (Lin.) and attentive probing (Att.) when adapting DINOv2 and OpenCLII.. Image-baselines benefit from using an attentive probing strategy. Results shown in gray are reported from the linear probe evaluation in Oquab et al.
22023
K400
SSv2
IN1K
Place205
iNat21
Method
Arch.
Lin.
Att.
Lin.
Att.
Lin.
Att.
Lin.
Att.
Lin.
Att.
DINOv2
ViT-g/14
78.4
83.4
38.3
50.0
86.5
86.2
67.5
68.4
85.7
88.8
OpenCLIP
ViT-G/14
78.3
81.8
35.8
34.8
86.2
85.3
69.8
70.2
76.0
83.6
Linear vs. Attentive probe attentive probe and multiple clips on the K400 and SSv2 downstream tasks. Additionally, Table 14 shows that attentive probing leads to better performance on average for DINOv2 and OpenCTIP models. Since attentive probing and multiclips eval improves the performance of all models, we use it as our default protocol in frozen evaluation.
Table 13 shows that V-JEPA and VideoMAE benefit from using non-linear
Table 15: Temporal Coverage on Kinetics-400. We evaluate the effect of temporal coverage on K400. We train an attentive probe on K400 using either clip (= seconds of video) clips (=s seconds of video). To sample N clips, we first divide a video in N equal-length lemporal segments and sample one clip at random per segment. The video encoder processes each clip parallel and all the encoder output tokens are concatenated at the input of the attentive probe. Increasing the temporal coverage from clip per video to 8 clips significantly improves the performance for both our VideoMAE baseline and V-JEPA.
Method Arch.
1 Clip
8 Clips
VideoMAE
VIT-L/16
69.4
77.8
V-JEPA
VIT-L/16
73.7
80.9
One Clip vs Multiple clips. We examine the impact of changing the temporal coverage of a model during downstream evaluation on K400 action classification. In Table [5] we evaluate VideoMAE and V-JEPA models using an attentive probe with access to either the feature map of 1 clip randomly sampled from the video, or the concatenatod feature map of clips randomly sampled from the vidco. To sample 8 clips from a video, we first divide the video into equal length temporal segments, and sample 1 clip random from each segment. A single clip corresponds to xe 2 seconds of video average, while 8 clips correspond to 16 seconds. The video encoders processes cach clip scparately to produce a clip-level feature map, which are then concatenated at the input to the attentive probe.
Increasing the temporal coverage from clip per video to clips improves the performance both V-JEPA and VideoMAE on K400 action classification. We therefore use the multiclip attentive probing setup as our default evaluation pipeline.
E.2 Sample Efficiency of pretraining
We compare the sample efficiency of pretraining various state-of tho-arrtth--tttt image and video models. Specifically, look at the number of samples (image video clips) processed by the network during pretraining, which
26

---

Under review as submission to TMLR
Ablating Number Masks Sample
Ablating Number of Blocks per Mask
Ablating Masking Ratio
Tomporal Musking Raiie
Kinetics 400
100%
Kinetics
Kinetics
Number of Masks per Samples
Number of Blacks per Mask
Spotial Masking Ratio
(a)
(b)
(c)
Figure 8: Masking Strategy Ablation. Evaluating a linear probe on a ViT-B/16 pretrained with V-JEPA on K400 under various 3D Multi-Block masking settings. examine the impact of (a) sampling several masks per video, (b) varying the number of blocks in a mask, and (c) varying the average spatial and temporal masking ratio. A temporal masking ratio of 100% extends the spatial mask across all the frames the clip. We find important to maintain a high spatial and temporal masking ratio during pretraining.
is larger than the size of the pretraining dataset for multi-epoch training. Notably, our results with V-JEPA are obtained while processing an order of magnitude fewer samples than previous methods, and notably two orders of magnitude fewer samples than OpenCIIP. We believe that further investment towards improving the video pretraining data distribution could lead substantial gains in downstream image and video tasks.
Table 16: Sample efficiency. We compare the sample efficiency of pretraining various state-of-the-ar image and video models. The #Samples Seen entry corresponds to the number of samples (image or video clips) processed by the network during pretraining, which is larger than the size of the pretraining dataset for multi-epoch training. The V-JEPA results in this paper are obtained while processing an order of magnitude fewer samples than previous methods.
Method
Arch.
Data
#Samples Seen
OpenCIIP
ViT--/14
LAIÓN-2B
39000
DINOv2
ViT-g/14
LVD 142M
1900M
VideoMAEv2
1/14
UnlabeledHybrid
1600M
V-JEPA
ViT-H/166s4
VideoMix2M
210M
E.3 Masking Strategy
An important component of the V-JEPA pretraining strategy is the 3D clip masking strategy. In this section, we detail 26 ablation experiments exploring different masks. For all the experiments, we pretrain a ViT-B/16 1 pretrained on K400. Figure X presents a summary of those results.
Figure 8c shows the effect of changing the spatial and temporal masking ratio. Figure Sb ablates the number of sampled blocks used to construct the masks given a fixed effective masking ratio of 90%. Finally, in Figure Sa we examine our multi-masking strategy and find that sampling two masks for each clip (long-range and short-range) to be more effective than sampling just single mask for each clip.
ratio of the area that covered by a mask on average for clip. Recall that each mask is constructed by sampling several (possibly overlapping) blocks and taking their union. We change the average spatial or temporal masking ratio by changing a block spatial or temporal size, as well as the overall number of blocks. We found that low spatial temporal coverage results in trivial prediction task, which degrades downstream performance. Based on those results, we sample masks that remove roughly 90% of the frame and extend along the entire temporal dimension of the clip by default.
In Figure Sc we explore different average spatial and temporal masking ratio, i.c. the
In Figure 8b we explore different block size given an effective spatial masking ratio of 90% and temporal ratio of 100%. We keep the masking ratio approximately constant by changing the block size and the number of block the same time. We find that sampling several blocks to perform better than sampling a single large block. Figure g visually illustrates the effect of sampling several smaller blocks to construct mask.
27

---

Under review as submission to TMLR
(a) Num. Blocks: 8, Spatial Block Size: 32 x 32
(b) Num. Blocks: 4, Spatial Block Size: 80 x 80
(c) Num. Blocks: Spatial Block Size: 160 160
Figure 9: Illustration of mask with number blocks and block size. Each mask constructed by sampling several (possibly overlapping) blocks and taking their union.
In Figure Sa we explore the effect of sampling various number of masks per sample. We find that sampling two masks for each clip, with different spatial block sizes for cach, to be more effective than sampling just a single mask. We hypothesize that this masking strategy induces complementary tasks. In our experiment, we use this as our default masks sampling.
F Predictor Visualization Details
To visualize V-JEPA predictions in pixel space, we the RCDM framework Bordes al. 20021) Given input pair (x, train generative diffusion model to reconstruct y from (& given x in representation space, and noisy version of f y + e, where is an additive noise vector.
prediction of y
Specifically, we train a decoder network D. to minimize the loss function DD, Pa(Es elli, where æ is a masked video and is a sequence of 16 frames at resolution 64 x 64 corresponding to the masked region. The parameters of the V-JEPA predictor Ps and encoder Eg are kept frozen when training the decoder. We use a V-JEPA VIT-H/ 16224 model.
Du is parameterized 3D- Unet composed by 4 downsampling blocks with 3 residual layers per downsampling block. The base channel dimension of D., is set to 192. We train our decoder for 450,000 iterations. All other hyperparameterss (architecture, optimization, noise schedule...) are identical to the one used in RCDM.
After training the decoder, one can subsequently feed the prediction vector of unseen test videos into the decoder along with various random noise vectors to generate several pixel-level visualizations of the predictions.
we drap the mask tokens the notation for simplicity.
28


Reviews:
[{"review_id": "mXwiQQBHBQ", "reviewer": "Reviewer_qEpG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strength**:\n1. The first paper to explore joint-embedding predictive architecture in video domain, it is a well-studied extension to I-JEPA [Assran et al, 2023). It studies different masking and probing strategies that were not used in the image domain and achieves good results on a wide range of image and video tasks.\n2. Compared to other self-supervised methods, the proposed V-JEPA has the  multiple advantages: a) Its training schedule is  relatively shorter. b) it achieves competitive results by attentive probing without the need to fine-tune the whole model, which saves the compute when transfer to downstream tasks. c) It achieves much better results on down-stream tasks in a low data-regime \n3. The method is clearly introduced and the findings are supported by thorough experiments and comparisons.\n\n### **Weakness**:\n1. It is not well discussed that why V-JEPA has to use attentive-probing to achieve good results on classification while  I-JEPA has good performance on classification just by linear probing.\n2. When the JEPA method is applied to video, apart from the tube masking, there seems to be very little adaptation to the method which considers or makes use of features of videos. Hence, it is hard to know whether the method is able to solve problems poses only in videos, for instance, motion estimation, object tracking etc. For example, I-JEPA and V-JEPA achieves similar results on Kinetics, and its performance on AVA is not much better than DinoV2.  The good performance on SSV2 seems to indicate its better motion modeling ability. However, it would be good to support it by evaluating on  a) classification on more challenging datasets (e.g., EpicKitchens, Ego4D), b) more other non-semantic tasks like tracking or spatio-temporal localization. \n3. It is not clear how much the model will benefit from more data. In Table 2, the average improvement by scaling the data from 700K to 2M is only 0.6.  \n4. There are some comparisons missing or experiments details need to be clarified -- see the ‘requested changes’ section for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the effectiveness of joint-embedding predictive architecture (JEPA) in self-supervised learning on videos. By evaluating the model on both image and video tasks, the authors demonstrate that the model trained with feature prediction is better in both compute efficiency and label efficiency when transfer to downstream tasks — V-JEPA model achieves competitive results by using attentive probing while pixel reconstruction models (e.g., MAE) needs fine-tuning. In addition, the fewer supervised labels used in transfer learning, the larger the gap is between V-JEPA and other pixel reconstruction models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Section 3.2**: It is unclear that how long-range and short-range mask are combined.  Do you take the union of the two mask to be applied to a single sample, or do you apply them separately to a duplicated sample and keep both of them in the same batch? Please specify it more explicitly. \n2. **Section 6**: *“We freeze the pretrained encoder and predictor networks and train a conditional diffusion decoder to map the V-JEPA predictions to interpretable pixels”*, please cite the conditional diffusion model you used, and add the implementation details to the Appendix.\n3. **Table 5 and Table 6**: Models are trained on different number of data. It will be good to add another line for V-JEPA trained on K400 only to have a fair comparison with models like Hiera and VideoMAE. \n4. **Table 5**: It’d be more complete to add VideoMAEv2 into the table as a stronger competitor than VideoMAE. The VideoMAEv2 fine-tuning results are only shown in Table 15, better to have it to Table 5 as well.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w5gCbtUI6G", "reviewer": "Reviewer_bA7B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- **S1.** The paper is overall clearly written and well explained. The authors do a good job motivating their approach and putting it in the context of prior work.\n- **S2.** The proposed approach seems to work well. The learned representations outperform or are competitive with prior work, and the \"attentive probing\" evaluation protocol also boosts performance (of their method and also baselines).\n- **S3.** The authors study a number of interesting questions through their experiments and ablations, such as the performance of representation prediction v.s. pixel prediction, and the label efficiency of different learned representations.\n- **S4.** All of the claims of the paper are precisely stated and seem well-supported by the evidence.\n\nWeaknesses:\n\n- **W1.** A few aspects of the motivation for attentive probing were difficult to understand, and the relationship between attentive probing and previous representation learning techniques could be better explained.\n- **W2.** Some details of the masking strategy and conditioning signals could be explained more precisely.\n- **W3.** The results in Table 2 and Section 4.2 seem to vary both dataset size and dataset composition together, which makes it hard to disentangle their effects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a method for self-supervised representation learning via a combination of masking and bootstrapped representation prediction. The proposed method involves masking spatial regions from a video, tokenizing the results, processing them with a vision transformer to obtain representations of each token, and then training a predictor to reconstruct the (moving-average) representations of masked-out patches from the kept patches. The authors conduct a wide set of experiments and ablations to understand the performance of their approach, and find that it outperforms previous techniques on motion-understanding tasks and is competitive with previous approaches for content-understanding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Important issues\n\nBefore recommending acceptance, I'd like the authors to address a few clarity issues:\n\n### Attentive probing\nSection 4.3 states that the representations may not be linearly separable because the prediction objective is unnormalized. However, I don't understand why being linearly separable should be related to the normalization of the objective. Linear separability is preserved under rescaling representations by a constant (or, indeed, any invertable linear operation). The authors must mean something else by \"unnormalized\", but I don't follow what it is.\n\nThe authors also state that the reason for using attentive probing instead of averaging is because averaging is a linear operation. But this motivation also doesn't make sense to me. Averaging seems to have more to do with aggregating information than it does with linear separability, so I don't see why the linearity of averaging is relevant.\n\nFinally, the attentive probing strategy proposed here seems somewhat similar to existing methods used for transformer representations, using e.g. a `[class]` (or `[CLS]`) token to extract an embedding (as in the [ViT](https://arxiv.org/abs/2010.11929) and [BERT](https://arxiv.org/abs/1810.04805v2) papers). I think this connection should be discussed more.\n\n### Masking strategy details\n\nI found the description of the masking strategy to be somewhat difficult to follow. I also didn't fully understand the difference between the \"random-tube\" and \"multi-block\" strategies; both of these use multiple \"spatial patches extended across the entire temporal duration\", right? What is the difference between a block and a tube?\n\nI think a figure comparing the different masking strategies in Section 4.4 would be useful.\n\n### What is $z$?\n\nFigure 2 in Section 3 states that there is an additional variable $z$ that \"provides the predictor with information about the transformation that computes $y$ from $x$.\" But I couldn't find a definition of $z$ for the proposed model. What is it?\n\nI notice that the appendix uses $z$ to refer to the encoded representation of $x$, but I don't think that's the same thing, since Figure 2 shows $z$ as a separate conditioning signal. (This notation should probably be modified to be more consistent.)\n\n---\n\n## Other suggestions\n\nI also think the paper could be improved by expanding on the below points, although I do not think these are necessary for acceptance.\n\n### Joint relationship between dataset size and diversity in Table 2 / Section 4.2\nIt seems that Table 2 and Section 4.2 vary the dataset size and composition jointly. The largest dataset is also the most diverse dataset, which includes three data sources. As such, the \"average\" gains could be due to the increased diversity, rather than size alone.\n\nI think it would be interesting to separate how much of the gains come from the diversity of the data, and how much from the size. These could be disentangled by subsampling all of the datasets, e.g. taking 25% or 50% of VideoMix2M, and seeing how that affects performance.\n\n### Dependence of bootstrapping procedure on the initial representation\n\nThe authors justify the V-JEPA bootstrapping procedure (in Section 3.1) by arguing that their objective leads the encoder to capture information about the video in order to minimize the conditional median-absolute-deviation of the masked tokens given the learned encoding.\n\nHowever, it seems that this argument relies on the initial target variable $Y$ capturing important features of the input. If $Y$ is invariant to some important property of the input, the encoder does not seem to have a reason to learn it.\n\nThis indicates that the initialization of the encoder would be important for the method to succeed. I'd appreciate some more discussion on this. How do you initialize the encoder to avoid missing important features of the input?\n\n### L1 loss\n\nI'm curious about the use of L1 regression in the objective. The authors mention stability; was this more stable relative to the L2 norm? (The L1 norm depends on the specific basis vector used, so I also wonder if this leads to any sort of sparsity or feature specialization.)\n\n### Masking before/after the encoder\nThe paper mentions that masking is done at the input when processing $x$, but at the output when processing $y$. It seems like this could be a potential distribution shift for the model, since (if I understand correctly) gradients are only calculated with respect to the embeddings of $x$, never for $y$. Do you observe any issues with distribution shift or length generalization due to feeding in more unmasked tokens?", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HafflKJSiy", "reviewer": "Reviewer_Lsst", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nI think I have already outlined the main strengths of the paper above in the contributions but to summarise: \\\n1.\tThe method is conceptually simple and involves feature prediction, without relying on other sources of supervision or auxiliary objectives. \\\n2.\tThe authors have done a very comprehensive evaluation on various downstream image and video tasks, demonstrating the versatility of the learned representations. \\\n3.\tThey have provided some ablations studies (and a very extensive appendix) \\\n4.\tComparison with state-of-the-art self-supervised video and image representation approaches \\\n5.\tQualitative analysis through visualisation, providing insights into the learned representations. \\\n\nWeaknesses: \\\n1.\tLimited discussion on the potential limitations or failure modes of the proposed approach; more related ablation studies would have helped. To be fair the authors have added a very comprehensive appendix, but I do not think it helps having a massive appendix with key results. \\\n2.\tThe paper primarily focuses on the evaluation of frozen backbones, with limited analysis of end-to-end fine-tuning performance (appendix). \\\n3.\tThe paper is relatively light on elaborating and expanding on hyperparameters on the performance  \n\nSome questions \\\n\n1.\tCould you provide more insights into the potential limitations or failure modes of the V-JEPA approach? \n2.\twhat about the fine tuning results? Wouldn't it more suitable to have them in the main paper? \n3.\tI appreciate that you have added the hyperparameters in the appendix, but similarly to point 2, maybe some of the details could have been included in the main paper. \n4.\tGiven the limited size of the pretraining dataset (VideoMix2M), how do you expect the performance of V-JEPA to scale with larger and more diverse video datasets? in the grand scheme of things, is this dataset large (e.g. akin to how we consider datasets like ImageNet.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary: The paper explores feature prediction as a stand-alone objective for unsupervised learning from video. It introduces V-JEPA (Video Joint-Embedding Predictive Architecture), a collection of vision models trained purely on a feature prediction objective without using pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision (which is one of the most interesting elements of this paper, arguably). The models are trained on 2 million videos collected from public datasets and evaluated on downstream image and video tasks. The results show that learning by predicting video features lead to versatile visual representations that perform well on motion and appearance-based tasks without adapting the model's parameters. \\\nIn terms of the main minor and major contributions: \\\n1.\tThe paper introduces V-JEPA, a relatively simple method based on feature prediction for unsupervised learning from video (largely inspired by Assran's and LeCun's I-JEPA work – to me very promising directions in general) \\\n2.\tThey show the effectiveness of feature prediction as a stand-alone objective for learning visual representations from video without other sources of supervision. \\\n3.\tThey evaluated V-JEPA on various downstream image and video tasks, showing its ability to learn versatile representations that perform well on motion and appearance-based tasks. \\\n4.\tThey run several comparisons between V-JEPA and other self-supervised video representation learning approaches (including Dino) and showing its advantages in terms of performance and efficiency. \\\n5.\tThe authors presented work demonstrating the effectiveness of design choices such as feature prediction vs. pixel prediction, pretraining data distribution, feature pooling strategy, and masking strategy. \\\n6.\tThey provided some qualitative analysis of the learned representations through some visualisations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tPlease include a dedicated section or subsection discussing the potential limitations or failure modes of the proposed approach, and outlining future research directions.\n2.\tPlease consider adding some fine tuning bits and hyperparemeters in the main paper.\n3.\tPlease include an ablation study or discussion on the impact of different architectures and hyperparameters on the performance of V-JEPA (I am not too invested in this, and to some extent some details are included in the appendix)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a good quality paper that can be a good reference for further work in representation learning from video. All reviewers suggest acceptance and are happy with the authors improvements post-review.
Decision: Accept (poster)