Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning
Authors: Shentong Mo
Shengbang Tong
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Joint-Embedding Predictive Architecture
Contrastive Self-supervised Learning
Primary area: machine_vision

Abstract: In recent advancements in unsupervised visual representation learning, the Joint-Embedding Predictive Architecture (JEPA) has emerged as a significant method for extracting visual features from unlabeled imagery through an innovative masking strategy. Despite its success, two primary limitations have been identified: the inefficacy of Exponential Moving Average (EMA) from I-JEPA in preventing entire collapse and the inadequacy of I-JEPA prediction in accurately learning the mean of patch representations. Addressing these challenges, this study introduces a novel framework, namely C-JEPA (Contrastive-JEPA), which integrates the Image-based Joint-Embedding Predictive Architecture with the Variance-Invariance-Covariance Regularization (VICReg) strategy. This integration is designed to effectively learn the variance/covariance for preventing entire collapse and ensuring invariance in the mean of augmented views, thereby overcoming the identified limitations. Through empirical and theoretical evaluations, our work demonstrates that C-JEPA significantly enhances the stability and quality of visual representation learning. When pre-trained on the ImageNet-1K dataset, C-JEPA exhibits rapid and improved convergence in both linear probing and fine-tuning performance metrics.

Connecting Joint-Emmeddee Predictive Architecture with Contrastive Self-supervised Learning
Shentong Mo Shengbang Tong²
1CMU, "YYU
Abstract
In recent advancements in unsupervised visual representation learning, the Joint- Embedding Predictive Architecture (JEPA) has emerged as significant method for extracting visual features from unlabeled imagery through an innovative masking strategy. Despite its success, two primary limitations have been identified: the inefficacy of Exponential Moving Average (EMA) from 1-JEPA in preventing entire collapse and the inadequacy of 1-JEPA prediction accurately learning the mean of patch representations. Addressing these challenges, this study introduces novel framework, namely C-JEPA (Contrastive-JEPAA which integrates the Image-based Joint-Embedeen Predictive Architecture with the Variance-Invariancee Regularization (VICReg) strategy. This integration is designed to effectively learn the variance/covariance for preventing entire collapse and ensuring invariance in the mean of augmented views, thereby overcoming the identified limitations. Through empirical and theoretical evaluations, our work demonstrates that C-JEPA significantly enhances the stability and quality visual representation learning. When pre-trained on the ImageNet-1 K dataset, C-JEPA exhibits rapid and improved convergence in both linear probing and fine-tuning performance metrics.
Introduction
Unsupervised learning of visual representations has recently seen remarkable progress, primarily due to the development of innovative architectures and strategies that exploit unlabeled imagery. Among these advancements, the Joint-Embeddee
I-JEPA
C-JEPA
VIT-B Linear Prob
Predictive Architecture (JEPA) [1, 2, 3] has distinguished self powerful approach. I-JEPA [2] leverages masking strategy to extract visual features, facilitat- ing significant strides in understanding and utilizing unlabeled visual data.
66
64
62
100
200
300
400
500
600
However, despite its successes, certain limitations within the JEPA framework have become apparent, particularly concerning its components I-JEPA Expo- nential Moving Average (EMA) and I-JEPA predic- tion capabilities. Specifically. 1-JEPA EMA has been found to be inadequate in preventing the issue of entire collapse [4, 5], while the I-JEPA prediction mechanism struggles accurately learn the mean of patch representations. These challenges not only hinder the performance of JEPA but also limit its applicability in broader contexts.
Pretrain Epoch
Figure 1: Our C-JEPA achieves faster and better convergence than I-JEPA.
CCoresponding author; shentongmo@gmail.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To address these limitations, we introduce a novel contrastive self-supervised learning framework based on JEPA, namely C-JEPA, which aims address the aforementioned challenges by incor- porating the principles of aaaneee--naariance-Cviaaaaa
Regularization (VICReg) [6]. VICReg's methodology adept at learning variance and covariance to avert entire collapse and ensure invari- ance for the mean of augmented views. By integrating VICReg with the Image-based JEPA, C-JEPA is designed to achieve faster and better convergence, as shown in Figure 1. In this paper, we aim to detail the theoretical underpinnings and empirical validations that substantiate the superionity of C-JEPA over previous self-supervised learning methods.
Our contributions are manifold and significant. Firstly, we identify and articulate the limitations inherent in the I-JEPA framework, specifically its EMA and prediction mechanisms. Secondly, we propose the C-JEPA framework as novel solution that synergizes JEPA with VICReg to address these limitations effectively. Thirdly, through rigorous empirical and theoretical evaluations, we demonstrate that C-JEPA not only mitigates the issues identified but also achieves superior performance metries when compared to existing frameworks. Particularly notable is C-JEPA's performance when pre-trained on the ImageNet-IKK IK dataset, where it shows fast and improved convergence in linear probing and fine-tuning scenarios. These results highlight C-JEPA's potential to set new benchmark in unsupervised visual representation learning, thereby contributing significantly to the field's advancement.
2 Related Work
Self-supervieed Learning. In the self-supervised literature, researchers aim to exploit the internal characteristics of data and leverage pretext tasks to train model. Recently, an unsupervised framework that learns effective views with data augmentation was proposed by Tian et al. [7] to reduce the mutual information between views. CMC [8] introduced multi-view contrastive learning framework with any number of views to learn view-agnostic representations. Another pretext task of solving jigsaw puzzles was developed in PIRL [9] to improve the semantic quality of learned image representations, achieving better object detection results than supervised pre-training. More recently, Masked image modeling (MIM) has been explored in many previous works [10, 11, 12, 13, 14] to reconstruct the masked image patch given the unmasked counterpart as clues. Some MIM approaches [10, 11, 12, 15, 16] designed customized masking strategies (i.e., random, block-wise) as pre-text tasks during pre-training. For example, block-wise masking introduced in BEiT [10] to learn transferrable visual representations by recovering discrete tokens of masked image patches. Given features extracted from the 25% unmasked patches, the seminal work, MAE [12] directly reconstructed missing pixels of 75% masked patches. SimMIM [14] randomly masked the input image with large square patch size (i.e., 32) and used a one-layer prediction layer after the encoder to predict RGB values of raw pixels. Other researchers [15, 16, 17, 18] started to leverage a teacher network like CLIP [19] or adversarial learning to generate the mask and supervision target.
Contrastive Learning. In the past years, contrastive learning has shown its effectiveness in self- supervised learning, where various instance-wise contrastive learning frameworks [20, 21, 22, 23, 24, 25, 26, 27, 4] and prototype-Ieve contrastive methods [28, 29, 30, 31] were proposed. The general idea of the iistance-wise contrastive learning is to close the distance of the embedding of different views from the same instance while pushing embeddings of views from different instances away. One common way is use large batch size to accumulate positive and negative pairs in the same batch. For instance, Chen el al. [20] proposed a simple framework with a learnable nonlinear projection head and large batch size to improve the quality of the pre-trained representations. To make the best use of a large amount of unlabelled data, they present a bigger unsupervised pre-training network and introduce distillation with unlabeled data in SimCLR v2 [21] to improve the performance in downstream tasks. Without involving negative instances, BOYL [22] trains the online network from an augmented view of an image to predict the target network representation of the same image under a different augmented view (positive instance). Another broadly-use approach [23] in the ICL literature is to apply a momentum encoder to update negative instances from a large and consistent dictionary on the fly. The dynamic dictionary was used with
encoder in MoCo
series [24, 23] to build a dynamic dictionary to update negative instances in a queue of considerable Invariance-Covariii
size. The to address shortcomings in self-supervised learning by enforcing stability through variance and covariance constraints.
(VICReg) [6] regularization strategy has been proposed

---

(a) I-JEPA
bb) SimSiam
similarity
similarity
stop grad
stop grad
Masked Patch
Random Patch
predictor g
predictor
encoder
encoder
encoder
encoder
imagex. x
image x
Figure 2: Illustration of I-JEPA (a) and SimSiam (b).
Joint-Embedding Predictive Architectures. The Joint-Embeddingg Predictive Architecture (JEPA) framework has shown considerable promise by enabling the prediction embeddings from a compatible signal [1]. Building on this, the Image- Joint- Embedding Predictive Architecture (I-JEPA) introduced by Assran et al. [2] utilizes a context encoder and target encoder, together with a predictor, manage representations under masking regime. This approach focuses learning embedding spaces directly, contrasting with other methods that operate on pixel or token spaces. While I-JEPA has been successful, it faces challenges such as the inefficacy of Exponential Moving Average (EMA) strategies in preventing model collapse and inaccuracies in predicting the mean of patch representations. These limitations hinder the stability and efficacy of the learned representations, prompting a need for enhanced methodologies. By maintaining sufficient variance in the embeddings and minimizing redundancy among features, VICReg supports robust feature learning. To address the limitations of I-JEPA, our work integrates VICReg into the JEPA framework, thereby aiming to prevent total model collapse and ensure invariance across different views of the same image.
3 Methodology
In this section, we present novel masked modeling framework designed for the joint-embedding predictive architecture to avoid entire collapsing and improve the mean of patch representations. Our key idea is integrate VICReg into the JEPA framework for alleviating entire model collapse and improving the invariance across different views of the same image. We first provide preliminaries in Section 3. then provide the theoretical and empirical connection between I-JEPA and SimSiam in Section 3.2, and finally connect I-JEPA with VICReg in Section 3.3 to show the benefit of reducing entire collapsing and learning the mean of patch representations.
3.1 Preliminaries
In this section, first describe the problem setup and notations, and then revisit I-JEPA [2]. SimSiam [4], and VICReg [6] for self-supervised image modeling-
Problem Setup and Notations. Given a dataset X' = { with images E Rexxx our goal to learn a neural network fo(:) to extract unsupervised representations from these visual samples.
Revisit I-JEPA [2]. Taking the self-supervised modeling as a joint-embedding predictive architecture, I-JEPA [2] utilized f-).) as context encoder, pair of neural network f-(-) a target encoder. A predictor (-(-) is applied to predict the target representations from M masked block patches (1)..... (M)) For target block by corresponding
target mask B, the predictor go
takes as input the output of the context encoder b and a mask token for each patch to predict
{m, JeB, and outputs the patch-level prediction (by, Jes, that is, (by, hee: ge((m, )es.))
3

---

The masking objective is optimized by the average &g distance between the predicted patch-level representations by and the target patch-level representation by which is formulated as:
CI-JEPA
[|b
by, l1..
(I)
JM
JEB,
where |AZ] denotes the total number of target blocks M, and B, is the generated mask corresponding to the i-th target block during training.
Revisit SimSiam [4]. For contrastive self-supervised learning. SimSiam [4] tried to obtain two augmented views 2; and 24.- Then they fed two views into a pair of neural networks fo(-) and fg (-)- To learn the invariance of two different views from the same image, a prediction MLP head, denoted as g, transforms the output z of one view and matches it to representations p of the other view. The overall objective of SimSiam is to minimize the distance between z and p for all random patches r in the same image, which is formulated as:
V Σ
LsamSian
Σ
Zry
Pr,112:
VV
i=1 JEP;
where [V] denotes the total number of augmented views V, and P, is the all random patches corresponding to the i-th view during training.
Revisit VICReg [6]. To prevent a collapse in which the encoders produce constant or non-informative vectors in Siamese net architecture, VICReg [6] introduced variance and covariance regularization terms based on the representations space in the invariance terms. Firstly, the variance regularization term v as hinge function on the standard deviation of the embeddings z along the batch dimension n to prevent collapse with all the embeddings mapped on the same vector, which is denoted as:
1 Σ d
u(z)
max -ee---Varraa Var(zz) +e),
1=1
where y denotes a constant value for the standard deviation in their experiments, and denotes a small numerical scalar to stabilize training. Secondly, the covariance regularization term e mini- mizes the sum of the squared off- off-diagonal coefficients encourage the off-diagonal coefficients of correlation matrix C along the batch dimension to be close to O, which is formulated as:
c(z)
Σ
Σ
z z)(z, - )},
where 1 Σ" Z This term is helpful to prevent different dimensions of the embeddings from encoding trivial information. Finally, the invariance term is defined to minimize the mean-squared Euclidean distance between each pair of vectors z and z,, from different views.
3.2 Connecting I-JEPA with SimSiam
In the realm of supervieee learning, both I-JEPA [2] and SimSiam [4] frameworks aim to extract robust representations from images, yet they approach the problem with distinct architectures and objectives. Here, we draw theoretical and empirical parallels between these methodologies, leveraging their unique approaches enhance the understanding of joint-embedding architectures.
Regarding theoretical connections, I-JEPA [2] uses a predictive model where the encoder and predictor work together to forecast masked parts of the input, relying partial views. Conversely, SimSiam [4] operates without explicit masking, utilizing dual augmentations of the same image to enforce consistency between the independently processed views. Both models share the underlying principle of minimizing the distance between certain representations. As shown in Figure 2, I- JEPA focuses the distance between predicted and actual masked patch representations, whereas SimSiam minimizes the distance between the two augmented views an image, thereby encouraging consistency across different transformations of the same data.
Meanwhile, empirical studies such as Tian al. [5] have shown that linear predictors in frameworks like BYOL [22], which are closely related to SimSiam, tend to learn alignment with the correlation

---

matrix of representations. This suggests that SimSiam's 's approach without a separate predictor could inherently align with the mean of augmented views, concept central I-JEPA's strategy of predicting masked patches. A study [32] in joint-embedding seff-supervised learning also indicates that these methods primarily focus on capturing the co-occurrence and distribution of image patches, which aligns with I-JEPA's objective of learning from masked representations.
3.3 Connecting I-JEPA with VICReg
VICReg [6] introduces variance and covariance regularization prevent the collapse of representa- tions in Siamese networks, ensuring that the model learns informative and diverse features across different dimensions. By integrating VICReg into I-JEPA, our objective to tackle the common challenges of model collapse and enhance the mean representation learning of image patches.
VICReg's variance regularization ensures that all dimensions of the embedding space contain mean- ingful variance, which crucial for preventing the model from collapsing to trivial solutions. I-JEPA, which aims to learn diverse patch representations, can benefit from such mechanism ensure that each masked patch contributes informatively to the overall representation. By minimizing the off-diagonal elements of the covariance matrix, VICReg encourages the features to be uncorrelated, enhancing the diversity of the learned features. This aspect can be particularly beneficial for I-JEPA, where diverse patch predictions are essential for effective representation learning. Literature on non-contrastive self-supervised learning [33] suggests that implicit variance regularization, as seen in VICReg, can facilitate the learning dynamics in joint-embedding architectures. Such regularization helps maintain balance between similarity and diversity in learned representations, which is crucial for effective self-supervised learning.
In the following, we consider linear predictor Wp E RMXM in I-JPEA, the masking objective with the average &a distance between the predicted patch-level representations Zy, and the target patch-level representations Zy as
M
L
IWWPyy - SG I12.
(5)
M
JEB
where |M] denotes the total number of target blocks M, and B, is the generated mask corresponding to the i-th target block during training. By connecting I-JEPA with SimSiam in Section 3.2, we can can diagonalize correlation matrix of Zy, over R: Cz,, UD where U is an orthogonal matrix whose columns are the eigenvectors of C2n, and Dc is the real-valued diagonal matrix of the eigenvalues 8k, k E [1, KJ. Given this eigendecomposition, the predictor is directly set Wp = UD?U", where is positive constant exponent applied element-wise De. The eigenvalues A of the predictor matrix Wp are then A = s Assume 2y the representations expressed in the predictor's eigenbasis, the asymmetric loss L can be formulated as:
M
K Σ
L= M
1
Σ
-SG(2 )11.
(6)
JEB,
Following non-contrastive self-supervised learning [33]. we can use NTK [34, 35] to characterize the
learning gradient dynamics of neural networks as Vz,, L (D z D, and the representational
dynamics of each mode independently follow gradient of the loss Van, b independent differential equations:
C. and decouple as K
d2y 1] D2n (1) = 1//x (Dig, - )12uy,
Oc
dt
By taking the expectation over the same masking blocks, we can have the dynamics for each mask patch yj as
dā 17 (1 Ax 2
(8)
dt
Note that when As < 1, d2y, has the same sign as & and they have opposite sign at X > 1. These convergent dynamics will push an eigenvalue A of one to prevent the collapse of each mode & in representation dynamics. When removing the stop-grad operator in L, we can have
5

---

Xx) By,ks where yk and &g always have opposite signs and the learned representations will become zero with exponentially decaying eigenvalues to collapse. Without the predictor Wp, we will have
d8 (1(11
džy, di = O, and the representations will not be updated.
Incorporating VICReg's 's regularization strategies into I-JEPA could prevent the entire collapsing of the model, especially when learning from a large and diverse dataset. This integration could also improve the granularity and utility of the patch-level representations by ensuring that each dimension of the embedding space remains informative and independent. Overall, the integration of VICReg and SimSiam principles into the I-JEPA framework offers promising avenues for enhancing the robustness and efficacy of self--uperrvieed learning models, particularly in tasks requiring nuanced understanding of complex visual content.
Experiments
4.1 Experimental setup
Datasets. Following previous methods [12, 2]. we use ImageNet-1K [36] for image classification, MS-COCO [37] for object detection and instance segmentation, and ADE20K [38, 39] for semantic segmentation. We closely follow previous work [40, 41], and adopt the Mask R-CNN [42] as the detector. The ViTs [43] backbone weights are initialized with weights pre-trained on ImageNet- IK using our C-JEPA. Following the settings in [12], we use the Semantic FPN and UPerNet approach [44] based on our ImageNet-1K are-trained ViTs for evaluation. For a fair comparison, we fine-tune the detector with the same learning rate in [12]. For video object segmentation, DAVIS-2017 dataset containing 60 training, 30 validation, and 60 testing videos. For low-level tasks, follow the previous work [2] and use Clevr/Count and Clevr/Dist on Clevr [45] dataset.
Evaluation Metrics. For image classification, we follow previous masked image modeling meth- ods [12, 2] to report the classification accuracy of linear probing and fine-tuning. For object detection and instance segmentation on MS-COCO, we apply A and A Aprask as metrics for the bounding boxes and the instance masks. mIoU results are reported to evaluate semantic segmentation on ADE20K. For video object segmentation on DAVIS-2017, we use Jabri-based (J&F) Jms Fm. as metrics to evaluate the quality of frozen representations of image patches by segmenting scenes with the nearest neighbor between consecutive frames. For object counting and depth prediction tasks on Clevr, object counting and depth prediction to evaluate the linear probing performance of our model.
Implementation. For input images, we resized the resolution to 224 x 224, ie., H = W = 224.
Following prior work [12, 2]. we apply patch size of 16, i.e., P 16. We use the tiny, small, base, and large models of VIT [43] architecture for experiments. We set the embedding dimension of the predictor to 384, and keep the number of self-attention heads the same as the backbone context-encoder. For the ViT-T/16, ViT-S/16, and ViT-B/16 context-encoder, we set the depth of the predictor as 6. For ViT-L/16 context-encoders, we set the depth of the predictor to 12. Following AdamW to optimize the context-encoder and predictor weights. We train model using the default batch size of 2048, and the learning rate linearly increased from 1c-4 to le-3 during the first 1.5 epochs of pre-training, and decay 1e-6 following cosine schedule. The weight decay is linearly increased from 0.04 to 0.4, and the target-encodee weights are initialized the same as the context-encoder weights, and updated via an exponential moving average. We momentum value of 0.996, and linearly increase this value to 1.0. For masking, we use the same strategy and settings I-JEPA [2] for possibly overlapping target block masks.
I-JEPA [2],
4.2 Experimental comparisons
In this work, we propose novel and effective framework for connecting Joint-Embedding Predictive Architecture with VICReg in non-contrastive self- self-supervised learning. order to demonstrate the effectiveness of the proposed C-JEPA, we comprehensively compare it to previous baselines [41, 10, 12, 46, 47, 2] on non-contrastive self-supervised learning and mask image modeling.
ImageNet-11 KK image classification. For image classification on the ImageNet- IK benchmark, we report the quantitative comparison results on linear evaluation and fine-tuning results in Table 1. We can observe that our C-JEPA achieves the best results in terms of ViT-B/16 compared to previous

---

Table 1: Comparison with prior work. We perform linear evaluation, fine-tuning, COCO detec- tion/segmentation, and ADE20K semantic segmentation on pre-trained ViT-B/16 models. We report linprob, fine-tune, Apbox, ApMask, and mloU to evaluate the quality of pre-trained representations. The best results are indicated in bold.
Method
Pretrain Epochs
linprob
fine-tune
Apbox
A Apzaak
mloU
DINO [41]
1600
78.2
82.8
50.1
43.4
46.8
BETT [10]
800
56.7
83.4
49.8
44.4
47.1
MAE [12]
1600
68.0
83.6
50.3
44.9
48.1
iBOT [46]
1600
79.5
84.0
51,2
44.2
50.0
data2vec [47]
800
60.8
84.2
48.2
I-JEPA [2]
600
72.9
83.5
49.9
44.5
47.6
C-JEPA (ours)
600
73.7
84.5
50.7
45.3
48.7
Table 2: Scaling up to Large Models. We perform linear evaluation, fine-tuning, video object segmentation, and low-level tasks on pre-trained ViT-L/16 models. We report linprob, fine-tune, (I (T&FF) m. Clevr/Count and Clevr/Dist metrics to evaluate the quality of pre-tree traineried representations. The best results are indicated in bold.
Method
Pretrain Epochs
linprob
fine-tune
(J&FF
Clevr/Count
Clevr/Dist
I-JEPA [2]
600
77.5
85.3
56.6 58.3
85.6
71.2
C-JEPA (ours)
600
78.1
86.2
86.8
71.6
masked image modeling approaches. Specifically, the proposed method outperforms the I-JEPA by 0.8 and 1.0 in terms of top-1 accuracy on linear evaluation and fine-tur fine-tuning settings. Meanwhile, we enjoy the advantage of fewer pre-training epochs compared to other masked image model frameworks.
COCO object detection and instance segmentation. For COCO object detection & instance seg- mentation benchmarks, we also report the quantitative comparison results in Table 1. As can be seen, we achieve significant performance gains of @ Aphox and 0.8@ Apnask on COCO object detection and instance segmentation compared to I-JEPA. We also achieve better results than DINO [41] and MAE [12] regarding both settings.
ADE20K semantic segmentation. For the ADE20K semantic segmentation, we report the quantita- tive comparison results in Table As be seen, the proposed C-JEPA outperforms I-JEPA [2], the current image-based joint-embedding predictive architecture by 1.1 @mloU. Also, we observed that our C-JEPA can achieve better performance than other masked image modeling baselines, in- cluding MAE [12] and data2vec [47]. These improvements suggest the importance of leveraging the VICReg's regularization capture better semanties for dense prediction tasks.
DAVIS video object segmentation. addition, we scale up our experiments from VIT-B to VIT-L and report the results in Table 2. The proposed C-JEPA still achieves better performance than I- JEPA [2] in terms of linear evaluation and fine-tuning settings. For DAVIS video object segmentation, method also shows significant and consistent gains shown in Table Compared I-JEPA, ours achieved the results gains of 1.7@ (J&F) on ViT-I/16.
Clevr object counting and depth prediction. We further measure the abilities of object-counting and depth prediction on Clevr benchmarks for our pre-trained models. Table shows linear evaluation results of C-JEPA the Clevr counting and depth benchmarks. Compared to I-JEPA [2], we achieve the results gains of Cleevr/Count and 0.4@ Clevr/Dist using ViT-L/16. These results further indicate the benefit of the proposed method in learning better representations than our vanilla I-JEPA baseline without VICReg regularization.
4.3 Experimental analysis
In this section, we performed ablation studies to demonstrate the benefit the proposed Vari- ance/Covariance and Invariance terms from VICReg. Here, we conducted extensive experiments on ImageNet-Ie 1k pre-trained ViT-B/16 to explore the impact of representation mean and collapse, and leamned meaningful qualitative patch-level representations.
7

---

Table 3: Ablation studies on component analysis for faster convergence. We perform ablation studies on Variance/Covariance and Invariance modules in VICReg using ViT-B/16 model. The best results are indicated in bold.
IJJEPA
VarvancelCovariance
Invariance
Backhone
Pretrain Epoch
linproh
(J F)_m
ViT-B/16
100
63.7
82.5
52.3
(mean)
V (collappee
X
VIT-B/16
100
68.3
83.2
54,6
VIT-B/16
100
69.5
83.6
55,2
VEMA for collapse)
(mmen)
VIT-B/16
100
67.6
82.8
53.9
Table 4: Ablation studies on component analysis for better convergence. We perform ablation studies on Variance/Covariance and Invariance modules in VICReg using ViT-B/16 model. The best results are indicated in bold.
I-JEPA
Variance/Covariann
Invariance
Backbone
Pretrain Epoch
liuprob
fine-tune
& F)_m
x
ViT-B/16
600
72.9
83.9
56,2
/ (mean)
(collapsee
X
ViT-B/16
600
73.5
84.3
56.9
VIT VIT-B/16
600
73.7
84.5
57.5
/EMAA for collapse)
((mea)
VIT-B/16
600
73.2
84.2
56.6
Variaace/Covariaacee and Invariance in VICReg, We first conducted an ablation study to determine the efficacy of incorporating Variance and Covariance terms, along with the Invariance term, in preventing model collapse and improving the fidelity of representation means. These experiments are crucial for understanding how each component contributes the overall stability and effectiveness of the learned embeddings. The results, as presented in Tables 3 and 4, provide clear insights into the performance enhancements driven by these modifications. The inclusion of the Variance and Covariance terms significantly accelerated the training process, as evidenced by the metrics in Table 3. Table 4 highlights improved accuracy and stability the embeddings, demonstrating the beneficial impact of the Invariance term in aligning the representation means across different mask blocks of the same images.
Representation Mean and Collapse. Further analysis focused on how well the C-JEPA framework mitigates the risk of model collapse and accurately learns the mean of patch representations: The integration of VICReg's ' terms with C-JEPA's architecture addresses the previously noted deficiencies I-JEPA's EMA component, which was prone to collapsing. The results (referenced in the last rows of Tables 3 and 4) underscore the robustness added by these regularization terms from VICReg.
Qualitative A Attention Visualizations. To complement quantitative findings, we also examined qualitative aspects through attention visualization techniques. Figures and ?? showcase the attention maps generated by both the base I-JEPA and the enhanced C-JEPA models. These visualizations illustrate the more focused and contextually relevant attention in C-JEPA, which correlates with the theoretical improvements expected from incorporating VICReg's terms. The attention maps further validate our approach by visually demonstrating the enhanced capability of C-JEPA to maintain stable and meaningful patch-level representations across various image contexts, an improvement over the base I-JEPA model. This qualitative evidence supports the quantitative improvements and highlights the benefits of our approach in capturing invariance.
5 Conclusion
In this work, we introduced C-JEPA, a novel enhancement to the Joint-Embeddiing Predictive Architecture that integrates the robust features of Variance-invarianee-CCoo (VICReg) to address critical limitations in the existing I-JEPA model. By refining the EMA and prediction components of I-JEPA, C-JEPA successfully prevents entire model collapse and more effectively learns the mean of patch representations, thereby advancing the state of unsupervised visual representation learning. Our theoretical analysis and empirical results have demonstrated the effectiveness of C-JEPA, particularly when pre-trained on the ImageNet- IK dataset. The framework shows marked improvements in convergence rates and performance in both linear probing and fine- tuning scenarios, outperforming existing methods. This underscores C-JEPA's capability to not only address the weaknesses of its predecessor but also to provide substantial improvements in learning quality and stability.
Regularization
8

---

Image
Target end I-JEPA
Target ene C-JEPA
C-JEPA
Image
Target I-JEPA
"brret Conteud
Image
Target I-JEPA
Target C-JEPA
C-JEPA
Figure 3: Qualitative visualization of learned attention maps using ViT-B/16 model. Columns for each sample denote the original image, attention maps from the target encoder I-JEPA [2], attention maps from the target encoder in our C-JEPA, and attention maps from the context encoder our C-JEPA. Our C-JEPA achieves much better attention maps.
Broader Impact
Our C-JEPA's adoption of VICReg principles enhances its versatility and applicability across various unsupervised learning contexts, making it valuable tool for tasks requiring robust and reliable visual representations. The implications of this research are significant, offering a pathway for future studies explore and expand upon the integration of contrastive and joint-embedding techniques. As move forward, further refinements and explorations into the scalability of C-JEPA more diverse and challenging datasets will be crucial. Additionally, investigating the adaptability of C-JEPA to different domains and its effectiveness in real-world applications will be essential to fully realize its potential. C-JEPA represents a significant step forward in the field of machine learning, particularly in the unsupervised learning of visual representations. It sets a new benchmark for future research and development in this area, promising enhanced performance and broader applicability in tackling complex visual understanding tasks.

---

References
[1] Yann LeCun. path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.
Open Review, 62, 2022. 1,3
[2] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint- embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 15619-15629, 2023. 1,3, 4, 6, 7, 9, 19, 20, 22,
23, 24
[3] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual repre- sentations from video. arXiv preprint 2oXiv:240.008777 2024. 1
[4] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750-15758, 2021. 2, 3,4
[5] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In Proceedings of the 38th International Conference Machine Learning (ICML), pages 10268-10278. PMLR, 2021. 1, 1,4
[6] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varianceiivarranc--ovvaaaa regular- ization for self-supervise learning. In Proceedings of International Conference on Learning
Representations (ICLR), 2022. 2, 3, 5,
[7] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 6827-6839, 2020.
[8] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Proceedings of European Conference on Computer Vision (ECCV), 2020. 2
[9] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant repre- sentations. In IEEF/CVF Conference on Computer Vision and Patern Recognition (CVPR), 2020.
[10] Hangbo Bao, Li Dong, and Furu Wei. Beit: BERT pre-training of image transformers. arXiv preprint 4rXiv:2106.08254, 2021. 2, 7
[11] Sara Atito, Muhammad Awais, and Josef Kittler. Sit: Self-supervised vision transformer. arXiv preprint arXiv:2 (xXiv:2004003600 2021. 2
[12] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.063/1 2021. 2, 6.
[13] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14668-14678, June 2022.
[14] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao. Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9653 9663, June 2022. 2
[15] Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, and Jinqiao Wang. MST: Masked sule-supervised transformer for visual representation. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021. 2
[16] Yuge Shi, N Siddharth, Philip HS Torr, and Adam R Kosiorek. Adversarial masking for self- supervised learning. Proceedings of International Conference on Machine Learning (ICMI., 2022. 2
10

---

[17] Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. MVP: multimodality- guided visual pre-training. In Proceedings of the European Conference on Computer Vision (ECCV), pages 337-353, 2022. 2
[18] Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Sem- MAE: Semantic-guide masking for learning masked autoencoders, In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00022 2021. 2
[20] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning (ICML), 2020.
[21] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020. 2
[22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 4
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020. 2
[24] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint 04X2v:2003.04297, 2020. 2
[25] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
[26] Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin. Zheng Zhang, and Han Hu. Parametric instance classification for unsupervised visual feature learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 15614-15624 2020. 2
[27] Hu Qianjiang, Wang Xiao, Hu Wei, and Qi Guo-Jun. AdCo: adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1074 1083, 2021.
[28] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), 2020.
[29] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In Proceedings of International Conference on Learning
Representations (ICLR), 2021. 2
[30] Xudong Wang, Ziwei Liu, and Stella X Yu. CLD: unsupervised feature learning by cross-level instance-group discrimination. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[31] Shentong Mo, Zhun Sun, and Chao Li. Siamese prototypical contrastive learning. In BMYC, 2021. 2
11

---

[32] Yubei Chen, Adrien Bardes, ZENGYT and Yann LeCun. Bag of image patch embedding behind the success of self-supervised learning. Transactions on Machine Learning Research, 2023. 5
[33] Manu Srinath Halvagal, Axel Laborieux, and Friedemann Zenke. Implicit variance regulariza- tion in non-contrastive SSL. In Proceedings ohirty-seeenn Conference on Neural Information Processing Systems (NeurIPS), 2023. 5,
[34] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Proceedings of Advances in Neural Information Processing Systems, 2018. 5, 14
[35] Jachoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019. 5, 14
[36] Jia Deng. Wei Dong, Richard Socher, Li-Jia. Li, Kai Li, and Li Fei-Fei. ImageNet: A Large- Scale Hierarchical Image Database. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255 2009, 19
[37] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pages 740-755, 2014. 6, 19
(38) Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122-5130, 2017. 6, 19
[39] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Se- mantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision (IJCV), 22:302-321, 2018. 6, 19
[40] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 6
[41] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-supe vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 6, 7
[42] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE/CVE International Conference on Computer Vision (ICCV), pages 2980-2983 2017. 6, 19
[43] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference on Learning Representations, 2021. 6
[44] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of European Conference on Computer Vision (ECCV), pages 432-448, 2018. 6, 19
[45] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901-2910, 2016. 6, 19
[46] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022. 6,
12

---

[47] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vee: general framework for supervvee learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022. 6,7 7
[48] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of International Conference on Learning Representations (ICLR), 2019. 17, 18
[49] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In Proceedings of International Conference Learning Representations (ICLR), 2017. 17, 18
[50] Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alex Sorkine- and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 000005, 2017. 19
13

---

Appendix
In this appendix, we provide the following materials:
proofs on the loss L in the predictor cigenspace and representation dynamics under L Section A,
derivation of learning dynamics without the stop-grad and predictor in Section B,
detailed loss functions and algorithm in Section C,
additional experimental results in small-scale models and VICReg hyperparameters in Section D,
additional visualizations on the learned attention maps in Section E.
A Proofs
A.1 Loss in the predictor eigenspace
We consider linear predictor Wp E RMxM M in I-JPEA, the masking objective with the average Eg dis- tance between the predicted patch-level representations zy, and the target patch-level representations
y;
C
|WW, (G,2,,)12
1= JEB
Assume 2y, the representations expressed in the predictor's 's eigenbasis. Since Wp E MxM M is symmetric matrix with eigendecomposition Wp = U U where U is an orthogonal matrix and UUT = I, we can substitute Wp in Eq.
M
1
I[W SG(Z),)) 11
M
JEB
IUDD Zy, AGGUUTZZ)))
MM
(10)
M
1
I1Dc - SG(22;)
M
JEB
M
K
Σ
| - SG
M
JEB
A.2 Representational dynamics under L
Following non-contrastive self-supervised learning [33]. we can use NTK [34, 35] to characterize the learning gradient dynamics of neural networks as V Vayg L (Dzy, z D, and the representational dynamies of each mode independently follow gradient of the loss Ve., L, and decouple as K independent differential equations:
dmyz.k
OL (t) 11 (2 D2,
Xn.Zys
(11)
By taking the expectation over the same masking blocks, we can have the dynamics for each mask patch y as
di dt
vata du 2
(12)
Suppose the parameters of the neural network are denoted as 0 with weights W M we follow Lemma proposed in [33] and have the empirical NTKθ()) Y)) in the orthogonal eigenbasis is equal
14

---

to the empirical NTKK(Y }) in the original basis. Each block of the full empirical NTK is denoted as (v yyy) (v) MM, where Y/ Yy denote the high-dimensionall inputs from mask patches j and q in the training samples. Iar E IR Mx is the identity. Based on this and the central limit theorem for an i..d standard Gaussian distribution, we can formulate the representational dynamics for each mask patch yj under the L as
d2 81
= (v., 2, L
dt
= = 8y) V Van, C- -n (. yg) Vang
= =nny y;) C - -
(1)) ya) V Van C
(13)
0L
=-1Va bng L -1 Abys
= - (Drby, - nny, De
where Dr is diagonal matrix composed of the eigenvalues dg. For Aehh entry of dbug. formulate the representational dynamics under the L as
dē V Aag.k u -1 1Bny N:
OC
(1)
dl
=n (N b 2y,,k) A
(14)
m (2 X Ag22y1.k)
Since the expectation over the same masking blocks 20, ks we can have the dynamics for each mask patch y as
d2y, n)u Au 2uy
(15)
dt
B Representation learning dynamics without the stop-grad and predictor
B.1 Learning dynamics of the loss L without the stop-grad
Recall we consider linear predictor Wp E M in I-JPEA, the masking objective with the average distance between the predicted patch-level representations Zy, and the target patch-level representations zy as
C
||W - SG
(16)
M
JEB
When removing the stop-grad operator in L, we can have the masking objective as
L
WP 2 112
M
jEB.
(17)
M
(2g
M
JEB;
15

---

The representational dynamics for each mask patch yj under the L is formulated as
y2y,
0L
(1)
dt
0Z by.k
0C
OC
(1) +
(1)
dz Us
(18)
(A t:
33 )x + (A A2 [(5,k))
= (A, - 1)
Since the expectation over the same masking blocks &yg, ko we can have the dynamics for each mask patch yj as
dž
-n((1 2x) 2g.k
(19)
dt
where dē & and ky always have opposite signs and the learned representations will become zero with exponentially decaying eigenvalues to collapse.
B.2 Learning dynamics of the loss L without the predictor
Without the predictor Wr, have the masking objective
M
4
SG(22,))
M
JEB.
(20)
M
K 12 SG(
M
JEB,
&
In this case, we have D 1 for all independent eigenvalues in the eigenbasis. Based on the representation dynamics for each mask patch yj as di
džy,, mdx(1 Au 2 we will have
džy,, k 0, and thus the representations will not be updated in the end.
dt
C Algorithm
In this section, we provide detailed algorithmic and implementation aspects of the Contrastive-JEPA (C-JEPA) framework, elaborating on the loss functions, pseudo-code, and experimental setup used throughout studies. These details aim enable reproducibility and deeper understanding of the modifications introduced in the C-JEPA compared to its predecessors.
C.1
Loss
The loss function for C-JEPA is combination of the original JEPA loss and the VICReg-inspired regularization terms, structured as follows:
L LJEPA Beierey LVicneg
(21)
where Bvicreg = 0.001 provides a scaling factor to balance the original JEPA loss with the new regularization terms introduced from VICReg. The VICReg loss component is defined as:
LVICReg = Baim * Leim + Batd * Latd + Beon Leov
(22)
with 3rmm 25, Bard 25, and Bor 1, which scale the similarity, standard deviation, and covariance regularization terms respectively, integral to maintaining the diversity and stability the leamedd representations.
16

---

C.2 Pseudo Code
The following pseudo-code outlines the batch-wise application of VICReg within the C-JEPA framework, providing clarity on the computational steps involved during training:
# Pseudo- code for batch-wise VICReg application in C-JEPA for each batch dataset:
# Forward pass through the context and target encoders
context_embeddings, target_embeddings encoce
# Apply VICReg terms
sim_loss oonputt.ssaailrrryyyyooooooooooo std_loss sumppteeettaaarrr...raaaiooooooe embeddingg)
target_embeddinge
cov loss compute_covariance__ loss hrrreeccccrrraneoooooooooootttto
# Combine losses
vicreg_loss beta_sim sim_loss beta_std std_loss beta_cov cov cov_loss total_loss jepa_loss (batch) bicr_g vicreg_loss
# Backward pass and update weights optimizer cetimmiizrzzeoogreee zero_grad
cotal_loss backward optimizer.step!
Algorithm outlines the procedure for applying VICReg in a batch-wise manner within the C-JEPA framework, focusing on multiple views in the context of contrastive learning.
Table 5: Pretraining setting for ViT-T. All mod- els trained for 100 and 600 epochs. value
Table Pretraining setting for ViT-S. All mod- els trained for 100 epochs.
config optimizer epochs
config optimizer epochs
value
AdamW [48]
AdamW [48]
100 600
100
learning rate
-3
learning rate
8e-4
weight decay
(0.04,0.4)
weight decay
(0.04, 0.4)
batch size
2048
batch size
2048
learning rate schedule warmup epochs encoder arch.
cosine decay [49]
learning rate schedule warmup epochs encoder arch.
cosine decay [49]
15
15
VIT-T
VIT-S
predicted targets
4
predicted largets
4
predictor depth
6
predictor depth predictor attention heads
predictor attention heads
12
12
predictor embedding dim.
192
predictor embedding dim.
384
17

---

Algorithm VICReg loss for our framework
Require: 2c: Context embeddings tensor of size [batch_size, num num_block, num ham_mask_pateer dimen-
sion]
Ensure: loss: Computed loss for the batch
function VICREGLoSS(2c, args)
2: 2C + MEAN &c, dim = 2)
> Average over the mask patches Apply projection to the embeddings
3: 2c PROJECTOR(20)
4: if 'vicreg' in args then
5:
criterion k- victeg_nriterron
6: else
7:
criterion +- CosineSimillrttyddim=))
8:
end if
9: loss +-
10:
num masks + SHAPE[:c)[1]
11:
for + 0 maum_maske
do
12:
for j 4 0 to num reu_masks- do
13:
if i H then
14:
20 i
15:
20g + 2c|:..1
if vicreg'' in args then ze_galher + GULLLAATHLLLYR
17:
18:
ze_galher; ze_galher E, i,:
19:
ze_galher, &- ze_gather F J.
loss + loss ORITERION(SC,) 20,, 2c 20. 2ctgherr,,,ggahhrr, else
21:
loss loss (EC, end if end if end for
23:
25:
26:
end for
27:
loss +- losss/num_maake x (num_maske 1))
return loss
29:
end function
Table 7: Pretraining setting for VIT-B, All mod- els trained for 600 epochs. config
Table 8: Pretraining setting for VIT-L, Trained for 600 epochs.
value
config
value
optimizer epochs
AdamW [48]
optimizer
AdamW [48]
600
epochs
600
learning
8c-4
learning rate
Sc-4
weight decay
0.4)
weight decay
(0.01, 0.4)
batch size
2048
batch size
2048
learning rate schedule
cosine decay [49]
learning rate schedule warmup epochs encoder arch.
cosine decay [49]
warmup epochs
15
15
encoder arch.
ViT-B
VIT-L
predicted targets
4
predicted targets
4
predictor depth predictor attention heads
6
predictor depth predictor attention heads
12
16
predictor embedding dim.
384
predictor embedding dim.
384
C.3 Implementation Details
For comprehensive implementation, we detail the specific settings used during our experiments:
Image Preprocessing: All input images were resized to resolution of 224 224 pixels.
Patch Size: Following prior work, patch size of 16 x 16 pixels was used.
18

---

Table 9: ImageNet-IK image classification. We perform linear evaluation pre-trained ViT- T/16 and ViT-S/16 models for image classification on ImageNet-1K benchmark. We report the Top-1 accuracies using knn, linprob and fine-tune settings evaluate the quality of pre-trained representations. The best results are indicated in bold.
Method
Backbone
Epochs
knn
linprob
fine-tune
VIT-T/16
100
15.16
22.
64.46
I-JEPA [2]
ViT-S/16
100
24.36
29.61
76.87
V7T-1716
100
19.18
23.31
66.77
C-JEPA (ours)
VIT-S/16
100
26.85
32.35
78.17
Table 10: COCO object detection and instance segmentation. We fine-tuned pre-trained ViT-T/16 and ViT-S/16 models to perform COCO object detection and instance segmentation. The Aphor and Aprask metrics denote the results of COCO detection, and COCO segmentation, respectively. The best results indicated in bold.
Method
Backbone
Pretrain Epoch
Apl
APhb
Aph
Apmk
Apmk
Apnk
VIT-T/16
100
30.5
52.6
32.8
29.1
49.5
30.6
I-JEPA [2]
ViT-S/16
100
36.9
58.5
39.2
35.2
56.3
37.3
VIT-T716
100
32.8
54.5
34.6
30.9
51.4
32.2
C-JEPA (ours)
VIT-S/16
100
38.5
59.7
40.3
37.1
57.5
39.2
Vision Transformer Configurations: We utilized tiny, small, base, and large models of the ViT architecture, with specific settings for each model scale provided in Tables 5, 6, and 8.
Optimizer:
The A optimizer was used for training, with specific learning rate schedules and weight decay settings described.
Batch Size and Training Schedule: A default batch size of 2048 was used, with learning rate and momentum adjustment schedule to optimize convergence.
To validate the efficacy of C-JEPA, employed several benchmark datasets and tasks:
Image Classification on ImageNet-1k [36]
Object Detection and Instance Segmentation on MS-COCO [37]
Semantic Segmentation on ADE20K [38, 39]
Video Object Segmentation on DAVIS-2017 [50]
Low-Level Tasks Clevr Dataset [45]
For each task, we adopted established methodologies and frameworks, such as Mask R-CNN [42] for object detection and Semantic FPN and UPerNet [44] for semantic segmentation, ensuring that our findings were comparable with mate-of-the-eee results.
D More Experiments
In this part, we provide additional experimental analyses to validate the effectiveness of the proposed C-JEPA framework further. These experiments encompass various downstream tasks, ablation studies, and the impact of different components and coefficients the model, specifically focusing on the Variance, Covariance, and Invariance terms from VICReg across different configurations of Vision Transformers.
Downstream Results on VIT-T and VIT-S. We evaluated the performance of pre- pre-trained ViT-T/16 and ViT-S/16 models across a series of downstream tasks to assess the quality of the representations learned by the C-JEPA framework. For image classification, we performed linear evaluations using knn, linear probing (linprob), and fine-tuning methods: The results top-1 accuracies, presented in Table 9, indicate the robustness and quality of the features extracted by pre- pre-trained models
19

---

Table 11: ADE20K semantic segmentation. We fine-tuned pre-trained ViT-T/16 and ViT-S/16 models to perform ADE20K semantic segmentation. The mloU, aAce, and mAcC metrics denote the results of ADE20K segmentation. The best results are indicated in bold.
Semantic FPN
UPerNet aAcc
Method
Backbone
Pretrain Epoch
mloU
aAcc
mAce
mIoU
mAcc
ViT-T/16
100
21.56
72.28
30.26
33.15
75.36
43.05
I-JEPA [2]
VIT-S/16
100
27.17
73.64
37.11
36.98
78.30
46.66
VIT-T/16
100
24.40
73.56
32.88
34.44
76.96
44.56
C-JEPA (ours)
VIT-S/16
100
31.36
76.27
41.34
38.68
79.07
48.86
Table 12: Ablation studies component analysis for faster convergence. We perform ablation studies on Varrance/Covariance and Invariance modules in VICReg using ViT-T/16 model. The best results are indicated in bold.
I-JEPA
VatiancelCovariance
Invariance
Backbone
Pretrain Epoch
knn
linprob
fine-tune
V
X
X
VIT-T/16
100
15.16
22.15
64.46
(mean)
/collapsse)
X
VIT-T/16
100
18.75
22.92
66.15
VIT-T/16
100
19.18
23,39
66.77
/EMAA for collapse)
X
V (mean)
VIT-T/16
100
18.57
22.57
65.92
under different evaluation settings. For object detection and instance segmentation, fine-tuned the pre-trained models to perform tasks on the COCO dataset. The average precision for bounding box detection (APboo) and mask segmentation (APpask) reported in Table 10, demonstrating the applicability of C-JEPA pre-trained models in complex vision tasks. The models were also fine- Union (mloU),
tuned for semantic segmentation tasks, and metrics such as mean Intersection average class accuracy (aAcc), and mean accuracy (mAcc) are detailed in Table 11, highlighting the effectiveness of approach in handling pixel-level classification tasks.
Ditiance/Cvvarinnee and Invariance in VICReg. components within the C-JEPA, we conducted focused ablation studies using the V model. For faster convergence, initial ablations in Table 12 analyze the impact the Cariaace/Covariance and Invariance modules on speeding up the model training. Further studies on better convergence, shown in Table 13, investigate how these modules contribute to achieving better training dynamics and model performance.
To dissect the contributions of individual VICReg
VICReg and Invariance Coefficients. For VICReg coefficients Iniereg, we varied the coefficients to understand their influence model training and final performance, with results compiled in Table 14. The impact of varying the invariance coefficient Aarm is explored optimize the balance between similarity and variance, with findings in Table 15.
These experiments underline the nuanced contributions of the VICReg components the overall efficacy of the C-JEPA framework. By systematically varying key parameters and assessing their impacts across a broad spectrum of tasks, we establish comprehensive understanding of the model's 's behavior and performance characteristics. This rigorous empirical foundation supports the theoretical enhancements proposed C-JEPA, affirming its potential advance the state-of-the-ar in unsupervised visual representation learning.
E More Visualizations
In order provide comprehensive understanding of the advancements offered by the C-JEPA framework, we include a detailed visual analysis alongside our quantitative evaluations. This section of the appendix is dedicated to showcasing attention visualizations that illustrate the practical impacts of the theoretical improvements achieved through the integration of VICReg components within the C-JEPA framework.
To better understand how the C-JEPA model processes visual information, we utilize attention visualization techniques. These techniques highlight the model's ability to focus on relevant features within an image, which crucial for tasks involving fine- fine-grained recognition and localization. The following figures 4, 5, 6, 7, and 8 provide visual comparison between the base I-JEPA and the
20

---

Table 13: Ablation studies on component analysis for better convergence. We perform ablation studies on Variance/Covarianee and Invariance modules in VICReg using ViT-T/16 model. The best results are indicated in bold.
I.JEPA
Variance(Covarianee
Invariance
Backbome
Epoch
linprbb
fine-
VTT-T/16
600
20.53
26.83
68.68
/ (mean)
(collapse)
K
VIT-T/16
600
25.87
29.65
71.06
VIT-T/16
600
27.61
31.86
71.52
/EMA for collapse)
((mean)
VIT-T/16
600
24.95
27.92
70.58
Table 14: Ablation studies on component analysis for VICReg coefficients. We perform ablation
studies on VICReg coefficients (-) using ViT-T/16 model. The best results are indicated in bold.
VICReg Soiereg
Backbone
Pretrain Epoch
(T&.F) m
Jm
Fm
0.0001
VIT-T/16
100
43.6
43.0
44.3
0,0002
VIT-T/16
100
45.7
45.3
46.1
0,0005
VIT-T/16
100
47.3
46.8
47.8
0.001
VIT-T/16
100
50.2
49.8
50.6
0.005
VIT-T/16
100
46.0
45.4
46.6
0.01
VIT-T/16
100
31.9
33.1
30.7
0.02
VIT-T/16
100
29.5
30.6
28.4
0.05
VIT-T/16
100
24.4
25.8
23.1
0.1
VYT-T/16
100
Collapsing
enhanced C-JEPA models. For each sample, columns denote the original image, attention maps from the target encoder in I-JEPA, attention maps from the target encoder in our C-JEPA, and attention maps from the context encoder our C-JEPA
These attention visualizations demonstrate the qualitative improvements in the C-JEPA's ability to focus and contextualize features compared the base I-JEPA model. The C-JEPA model produces more focused and contextually relevant attention maps, which is direct result of the enhanced model architecture that incorporates VICReg's variance, covariance, and invariance regularization strategies. The enhanced attention capabilities of C-JEPA facilitate better feature extraction from complex visual inputs, which is critical for improving performance on downstream tasks. The more detailed and precise attention maps correlate strongly with the theoretical enhancements integrated into the C-JEPA framework, such as improved handling of variance and invariance across different visual domains.
The attention visualizations provided in this section validate the theoretical improvements posited in the earlier sections of this paper. They offer qualitative evidence that supports the quantitative results, underscoring the benefits of the C-JEPA model in capturing more nuanced and robust representations. Overall, these visualizations highlight the C-JEPA's ability to maintain stable and meaningful patch- level representations across variety of image contexts, marking significant advancement over its predecessor.
21

---

Table 15: Ablation studies component analysis for invariance coefficients. We perform ablation
studies on invariance coefficients (-) using ViT-T/16 model. The best results are indicated in bold.
IJJEAA Varinnce/Covariane
Invariance
Backhone
Pretrain Epoch
knn
linprob
fine-tune
/(25)
V/T-1716
100
18.03
21.72
65.68
/(20)
VIT-T/16
100
18.35
22.25
65.73
X
7(15)
V/T-T/16
100
18.57
22.53
65.92
v(10)
VII-1/16
100
18.45
22.36
65.81
Image
Target enc IJERA
Targel enc C-JEPA
Context end C-JEPA
Image
IJJEPA
Target enc C-JEPA
C-JEPA
Image
Targel enc IJEPA
Target enc C-JEPA
Context end C-JEPA
Figure 4: Qualitative visualization of learned attention maps using VIT-B/16 model. Columns for each sample denote the original image, attention maps from the target encoder in I-JEPA [2], attention maps from the target encoder in our C-JEPA, and attention maps from the context encoder in our J-JEAA Our C-JEPA achieves much better attention maps.
22

---

Image
Target
Image
Target end IJEPA
Target enc C-JEPA
Context end C-JEPA
Image
Targer
Figure 5: Qualitative visualization of learned attention maps using ViT-B/16 model. Columns for each sample denote the original image, attention maps from the target encoder I-JEPA [2], attention maps from the target encoder in C-JEPA, and attention maps from the context encoder in our C-JEPA. Our C-JEPA achieves much better attention maps.
Image
Targel enz Targel end Cantext enu IJPPA
C-JEPA
C-JEPA
Image
Target and Target one Context ene IJEPA
C-JEPA
C-JEPA
Image
Targel enz: LIEPA
Targel enc Contest enc C-JEPA
C-JEPA
Figure 6: Qualitative visualization of learned attention maps using VIT-B/16 model. Columns for each sample denote the original image, attention maps from the target encoder in I-JEPA [2]. attention maps from the target encoder in our C-JEPA, and attention maps from the context encoder in our J-JEAA Our C-JEPA achieves much better attention maps.
23

---

Target enc Target IJJEAA
Context enc C-JEPA
Targer
Target
Image
IJEPA
C-JEPA
C-JEPA
C-JEPA
Image
I-JEPA
C-JEPA
C-JEPA
Figure 7: Qualitative visualization of learned attention maps using ViT-B/16 model. Columns for each sample denote the original image, attention maps from the target encoder in I-JEPA [2]. attention maps from the target encoder in our C-JEPA, and attention maps from the context encoder in our C-JEPA Our C-JEPA achieves much better attention maps.
Image
Target end
IJPPA
Target eno Conteat ene C-JEPA
C-JEPA
Image
Target Target I-JEPA
C-JEPA
C-JEPA
Image
Target enc
IJJEPA
Tanget enc C-JEPA
Context enc C-JEPA
RT
Figure 8: Qualitative visualization of learned attention maps using ViT-B/16 model. Columns for each sample denote the original image, attention maps from the target encoder in I-JEPA [2].
attention maps from the target encoder in our C-JEPA, and attention maps from the context encoder in our C-JEPA. Our C-JEPA achieves much better attention maps.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See our discussion in the limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
We do not have theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
See the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
hatpe//CeeSSueeeeeeeeeeee wwwiiisssooolioooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct guides/ /CodeSubmic /tte:///ggiiieeeeeeeuuubiiiiii saissnsillllo onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //nuurrrss.... httpplic//thissuuieeeee.
Answer: [Yes]
Justification:
We have conformed in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See our discussions in the broader impact section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
28

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Section 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
29

---

Answer:
[NA]
Justification:
We do not have new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not have experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "aqHxtZCEWh", "reviewer": "Reviewer_aesY", "summary": "The paper presents C-JEPA, a novel framework integrating VICReg into the Image Joint-Embedding Predictive Architecture (I-JEPA) to address its limitations in preventing model collapse and learning mean patch representations. Empirical and theoretical evaluations demonstrate that C-JEPA enhances the stability and quality of visual representation learning, showing superior performance across multiple benchmarks.", "questions": "Can you provide more details on the computational overhead introduced by the VICReg integration?\nHave you considered testing C-JEPA on more diverse datasets beyond ImageNet-1K and more diverse downstream tasks?", "limitations": "The authors have acknowledged the potential computational complexity and the need for further testing on diverse datasets. They have made significant efforts to address the limitations of C-JEPA, and their proposed future work aims to explore these aspects further.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Rhe paper offers an innovative integration of VICReg with JEPA is innovative and addresses critical limitations in existing frameworks.\nComprehensive validation i achieved through empirical evaluations and comparisons with state-of-the-art methods.\nThe paper offers strong theoretical foundation supporting the benefits of the proposed method.\nThe  results support performance improvements in multiple tasks and benchmarks.\nAblation studies and qualitative visualizations provide deeper insights into the method’s effectiveness.", "weaknesses": "The additional regularization terms may lead to increased computational overhead.\nFurther testing on larger and more diverse datasets is needed to confirm scalability and generalization.\nWhile the results are promising, testing across more varied domains would strengthen the conclusions.d \nThe new combination of established SSL methods (I-JEPA and VICReg) comprises an incremental development in hte fiels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the valuable comments and answer the raised questions below.\n\n> Computational Overhead\n\nWe conducted a series of performance evaluations comparing the computational costs between I-JEPA and C-JEPA. The results in the Table below detail the runtime and resource utilization, demonstrating that the increase in computational overhead is counterbalanced by significant improvements in model stability and performance.\n\n| Method            | Pre-train Epochs | Max Memory  per GPU | Running Time  per Step |\n|-------------------|------------------|---------------------|------------------------|\n| I-JEPA            | 600              | 21.9G               | 606.4 ms               |\n| C-JEPA (ours) | 600              | 21.9G               | 607.5 ms               |\n\n> Testing on Larger Datasets\n\nWhile our current evaluations provide a robust foundation for the efficacy of C-JEPA, we recognize the importance of scalability and generalization across more diverse datasets. Future work will extend these evaluations to include larger datasets from varied domains, enhancing our understanding of the framework's applicability and robustness in broader contexts.\n\n> Clarification on Integration of I-JEPA and VICReg\n\nThe integration of VICReg into the I-JEPA framework represents a significant advancement beyond mere incremental development. We also provide a deep theoretical analysis that elucidates the synergistic effects of this integration, addressing critical weaknesses in existing models and setting a new benchmark for stability and performance in visual representation learning.\n\n> Testing on More Diverse Tasks\n\nC-JEPA has been rigorously tested across multiple standard benchmarks such as ImageNet-1K, COCO, ADE20K, DAVIS, and Clevr, covering a wide range of visual tasks from image classification to video object segmentation. Plans to include datasets from audio and video domains are underway, aiming to diversify the testing landscape further and demonstrate the adaptability of C-JEPA to various modalities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bQtrobqiPp", "reviewer": "Reviewer_Rspm", "summary": "The paper introduces C-JEPA, an enhancement to the Joint-Embedding Predictive Architecture incorporating Variance-Invariance-Covariance Regularization (VICReg) for non-contrastive self-supervised learning. This approach addresses limitations such as model collapse and inaccurate mean patch representations, enhancing stability and learning quality. C-JEPA shows improved performance across diverse tasks, including image classification, object detection, semantic segmentation, and video object segmentation.", "questions": "Please refer to the weakness.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. C-JEPA's innovative combination of VICReg with self-supervised learning architectures addresses critical issues like model collapse and enhances unsupervised visual representation learning.\n2. The framework is supported by extensive empirical evidence and theoretical analysis, demonstrating superior performance compared with existing methods on multiple datasets and diverse tasks.", "weaknesses": "1. The framework would benefit from comparisons with latest leading methods like Dinov2[1], Mocov3[2], and IWM[3] to benchmark against current advancements.\n2. The paper claims that the inclusion of Variance and Covariance terms enhances training speed and stability (lines 291–295). However, the basis for these conclusions is not clearly articulated, necessitating further explanation to validate these claims.\n3. While the paper presents results from pre-training on image dataset like ImageNet, its applicability to video-related tasks can be explored. An adaptation on video JEPA methods, such as V-JEPA[4] for video domains, could significantly extend the framework's utility.\n4. There are typographical and formatting errors, such as the unexplained markers in line 302.\n\n[1] Oquab, Maxime, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023).\n[2] Xie, Zhenda, et al. self-supervised learning with swin transformers. arXiv preprint arXiv:2105.04553 (2021).\n[3] Garrido, Quentin, et.al. Learning and Leveraging World Models in Visual Representation Learning. arXiv:2403.00504 (2024)\n[4] Bardes, Adrien, et.al. Revisiting Feature Prediction for Learning Visual Representations from Video. arXiv:2404.08471 (2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the valuable comments and answer the raised questions below.\n\n> Comparison with More Methods\n\nTo demonstrate the competitiveness of C-JEPA, we included our comparative analysis to include recent advancements in the field such as DINOv2, MoCo v3, and IWM. However, the comparison with DINOv2 is unfair, as it was trained on a larger dataset (collected LVD-142M). While acknowledging differences in training datasets and scales, our results indicate that C-JEPA achieves comparable or superior performance than MoCo v3 and IWM, particularly in scenarios where model stability and unsupervised learning quality are critical. \n\n| Method  | Backbone | linear prob | finetune |\n|---------|----------|-------------|----------|\n| MoCo v3 | ViT-L    | 77.6        | 84.1     |\n| IWM     | ViT-L    |     --        | 85.4     |\n| C-JEPA (ours)  | ViT-L    | **78.1**        | **86.2**     |\n\n> Clarification on Training Speed and Stability\n\nThe inclusion of Variance and Covariance terms in C-JEPA has demonstrably enhanced training speed and stability. Empirical results show that C-JEPA reaches comparable performance metrics to I-JEPA in significantly fewer epochs (100 vs. 400). \n- Training speed: C-JEPA achieves fast convergence, where linear probing at 100 epochs is close to I-JEPA at 400 epochs.\n- Stability: C-JEPA achieves better performance, as linear probing at 400 epochs is much better to I-JEPA at 400 epochs.\n\n> Applicability to Video-Related Tasks\n\nThe adaptability of C-JEPA to video domains represents a promising direction for future research. Preliminary theoretical considerations suggest that the integration of VICReg components could be similarly beneficial in video-based JEPA frameworks (V-JEPA). We plan to explore this in future work, aiming to extend the utility of C-JEPA to dynamic visual contexts and enhance learning from video data.\n\n> Formatting and Typo\n\nThanks for pointing this out. We have removed the question marker in line 302.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2JtlMWceJ", "reviewer": "Reviewer_1VXy", "summary": "This paper proposes a novel contrastive self-supervised learning framework based on JEPA, namely C-JEPA. The main idea of C-JEPA is to address the limitations of the I-JEPA, especially the limited prevention of collapse with EMA, by incorporating the principles of VICReg. The authors demonstrate the effectiveness of C-JEPA through rigorous empirical and theoretical evaluations. They show that C-JEPA achieves superior performance metrics compared to existing frameworks with faster and better convergence.", "questions": "Please address my concerns in the Weaknesses.", "limitations": "They did not address the limitations and potential negative societal impact of their work in the paper.\n\nThey must include the limitations in the final manuscript, e.g., the proposed approach requires a large computation and a large network capacity, therefore raising environmental concerns, e.g., carbon generation [1].\n\n[1] Schwartz, Roy, et al. \"Green ai.\" Communications of the ACM 63.12 (2020): 54-63.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "**[S1]** The paper is well-motivated that I-JEPA has room for improvement regarding the risk of model collapse and challenges in learning the mean of patch representations. They demonstrate that the proposed approach can effectively address the problem.\n\n**[S2]** The paper shows that the proposed approach consistently and significantly improves the performance of I-JEPA on various downstream tasks and shows the scalability of the approach.\n\n**[S3]** The overall writing is smooth and easy to follow.", "weaknesses": "**[W1]** Though the paper shows a strong performance, I think the contrastive learning on JEPA lacks novelty. The proposed approach is a simple combination of I-JEPA and VICReg’s regularization strategy.\n\n**[W2]** Does C-JEPA perform better than I-JEPA with more pre-training, e.g., 800 or 1200 epochs? In figure 1, it seems that the slope of I-JEPA is much larger than C-JEPA.\n\n**[W3]** Invalid citation: Line 302.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the valuable comments and answer the raised questions below.\n\n> Clarification\n\nWhile C-JEPA leverages components from both I-JEPA and VICReg, its novelty lies in its theoretical grounding and practical implementation, which specifically addresses and mitigates I-JEPA's limitations. The integration is not merely additive but synergistic, optimizing the prevention of model collapse and enhancing the learning of mean patch representations in ways that neither approach could achieve independently.\n\n> Performance with Extended Pre-training\n\nTo address concerns about C-JEPA's long-term performance, we extended our pre-training evaluations up to 800 and 1200 epochs. The results in the Table below show that C-JEPA not only maintains but also enhances its performance advantage over I-JEPA, further validating its effectiveness and robustness in prolonged training scenarios. \n| Epochs | I-JEPA | C-JEPA (ours) |\n|--------|--------|--------|\n| 800    | 73.3   | **74.2**   |\n| 1200   | 73.9   | **75.0**   |\n\n> Typo\n\nThanks for pointing this out. We will fix this typo.\n\n> Potential Negative Societal Impact and Limitations\n\nThanks for your suggestion. We have claimed this in the section Broader Impact at the end of page 9. We will also include environmental concerns in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary: The paper presents an improvement to I-JEPA to improve representation learning quality. The authors observe that I-JEPA has limitations in terms of model collapse and learns mean patch representations. The authors connect I-JEPA to VICReg and SimSiam to address both these drawbacks respectively. The method is tested on ImageNet.

Positives:
1. Clearly written paper with a clear motivation and method section. It was easy to follow the problem that the authors were trying to solve and how they solved it.
2. Strong ablation studies strengthen the main claims in this work
3. The final method improves over the recent I-JEPA work and the authors demonstrate this improvement across multiple model sizes, and downstream tasks.
4. The method holds gains even after longer training (800 and 1200 epochs) which shows this technique isn't merely accelerating convergence. Please add this to the paper appendix.

Negatives:
1. Missing comparisons to stronger baselines like DinoV2. This would admittedly require training on much larger pretraining datasets.

Reviewers:
The reviewers are positive about this work and maintain the positive ratings after the author rebuttal.

Decision:
Given the reviewer recommendations and results of this paper, it should be accepted at NeurIPS. 
Further, given that it shows strong empirical fixes to I-JEPA this work should be made a spotlight.
However, since the proposed improvements are limited to a single method (rather than all non-contrastive methods like BYOL, DINO), the AC does not recommend it to be an oral.
Decision: Accept (spotlight)