One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation
Authors: Xiaoyu Yang
Lijian Xu
Hongsheng Li
Shaoting Zhang
Venue: ICML 2025 poster (2025)
Keywords: contrastive learning
self-supervised pre-training
Primary area: deep_learning->selfsupervised_learning
TL;DR: We propose Occluded Image Contrastive Learning (OCL), a scalable method that contrasts masked image patches to learn semantic concepts efficiently without hand-crafted augmentations.

Abstract: This paper proposes a scalable and straightforward pre-training paradigm for efficient visual conceptual representation called occluded image contrastive learning (OCL). Our OCL approach is simple: we randomly mask patches to generate different views within an image and contrast them among a mini-batch of images. The core idea behind OCL consists of two designs. First, masked tokens have the potential to significantly diminish the conceptual redundancy inherent in images, and create distinct views with substantial fine-grained differences on the semantic concept level instead of the instance level. Second, contrastive learning is adept at extracting high-level semantic conceptual features during the pre-training, circumventing the high-frequency interference and additional costs associated with image reconstruction. Importantly, OCL learns highly semantic conceptual representations efficiently without relying on hand-crafted data augmentations or additional auxiliary modules. Empirically, OCL demonstrates high scalability with Vision Transformers, as the ViT-L/16 can complete pre-training in 133 hours using only 4 A100 GPUs, achieving 85.8\% accuracy in downstream fine-tuning tasks. Code is available at https://github.com/XiaoyuYoung/OCL.

One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation
Xiaoyu Yang'
Lijian Xu
Hongsheng Li
Shaoting Zhang" 24
Abstract
1. Introduction
Self-supervised learning (SSL) key approach for build- ing world models, especially for pre-training vision models (Chen et al., 2020a; He et al., 2022; Bao et al., 2021; Rad- ford et al., 2021; Wang et al., 2023; Yang et al., 2024b), Its strength lies in learning versatile visual representations without relying on human annotations. Currently, the two main paradigms in visual SSL are Masked Image Mod- cling (MIM) (He al., 2022: Wang et al., 2023; Kong et al., 2023; Zhang et al., 2022; Gupta al., 2023) and Contrastive Learning (CL) (Chen et al., 2020a). Both have shown strong scalability, particularly for Vision Transform- ers (ViTs) (Dosovitskiy et al., 2021).
This paper proposes a scalable and straightfor- ward pre-training paradigm for efficient visual conceptual representation called occluded image contrastive learning (OCL). Our OCL approach is simple: we randomly mask patches gener- ate different views within image and contrast them among mini-batch of images. The core idea behind OCL consists of two designs. First, masked tokens have the potential to significantly diminish the conceptual redundancy inherent in images, and create distinct views with substan- tial fine-grained differences on the semantic con- cept level instead of the instance level. Second, contrastive learning is adept at extracting high- level semantic conceptual features during the pre- training, circumventing the high-frequency inter- ference and additional costs associated with image reconstruction. Importantly, OCL learns highly semantic conceptual representations efficiently without relying hand-crafted data augmenta- tions or additional auxiliary modules. Empiri- cally, OCI. demonstrates high scalability with Vision Transformers, as the ViT-L/16 can com- plete pre-training in 133 hours using only 4 A100 GPUs, achieving 85.8% accuracy in downstream fine-tuning tasks. Code is available at https: / //github.ce hww.hh...oooccecc wwtpttww......./tcottttoooooaoee wwt.//iaaaooooo//oeoooaoa
Despite the success of existing methods, both MIM and CL struggle to achieve efficient visual representation. MIM focuses heavily on pixel-level reconstruction, which often prioritizes local details over high-level semantic concepts. Similarly, CL suffers from conceptual redundancy, where transformed images may lack meaningful differences. This raises a critical question: How can we bridge the gap be- tween efficient visual representation and effective concep- tual pre-training, overcoming the limitations of current MIM and CL approackes?
To address this question, we first revisit the two main pre- training paradigms: MIM and CL. MIM learns visual rep- resentations by reconstructing masked image patches (see 1bg. Notable examples include BEiT (Bao et al., 2021) and MAE (He et al., 2022). MAE highlights that images contain significant semantic redundancy, meaning only ba- sic understanding objects and scenes is needed predict missing patches from their surroundings. However, pixel- level reconstruction is too detailed for pre-training vision models. It focuses excessively on high-frequene, details and local features, conflicting with the goal of pre-training: learning high-level semantic concepts. While this detailed task helps models learn visual representations, it comes at the cost of pre-training efficiency.
Xiaoyu Yang conducted this work during an internship at Shen- zhen University of Advanced Technology. 'Shenzhen University of Advanced Technology, Shenzhen, China 2 Centre for Percep- tual and Interactive Intelligence, the Chinese University of Hong Kong, China 'Department of Electronic Engineering, the Chi- nese University of Hong Kong, Hongkong, China 'SenseTime Research, Shanghai, China. Correspondence to: Lijian Xu <bmexlj@ mamexljj@mmil.comm gomi.
In CL, the core idea is simple: maximize agreement be- tween different views of the same image (see Fig. la). Pop- ular methods like SimCLR (Chen et al., 2020a;b), MoCo v3 (Chen et al.), and DINO (Caron et al., 2021) use com-
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Occlusion- Based Contrastive Learning for Efficient Visual Representation
plex pre-processing and auxiliary networks create distinct views of image. The main challenge lies in optimiz- ing the agreement between these views. For CL to work well, distinct views are essential. However, this is difficult because images often have conceptual redundancy. Addi- tionally, the dependency on large-scale batches to generate sufficiently diverse negative samples imposes stringent com- putational constraints, notably escalating computational and time consumption overhead during training.
level hand-crafted data augmentation to generate dis- tinct views, OCL diminishes the conceptual redun- dancy inherent in images and improves efficiency.
2.
Our approach eschews the reconstruction of masked images in favour of leveraging contrastive loss to steer the entire model. Independently of additional auxiliary modules, OCL is adept at extracting high-level seman- tic concept features from images more efficiently.
Driven by this analysis, we found that these two paradigms can complement each other: masked tokens have the po- tential to significantly diminish the conceptual redundancy inherent in images, whereas contrastive learning is adept at extracting high-level semantic features during the pre- training phase. Thus, we present novel and straightforward paradigm for self-supervised visual representation learning: occluded image contrastive learning (OCL). OCL tackles the above issues systematically: 1) Masked image tokens offer diverse views of a single image with substantial fine- grained conceptual differences. II) Contrastive learning enables pre-training to concentrate exclusively on the high- level semantic information contained within images while disregarding high-frequency redundancies. IIII The pro- posed paradigm obviates the need for auxiliary modules and expedites the efficient extraction of model features.
3. Extensive experiments are conducted to verify the effi- ciency and scaling capability of our method. ViT-L/16 can complete pre-training in 133 hours using only 4 A100 GPUs with 85.8% accuracy in fine-tuning. Ad- ditionally, we have structured ablation experiments to delve into the implications of different configurations within OCL, with particular focus on the need of the MLP head in contrastive learning.
2. Methodology
2.1. Architecture
With an input image IG E Hx it is reshaped into a sequence of 2D patches Tp E R R (PP..C)) where (H, W) denotes the original image resolution, C' is the number of channels, P represents the patch size, and N HW/ p2 indicates the number of patches. Subsequently, a linear projection is employed on Zp to transform it into D dimen- sions, yielding patch embeddings x E NxD. Following the MoCo v3 (Chen et al.), the linear projection is initial- ized using the Xavier uniform method and remains fixed throughout pre-training to mitigate potential instability in VIT caused by the large batch size. Thereafter, fixed position embeddings p E IR (g(N (1xD are incorporated into the patch embeddings to preserve positional information, employing sinusoidal positional encoding.
OCL has a particularly simple and straightforward work- flow, as presented in Fig. Ic. Here is how it works: Firstly, we mask a batch of images with a high rate, dividing vis- ible patches within one image into two oon-overlapping groups. In succession, the pre-train model extracts the fea- tures of these two groups of batch image tokens, respectively. Subsequently, contrastive learning is employed to predict the correct pairings for a batch of visible image tokens. Positive samples are different visible tokens in the same image, while negative samples are from different images of the mini-batch. Finally, inspired by T-distributed classi- fier (Yang et al.; 2024a), we the T-distributed spherical loss to constrain the inter-class margins in the pre-training. Comprehensive experiments demonstrate the scalability and efficacy of our approaches, where ViT-L/16 can complete pre-training in 133 hours using only 4 1100 GPUs and at- tain an 35.8% top-1 accuracy in fine-tuning classification. In particular, our model stands out from other pre-trate greitrann methods as it operates without the need for auxiliary mod- ules or hand-crafted data augmentation to generate diverse views.
A fttrr random masking patches, visible patches that retain the original image position information are divided into two non-overleaping groups: zy E RT*D and IT E R where n denotes the number of visible patches. Each group adds an independent [CLS] token Tets E RRD to aggregate the information of each group. Moreover, [CLS] tokens of each group will add the position embeddings Po- In succes- sion, ViT (Dosovitskiy et al., 2021) is utilized as our encoder. zy = : ① p and 21 = [x 2 are the input of pre-training ViT, where @ denotes element-wise plus, and f() represents the ViT. Consequently, image feature tokens extracted by ViT are symbolized as y = f(2), where y/ E (=+1) Furthermore, we only preserve [CLS] to- kens Yete E R of both groups for contrastive learning, as it aggregates the high-level semantic information of each image. It is noteworthy that, unlike existing methodologies, we abstain from employing auxiliary modules. This allows our model to more easily extract image features and reduce
In summary, our paper mainly makes the following contri- butions:
1. We endeavour to explore an alternative of using masked images to create diverse views with fine-gran -heggrained con- ceptual differences for contrastive learning. By forgo- ing the conventional approach of employing instance-
2

---

Declusion-Banee Contrastive Learning for Efficient Visual Representation
Agreement
Reconstructed
Image
Head
[3.9]
Model
Decoder
Model
Botch Occluded
Tokens (Non overlapping)
Model
D-1
Aegneesaion
Batch Images
DrU.
Dr
Batch Leaf Tokens
Contrastive Learning
Image
Masked Image
(a) CL
(b) MIM
(c) Our occluded image contrastive learning
Figure 1. Comparison between different pre-training paradigms. The Model in blue is the pre-training model, and the orange modules indicate auxiliary modules. (a) Contrastive Learning (CL) endeavours to maximize agreement between different views of an image. (b) Masked Image Modeling (MIM) aims restore masked image patches. (c) Our occluded image contrastive learning: Through non-overlapping occluding, distinct tokens within an image are categorized as intraclass, while across-image tokens within batch are viewed interclass. Our objective is to enhance intraclass compactness and interclass separability through contrastive learning approach. Just as single leaf can tell the coming of autumn, we believe that small area the image contains the majority of the meaning of the entire image.
the consumption of computing resources.
semantics. The concepts within an image exhibit distinct conceptual characteristics, yet they are all interconnected with the overall meaning of the image, albeit to varying extents. Thus, masked images are used to create views that contain semantic distinctions of concepts, and mitigate the conceptual redundancy present in images, as presented in Fig. 2.
Globoal Mosking
Contrastive Views with Diffrrent grained
We apply the masking strategy of random sampling. the same the MAE, which samples random patches without replacement following the uniform distribution. Beyond that, a low masking ratio (e.g. 30%) is implemented overall, but leads to higher masking ratio (e.g. 65%) for indi- vidual contrastive branches, as illustrated in Fig. The low global masking rate helps the model capture the over- all image structure and the relationships between different concepts within mini-batch. During each forward pass, the model only a small portion of the visible tokens. This creates diverse views with fine-graines semantic differ- ences, making the task more challenging and reducing the risk of the model relying on simple patterns or redundant information. The high masking ratio for each contrastive branch ensures that the task cannot be solved by simply ex- trapolating from basic image transformations. This reduces conceptual redundancy and forces the model to focus on localized features. Additionally, by training only few parts of the image, the masking strategy also improves scal- ability and efficiency, making it easier to handle the large batch sizes required for contrastive learning.
Carpet, Lend, Furry
Leg. Ear. Body
Disagreement
Agreement
Dogg
Eye, Ear, Head, Body
Ear, Body, Background
Figure 2. A toy example of masked images for conceptual con- trastive learning. The low global masking ratio aids the model in capturing comprehensive information from the image and un- derstanding the interconnectedness of various concepts within mini-batch. Beyond that, each contrastive branch has a higher masking ratio, generating diverse views with different semantic concepts for contrastive learning and diminishing conceptual re- dundancy within the image.
2.2. Occluded Image for Conceptual Pre-training
Despite the strides made in instance-wise contrastive leam- ing (Chen et al., 2020a; Caron et al., 2021), the differences between various views primarily manifest at the pixel level, instead semantic concept disparities. In this context, a patch is abstracted concept containing fine-graine
To divide visible patches into two separate, con-overlapping groups,
random sampling strategy similar the masking approach described earlier. This helps avoid poten-
3

---

Occlusion- Bassed Contrastive Learning for Efficient Visual Representation
tial biases and ensures that the central positions of the visi- ble patches are consistent across both groups. Meanwhile, non-overlapping patches present a challenging scenario, im- peding the model from relying solely on analogous patches for inference.
Algorithm I Pytorch-like pseudo-code for the core of an implement of OCL
(BB, DJ patch embeddings
# mask image and split into two non-overleaping groups
2.3. Efficient Contrastive Learning
# x [BX2, n, L]
x masking (x, ratio 0.3)
We randomly select mini batch of B instances and estab- lish the contrastive prediction task on the visible token pairs extracted from this mini-batch, yielding a total of x B data points. Feature tokens from different groups within the same image are treated as positive pairs, while tokens from different images in the same mini-batch are treated negative pairs. We concurrently maximise the similarity of B positive pairs while minimizing the similarity of B B negative examples to drive the network.
# extract [CLS] token of both groups using pre-training model
model (x) # [Bx2, 1, L]
x.reshape (-1, 2, x.shape (-1])
# L2 normalize the [CLS] token of each group
m, n = x(;, 0], x[;,1] [B, L] m m/m. norm (dim=-1, ( keepdim=True) n norm (dim=-1, xeepdim=True) # compute the scaled pairwise T-SP similarities
In terms of similarity computation, we introduce T- distributed spherical (T-SP) metric (Yang et al.; Kobayashi) to significantly promote the intraclass compactness and interclass separability of features. Given [CLS] tokens of' RDD and vf E R of both non-overlapping groups, the cosine distances between sf and ff are:
sim mn = compute tSP (m @ n.T)
simmn So (n @ m.T)
# symmetric loss function labels torch. torch.arange (B)
loss= (F. cross entropy (sim mn, labels) +F #F.cross.entrope (sim.nm, labels) /2
y T y
cosLv ,,f)
lo llf1l
and the T-SP similarity is defined follows:
based on the MAE code, mainly involving the process subse- quent to the acquisition of image feature embeddings from the encoder. First, we randomly mask a subset of embed- ded patch tokens with a low masking ratio. In succession, listed tokens are shuffled randomly and divided into two non-overlapping groups. Following MAE (He et al., 2022), within positional and [CLS] embeddings, lists of tokens are encoded by the VIT. It is noteworthy that we obtain the encoded [CLS] token from the ViT directly, without the
1 COSTU
Lyl w 0.5 1+ (1 COSLU)
(2)
where 2 0 denotes the concentration hyperparameter of T-SP metric. As к decreases, the similarity function becomes more condensed, where only two tokens in close proximity are deemed positive examples. Besides, we add a trainable temputare parameter 7 to effectively scale the different samples. Thus, the loss function for a positive pair is:
incorporation of additional auxiliary modules, even a linear head or lightweight decoder.
Subsequently, the obtained [CLS] tokens within each group are I.2 normalized, and then the T-SP metric is applied to calculate the similarity between tokens from each group. Finally, simple cross-entropy loss is calculated symmet- rically to drive model training, enhancing the intraclass conceptual compactness within an image and the interclass semantic separability across images.
Clul .
exp (sim ( v/.) x T)
(3)
lo
Ex=1 If [eae exp (simrsp (u/ v/) x T)
where If ((++))) E {0, }} is an indicator function evaluating to I if k a i. Finally, inspired by CLIP (Radford et al., 2021), we optimize a symmetric loss over these similarity scores within a mini-batch exhibited in Algorithm I.
3. Experiments
We perform self-supervised pre-training on the ImageNet- IK (Russakovsky et al., 2015) dataset with the resolution of 224x224. By default, ViT-B/16 and ViT-L/16 (Dosovitskiy et al., 2021) are leveraged as the backbone architecture with 800 epochs for pre-training and 40 epochs for warm-up.
2.4. Simple Implementation
The implementation of our OCL pre-training is efficient and involves minimal specialized operations. As pseudo code depicted in Algorithm 1, we make only minor modifications

---

Occlusion- Based Contrastive Learning for Efficient Visual Representation
Model
Blocks
Dim
Heads
Params
VIT-8/14
ViT-B/16
12
768
12
86M
VIT-1116
VITAL/16
ViT-L/16
24
1024
16
304M
Linear Probing Top-1 Accuracy
78
(200
VIT-R116
VVE--1/14
ViT-h/16
32
1280
16
632M
76-
Table 1. Configurations of Vision Transformer (Dosovitskly al., 2021) models in our experiments. Block denotes the her of transformer blocks, with dim representing the input/output channel dimension of all blocks. Hcads the number of heads in multi-hear attention modules. We also provide the parameter sizes of different models.
VTT-8/11
VVT-B/16
72-
70-
I-JEPA
MoCo
ViT-B/16 applies the overall masked ratio of 0.3, while ViT- L/16 is set 0.4. The initial base learning rate is 1.5 10-4 Like other contrastive learning (Chen et al., 2020a; Radford et al., 2021), our method relies on large effective batch size: 9,600 for ViT-B/16 and 2,048 for ViT-L/16. Besides, R is set to 64 in the T-SP metric for computing similarity by default. Our implementation is based on MAE (He et al., 2022), with further details provided in the Supplementary Material.
68
VTT-8-16
OCL
500
1000
1500
2000
Pre-training GPU Hours
Figure 3. Efficiency and Scaling, MAE (Il et al., 2022), I-JEPA (Assran et al., 2023) and MoCo v3 (Chen et al.) are opted for comparison. All methods Top-1 accuracy (Acc) as the metric, and the pre-training GPU time with A100 hour as indicator. The pre-training epochs (denoted as ep) and model architecture are also exhibited.
evaluated hy linear prohing with
In terms of supervised validation, OCL is evaluated through end-to-end fine-tuning and linear probing on the ImageNet- Ik dataset for classification with 100 epochs for ViT-B/16 and 50 for ViT-L/16, following common practices (Caron et al., 2021; He et al., 2022; Wang et al., 2023). Top-1 accuracy is utilized to verify the performance of different methods.
Compared to ViT-B/16, OCL achieves nearly 4% improve- ment in linear probing with ViT-L/16, surpassing MoCo v3. It implies that efficiently train larger models to achieve better performance, within acceptable timeframe.
3.2. Ablation Studies
3.1. Scalability
3.2.1. MASKED RATIO
To demonstrate the scalability of our OCL for efficient con- ceptual pre-training, we access the efficiency and scaling of our model in Fig.3. It illustrates the pre-training hours related to various model sizes for different methods, with linear probing accuracy. OCL is highly scalable compared to previous methods, requiring less computational resources while achieving strong results and without relying on hand- crafted data augmentations. Unlike reconstruction based approaches like MAE and I-JEPA, OCL needs fewer training epochs and avoids the need for pixel-level reconstruction, significantly improving training speed. In contrast to con- trastive learning methods such as MoCo v3, which depend on handcrafted augmentations and complex architectures to generate and process multiple image views, OCL eliminates the need for auxiliary modules like momentum encoders. This simplicity makes OCL's framework more efficient and accelerates pre-training. For example, when scaling up from ViT-B/16 to ViT-L/16, OCL requires far less additional pre- training time compared to MoCo v3.
Firstly, we conduct ablation experiments discuss the im- pact of the overall masked ratio the performance of con- ceptual pre-training, revealed in Table 2. To ensure clarity in presentation, we offer the visible ratio of each contrastive branch within a single forward pass of the mini-hatch, which is a crucial factor for the cross- cross-similaris similarity within our OCL framework. Likewise, the effective batch size is intertwined with the scalable masked ratios, with larger batch sizes show- casing performance enhancements for the model, verified in Section 3.2.2. Thus, we attribute this improvement to the masked ratio.
Regarding large models such as ViT-L/16 Table 2, the increment of the overall masked ratio from 0.2 to 0.4 could significantly diminish conceptual redundancy and increase the effective batch size, thereby resulting in performance improvement. Nevertheless, with continued rise in the masked ratio, the visible patches of images in one forward process of the model diminish incrementally. Despite in- creases batch size, the precise extraction of conceptual information from the images becomes compromised, result- ing in degraded performance. Furthermore, as displayed
Scaling model size. Moreover, our model leverages a scal- able model size, resulting in more substantial performance enhancements with the larger model as illustrated in Fig.3.
5

---

Occlusion- Based Contrastive Learning for Efficient Visual Representation
Overall Masked Ratio
Forward Visible Ratio
reveals that a larger effective batch size correlates with im- proved model performance and efficiency, aligning with the consensus within the community (Goyal et al., 2018; You et al., 2019), We attribute this to the statistical advantage of expanded negative sample pools, which achieve broader coverage of the latent feature space. Such comprehensive sampling sharpens inter-class separation by refining deci- sion boundaries during contrastive optimization. However, limited by computational resources, we are unaware of the maximum capacity of the effective batch size.
Bsz.
LIN
FT
ViT-1/16
0.2
0.4
1,024
74.2
84.4
0.4
0.3
2,048
77.9
85.8
0.8
0.1
7,200
69.1
83.0
ViT-B/16
0.3
0.35
9,600
74.2
83.4
0.6
0.2
9,600
71.0
82.8
0.8
0.1
12,800
65.7
81.6
3.2.3. MLP HEAD IS NOT YOU NEED
Table Ablation evaluation experiments on masked ratio. The results are based on the ImageNet- KK with the Vit-B/16 and ViT- L/16. All methods are evaluated by linear probing (LIN) and fine-tuning (FT). We provide the visible ratio of each branch for contrastive learning according to the masked ratio. Correspond- ingly, give different effective batch sizes (Bsz.) related the overall masked ratio. The resolution of images is fixed to 224x224. Top-1 accuracy is used as the metric.
MLP Head
FLOPs
Param.
Hours
Bsz.
LIN
FT
w/o
42.3
303
533
2,048
77.9
85.8
w/o
42.3
303
559
1,800
77.4
85.7
2-layer 3-layer
43.6
305
600
1,800
76.7
85.6
43.7
306
611
1,800
76.6
85.6
Table 4. Ablation experiments on MLP head. The results are based on the ImageNet- 1K with the ViT-L/16. All methods are evaluated by FLOPs/G, Parameters (Param. VM, pre- pre-training hours (Hours), effective batch size (Bsz.), linear probing (LIN) and fine- tuning (FT). Correspondingly, we give different effective batch sizes related to the MLP head within the pre-trained model. Be- sides, pre-training hours on A100 are provided. The resolution of images is fixed to 224x224. Top-1 accuracy is used as the metric.
in ViT-B/16 of Table 2. once the batch size surpasses its threshold, an excessively high masked ratio can impair the model performance, transitioning from reducing conceptual redundancy damaging essential semantic information.
3.2.2. LARGE BATCH SIZE FOR EFFECTIVE CONTRASTIVE REPRESENTATION
Effective Batch Size
Pre-training
In many contrastive learning methods (Chen et al., 2020a; Chen et al.), the ViT model is typically paired with an MLP head (Taud & Mas, 2018) to assist with the pretext task. The ViT learns semantic features from the image, while the MLP head handles the pretext classification task, en- hancing the ViT's learning capabilities, shown in carlier work. However, traditional contrastive learning methods rely on handcrafted data augmentations to generate and pro- cess multiple image views at the instance level. In contrast, OCL explores an alternative approach: using masked im- ages to create diverse views with fine -raigrriiddd conceptual differences the token level. As a result, OCL does not require MLP head for instance-level classification tasks.
LIN
FT
Hour
2,048
77.9
85.8
533
4,096
77.7
85.7
533
1,800
77.4
85.7
559
1,024
74.4
84.9
586
512
71.6
84.6
613
Table Ablation evaluation experiments on batch size. The results are based on the ImageNet-1K with the VIT-L/16. All methods are evaluated by pre-training hours, linear probing (LIN) and fine-tuning (FT). Mcanwhile, pre-training hours on A100 are provided. The resolution of images is fixed to 224x224. Top-1 accuracy is used as the metric.
The ablation experiments on the MLP head involve test- ing different configurations, including a 2-layer MLP, a 3-layer MLP, and no MLP head (see Table 4). The de- sign of the MLPs follows MoCo v3 (Chen et al.), where the hidden layers of both the 2-layer and 3-layer MLPs are domensimelsi and employ the GELU activation function (Hendrycks & Gimpel, 2023). The output layers of both ML.s are 512-dimensional and do not use GELU. Addition- ally, all layers in the MLPs incorporate Batch Normalization (BN) (loffe, 2015), consistent with the methodology in Sim- CLR (Chen et al., 2020b). Due to GPU memory constraints, the batch size was adjusted to 1,800 for validation. From
VIT models (Dosovitskiy et al., 2021) are inherently com- putationally intensive, and training with large batches is a preferred strategy for handling large ViT models. Moreover, sizable batch size is advantagcous for achieving accuracy in contemporary self-supervised learning techniques. In particular, concerning contrastive learning methodologies that heavily lean on large batch sizes, ablation experiments are conducted to ascertain the influence of batch size on our occluded image contrastive learning approach, as shown in Table 3. The utilization of the ViT-L/16 model for validation
6

---

Occlusion- Baased Contrastive Learning for Efficient Visual Representation
the fine-tuning results presented in Table 4, OCL capable of operating effectively without MLP head. Unlike tra- ditional contrastive learning approaches, the absence of an MLP head does not degrade the model's performance. On the other hand, the inclusion of additional auxiliary modules, such as deeper MLP heads, increases both the cost and time required for pre-training. However, the benefits of these modules do not outweigh the performance gains achieved by simply increasing the batch size.
ever, if kappa becomes too large, the model converges slowly, which negatively impacts the overall effectiveness of the pre-training process.
3.3. Comparisons with previous results
Aug.
Ep.
FLOPs
Param.
LIN
FT
Masked Image Modeling
BEiT
w/o
800
17.6
87
83.2
T-SP Loss
MLP Head
LIN
FT
MAE
w/o
1,600
17.5
86
68.0
83.6
CAE f
w/o
1,600
17.5
86
70.4
83.9
77.0
85.5
I-JEPA
w/o
600
17.5
86
72.9
V
72.4
85.1
Contrastive Learning
X
77.9
85.8
X
X
61.3
82.6
DINO
w/
1,600
74.7
171
78.2
82.8
MoCo v3
w/
600
74.7
171
76.7
83.2
Table 5. Ablation experiments on the relationship between ML.P head and T-SP loss. The results are based on the ImageNet- IK with the ViT-L/16. All methods evaluated by linear probing (LIN) and fine-tuning (FT). The resolution of images is fixed to 224x224. Top-1 accuracy is used as the metric.
Masked Image Modeling with Contrastive Learning
SiameseIM
w/
1,600
16.3
88
78.0
84.1
ccMIM
w/o
800
39.1
86
68.9
84.2
ConMIM
w/
800
17.5
86
85.3
MiXMAE
w/o
600
15.6
88
61.2
84.6
iBOT
w/
1,600
17.5
86
79.5
Besides, Table provides the ablation experiments about the relationship between the MLP head and T-SP loss. It demonstrated that T-SP loss is utilized to constrain the similarity distributions between two views, enforcing more elaborate feature extraction and representations through ex- plicit probabilistic margin compression. In contrast, the MLP head operates through implicit feature projection into a lower-dimensional space. While their mechanisms differ, both components contribute to efficient representations in pre-trained models.
OCI
w/o 800
12.0
86
74.2
83.4
Table 7. Comparison with previous methods on ImageNet-1K classification with VTT-B model. All methods are evaluated by Aug., Epoch (Ep.), FLOPs/G, Parameters (Param.J/M, linear prob- ing (LIN) and fine-tuning (FT). The resolution of images is fixed to 224x224. Aug. indicates the utilization of handerafted view data augmentation during pre-training. FL.PPs/G is utilized to show the runtime and computational resources the pre-training. Param./M is caleulated for the encoders of the pre- pre-training model, following the MixMAE. denotes the results are copied from the MixMAE. Top-1 accuracy (Acc) is used the metric.
3.2.4. CONCENTRATION OF SIMILARITY METRIC
kappa
4
16
32
64
128
To demonstrate OCL's ability to learn high-level conceptual representations without relying on handcrafted data augmen- tations, we compare its performance on linear probing and fine-tuning tasks under pre-training on ImageNet- lk Im- portantly, while masked images are used to reduce semantic redundancy, OCL avoids reconstructing masked images. In- stead, it uses contrastive loss to guide the network's learning process. As result, OCL is categorized contrastive learning method.
FT
84.8
85.1
85.6
85.8
85.5
Table Ablation experiments on concentrate parameter of kappa. The results are based on the ImageNet- IK with the ViT- L/16. Fine-tuning (FT) results are provided. The resolution of images is fixed to 224x224. Top-1 accuracy is used as the metric.
Following the approach of T-SP (Yang et al.), we conduct ablation experiments to evaluate how different concentra- tions of the T-distributed adapter affect the pre-training model's performance. The results are shown in Table 6. This study involves five specific degrees of concentration, namely x 4, 16, 32, 64, and 128. From the Table 6, we can infer that kappa increases, the pre-training perfor- mance of the model improves progressively until reaching K = 64 with the fine-tuning result of 85,8 under ViT-L/16. We explain that as kappa increases, the model must extract more precise semantic concepts from positive samples, cre- ating more challenging pretext task for pre-traine pre-training. How-
Table.7 exhibits the performance of our method under the fine-tuning and linear probing for ImageNet- Ik classifica- tion. Of greater significance, our FLOPs of 12.0 exceed those of the second-best method MixMAE (Liu al., 2023a) with 15.6 G, improving efficiency by about 23%. Concern- ing DINO (Caron et al., 2021) and MoCo v3 (Chen et al.), they leverage student-teacher dual networks to pre-train, leading to higher FLOPs and Parameters of the pre-training encoder. Our OCL method has demonstrated outstanding performance contrastive learning, notably obviating the
7

---

Occlusion-Bane Based Contrastive Learning for Efficient Visual Representation
need for intricate augmentations across multiple views. It validates the practicality and efficacy of masked images for generating diverse views encapsulating distinct fine- grained semantic concepts, which diminishes conceptual re- dundancy and expedites the conceptual pre-training process. Beyond that, it also stands out the significant competency of contrastive learning in extracting high-level semantic concepts.
robustness of our method.
Concerning the fusion of MIM and CL, significant endeav- ours (Tao et al., 2023; Yi et al., 2023; Zhou et al., 2022) are made to enhance pre-training performance on down- stream tasks. However, they result in inefficiencies due to increased training iterations, augmented data, auxiliary mod- ules, and diverse loss combinations. For instance, Siame- seIM (Tao et al., 2023), ConMIM (Yi et al., 2023) and iBOT (Zhou et al., 2022) leverage both hand-crafted data augmen- tation and more training epochs. Despite our model also integrating masking strategy with contrastive learning, it does not rely on hand-crafted view data augmentations and additional auxiliary modules, reconciling the divergence between efficient visual representation and effective concep- tual pre-training. Besides, our model OCL also achieves competitive results on downstream tasks of linear probing and fine-tuning, and more importantly, shows promising scaling behavior.
COCO
ADE20k mloU
Methods
Epochs Apb Apm
ViT-B/16
Supervised (Tao et al., 2023)
300
47.9
42.9
47.4
DINO (Caron et al., 2021)
800
50.1
43.4
46.8
iBOT (Zhou et al., 2022)
1600
51.2 44.2
50.0
DenseCL (Wang et al., 2021)
400
46.6 41.6
44.5
MoCo-v3 (Chen et al., 2021b)
600
47.9 42.7
47.3
BEiT (Bao al., 2021)
800
49.8 44.4
47.1
MAE (He et al., 2022)
400
50.6 45.1
45.0
Beyond that, we conducted supervised fine- tne-ttunig on the COCO dataset for object detection and instance segmenta- tion using the Mask RCNN (He et al.) framework, with our pre-trained encoder serving as the backbone. We follow the setup of MixMAE (Liu et al., 2023a), with the window size to 16 x 16 to align with the 1024 x 1024 input image reso- lution. In terms of semantic segmentation on the ADE20k dataset, We use the UperNet (Xiao et al.) framework with our pre-trained encoder as its backbone, with the changed window size as mentioned above. The results are shown in Table 8, that our model achieves competitive results com- pared to other models with efficient pre-training progress. Compared to the contrastive learning method of MoCo-v3, our method obtains 3.6% improvement of APh under ViT- B/16, especially without view data augmentations. Besides, the consistent performance gains on ADE20K semantic segmentation, achieving +7.1 mIoU improvements from ViT-B to ViT-L, empirically validate our method's scala- bility across model capacities while maintaining parameter efficiency
MAE (He et al., 2022)
1600 51.6 45.9
48.1
GreenMIM (Huang al., 2022)
800
50.0 44.1
EVVIT (Li et al., 2021a)
300
47.3
MixMAE (Liu et al., 2023a)
300
52.3 46.4
49.9
MixMAE (Liu et al., 2023a)
600 52.7 47.0
51.1
SiameseIM (Tao et al., 2023)
400
50.7 44.9
49.6
SiameselM (Tao et al., 2023)
1600
52.1 46.2
51.1
SimMIM (Xie et al., 2022)
300
51.1
45.4
48.9
OCL
800
51.5
45.5
46.1
ViT-L/16
MoCo-v3 (Chen et al., 2021b)
BEiT (Bao et al., 2021)
800
53.3
47.1
53.3
MAE (He et al., 2022)
1600
53.3
47.2
53.6
SimMIM (Xie et al., 2022)
800
53.8
53.6
MixMAE (Liu et al., 2023a)
600
54.3
48.2
53.8
OCL
800
53.2
47.0
53.2
Table 8. Comparison with previous methods on various down- stream tasks, including object detection and segmentation COCO and ADK20K. We report Ap (AP") and Ap (AP")) on COCO, and mIoU ADE20K. Arch. represents the model architecture, where VIT-B/16 and V/T-L/16 are utilized to validate the performance of various methods.
Moreover, we conduct experiments in the domain general- ization setting following the SiameselM (Tao et al., 2023), as shown in Table 9. Experiments are conducted on ImageNet- A (Hendrycks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a) and ImageNet-Sketch (Wang et al., 2019), with Ima- geNet (Russakovsky et al., 2015) as the source dataset. Our method achieves results with a competitive edge, demon- strating the advantages of our model in generalization and ro- bustness. Notably, our framework achieves competitive per- formance with significantly fewer training epochs than con- ventional contrastive approaches, demonstrating remarkable training efficiency while maintaining stable convergence- critical advantage for scaling large-scale datasets and complex architectures.
Compared to MIM methods, due to the absence of pixel- level image reconstruction in OCL, the training process is finished in 800 epochs, in contrast to the 1600 epochs re- quired for MAE (He et al., 2022), CAE (Chen et al., 2024). It corroborates the contribution of our methodology from the perspective of efficiency and high-level semantic con- cept extraction. Moreover, our OCL method is superior to BEiT (Bao et al., 2021) regarding the fine-tuning and linear probing results, also confirming the competency and
8

---

Occlusion- Based Contrastive Learning for Efficient Visual Representation
Methods
Epochs
IN-A
IN-R
IN-S
References
MSN (Assran et al., 2022b)
1200
37.5
50.0
36.3
Assran, M., Caron, M., Misra, I., Bojanowski, P., Bordes, F., Vincent, P., Joulin, A., Rabbat, M., and Ballas, N. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pp. 456-473. Springer, 2022a.
iBOT (Zhou et al., 2022)
1600
42.4
50.9
36.9
DenseCL (Wang et al., 2021)
400
30.8
43.8
29.9
MoCo-v3 (Chen et al., 2021b)
600
32.4
49.8
35.9
MAE (He et al., 2022)
1600
35.9
48.3
34.5
SiameseIM (Tao et al., 2023)
1600
43.8
52.5
38.3
Assran, M., Caron, M., Misra, I., Bojanowski, P., Bordes, F., Vincent, P., Joulin, A., Rabbat, M., and Ballas, N. Masked Siamese Networks for EffiieEff Learning. In Avidan, S., Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Computer Vision ECCV 2022, pp. 456-473. Springer Nature Switzerland, 2022b. ISBN 978- 3-031-19821-2 -22 doi: 10. 111007/978-3-03--19881111
OCL
800
42.2
52.3
37.6
Table Comparsion with previous methods on generalization capability and robustness on ImageNet-A (IN-A), ImageNet- (IN-R) and ImageNet-Sketch (IN-S) datasets. ViT-B/16 is utiliize as the backbone to validate the performance of various methods, T accuracy used as the metric.
Assran, M. Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-Supervised Learning From Images With a Joint- Embeddig Predic- tive Architecture. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
4. Conclusions and Outlooks
In this paper, we introduce OCL, novel, simple, and effec- tive pre-training paradigm for visual conceptual representa- tion. Our approach uses a masking strategy to generate di- verse views with fine-grained semantic differences, enabling contrastive learning to classify and learn agreements within mini-batch. This design eliminates the need for I) ad- dressing semantic conceptual redundancy within images, II) reconstructing images, III) hand-crafted data augmentations, and IV) additional auxiliary modules, thereby improving efficiency and scalability, Experiments showcase the effi- ciency and scalability of our method, yielding competitive results compared to previous approaches. Additionally, ab- lation experiments provide insights that could inspire future pre-training paradigms.
Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M. Data2vec: A general framework for self-supervised learning in speech, vision and language. In Interna- tional Conference on Machine Learning, pp. 1298- 1312. PMLR, 2022.
Baevski, A., Babu, A., Hsu, W.-N., and Auli, M. Efficient self-supervised learning with contextualized target repre- sentations for vision, speech and language. In Interna- tional Conference on Machine Learning, pp. 1416-1429. PMLR, 2023.
We hope that our work will inspire future advancements in contrastive pre-trnining paradigms, specifically efficient vi- sual representation. Our further work will focus on enhanc- ing computational efficiency for Illlion-parameter models without sacrificing representation quality.
Bao, H., Dong, L.,, Piao, S., and Wei, F. BETT: BERT Pre- Training of Image Transformers. In International Confer- ence Learning Representations, 2021. URL https: //openreview.r net htte//foummiccc BhZS25904.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging Prop- erties in Self-Supervised Vision Transformers. In Proceedings of the IEEF/CVF International Con- ference on Computer Vision, pp. 9650-9660, 2021. URL https: https://openaccees: wtw.s/cccccccst......coc www.ch. content hattea///ccco222//ht////rooooonn Emeraring___eeeinnn__ Properties in Sttprriiessii_SSsee-uueeeeeeeeee Transformers_icc___oe
Impact Statement
This paper presents work whose goal is to advance the field of machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Vision_
Acknowledgment
This research was partially supported by the Centre for Per- ceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK. Shaoting Zhang and Hongsheng Li are Principal Investiga- tors of the Centre of Perceptual and Interactive Intelligence (CPII) under the InnoHK
Chen, T., Korbbiihh, S., Norouzi, M., and Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th Interna- tional Conference on Machine Learning, pp. 1597-1607. PMLR, 2020a. URL ht https: /ttaa://proceeee /ppppoeeeeinggas.ceeeeepp.c..aae mllr. press/v119 chen20 htmll.

---

Occlusion-Bane Based Contrastive Learning for Efficient Visual Representation
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. Big Self-Supervised Models are Strong Semi- Supervised Learners, October 2020b.
He, K., Gkioxari, G., Dollar, P., and Girshick, R. Mask R-CNN. In Proceedings of the IEEE International
Conference on Computer Vision, pp. 2961-2969, URL (0tccc_201//hmmmm/ Staettmmeeeeeae o-t_
Chen, X., Xie, S., and He, K. An Empirical Study of Train- ing Self- SelerSuuee Vision Transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9640-9499, / thecvf thecof..com/contsen//ccvvo/////1 Chee__aih.Bmppttrraaaaasttetoesp
content_iccv_2011 IcCV_2017_papeer html.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 16000-16009, 2022. URL
Self sellf=-Sppervisseddiliiiooonnppe IcCV_2021_papezz html.
theoff.
com/ http//nnteet////332//t//////// Masked__as_d____ Auttomnoodera__rre__calabbbb. Learmers_cvPR__22__pppp
Chen, X., Xie, S., and He, K. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 9640-9649) 2021a.
Hendrycks, D. and Gimpel, K. Gaussian Error Linear Units (GELUs), June 2023.
Chen, X., Xie, S., and He, K. An Empirical Study of Training Self-Supervised Vision Transform- ers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 9640- 9649, 2021b. URL https::/epeeaccees thecvf. ctt.////oooeee////////////t/ Chen_ htte___w__mmpirrallsssttteeeorr_
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, E,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The Many Faces of Robustness: Critical Analysis of Proceedings of the IEEE/CVF International Confer- ence Computer Vision, pp. 8340-8349, 2021a.
Generalization. In
Self htt.-SSuerrraeeeessiioonntteaeee html.
URL https:// htwp:peennnccesssse.eeeeeeeeeeee com/ content//Tccv20211/120 conteent//ECc/000///hh////eeeet Ltt__Fccce____oouuuutssssss_CCC_ 2021 www.ppapr.htt.
Chen, X., Ding, M., Wang, X., Xin, Y., Mo, S., Wang, Y., Han, S., Luo, P., Zeng, G., and Wang, Context Autoencoder for Self-supervised Representation Learn- ing. International Journal of Computer Vision, 132 (1):208-223, January 2024. ISSN 1573-1405. doi: 10.10/ 1263-023-01852-44
Hendrycks, D.. Zhao, K., Basart, S., Steinhardt, J., and Song, D.. Natural Adversarial Examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b. URL. https::/ www.ccccsssss......cocoo.o thecvf. com/ http://ntenn//eeaa22////a////eee Natural hawaaaaadddveerrrrlaaaaaaappp//
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.. Heigold, G., Gelly. S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021. URL http:
2021 papl_ppper: html.
//arxiv. org/ abs 1110111192200
Huang. L.,, You, S., Zheng, M., Wang, F,, Qian, C., and Yamasaki, T. Green Hierarchical Vision Transformer for Masked Image Modeling. 35:19997-20010, 2022.
Gao, P., Lin, Z., Zhang, R., Fang, R., Li, H., Li, H., and Qiao, Y. Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking. 132
(5):1546-1556 2024. ISSN 1573-1405. doi: 10.1007/ $11263-023-011888-4 URL https: www...org....../oooooo 1007/s1122 023 0003--1800 01898-4.
Huang, Z., Jin, X., Lu, C., Hou, Q., Cheng, M.-M., Fu,
D., Shen, X., and Feng. Contrastive masked autoen- coders stronger vision learners. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Goyal, P., Dollár, P., Girshick, R. Noordhuis, P., Wesolowski, I.., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, Large Minibatch SGD: Training ImageNet in Hour, April 2018
loffe, S.
Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv: 1502.0316. 2015.
Gupta, A., Wu, J., Deng, J., and Fei-Fei, L. Siamese Masked Autoencoders, 2023. URL /arpiv hrgg/abs/en a30//305.... 2305. 14344.
Jiang, Z., Chen, Y., Liu, M., Chen, D., Dai, X., Yuan, L., Liu, Z., and Wang. Z. Layer Grafted Pre-training: Bridg- ing Contrastive Learning And Masked Image Modeling
10

---

Occlusion-Basee Contrastive Learning for Efficient Visual Representation
For Label-Efficient Representations. In The Eleventh International Conference Learning Representations, September 2023.
Efficient Pretraining of Hierarchical Vision Transform-
In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 2023a. URL https::/openacceeeet Tteerft..oo//oottte///////22///2 Mi MixMAE MixEAE LiAaMhAMMueuuannnnlsseeeetttttot for for__fficieen__reerraiiiig.. html.
Jing. L.., Zhu, J., and LeCun, Masked siamese con- vnets: Towards an effective masking strategy for general- purpose siamese networks.
Heerarchical
Kobayashi, T. T-vMF Similarity For Regularizing Intra-Class Feature Distribution. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 6612-6621. IEEE. ISBN 978-
Liu, W., Chen, Y., Yue, X., Zhang, C., and Xie, S. Safe multi-view deep classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8870-8878, 2023b.
1-66544-509-2. doi: 10.1099CVVP44447.20011 00655. URL http::ss///eeeepprreeee. ieee. org/ (ocument/9578211
Liu, W., Chen, Y., and Yue, X. Building trust in decision with conformalized multi-view deep classification. In ACM Multimedia 2024, 2024.
Kong, L., Ma, M. Q., Chen, G., Xing, E. P., Chi, Y., Morency, L.-P., and Zhang, K. Understanding Masked Autoencoders via Hierarchical Latent Variable Models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Putlern Recognition, pp. 7918-7928, 2023. URL https: https:// .tw.aaaacccsss......eeeeeeeeeeee thecvf. can//ooneen/////RR/////////////
Liu, W., Chen, Y., and Yue, X. Enhancing multi-view clas- sification reliability with adaptive rejection. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 18969-18977, 2025.
understandinggMsskkeLLLtttoonon
Mishra, S., Robinson, J., Chang, H., Jacobs, D., Sarna, A., Maschinot, A., and Krishnan, D. A simple, efficient and scalable contrastive masked autoencoder for learning visual representations. arXiv preprint 100v:2220011670. 2022.
vi iia_Hieeracchiccl_aaateette Models_CVPR_200 200e____vR_22333ppper.....
Li, C., Yang. J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., and Gao, J. Efficient Self-supervised Vi- sion Transformers for Representation Leamiing In In- ternational Conference on Learning Representations, 2021a. URL id=fVu3o YUGQK.
Qi, Z., Dong, R., Fan, G., Ge, Z., Zhang, X., Ma, K., and Yi, L. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In International Conference on Machine Learning, pp. 28223 28243.. PMLR, 2023.
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., and He, K. Scaling eegguage-image pre-training via masking. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13390-234400 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Trans- ferable Visual Models From Natural Language Super- vision. In Proceedings of the 38th International Con- ference on Machine Learning, pp. 2021. URL / http://ppcceeee.eee/ee/e mlr. p.l..spss//// v139 11339/adford21e1 html.
Li, Z., Chen, Z., Yang, F., Li, W., Zhu, Y., Zhao, C., Deng, R., Wu, L., Zhao, R., Tang, M., and Wang, J. MST: Masked Self-Supervised Transformer for Visual Repre- sentation. In Advances in Neural Information Process- ing Systems, volume 34, pp. 13165-13176. Curran Asso- ciates, Inc., 2021b.
PMIR,
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.. Microsoft coco: Common Objects in Context. In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), Computer Vision ECCV 2014, Lecture Notes in Computer Science, pp. 740-755. Springer International Publishing, 2014. ISBN 978-3-319-106022 doi: 10. 10.100997----319-21 10602-1-1 48.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., and Bernstein, M. Imagenet large scale visual recognition challenge. International journal of computer vision, 115 (3)2211225,, 2015.
Tao, C., Zhu, X., Su, W., Huang, G., Li, B., Zhou, J., Qiao, Y., Wang, X., and Dai, J. Siamese Image Modeling for Self-Supervised Vision Representation Learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 2132-2141, 2023.
Liu, J., Huang, X., Zheng, J., Liu, Y., and Li, H.
MixMAE: Mixed and Masked Autoencoder for

---

Occlusion-Basse Contrastive Learning for Efficient Visual Representation
Taud, H. and Mas, J. Multilayer Perceptron (MLP). In Camacho Olmedo, M. TT, Paegelow, M., Mas, J.-F., and Escobar, F. (eds.), Geomatic Approaches for Modeling Land Change Scenarios, pp. 451-455, Springer Interna- tional Publishing, Cham, 201 8. ISBN 978-3-319-6098 doi: 10.
Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Computer Vision ECCV 2022, pp. 523-539, Cham, 2022. Springer Nature Switzerland. ISBN 978-3- 031-19803-8 doi: 10.
Yang, J., Guo, S., Wu, G., and Wang, L. CoMAE: Sin- gle Model Hybrid Pre-training on Small-Scale RGB-D Datasets, February 2023a.
Wang, H., Ge. S.. Lipton, Z., and Xing, E. P. Learn- ing Robust Global Representations by Penalizing Local Predictive Power. In Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https: httpttpesp/ppcccediiggg.s neurips. seuzipes.c//aaper//////asss Beeffccc800776644452222233622223 html.
Yang. X., Chen, Y., Yue, X., Xu, S., and Ma, C. T-distributed Spherical Feature Representation for Imbalanced Classi- fication. 37(9):10855-00833
ISSN 2374-3468 doi: 10. (009/aaii.v3779.2200 26284. URL httttttttt/ssssaaeeeee /AAAAAAAIAA
Yang, X., Lu, J., and Yu, E. Adapting multi-mode large language model to concept drift from pre-training wards. arXiv preprint arXiv:2405. /3459, 2024a. URL haaps://aaxiv.../ wttpxvv..rrr/ab//20011...0
Wang, H., Fan, J., Wang. Y., Song, K., Wang, T., and Zhang, Z.-X. DropPos: Pre- Training Vision Transformers by Re- constructing Dropped Positions. 66:46134-466
2023.
Wang, X., Zhang, R., Shen, C., Kong, T., and Li, L. Dense Contrastive Learning for Self-Supervised Visual Pre- Training. pp. 3024-3033, 2021. URL https:
Yang, X., Xu, L., Sun, H., Li, H., and Zhang, S. Enhancing visual grounding and generalization: A multi-task cycle training approach for vision- models. arXiv preprint arXiv:2311.12322 2024b. URL https:// a arxiv.orgh 2tt.//1111.11311
Propenaccess.theete cttp://ff..oo//oonttt////
CwPR2022//htmm//aang_eeness contrneaeeeeeeecooorrrttteieeee
Stt.rrsssees--spperisseeeeeeee Visuall Pre- Pre-Training_
Yang, X., Lu, J., and Yu, E. Adapting multi-modal large language model to concept drift from pre-training on- wards. In The Thirteenth International Conference on Learning Representations, 2025a. URL https: / //openzeview.co net wtt.:/rrrddddd22222----
pttp1__ppperemlll. html.
Wei, Y., Gupta, A., and Morgado, P. Towards latent masked image modeling for self-supervised visual representation learning. In European Conference on Computer Vision, pp. 1-17. Springer, 2025.
Yang, X., Lu, J.,, and Yu, E. Walking the tightrope: Disentan- gling beneficial and detrimental drifts non-stationary custom-tuning. arXiv preprint arXiv:2505. 13081, 2025b. URL https: arxiv. .rg. /ttg//b/////// 200.. 13081.
Wu, Z., Lai, Z., Sun, X., and Lin, S. Extreme masking for learning instance and distributed visual representations. arXiv preprint ar arXiv:2206.0466
2022.
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified Perceptual Parsing for Scene Understanding. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 418 434. URL https: openaccess. thecvf. cwm http///ontttcccceeeeeeeeee LB____cV___0000 1 8 //html/
Yang, X., Lu, J., and Yu, E. Causal-informed contrastive learning: Towards bias-resilient pre- training under con- cept drift. arXiv preprint arXiv:2502 2025c. URL https: / //axxivvvv.//////////a././aaa orw. abs
Litee_iao__Uniie Bnaio__nniieed_ePeeptttaaall Per:eptual_Parsingg ECCV_2018_ htmm. html.
Yang, Y., Huang, W., Wei, Y., Peng. H.. Jiang, X., Jiang, H., Wei, F., Wang, Y., Hu, H., Qiu, L., al. Attentive mask clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2771-2781, 2023b.
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q. and Hu, H. SimMIM: A Simple Framework for Masked Image Modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022. URL http::ttp//tc httpt::psp///peaccssss.sss thecvf. com/ http://cnceen//c//22///ttttl pp. otw..//Se.mpeeeeraaerrrr...ooooo
Yi, K., Ge, Y., Li, X., Yang, S., Li, D., Wu, J., Shan, Y., and Qie, X. Masked Image Modeling with Denoising Contrast. In The Eleventh International Conference on Learning Representations, September 2023.
You, Y., Li, Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsich, C.-J. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. In International Conference on Learning Representations, September 2019.
Image_Moder httpe_/ooooeeiigg_______022200_0
Yang, J., Chen, H., Liang, Y., Huang, J., He, L., and Yao, ConCL: Concept Contrastive Learning for Dense Pre- diction Pre-training in Pathology Images. In Avidan, S.,
12

---

Occlusion-Bare Based Contrastive Learning for Efficient Visual Representation
Zhang, Q.. Wang, Y., and Wang, Y. How Mask Matters:
Towards Theoretical Understandings of Masked Autoen- coders. 35:27127-271399 2022.
Zhang, S., Zhu, F., Zhao, R., and Yan, J. Contextual Image Masking Modeling via Synergized Contrasting without View Augmentation for Faster and Better Visual Pre- training. In The Eleventh International Conference Learning Representations, September 2023.
Zhao, Y. Wang, G., Luo, C., Zeng, W., and Zha, Z.-J. Self-Supervieed Visual Representations Learning by Contrastive Mask Prediction. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 10160-10169. URL https:
// LccV202//Etti///aaaaaa-----rrree
Vaaaal_Bppeeaattttcooo__eeeraaa Contrrass.vee.Mekk.Prrccctooooo pam.r.html. html.
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene Parsing Through ADE20K Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 633-641. URL https: http///ppaaccssccceooccc thevff......ooc com/ hattent_orpr_2011/httmh/hhouu Berrshng__hcoogtt...PPP20111pppe
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A. and Kong, T. Image BERT Pre-training with On- line Tokenizer. In International Conference on Learning Representations, October 2022.
13

---

Occlusion-Bal Based Contrastive Learning for Efficient Visual Representation
A. Related Works
The remarkable success of large-scale models across diverse domains has catalyzed a paradigm shift in machine learning (Yang et al., 2025b;a), with pretraining emerging as a cornerstone methodology. However, visual representation learning remains constrained by prohibitive computational requirements compared to its NLP counterparts, leading to more research concerning the efficiency of the visual mre-training. In this section, will review the three main vision pre-training methods, namely masked image modeling, contrastive learning and their combination. Besides, the difference between ours and theirs is also provided.
A.I. Masked Image Modeling
Inspired by masked language modeling in NLP, the core of masked image modeling is to predict the masked part of the input image. Among them. BEiT (Bao et al., 2021) tokenizes image patches through the reconstruction of the individual image using dVAF, and then predicts the tokens of the masked patches to learn visual representation. Similarly, MAE (He et al., 2022) utilizes a high masked ratio (75%) to corrupt the image and directly reconstruct the pixel-level masked image patches. Subsequently, numerous studies have referenced this paradigm for pre-training endeavours, including DropPos (Wang et al., 2023), U-MAE (Zhang et al., 2022) and CAE (Chen et al., 2024). DropPos (Wang et al., 2023) incorporates position reconstruction to bolster the spatial awareness of ViTs. U-MAE (Zhang al., 2022) introduces uniformity loss regularization to the MAE loss to further encourage the feature consistency of the pre-training, and addresses the dimensional feature collapse. CAE (Chen et al., 2024) decouples the learning processes for image representation and pretext tasks, enabling the pre-trained model to prioritize image representation while disregarding the pretext task. Additionally, following the successful implementation of MAE, its applicability has been extended to diverse disciplines, such as SiamMAE (Gupta et al., 2023) and MR-MAE (Gao et al., 2024). Similarly, I-JEPA (Assran et al., 2023) predicts the feature encoding of the contextual region via MIM, and employs contrastive learning to align the features of neighbouring regions the feature level.
Besides, latentMIM (Wei et al., 2025) tackles key training challenges in Latent MIM, showcasing its ability to produce spatially diverse, high-level semantic representations. Context Autoencoders (Chen et al., 2021a) employ an encoder-decoder framework optimized with a combined reconstruction loss and alignment constraint, ensuring predictable representations of missing patches. data2vec (Baevski et al., 2022) predicts representations of missing patches using an online target encoder, eliminating the need for handcrafted augmentations and achieving strong performance across vision, text, and speech modalities. Its successor, data2vec-v2 (Baevski al., 2023), further explores efficient architectures for multimodal learning.
A.2. Contrastive Learning
Among these methods, SimCLR (Chen et al., 2020a) relies on complex pre-processing techniques to create distinct views of an image, aiming to make the task challenging enough to learn effective visual representations during pre-training. However, requires large batch sizes generate negative sample pairs, leading to long training times and high computational costs. DINO (Caron et al., 2021) uses a
architecture to extract visual representations from different views. Similarly, MoCo v3 (Chen et al.) employs momentum updates to optimize its auxiliary network. These methods highlight that the core of contrastive learning lies in creating views with significant differences. However, this is challenging due to the inherent semantic redundancy in images. Additionally, ConCL (Yang et al., 2022) generates distinct concepts by cropping images and applies contrastive learning within a teacher-student framework, specifically for pre-training on pathological images.
DenseCL (Wang et al., 2021) introduces the concept of dense contrastive loss, which calculates the contrastive loss between dense feature vectors generated by the dense projection head the local feature level, contrasting with traditional contrastive learning methods that operate at the global feature level. Moreover, MaskCo (Zhao et al.) utilizes the teacher-student network for generating image features of query and keys. The key features of different images are used to query certain images to generate positive and negative examples for contrastive learning. Furthermore, MSN (Assran et al., 2022b) leverages data augmentation to create two views known as the anchor view and the target view. Subsequently, a random mask is applied to the anchor view, aligning the representation of the masked anchor view with the clusters of the unmasked target view. Besides, ResilientCL(Yang et al., 2025c) leverages causal inference to analyze bias sources in contrastive pre-training's momentum updates and proposes a causal interventional objective to address distribution shifts. Moreover, building vision-specific uncertainty principles, multi-stage framework (Liu et al., 2023b; 2024; 2025) employs learnable concept lattices to stabilize pretraining under covariate shift.
14

---

Occlusion- BBased Contrastive Learning for Efficient Visual Representation
A.3. Combination between MIM and CL
There are also lot of efforts bridge the gap between mask image modeling and contrastive learning, where the integration of teacher-student models emerges as the prevailing approach. iBOT (Zhou et al., 2022) employs the teacher-student network to independently encode the two augmented views, with the student network processing masked images. The objectives of MIM and CL are jointly trained for self-distillation. Likewise, MST (Li et al., 2021b) introduces a masked token strategy leveraging multi-head self-attention maps, which selectively mask the tokens of the student network based on the output self-attention map of the teacher network, ensuring vital foreground remains intact. Similarly, SiameseIM (Tao et al., 2023) employs a Siamese network featuring two branches. The online branch encodes the initial view and predicts the representation of the second view based on their relative positions. Meanwhile, the target branch generates the target by encoding the second view. MSCN (Jing et al.) generates multiple augmented views from input images and applies random masking. These masked views are encoded using standard ConvNet, with representations optimized through a joint-embedding loss. RECON (Qi et al., 2023) integrates generative and contrastive learning paradigms via ensemble distillation, where generative student model guides a contrastive student to unify both upproaches. CMAE (Huang et al., 2023) employs a dual-branch architecture: an online branch with an asymmetric encoder-decode for reconstructing masked images, and a momentum branch with a momentum encoder for contrastive learning on full images. This design enables holistic feature learning and enhanced discrimination.
ccMIM (Zhang et al., 2023) leverages a contrastive loss aid the reconstruction task regularizer, facilitating the extraction of image-wide global information from both masked and unmasked patches. Likewise, ConMIM (Yi et al., 2023) produces simple intra-image inter- titrrpptch contrastive constraints as the sole learning objectives for masked patch prediction, and strengthens the denoising mechanism with asymmetric designs to improve the network pre-training. Additionally, CoMAE (Yang et al., 2023a) also applies CL to assist cross-modal MIM tasks. Besides, LGP (Jiang et al., 2023) integrates MIM and CL in a sequential cascade manner: early layers are first trained under one MIM loss, on top of which latter layers continue be trained under another CL loss.
A.4. Differences of OCL from Existing Methods
Masking has been adopted as an effective data augmentation technique to enhance training efficiency in several studies (Li et al.. 2023; Yang et al., 2023b; Mishra et al., 2022: Wu et al., 2022; Assran et al., 2022a). Previous studies have demonstrated its effectiveness in improving model performance and training efficiency (Li et al., 2023; Yang al., 2023b). ExtreMA (Wu et al., 2022) employs random masking computationally efficient augmentation for Siamese representation learning, accelerating learning and enhancing performance on large datasets. MSN (Assran et al., 2022a), the most relevant work to ours, generates two image views: a masked anchor view and an unmasked target view, aiming to cluster their representations. CAN (Mishra et al., 2022) integrates contrastive learning, masked auto-encoding, and diffusion denoising into a unified framework. Our approach distinguishes itself from previous hybrid methods by achieving superior performance with a better performance-efficeen
trade-off. Specifically, our masking strategy generates semantically diverse views and leverages contrastive learning promote classification agreement within mini-batches. This approach eliminates: (I) semantic redundancy, (II) image reconstruction, (III) hand-crafted augmentations, and (IV) additional auxiliary modules, resulting in enhanced efficiency and scalability.
B. Experiments
In this section, we first introduce the utilized datasets for pre-training and various downstream tasks. Subsequently, implementation details are provided. Finally, we present more results on downstream tasks, such as object detection, segmentation and domain generalization.
B.1. Datasets
The ImageNet- Ik dataset (Russakovsky et al., 2015) is a widely used image dataset consisting of 1.28M labeled images across 1 categories, with 50K validation images and 100k test images. The dataset has been instrumental in advancing computer vision research by providing large-scale benchmark for image classification tasks. By leveraging the vast amount of labeled images in ImageNet-Ik, Ik, self-supervised models can learn rich representations of visual data in an unsupervised manner, which can then be fine-tuned on downstream tasks with smaller labeled datasets. Besides, ImageNet-A (Hendrycks et al., 2021b), ImageNet-R (Hendryeks et al., 2021a) and ImageNet- Skeett (Wang et al., 2019) are leveraged for validation
15

---

Occlusion-Base Contrastive Learning for Efficient Visual Representation
of the generalization capability and robustness of the vision model, with the training source of ImageNet.
MSCOCO (Lin et al., 2014) dataset is a large-scale dataset widely used for object detection and instance segmentation tasks created by Microsoft Research Asia. The COCO Detection dataset contains more than 330K images, including more than 1.5M labeled object instances for 80 different categories. Each object instance is labeled with category, bounding box. and segmentation mask information.
The ADE20K (Zhou et al.) semantic segmentation dataset comprises over 20K scene-centric images for 150 semantic categories, meticulously annotated with pixel-level object and object parts labels.
B.2. Implementation Details
We employ ViT-Base and ViT-Large as our visual backbones, respectively, Among them, Vit-Base consists of 12 transformer encoder layers and an FFN intermediate size of 3,072. The hidden dimensions of the ViT-Base are 768, with 12 attention heads. The number of parameters is about 86 million. The input image size is set to 224 x 224. In terms of ViT-I/16, ViT-L/16 consists of 24 transformer encoder layers and an FFN intermediate size of 4,096. The input image size is set to 224 x 224, with patch size of 16 16. The hidden dimensions of the ViT-Large are 1 1,024, with 16 attention heads. And, the number of parameters is about 307 million.
Table 10. The pre-training hyperparameters.
Table 11. The fine-tuning hyperparameters.
ViT-B/16
ViT-L/16
VIT-B/16
VIT-L/16
Training Epochs
800
800
Training Epochs
100
50
Warmup Epochs
40
40
Warmup Epochs
5
Optimizer
AdamW
A
Optimizer
AdamW
AdamW
Base Learning Rate
1.5e-4
1.5e-4
Base Learning Rate
5e-4
1e-3
Learning Rate Decay
Cosine (0.9, 0.95)
Cosine
Learning Rate Decay
Cosine (0.9, 0.95)
Cosine
Adam
(0.9, 0.95)
Adam B
(0.9, 0.95)
Weight Decay
0.05
0.05
Weight Decay
0.05
0.05
Batch Size
9,600
2,400
Batch Size
1.024
1,024
In terms of the pre-training progress, the hyperparameters are presented in Table 10. We utilize the A optimizer, which is configured with a cosine annealing schedule as the learning policy. The initial base learning rate is set to 1.5 x 10-1, and the AdamW optimizer is employed with hyperparameters = (0.9, 0.95). Additionally, we set the weight decay to 0.05 without dropout. We use the strategy of cosine learning rate decay, with 40 warm-up epochs. Unless otherwise specified, the pre-training of our vision language model consists of 800 epochs, executed on 2 NVIDIA A100 GPUs.
Concerning the downstream tasks of fine-tuning and linear probing ImageNet, the hyperparameters are shown in Table 11 and Table 12.
Table 12. The linear probing hyperparameters.
ViT-B/16
ViT-L/16
Training Epochs
90
50
Warmup Epochs
10
10
Optimizer
LARS
LARS
Base I Learning Rate
0.1
0.1
Learning Rate Decay
Cosine
Cosine
Weight Decay
0.0
0.0
Batch Size
16,384
1,024
16


Reviews:
[{"review_id": "PTenEtWpW5", "reviewer": "Reviewer_FSbF", "summary": "This work proposes a contrastive self-supervised training method that relies on masking. After a global masking step, two nonoverlapping sets of patches are selected to create two views which are then aligned to each other and contrasted to all views of other samples in the minibatch. Noticeably, no contrastive MLP-head is used. Despite its simplicity regarding architecture and augmentations, the proposed OCL method performs reasonably well both in fine-tuning and linear probing evaluations. Additionally, the authors report a significant decrease in training duration compared to multiple other efficient methods, e.g. MAE, I-JEPA. The method applies a loss (T-distributed spherical  metric) that introduces an additional concentration parameter.\n\n----\n\nupdate after rebuttal: The authors have addressed some of my concerns and answered the questions. Therefore, I will keep the positive score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful and detailed review. Your positive feedback serves as a great source of encouragement for our team, especially recognizing the strengths: 1) efficient and simple framework, 2) not relying on specific masking strategies and 3)  higher level of semantic abstraction than standard MIM methods. Furthermore, we have diligently responded to each of your questions and incorporated your feedback into our manuscript revisions:\n\n1. **(Weakness 1) efficiency about epochs and views generations** Thanks very much for detailed and thoughtful review. We sincerely apologize for the typo in Table 6, that we actually used 800 epochs for pre-training, which is consistent with our Table 7, Table 8 and the description in line 380 of the paper. **We have corrected this typo and double-checked the entire manuscript to make sure there are no further typos.**\n\n    Besides, we are sorry for the misunderstanding of the efficiency of views generation. **We do take into account the time it takes to generate two views.** As illustrated in Algorithm 1, we conduct generations of two views in torch module MiCLAutoencoderViT (code in models_mae.py) with CUDA acceleration, instead of image pre-processing using CPU like DINO and MoCo v3. Thus, our method has considered the efficiency of views generation. And **we have clarified this misunderstanding in our revised manuscript.**\n\n2. **(Weakness 2) runtime and training epochs.** Many thanks for your kind suggestions. To better demonstrate the efficiency of our model, we conduct the table following the MixMAE.\n\n    https://s2.loli.net/2025/03/31/RfKjqYMtAxoVQHC.png (Same Table as in Reviewer 2yVR, sorry for limited chars.)\n\n    **From the table, our FLOPs of 12.0 G significantly exceed those of the second-best method, MixMAE (15.6 G), improving efficiency by about 23%.** Concerning DINO and MoCo v3, they leverage student-teacher dual networks to pre-train, leading to higher FLOPs and Parameters of the pre-training encoder. **We have revised the paper and added this table to illustrate the efficiency of our method more clearly.**\n\n3. **(Weakness 3 & Question 2) the sources of the speedup.** We sincerely appreciate your kind and insightful suggestions. The acceleration of our method primarily stems from two key factors:\n    - First, **the masking strategy significantly reduces the number of tokens processed by the ViT for a single image.** With a ratio of 0.4, only a part of tokens participate in pre-training, dramatically decreasing the computational load required by the ViT.\n    - Second, **the contrastive framework eliminates the need for additional modules to reconstruct the image.** Traditional MIM methods and most MIM-CL hybrid approaches employ a transformer decoder for image reconstruction, which incurs substantial computational overhead. In contrast, our OCL requires neither a decoder nor an MLP head, thereby reducing pre-training computations and enhancing efficiency.\n\n    As shown in Table of Answer 2, **our method requires significantly less training time per epoch compared to MAE and I-JEPA.** Specifically, OCL achieves 12.0G FLOPs, surpassing MAE of 17.5G by approximately 31% through elimination of decoder dependency. Moreover, I-JEPA’s higher FLOPs (29.8 G) result from its dual encoders (context and target) to generate different views. **We have revised the paper to provide a more detailed explanation of these acceleration mechanisms.**\n\n4. **(Weakness 4) frozen patch layer.** Sincerely thanks for your thoughtful and helpful suggestions. We did experience training failures. While **we were more concerned about the efficiency of the model's pre-training**, so we simply changed the random seed and retrained. **We will continue to explore solutions to model failures in future research.**\n\n5. **(Question 1) global masking strategy.** Thank you for valuable suggestions. **The global masking strategy is designed to enhance representation learning by addressing semantic redundancy in images.** As highlighted by the MAE and discussed in line 32 of our paper, images inherently carry redundant semantics. Global masking prunes unnecessary patches, forcing the model to abstract high-level semantic patterns. **We have revised the paper to provide a detailed explanation of the global masking strategy.**\n\n6.  **(Question 3) the relationship between MLP head and loss.** Thanks for valuable suggestions. Though distinct in mechanism, both T-SP loss and MLP head enhance visual representations. **We have added additional ablation experiments and related discussion to validate the relationship between MLP head and T-SP loss.** \n\n    | T-SP Loss | MLP Head | LIN| FT|\n    |--|--|--|--|\n    |√|√|77.0|85.5|\n    |x|√|72.4|85.1|\n    |√|×|77.9|85.8|\n    |×|×|61.3|82.6|\n\nThanks again for your valuable time and careful review. Your feedback is immensely valuable to us. If you find our response satisfactory, could you please consider helping us improve our rating score?", "claims_and_evidence": "Claims are that OCL is a highly efficient SSL method that performs nearly on-par with SOTA methods. Evidence is provided in various benchmark evaluations.", "methods_and_evaluation_criteria": "Imagenet classification evaluation is the common way to evaluate self-supervised methods. Additionally, semantic segmentation and object detection performance is evaluated on ADE20k and COCO, respectively. Robustness is evaluated on a few ImageNet-like benchmarks (A,R and S). Overall, a solid set of evaluation experiments.\n\nAblation and sensitivity studies are mainly performed on ImageNet1k.", "theoretical_claims": "no theoretical claims.", "experimental_designs_or_analyses": "I checked the performance and ablation experiments and did not find any issues.", "relation_to_broader_scientific_literature": "OCL is another small step towards improved SSL training efficiency and less reliance on hand-crafted augmentations.", "essential_references_not_discussed": "none.", "other_strengths_and_weaknesses": "Strengths:\n- Efficent and simple. The result without MLP head is quite astounding.\n- Only needs very basic augmentations, like MAE. Masking is performed randomly, which is a strength compared to methods that rely on specific masking strategies that are tailored to Imagenet, e.g. I-JEPA.\n- Like other joint embedding methods, it is shown that the learned features already have a higher level of semantic abstraction than standard MIM methods.\n\nWeakness:\n- Results are good, but not ground breaking. Them main argument for OCL is its reported efficiency. The number of reported epochs varies between 800 (line 380) and 1600 (Table 6.). Unless the authors already take the 2 generated views into account, and report effective epochs, this does not fit. Furthermore, there is not a single table that holds both, runtime and training epochs.\n- Given the similarity in effective masking ratio to MAE and the need to create 2 views, the source of the speedup is not that easy to explain. see questions.\n- Using the strategy from MoCo V3 to freeze the patch layer reduces the number of learnable parameters and has shown benefits with regard to overfitting. On the other hand it suffers from the same problem like e.g. max-pooling, it often works well, but sometimes fails. No study about the effect of random initialized untrainable patch creation is performed.", "other_comments_or_suggestions": "none.", "questions_for_authors": "Why do you apply this global mask, why is it applied to all samples in a mini-batch. Does it have efficiency reasons or does it benefit learning?\n\nLooking at the masking ratios and the number of epochs, I cannot spot the source of the speedup compared to MAE. Compared to I-JEPA it is even harder to find the edge of OCL. You mention the simplicity of the architecture and no MLP-head, is an effective epoch (sample times views) of OCL that much faster to train than an epoch of MAE or I-JEPA? \n\nThere are studies about the concentration parameter, but no ablation about the loss itself. How is the loss and the ability to not use MLP heads related?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "16D4FD4Q3t", "reviewer": "Reviewer_4wwQ", "summary": "The paper introduces Occluded Image Contrastive Learning (OCL), a novel self-supervised learning (SSL) paradigm for efficient visual representation. OCL combines the strengths of Masked Image Modeling (MIM) and Contrastive Learning (CL) by using random masking to create diverse views within an image and contrasting them within a mini-batch. The key innovation lies in generating fine-grained semantic differences through masking, which reduces conceptual redundancy and avoids the need for hand-crafted data augmentations or auxiliary modules. The authors demonstrate that OCL is highly scalable, achieving competitive results on downstream tasks like ImageNet classification, object detection, and segmentation, while significantly reducing pre-training time and computational resources.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review, especially for summarizing the strengths of our work, including 1) well-motivated method 2) well-written paper and easy to follow and 3) comprehensive ablation studies. In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly.\n\n1. **(Weakness 1) computational costs.** Thanks very much for your detailed review. Concerning traditional contrastive paradigms such as MoCo v3 and DINO, they use student-teacher dual networks as two branches to process distinct views, leading to almost double the computation cost for one image. However, **we did not use two networks as two branches, while we utilized two different parts of an image as two branches.** Thus, two branches of contrastive learning in our method will not bring additional computation costs. Cooperating with the masking strategy, we can further reduce the amount of computation for an image. Moreover, our model does not depend on an additional transformer decoder to reconstruct the image, leading to less computation. **In summary, we use the above contributions to significantly reduce the computational costs of the pre-training paradigm and improve the efficiency.**\n\n     - To better demonstrate the efficiency of our model, we conduct the table following the MixMAE.\n\n        https://s2.loli.net/2025/03/31/RfKjqYMtAxoVQHC.png (Same Table as in Reviewer 2yVR, sorry for limited chars.)\n\n        **From the table, our method achieves 12.0 G FLOPs, surpassing the second-best method (MixMAE at 15.6 G) and reducing computational costs by approximately 23%.** \n    - Moreover, **we have presented Figure 3 in our manuscript to illustrate the efficiency and scaling ability of our model, and discussed the computational costs in line 248.** OCL is highly scalable compared to previous methods, requiring less computational resources while achieving comparable and competitive results and without relying on handcrafted data augmentations.\n        https://s2.loli.net/2025/03/31/VqHwXUrMgpcGTt1.png\n    - Furthermore, **we have also conducted ablation experiments on mask ratio and discussed it in section 3.2.1. MASKED RATIO of our manuscript.** The results demonstrate that a lower masking ratio (0.4) optimally balances computational efficiency and visual representations.\n        https://s2.loli.net/2025/03/31/VfcqPNk7SXuoIjO.jpg\n    - Besides, **We also provide the runtime with FLOPs and parameters for Table 4** to better illustrate the ablation of the MLP head from the efficiency perspective.\n    | MLP Head | FLOPs(G) | Param. (M) | Pre-training Hours | Eff. Bsz. | LIN  | FT   |\n    |--|--|--|--|--|--|--|\n    | w/o      | 42.3 | 303 | 559 | 1,800 | 77.4 | 85.7 |\n    | 2-layer  | 43.6 | 305 | 600 | 1,800 | 76.7 | 85.6 |\n    | 3-layer  | 43.7 | 306 | 611 | 1,800 | 76.6 | 85.6 |\n\n    **We have added the discussion about the computational costs of two branches in our contrastive paradigm, and added the Table of pre-training details of the ViT-B model to show the runtime and effective pre-training epochs, and revised our Table 4 to add FLOPs and Parameters.** Thanks for your constructive suggestions.\n\n2. **(Weakness 2) fine-tuning and linear-probing results.** Many thanks for your sincere review. We are sorry for the misunderstanding of our theme in this paper. Our goal is to provide a new pre-training paradigm for efficient visual representation with affordable training time and reasonable computation cost. **Actually, our method reduces computational costs while maintaining efficiency.** From the FLOPs comparison in the table above, we can see that our FLOPs of 12.0 G significantly exceed the second-best MixMAE with 15.6 G, **improving the efficiency by about 23%.** With such significant improvement in efficiency, our method achieves comparable and competitive results of fine-tuning and linear-probing to previous SOTA contrastive learning and MIM methods. **We have added a more detailed discussion about the fine-tuning and linear-probing results to clarify the misunderstanding.** Thanks for your sincere review.\n\n3. **(Weakness 3) detection and segmentation results.** Thanks for your comments. We found it is quite similar to the weakness 2. We provide a variety of downstream tasks to show that our pre-trained model has the generalization ability competitive with previous SOTA methods. **However, our core contribution lies in proposing a simple pre-training paradigm that balances efficiency and performance, no bells and whistles.** We hope it helps us train larger vision models and replicate the success of LLM. **We have also added a discussion about the relationship between downstream tasks and our theme.** Thanks for your sincere review.\n\nWe appreciate your time and thorough review. Your feedback is highly valuable to us, and we welcome further communication from you. If you are satisfied with our response, we would be grateful for your support in enhancing our rating score.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed method is well-motivated.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "I believe the experiments are not convincing enough.", "relation_to_broader_scientific_literature": "The paper builds on existing work in Masked Image Modeling (MIM) (e.g., MAE, BEiT) and Contrastive Learning (CL) (e.g., SimCLR, MoCo v3), but distinguishes itself by combining the strengths of both paradigms without relying on hand-crafted augmentations or auxiliary modules.", "essential_references_not_discussed": "This paper has considered most of the relevant works as far as I know.", "other_strengths_and_weaknesses": "Strength:\n\n1. The paper is well-written and easy to follow.\n\n2. The ablation studies are comprehensive.\n\nWeaknesses:\n\n1. In line 134, only a low masking ratio is adopted, and the contrastive learning requires two branches, which may take much more computational costs?  However, no effective training epoch or running time is provided in Table 4 for fair comparisons.\n\n2. Although more computational costs are consumed, both the fine-tuning and linear-probing results are comparable to previous SOTA contrastive learning and MIM methods.\n\n3. The detection and segmentation results on COCO and ADE20k datasets are also lower than previous SOTA methods.", "other_comments_or_suggestions": "It's better to provide training wall-clock time or computational costs of the proposed methods.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWOuWL2Jkh", "reviewer": "Reviewer_2yVR", "summary": "The work propose occlusion-based contrastive learning with masked image modeling approach. It compares against iBot, MAE, I-JEPA and other relevant SSL methods. Achieves competitive results with less time needed for training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your sincere review, especially for summarizing the strengths of our work, including 1) interesting and well-written work, 2) clear presentation of the claims and confirmed by the results, 3) novel training method of occlusion that is easy to follow and 4) good experiments with common benchmarks, additional promising ablations and generalization validations. In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly.\n\n\n1. **(Comment & Question) occlusion time.** Thanks so much for your sincere and constructive suggestions. Different from the standard augmentations used in traditional contrastive pre-training, we use the torch module with CUDA acceleration to implement the occlusion operations for our method. **Quantitatively, our whole model has a floating point of operations (FLOPs) of 12.03 G, while the operation of occlusion only has 0.12 G FLOPs, accounting for about 1%.** It is confirmed that occlusion operations take up very few computation resources and runtime but bring great efficiency improvements.\n\n    Besides, **to better demonstrate the efficiency of our model, we conduct the table following the MixMAE to illustrate the comparison with previous methods on ImageNet-1K classification with ViT-B model.** All methods are evaluated by Aug., Epoch (Ep.), FLOPs/G, Parameters (Param.)/M, linear probing (LIN) and fine-tuning (FT). The resolution of images is fixed to 224×224. Aug. indicates the utilization of handcrafted view data augmentation during pre-training. FLOPs/G is utilized to show the runtime and computational resources of the pre-training. Param./M is calculated for the encoders of the pre-training model, following the MixMAE. $\\dagger$ denotes the results are copied from the MixMAE. Top-1 accuracy (Acc) is used as the metric.\n\n    |                                                       | Aug. |  Ep.  | FLOPs (G) | Param. (M) |  LIN  |   FT  |\n    |-------------------------------------------------------|:----:|:-----:|:---------:|:----------:|:-----:|:-----:|\n    | **Masked Image   Modeling**                           |      |       |           |            |       |       |\n    | BEiT $\\dagger$                                        |  w/o |  800  |   17.6    |     87     |   -   | 83.2  |\n    | MAE $\\dagger$                                         |  w/o | 1,600 |   17.5    |     86     | 68.0  | 83.6  |\n    | CAE $\\dagger$                                         |  w/o | 1,600 |   17.5    |     86     | 70.4  | 83.9  |\n    | I-JEPA                                                |  w/o |  600  |   17.5    |     86     | 72.9  |   -   |\n    | **Contrastive Learning**                              |      |       |           |            |       |       |\n    | DINO                                                  |  w/  | 1,600 |   74.7    |     171    | 78.2  | 82.8  |\n    | MoCo v3                                               |  w/  |  600  |   74.7    |     171    | 76.7  | 83.2  |\n    | **Masked Image   Modeling with Contrastive Learning** |      |       |           |            |       |       |\n    | SiameseIM                                             |  w/  | 1,600 |   16.3    |     88     | 78.0  | 84.1  |\n    | ccMIM                                                 |  w/o |  800  |   39.1    |     86     | 68.9  | 84.2  |\n    | ConMIM                                                |  w/  |  800  |   17.5    |     86     |   -   | 85.3  |\n    | MixMAE $\\dagger$                                      |  w/o |  600  |   15.6    |     88     | 61.2  | 84.6  |\n    | iBOT $\\dagger$                                        |  w/  | 1,600 |   17.5    |     86     | 79.5  |   -   |\n    | OCL                                                   |  w/o |  800  |   12.0    |     86     | 74.2  | 83.4  |\n\n    **From the table, our method achieves 12.0 G FLOPs, surpassing the second-best method (MixMAE at 15.6 G) and reducing computational costs by approximately 23%.** It is verified that our occlusion operation achieves considerable enhancement of efficiency.  **We have revised our paper and added more detailed descriptions about occlusion time.** Thanks for your sincere suggestion.\n\n\nThank you once more for generously dedicating your time to provide a thoughtful review. Your feedback is tremendously valuable, and we are open to hearing from you at any time. If you find our response satisfactory, we would greatly appreciate your assistance in improving our rating score.", "claims_and_evidence": "clearly stated and confirmed by the results\n\nOne of them is usage of occlusion into MIM training, which is well designed for MIM framework rather than manual augmentation selection.   The OCL is able to extract better high level concepts.", "methods_and_evaluation_criteria": "correct and appropriate\n\nmethods based on contrastive learning and MIM are commonly used, however using occlusion for training is novel\n\nStandard benchmark are used such as ImageNet and architecture such as ViT-L/16. Additionally tasks such as linear probining and fine tuning are reported. Additional benchmarks on COCO and ADEK are used.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "following the good practices and other works\n\nCommon benchamrks such as ImageNet, ADEK, COCO and tasks such as fine tuning and linear probing. Additional ablations on times needded to train are provided and looks promising. Additionally generalization capabilites are tested on different variants of ImageNet.", "relation_to_broader_scientific_literature": "provided extensively, including SimCLR, iBot, I-JEPA and MIM.", "essential_references_not_discussed": "it is discussed properly", "other_strengths_and_weaknesses": "it is interesting and well written work, clear presentation of the claims and methodology that is easy to follow", "other_comments_or_suggestions": "a discussion of how much time is needed for occlusion in this setup? Not whole pretraining, but I would like to see what kind of fraction of time in pretraining is used on occlusion when compared to standard augmentations.", "questions_for_authors": "how much time is needed for occlusion in this setup in training compared to non-occluded images?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presented a method for visual representation learning with occlusion-based contrastive learning. A new view selection method was proposed  Experiments on downstream tasks show the effectiveness and efficiency of the proposed method.

This paper received a consistent recommendation, with *3 Weak Accept* from the reviewers. The reviewers acknowledge the proposed simple idea and its efficiency, and raised concerns about experimental details, limited performance over previous methods, and other technical issues.
The authors provided a rebuttal, which addressed the major concerns raised by the reviewers.

Overall, this paper presented an interesting idea, with contributions that could be of interest to the audience at ICML. The AC is happy to recommend it being accepted to present at the conference.
Decision: Accept (poster)