Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations
Authors: Ziyu Jiang
Yinpeng Chen
Mengchen Liu
Dongdong Chen
Xiyang Dai
Lu Yuan
Zicheng Liu
Zhangyang Wang
Venue: ICLR 2023 poster (2023)
Keywords: Mask Image Modeling
Contrastive learning

Abstract: Recently, both Contrastive Learning (CL) and Mask Image Modeling (MIM) demonstrate that self-supervision is powerful to learn good representations. However, naively combining them is far from success. In this paper, we start by making the empirical observation that a naive joint optimization of CL and MIM losses leads to conflicting gradient directions - more severe as the layers go deeper. This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer. Inspired by experimental observations, we find that MIM and CL are suitable to lower and higher layers, respectively. We hence propose to combine them in a surprisingly simple, ``sequential cascade'' fashion: early layers are first trained under one MIM loss, on top of which latter layers continue to be trained under another CL loss. The proposed Layer Grafted Pre-training learns good visual representations that demonstrate superior label efficiency in downstream applications, in particular yielding strong few-shot performance besides linear evaluation. For instance, on ImageNet-1k, Layer Grafted Pre-training yields 65.5% Top-1 accuracy in terms of 1% few-shot learning with ViT-B/16, which improves MIM and CL baselines by 14.4% and 2.1% with no bells and whistles. The code is available at https://github.com/VITA-Group/layerGraftedPretraining_ICLR23.git.


LAYER GRAFTED PRE-TRAINING: BRIDGING CON- TRASTIVE LEARNING AND MASKED IMAGE MODEL- REPRESENTATIONS
ING FOR
Ziyu Jiang ; Yinpeng Chen', Mengehen Liu', Dongdong Chen', Xiyang Dait,
Lu Yuan', Zicheng Liu', Zhangyang Wang
Texas A&M University *Microsoft University of Texas Austin
ji aanzi edu, at attlasaang@uteess .
yy sheche mengeliu, dochen, xidai, luyuan, 2 eliu/emlczosoft.com com
ABSTRACT
Recently, both Contrastive Learning (CL) and Mask Image Modeling (MIM) demonstrate that self- superviser is powerful to learn good representations. How- ever, naively combining them far from success. In this paper, we start by making the empirical observation that naive joint optimization of CL and MIM losses leads to conflicting gradient directions- more severe as the layers go deeper. This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer. Inspired by experimental observa- tions, we find that MIM and CL. suitable lower and higher layers, respec- tively. We hence propose to combine them in a surprisingly simple, "sequential cascade" fashion: early layers are first trained under one MIM loss, on top of which latter layers continue to be trained under another CL loss. The proposed Layer Grafted Pre-training learns good visual representations that demonstrate superior label efficiency in downstream applications, in particular yielding strong few-shot performance besides linear evaluation. For instance, on ImageNet- Ik, Layer Grafted Pre-training yields 65.5% Top-1 accuracy in terms of 1% few-shot learning with ViT-B/16, which improves MIM and CL. baselines by 14.4% and 2.1% with no bells and whistles. The code is available at https://github: ICLR23.cit
com/VITA CennTTTTA.ccceg/geeeraatttetttrr
INTRODUCTION
Self-supervision has demonstrated undoubted power in learning strong visual representation. with two mainstream representative methods: Contrastive learning (CL) (Chen et al. 2020b He et al.)
2020 Chen et al. 2020d 2021 Grill et al. 2020 Caron al. 2021), and Mask Image Modeling (MIM) (Bao et al. 2021 He et al. 2021 Xie et al. 2022,Dong et al. (002)22022 The two methods
follow different mechanisms, and often manifest different strengths. Generally, CL performs the instance-level task that pulls augmented views from the same image to be similar while pushing different images to distribute diversely, making it versatile at learning semantic-aware clustering structures across images. In contrast, MIM draws inspiration from BERT (Devlin et al. 2018) and performs masked token or pixel reconstruction that facilitates the learning of rich local structures within the same image. In particular, although the latter one, MIM, has recently surpassed CL on the fine-tuning performance of many datasets, CL often remains to be a top competitor in data-scarce, few-shot downstream applications (Chen et al. 2020c Tian al. 2020).
A natural question then follows: are CL. and MIM indeed complementary to each other, and is there way to best combine their strengths?. One immediate, conceptually simple idea is to refer to multiple task learning (MTL) and jointly optimize the two losses on top of the same backbone. Unfortunately, our preliminary experiment (See Section 2.2) shows that such a vanilla combination fails to improve over either baseline, in fact often compromising the single loss's 's performance. A deeper dive reveals that the two losses, when being optimized together, will incur increasingly severe
Part of this work was conducted during summer internship Microsoft.

---

conflicts in the gradient directions, the layers go deeper (see Figure IT. That causes considerabl hurdles for the ppre-)training to effectively proceed.
We are then inspired to ask: the two losses conftict when both are placed at the end, how about placing them differently, such as appending them to different layers? Based on experimental ob- servations, it appears that lower layers tend to learn better from the MIM loss in order capture local spatial details; while higher layers tend to benefit more from the CL loss in order to learn semantically-awwe grouping and invariance. Inspired by so, we propose a simple MIM-CL Graft- ing idea to combine the bests of both worlds: (step i) first training the lower layers with MIM loss and fixing their weights, on top of which (step ii) higher layer weights continue to be trained un- der another CL loss. This simple cascaded training idea neatly separates MIM and CL losses to avoid their conflicts against each other if placed together; each loss also strategically placed to pre-training its most suitable portion. Practically, we "'smooth out" the grafting by allowing lower layers be slowly tuned in step ii, Our ablation experiments also find that the order of graft- ing matters, i.e., reversing MIM/CL loss locations and performing CL-MIM will considerably damage the performance. The contributions of this paper are summarized as follows:
We propose Layer Grafted Pre-training, principled framework to merge MIM and CL. improving representation learning beyond both, with no bells and whistles.
We investigate the different preferences of lower and higher layers towards CL and MIM losses, and show the order of grafting to matter.
Despite its embarrassing simplicity, the proposed Layer Grafted Pre-training demonstrates more desirable representation quality, and consequently superior label efficiency in down- stream applications, yielding strong few-shot performance besides linear evaluation. For example, we achieve [65.5%. 77.8%. 77.7%] terms of [1% few-shot, 10% few-shot, linear evaluation] performance, improving over MIM and CL baselines by [14.4%, 4.5%, 9.7% and [2.1 %% 2.4%, 1.0%], respectively.
2 METHOD
2.1 PRELIMINARY AND OVERVIEW
In Contrastive Learning (CL), the learning target is to pull the positive pairs together in the feature space while pushing negative pairs apart. Formally, the loss can be defined as:
M(Ui, V-, T)
1 N
Σ 1=1
log exp (v: uz / + Eoj EV exp (vi . 10, /T/T)
exp (or U,51 /T)
(I)
where (v,, w represents features of the positive pairs while (u,, v, means features of negative pairs. Also, V is the pool of negative features. т denotes the temperature. N is the number of samples. In practice, the positive pairs are often the different augmented views from the same image while the negative pool is composed by all the views from different images (Chen et al. 2021).
On the other hand, Mask Image Modeling (MIM) learns to reconstruct corrupted image where some parts of the image or feature map are masked out. The learning target can be formulated as:
C(z,, M) = N Σ D(d(f(Ma;)),.)
(2)
where I; and M are input images and randomly generated masks, respectively. and d represent the encoding and decoding functions, respectively. d(f(Mx,) is the generated image conditioned by masked image Mr,, D measures the difference between d(f (M()) and the original image 2;.
Overview. In the following parts of this section, we first introduce our preliminary exploration on the MTL of MIM and CL tasks in Section 2.2 which reveals the existence of the conflicting gradient direction. Afterward, in Section 2.3] we provide a simple separating idea towards mitigating the conflicts, which further leads to the proposed I. Layer Grafted Pre-training in Section 2.4
2

---

0.15
0.10
CaN
0.05
0.00
0.05
-0.10
Block 1
Block 2
Block
Block 4
Block 5
Block
Block 7
Block 8
Block
Block 10
Block 11
Block 12
Block Index
Figure 1: The box plot of CMIM,CL CL (œ) across different blocks for MTL combination of MIM and CL. This is measured on training datasets when the network is trained to 100 epochs (total 300 epochs). The red dash line indicates the linear regression of median numbers.
2.2 CONFLICTS PREVENT MULTI-TASK LEARNING FROM WORKING
Our first step towards answering the question of whether CL and MIM can complement each other examine the most straightforward and conceptually simple idea Multi-Task Learning (MTL) combination. Specifically, each iteration of MTL is composed of two steps. Firstly, the images are augmented twice for computing the CL loss following Moco V3 Chen et al. (202T). Then, the image with minimal augmentation would be utilized for computing MIM loss following MAE He al. (202T). These two losses share the same encoder and would be added together as the final loss.
As summarized in Table I MTL only yields marginal performance improvement of 0.4% on linear evaluation compared to the MIM baseline. However, 8.3%). Moreover, on both 1 1 few-shot and fine-tuning performance, MTL is even inferior to both MIM and CL baselines. Similar observations were also made by Wang et al. (2022a).
still much lower that the CL baseline (-
We further conjecture that the conflicts between these two targets in the MTL combination is the cause of the bad performance. To verify it, we design gradient surgery experiment by computing the cosine similarity between gradients of two tasks following Yu et al. 2020). Formally, the cosine similarity calculated as follows:
CMIM,CL VOLNIM (x) T VoLai (4)
(3)
V (x) V (z)
where LMIM and LeL denote the losses for MIM and CL, respectively. r batch of input samples. We measure the distribution of CNIM, (x)) across different layers of pre-trained MTL model. As shown in Figure 1 there always exist negative values for CMIM,CI (1)). where the MIM and are optimized in opposite directions. Moreover, the gradient direction varies across layers more severe as layers go deeper.
Also, the conflicts can be reflected in two losses' contradictory targets enforce. The MIM loss, for instance, requires that the reconstruction have the same brightness, color distribution, and positions as the input image, therefore the model needs to be sensitive to all these augmentations. Conversely, CL loss is designed to ensure that the model remains invariant regardless of different augmentations.
2.3 ADDRESSING THE CONFLICTS VIA SEPARATING
Given the conflicts the MTL combination, we ask the following question: the two losses conflict when both placed at the end, how about placing them differently. such appending them dif- ferent layers? Fortunately, recent empirical evidence suggests that CL and MIM may favor different pre-training methods. For MIM, Wang et al. (202cc) points out that, when only the pre-trained lower
3

---

MIM + CL Grafting
CL MIM Grafting
Layer Grafted Pre-training
MIM loes
CL loss
MIM loss
Step
CL loss
MiM loss
CL loss
Step
Learning Rate Decay
The pipelines the MIM-CLL CL-MIM Grafting, and Layer Grafted Pre-training. The former two are employed for preliminary experiments. The latter one is the final adopt pipeline, which is the 'smooth out' version of MIM-CL Grafting.
Figure 2:
Table 1: Illustration of preliminary study experiments' performance on ViT-B/16. Linear, 1% and Fine-tuning denote linear evaluation, 1% few-shot and fine-tuning performance, respectively. The performance of MIM and CL. are from MAE (He et al.] (021)) and MocoV3 (Chen et al.) (202)), re- spectively. MTL combination denotes the Muti-Task Learning (MTL) Combination of MIM and CL. MTL combination is pretrained for 300 epochs. For step of MIM-CL and CL-MMIM Grafting, directly adopt the pre-trained model of MAE and MoCo V3, respectively. Step 2 of MIM-CL and CL-MMIM Grafting is trained for 100 epochs.
Method
Linear
1%
Fine-tuning
MIM (MAE)
68.0
51.1
83.6
CL (Moco V3)
76.7
63.4
83.2
MTL. combination
68.4
47.6
81.0
CL MIM Grafting
65.5
32.5
82.5
MC Grafting
74.5
56.5
83.6
layers are retained while the higher layers are reset to random initialization, most of the gain still preserved for downstream fine-tuning tasks. Based this observation, the lower layers appear to be a key element in MIM. On the other hand, Chen et al. (202T) finds CL to be ineffective and even unstable for training the projection layer, the earliest layer of ViT (Dosovitskiy et al.) (020).. Fixing its weight to be random initialization can even yield significantly higher performance. Additionally, CL excels semantic concepts, which happen often higher layers of the neural network.
Driven by the above analysis, we propose a simple MIM-CI Grafting framework. As shown in Figure 2 MIM+CL Grafting can be separated into two steps: (step i) the lower layers are first trained with MIM and then fixed, on the top of which (step ii) higher layers continue to learn with CL. Despite the simplicity of this framework, yields promising preliminary results as shown in Table exceeding the MTL combination by [6.1%, 8.9%, 2.6%] for [linear evaluation. 1 % few- shot, Fine-tuning] performance, respectively. In contrast, when the order of two tasks reversed, the resulting CL-MMM Grafting, would suffer a dramatic drop in performance, which is even lower than the MTI combination by [2.9%. 15] terms of [linear evaluation, 1% few-shot] perfor- mance, respectively. The huge gap between CL-+MIM and MIM-CL Grafting further confirms the preference of MIM and CL towards lower and higher layers, respectively.
The example discussed at the end of Section 2.2 can also explain why this preference difference happens: The two different prior knowledge types requested by MIM and CL, while seemingly at odds, may work together different layers of the model. For example, the sensitivity to augmenta- tions can helpful for recognizing the local feature with strong color patterns (Xiao et al., 2020) in the lower layers (i.e. the fur of leopards). Meanwhile, for consistent semantic understanding.
4

---

the influence of the lightness difference should be eliminated when it comes to understanding the context feature at higher layers.
2.4 LAYER GRAFTED PRE-TRAINING
To fully unleash the power of Grafting, we 'smooth out' the boundary of MIM-+CL grafting to avoid sudden change in the feature space. Specifically, rather than fixing the lower layers, we assign them small learning rate. The resultant method, termed Layer Grafted Pre-training, is shown in Figure 2 In Section 3.3, we also explore other LR choices and our results indicate that employing small and large LR for lower and higher layers, respectively, yields the best performance.
features (i.e. the colors) in the lower layers with MIM while learning semantic alignment in the higher layers with CL. The proposed Layer Grafted Pre-training enables the learning of strong visual representations. It not only provides strong inter- class variance that helps cluster but also benefits the intra-class variance by keeping the diversity of samples in the early features.
By effectively capturing the auggentation--ensiine
EXPERIMENT
3.1
SETTING
General. We conduct all the experiments on ImageNet-Ik (Deng et al. 2009) with Nvidia V100 GPUs. The code is implemented with Pytorch (Paszke et al., 2019).
Backbone. We adopt the standard VTT-B and ViT-L architecture (Dosovitskiy et al. 2020) with the token size of 16x 16. The VTT-B is by default employed unless specified. When training with CL loss, we employ the projection and prediction head following Moco V3 (Chen et al. 202I). The settings for pre-training and evaluation protocols can be found Appendix A.5
3.2 COMPARISON WITH STATE-OF-THE-ART METHODS
We start by verifying the effectiveness of the proposed Layer Grafted Pre-training by comparing with state-of-the-arr methods. As shown in Table 2 in VIT-B/16, compared to the employed MIM and CL baselines, the proposed Layer Grafted Pre- Pre-training leads to a consistent improvement. For instance, it improves MAE and Moco V3 by [9.7%, 14.4%, 4.5%) and [1.0%, 2.1%, 2.4%) for [linear evaluation, 1% few-shot, 10% few-shot), respectively.
Compared to close competitors which also attempt combine MIM and CL, the proposed Layer Grafted Pre-training surpasses iBoT by 1.5% for linear evaluation performance. Compared to SIM, the proposed Layer Grafted Pre-training yields improvement of 1.1% and 0.2% for linear evalu- ation and 1 % few-shot learning performance, respectively.
Table 2: Comparison with State-of- The-hf-rTtttss on ViT-B/16 and ViT-L/16. Linear, 1% and 10% denote the top-1 accuracy (%) of linear evaluation, 1% and 10% few-shot learning, respectively. t: We employ the result of iBoT without augmentations from Zhou et al. (021) for fair comparison.
Bone
Method
I Linear
1%
10%
MAE (ee et al.
2021
68.0
51.1
73.3
Moco V3
Chen al.
202
76.7
63.4
75.4
VIT-B/16
iBoTt
Zhou al.
2021
76.0
SIM
lan el a 2022
76.4
65.3
C-MAE
(Hung al.
2022
73.9
65.3
77.3
MimCo 7hou et al. 2022
70.2
62.7
Layer Gralted Pre-trammig (Ours)
77.7
65.5
77.8
MAE (He et al.
2022
75.8
55.2
78.7
VIT-L/16
Moco V3 (Chen al.
2021
77.6
Layer Grafted Pre-traimng (Ours)
81.0
69.3
80.1
5

---

(a) Moco V3
(b) I Layer Grafted Pre-training
Figure 3: t-SNE (Van der Maaten & Hinton 2008 visualization for feature distribution of Moco V3 and Layer Grafted Pre-training Different colors represent different classes. Best viewed in color.
Our method also demonstrates good scalability toward larger models size. For instance, when scal- ing from ViT-B/16 ViT-L/16, Layer Grafted Pre-training further improves accuracy by [3.3%, 3.8%, 2.3%] in terms of [linear evaluation, 1% few-shot, 10% few-shot], respectively. Remarkably, gap above Moco V3 linear evaluation performance also increases from 1.0% to 3.4%.
We further qualitatively evaluate the representations learned by the proposed Layer Grafted Pre- training using 1-SNE (Van der Maaten & Hinton, 2008). As shown Figure 3bb the proposed Layer Grafted Pre-training shows better inter-class variance. For example, the categories represented by pink (+) and light blue (o) points are hard to separate given they are very close with each other in Figure 3a In contrast, for representation of the proposed Layer Grafted Pre-training, they form two clusters with clear boundary Figure 3b Besides, the proposed Layer Grafted Pre-training also
Aocuno %.
1540
15eb
Feat State Lp
Part Blgge UR
Lied
1
1.5p6
1.50
Sussind Stage UA
Sucsvid Stage Ul
Becorid Stage 10
1.54
155p4
1264
(a) Linear, 3rd Stage LR: 1.5e-
Linear, 3rd Stage LR: 1.5e-5
(c) Linear, 3rd Stage L.:: 1.5e-6
83.5
22.5
Firt Suage
Fort 130b Lp
Sungr
5
1it Swcond
Lie Secund
1ime Secood
15ed
1ted
15e4
(d) Tuning, 3rd Stage LR: 1.5c-4
(c) Tuning, 3rd Stage LR: 1.5c-5 (f) Tuning, 3rd Stage LR: 1.5c-6
Figure 4: Illustration the LR grid search results for different stages in terms of linear evaluations and fine-tuning performance. The grid is 1..5-- 1.5e44 for each stage. [(a), (b), (c)) and [(d), (e), (f)) denotes the linear evaluation and fine-tuning performance with third stage of [1.3e- 4, 1.5e-5, 1.5c-6], respectively. ((a), (b)] and ((e). (D) share the same color bar with (c) and (g), respectively. That of (d), (c) and (f) are also the same. The tested points are highlighted with blue dots in each plot. Best view in color.

---

shows better intra-varianan variance: the red (o), green (o) and yellow (+) points of Moco V3 collapse to smaller region than the proposed Layer Grafted Pre- tre-training.
3.3 LR SEARCH LAYER GRAFTED PRE-TRAINING
We further verify if small and large LR work the best for the proposed Layer Grafted Pre-training, Specifically, we study the performance with different I.R settings on the three stages on ViT-B/16 (Refer to fine-tuning part of Appendix A.3 the definition of stages), where each stage is searched with a grid of 1.55-6,1..e-5,1.55. 1.5c-6 to 1.5e-4 for the third stage, both linear evaluation and fine-tuning performance achieve improvement. Taking linear evaluation as an example, as shown in Figure 4a 4b and 4c when the LR of the third stage increase from 1.5e-6 1.5e-5 and 1.5e-4, the performance range could improve from 2.5%-63.05 5300% to 64.8%-70. 30%%%. and 73.7%-75. 1..11.,, respectively. In contrast, big LR of the first stage would lead drop in performance. For instance, in terms of the linear evalu- ation performance with third stage LR of 1 1.5e-4, as shown Figure 4al the performance ranges 74.9%-75.11
As demonstrated in Figure 4 when the LR increase from
14.8%-75.0% and 73.7%-73.8%% for first stage LR of 1.5c-6, 1.5e-5 and 1.5e- 4, respectively. The LR of the second stage is not as sensitive as that of the first or third stage. The trend of fine-tuning performance also similar that of the linear evaluation performance. The preference for larger LR for higher layers indicates that they can ben- efit by performing CL. Meanwhile, lower layers prefer smaller LR means that keeping MIM features these layers can be helpful.
60.0
25
3.4 MORE ABLATIONS
10.5
Mid
Layer Sraitedd Protraingg
Fine-tuning Performance Comparison. We also com- pare State-of- The-Arterrt methods for fine-tuning perfor- mance. shown in 3 the proposed Layer Grafted Pre-training yields competitive fine-tuning performance of 83.9%, which 0.3% and 0.7% higher than the em- ployed MIM (MAE) and CL (Moco V3) baselines, re- spectively. Moreover, Layer Grafted Pre-training also surpasses SIM by 0.1%.
The number Toiing
Figure 5: Comparison between the pro- posed Layer Grafted Pre-training and MAE under different numbers of fix- ing blocks on ViT-B/16 in terms of fine-tuning performance. The training dataset is the full ImageNet-lk. Best view color.
Partial Fine-tuning, Follow MAE (He et al. 2021) we , evaluate the performance of the proposed Layer Grafted Pre-training with different number of fixing blocks. As illustrated in Figure Layer Grafted Pre-training consistently yields higher performance than MAE. And this gap continue to grow larger when more layers are fixed, indicating the superiority of the representations learned by the proposed method.
Sawanie--nnaaiinee-CCCe Covariaacc Analysis. A study of the Inviuncce-Ivviinneeeee tern for the output each block is conducted in order better understand the Layer Grafted Pre- training. As illustrated in Figure G
Invariance-Covarianne pat-
we find that the VIC pattern of Layer Grafted Pre-training tends
Table 3: Top
Fine-tuning performance comparison.
BackBone
Method
Fine-tuning
MAE (H et al. 2021
83.6
Moco V3 Chen et al. 2021
83.2
ViT-B/16
SIM (Tao al. 202
83.8
ConMIM (Yi et al. 2022
83.7
Layer Grafted Pre- rre-tramnng (Ours)
83.9
MAE He al. 2021
85.9
ViT-L/16
Moco V3 CCen el al. 2021
84.1
Layer Grafted Pre-traming (Ours)
85.9
7

---

1.242
MAE
MAE
3.888
3.288
0.035
-0-
RME Tio bio
Layor Grafted Protrshing
NIE Fre-nne
Layo Grufted treiiiing
3.534
0.024
3.230
0.6033
0.00
0.002
3.228
6:44
0.22
0.030
6.00
10
(a) Variance
(b) Invariance
(c) Covariance
3.040
00077
Noup VS
0.0035
Mirn Vi Rnsene
Laysr Crufted mobasinng
Nocs 12 Rre-tune
Layal Grafted the reining
3.536
0 0.6025
2.332
0.0033
0.08
0.00
0 00022
2.522
5.04
0.04
0.000
0.02
(d) Variance
(c) Invariance
(f) Covariance
Figure 6: The Intinncce--varinneeeee Invariance-Covariance (VIC) analysis for different methods. VIC are com- puted on ViT-B/16 following [ardes et al.] (202T). The input features are first averaged over all tokens and then normalized remove the effect from magnitude. [(a), (d)), [(b), (e)] and [(c), (f)
study variance, covariance and invariance, respectively. Best view in color.
Table 4: Layer Grafted Pre-training on ViT-B/16 with VICReg (Bardes et al. 202 I). We train VIT- B/16 with VICReg for 100 epochs the baseline. For Layer Grafted Pre-training VICReg, the CL loss of stage ii is replaced with VICReg loss.
Method
Linear
Fine-tuning
VICReg (Bardes et al. 2021)
70. 10.1%
81.2%
Layer Grafted Pre-training VICReg (Ours)
74.9%
83.6%
to be similar to that of fine-tuning. In the MAE case, the VIC curve of Layer Grafted Pre-training closely matches that of MAE Fine-tune, much closer than MAE itself. The similarity between the proposed Layer Grafted Pre-training and fine-tuning in the VIC pattern also explains the high few-shot performance: weights of the pre-training do not need to change substantially to match the downstream task.
Layer Grafted Pre-training with VICReg. We further examine the generalizability of the pro- posed idea on different pre-training method VICRegn (Bardes al. [202T) As shown in Table when replacing the CL loss with the VICReg loss, the proposed Layer Grafted Pre-training still yields strong performance, surpassing the VICReg baseline by [4.8%, 2.4%] for [linear evaluation, fine-tuning] performance, respectively.
4 RELATED WORKS
Contrastive Learning. CL performs instance classification tasks by contrasting positive pairs
against negative pairs (Chen et al. 2020b He et al. 2020; Zhuang et al. 2019 Dosovitskiy et al,
2014). Other close works also explore learning without negative samples (Grill al. 2020 Bardes
et al. 2021; Caron et al. 2021 Zbontar et al. 2021 Chen & He 2021 and the clustering based
approachs (Caron et al. 2020).
8

---

One common merit of these methods is their strong performance learning good representa- tions, which shows strong clustering pattern (Dwibedi et al. 2021) and leads to state-of- of-the-art few-shot/emii-suprrvie
performance (Chen al. 2020et Tian et al. 2020; Li et al. 2021 Jiang eet al. 2022 You et al., 2022). However, they contain an implicit assumption that the features should be invariant heavy augmentations, which, however, could further lead to worse performance when the downstream performance violates it (Xiao et al.) 2020). The proposed Layer Grafted Pre-training address this via leveraging MIM for processing the features of lower layers.
Mask Image Modeling, Mask Image Modeling (MIM) is inspired by the success of BERT (Devlin at. 2018) Natural Language Processing WLP)..GPT (Chen al. 2020a) begins the exploration
of this idea in Computer Vision (CV). The emergence of ViT (Dosovitskiy et al., (200)) further
shrinks the gap backbones between CV and NL., motivating more researchers to delve into this
direction. Beit (Bao et al.] 2021 Peng et al. 2022). FaskFeat (Wei et al. (022) and Peco (Dong
al., 2021) focus on predicting tokens. MAE (He et al.] (021) and simMIM (Xie et al.] 2022)
further show the possibility of directly reconstructing the original pixels. Following works (Dong
ct al. 2022 Chen ct al. 2022 Wang et al. (022b)) continue to improve the performance or extend to other modalities. However, MIM achieves the most success in fine-tuning with enough data points.
For downstream tasks with limited data, MIM fails to surpass CL given the lack of linear separability for its representations (Tao al. 2022). We address this drawback by employing CL for learning the semantic alignment for higher layers.
Bridging Contrastive Learning and Mask Image Modeling. Only recently, researchers begin to explore the potential of combining MIM and CL. iBoT (Zhou et al. [021]. one of the pioneers in this direction, proposes switch the modeling of images the modeling of features. Some
concurrent works also follow this self-distillation paradigm (Tao et al. 2022 Assran et al.) 2022)
However, just like CL, this paradigm still relays on the involvement of strong augmentations avoid collapsing, which could lead to suppressinn of some features (i.c. color) (Xiao et al. 2020). In contrast, we employ MAE (He et al. 202T), an image modeling framework that free from strong augmentations. Besides, previous combination works treat the network as whole while we provide a new laser-wise perspective.
Comparison to other multiple-step pre-training tasks. One may relate the proposed method with previous multiple-step pre-training tasks like intermediate fine-tuning ( (eg., finetuning a MIM model using ImageNet22kk and transfer to ImageNetlk (Bao al. 2021)) self-supervised fine- tuning like Reed et al. (2022). The main differences lie in two aspects: (1) The key innovation of the proposed method reveal and utilize the layerwise difference between MIM and CI.. In contrast, intermediate finetuning and self-supervised fine-tuning are treating the model as a whole; and (2) While intermediate finetuning and self-supervised are designed for the same pretraining methods across different domains, the proposed method is devised for different pretraining methods in the same domain.
CONCLUSION
In this work, we propose Layer Grafted Pre-training, simple yet principled method for under- standing and bridging two popular types of supersupe learning methods Mask Image Mod- eling (MIM) and Contrastive Learning (CL). Our work provides simple remedy to the conflicts between MIM and CL and further reveals the different preferences of these two methods toward different parts of the neural network. It advances the quality of the self-supervised representations and achieves strong performance on linear evaluation and few-shot performance. Potential future work includes assessing or extending the proposed method to real-world unlabeled data with more challenges such as long-tail distribution or imbalance Jiang et al. (202T).
REFERENCES
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning, arXiv preprint arXiv:2204.0711 00.4411,, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2/066 08254, 2021.

---

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: for self-supervised learning. arXiv preprint arXiv:2105.04906 2021.
regulaiiztionn
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances Neural
Information Processing Systems, 33:9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference machine learning, pp. 1691 1703. PMLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information pro- cessing systems, 33:22243-22255, 2020c.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning, arXiv preprint arXiv:2202.03006 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 55750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020d.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- suff-vupee vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 9640-9649, 2021.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D) Manning. Electra: Pre-training text encoders discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-traiming of vision transform- ers. arXiv preprint arXiv:21 2.. 127 2021.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. In ECCV,
Alexey Dosovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems, 27, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19299 2020.
10

---

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9588-9597, 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.0260 2017.
Jean-Bastiens Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in Neural Information Processing Systems, 13:21271-21284, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 9729-972 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. arXiv preprint arXiv:2207. 13532, 2022.
Ziyu Jiang, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning, pp. 4927-4939. PMLR, 2021.
Ziyu Jiang, Tianlong Chen, Xuxi Chen, Yu Cheng, Luowei Zhou, Lu Yuan, Ahmed Awadallah, and Zhangyang Wang. Dna: Improving few-shot transfer learning with low-rank decomposition and alignment. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27,27, 2022, Proceedings, Part XX, pp. 239-256. Springer, 2022.
Suichan Li, Dongdong Chen, Yinpeng Chen, Lu Yuan, Lei Zhang, Qi Chu, Bin Liu, and Nenghai Yu. Improve unsupervised pretraining for few-label transfer. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10201-10210, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1rXiv:1711.05101, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buse Buc, F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei, Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
Colorado J Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, et al. supf-super pretraining im- proves sulf-supe pretraining. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pp. 2584-2594, 2022.

---

Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for sepf-supe vision representation learning. arXiv preprint arXiv:2206.01204, 2022.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, pp. 266-282. Springer, 2020.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. arXiv preprint arXiv:2204.07118, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911), 2008.
Luya Wang. Feng Liang, Yangguang Li, Wanli Ouyang, Honggang Zhang, and Jing Shao. Repre: Improving self-supervised vision transformer with reconstructive pre-training. arXiv preprint arXiv:200.006857, 2022a.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In IEEE Con- ference Computer Vision and Pattern Recognition (CVPR 2022), 2022b.
Shaoru Wang, Jin Gao, Zeming Li, Jian Sun, and Weiming Hu. A closer look at self-supervised lightweight vision transformers. arXiv preprint arXiv:2205. 14443, 2022c.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yual Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14668-14678, 2022.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint (xXi::0008.05658 2020.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.
Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. arXiv preprint arXiv:2205. 09616, 2022.
Chenyu You, Weicheng Dai, Fenglin Liu, Haoran Su, Xiaoran Zhang, Lawrence Staib, and James Duncan. Mine your own anatomy: Revisiting medical image segmentation with extremely limited labels. arXiv preprint arXiv:2209.13477 2022.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824 5836, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 1..0022111.07832, 2021.
Qiang Zhou, Chaohui Yu, Hao Luo, Zhibin Wang, and Hao Li. Mimco: Masked image modeling pre-training with contrastive teacher. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4487-4499 2022.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6002-6012, 2019.
12

---

Table 5: Comparison between MIM (MAE) and CL (Moco V3) for frozen features from blocks [6,9,12], on top of which we employ various numbers of blocks (#Blocks) for fine-tuning, block indicates that only linear classification head is employed (identical to linear evaluation). Top accuracy ImageNet-1K IK is reported and the best performance under each setting is highlighted with bold text.
The block index of the feature
#Blocks
12
MIM
CL
MIM
CL
MIM
CL
38.9%
43.6 %
59.3%
65.5 %
68.0%
76.7 %
70.1%
72.7%
77.8%
78.1%
78.9%
79.1 %
76.3%
76.8% %
80.2%
79.2%
80.6%
79.7%
78.5%
78.1%
81.2%
79.4%
81.4%
79.2%
A APPENDIX
This appendix contains the following details that could not include in the main paper due to space restrictions.
A.1 MORE ANALYSIS FOR THE LAYER-WISS WIIE DIFFERENCE BETWEEN MIM AND CL
We provide analysis further understand the layerwise difference between MIM and CL. We start by analyzing the average attention dis-
110
tance across different layers. As shown in Fig- ure 7 on the one hand, for the deep layers (i.e.
CL Layer Grafted Pre-training
from 8th to 12th blocks), the average attention distance of CL would keep increasing, where
80
the aggregation of local features is likely to happen. contrast, the average attention dis- tance of MIM keeps the same level for the deep layers. On the other hand, the average attn dis- tance of shallow layers (i.e. 1st and 2nd blocks) of CL is much larger than that MIM, which may distract the model from extracting local features. The proposed method combines the lower and higher layers' patterns of MIM and CL, respectively, forming a gradually increas- ing attention distance pattern. Remarkably, this pattern echos the philosophy of gradually in- creasing receptive field for designing network architecture He et al. (2016): Liu al, (021).
70
50
40
10
Figure 7: Demonstration average attention dis- tance (Dosovitskiy et al. (020) for MIM (MAE), CL (Moco V3) and Graft in terms of the across all attention heads.
Secondly, we study the different properties of features across different layers. Specifically, in Im- ageNet IK, we turn several random initialized blocks on top of features from different layers. As demonstrated in Table 5] when only fine-tuning the classification head B#olk 0), the perfor- mance of MIM is much lower than CL, indicating that the feature of CL more close to semantic representation. By contrast, when inereasing the turnable blocks, the performance of MIM would significantly increase and even surpass CL, demonstrating the features MIM better encode the local features. The potential of these local features be stimulated by adding enough modules to aggregate them. The proposed Layer Grafted Pre-training employs MIM for producing high-quality early features while utilizing the CL in higher layers for aggregation.
A.2 DISCUSSION AND COMPARISON WITH CONCURRENT WORKS
In this section, we discuss the difference between the proposed method with four concurrent
works (Tao al. 2022 Huang et al. 2022 Zhou al. 2022 Yi al. 2022) and provide more
13

---

0.10
0.05
0.00
-0.05
Block
Bleck 2
Block
Block
Block
Block
Block
Block 8
Block
Block 10
Block 11
Block 12
200 epochs
0.075
0.050
0.025
0.000
CHO.,
-0.025
-0.550
Block 1
Block
Block 3
Black
Black
Black
Black 7
Black 8
Block
Block 10
Black 11
Block 12
Black Index
(b) 300 epochs
(M (x) across different blocks for MTL combination of MIM and CL. This is measured on training datasets when the network is trained to (a) 200 epochs and (b) 300 epochs (total 300 epochs). The red dash line indicates the linear regression of median numbers.
Figure 8: The box plot of CMIM.IL.
comparisons. To combine the strength of MIM and CL. SIM (Tao et al. 2022) proposes to pre- dict the dense representations of an augmented view for enforcing semantic alignment and spatial sensitivity simultaneously. CMAE (Huang et al. 2022) proposes two new components: pixel shift and feature decoder. MimCo (Zhou et al. (022) utilizes the CL pre-trained model as teacher and performs patch-level and image-level reconstruction tasks. ConMIM (Yi et al. 2022) utilizes contrastive constraints to produce dynamic masked prediction target. The difference between the proposed Layer Grafted Pre-training and these works are as follows:
While all the concurrent works treating the network whole, reveal the different preferences of MIM and CL towards their internal different layers, which motivates us a design novel layerwise method.
Our method employs the original design of MIM and CL, which not only is simple but also enables an apple-to-apple comparison. In contrast, It's non-straightfooraard to tell whether the improvements of the concurrent works are from the newly introduced module or the original MIM/CL design.
The proposed Layer Grafted Pre-training provides in-depth analysis of the reason why MIM and CL cannot be directly combined together through gradient analysis.
Also, here we further analyze why the proposed method fails surpass the concurrent work C- MAE (Huang et al., 2022) in terms of fine-tuning performance. One possible reason lies in whether the masked view employed for contrastive learning. C-MAE is contrasting masked and full image while the proposed method contrasting two full images following Moco V3. The difference in design further leads to different strengths: On the one hand, empirical results highlight that the
14

---

Table 6: ADE20K semantic segmentation comparison using UperNet with different pre-training methods.
Method
mloU
MAE (He et al.
2021)
48.1
Moco V3 Chen et al.
2021
47.3
Layer Graftd Pre-traming (Ours)
48.7
masked view can benefit the downstream fine-tuning task (Touvron et al. 2022 He et al.) 2021] which may be because it helps to learn the correlation between sparse patches that cannot be built under full view. On the other hand, contrasting full images leads to smaller gap with downstream tasks and thus benefiting the downstream few-shot and linear evaluation tasks.
A.3 MORE GRADIENT ANALYSIS RESULTS
We measure the distribution of CMIM,CI (z) across different training epochs and confirmed that the statistical results persist the same across the entire training process, rather than just few specific epochs. As two examples, Figures Sa and 8b show that a considerable part of values is negative in the 200 and 300 epochs.
A.4 TRANSFER LEARNING RESULTS
We further evaluate the proposed method for the standard transfer semantic segmentation task. As shown in Table 0 on ADE20K with UperNet, the proposed method achieves higher performance than both MAE and Moco V3.
A.5 MORE SETTINGS
Pre-training. We by default adopt MAE (He al. 202I) and Moco V3 (Chen al. 2021) as our MIM and CL frameworks, respectively. For the first step of Layer Grafted Pre-training, since it identically follows the original MIM pipeline, we directly adopt the pre-trained model of MAE (He al.) 2021) and conduct the second step, where initialize the network with the MIM pre- treitrained model and train with Moco V3 (Chen et al. (02T)) for 300 epochs. For LR, We first split the network into three stages. Each of them contains the same number of blocks. (i.e. 4 and 8 blocks for VIT-B and ViT-L, respectively.) Then, the base LR of the first and second stages (corresponding to lower layers) is assigned 1.5e-5 while the third stage is set 1.5e-4 by default. In the second stage of ViT-L, we further ensure the early layers of the resultant model are close to the MIM pre-training by minimizing the 1 distance between the first 12 layers between them (Refer Section A.6 for more ablations). Other settings are identically followed from Moco V3 (Chen et al. 2021).
Fine-tuning, For fine-tuning, we train with AdamW (Loshchilov & Hutter 2017) for 100 epochs following MAE (He et al. 2021). We employ base LR of 5c-4 with linear scaling rule (Goyal al. 2017). The layer-wise LR decay ratio set as 0.6 (Clark et al. 2020). For other settings such as data augmentation, LR scheduling or weight decay, we identically follow He et al. (2021).
Few-shot Learning. We conduct few-shot learning with 1% or 10% available labels. The sub- sampling splits are adopted from Chen et al. (2020c). For 1 1% few-shot evaluation, following Caron al. (2021), we first generate frozen features on training images without data augmentation, on top of which logistic regression classifier is trained for prediction. For 10% semi-supervised learning. we train from the first layer the projection head following Chen et al. (2020c). We train for 400 epochs with initial base LR of 3e-5. Other settings are identical to the fine-tuning.
Linear Evaluation. For linear evaluation, we train a linear classifier on top of frozen pre-train features to measure the quality of the visual representations following common practices (Chen ct al. 20206). Following Moco V3 (Chen et al. 202I), the classifier is trained for 90 epochs with SGD optimizer and weight decay of 0. The LR is swept for each case.
15

---

Table Ablation study for 12 regularization on VIT-L. Top accuracy on ImageNet-I IK is reported and the best performance under each setting is highlighted with bold text.
t regularization
Linear
1%
10%
Finetune
X
80.5
68.9
79.8
85.7
81.0
69.3
80.1
85.9
A.6 ABLATION STUDY FOR 1 REGULARIZATION
In this section, we ablation study the effectiveness of 1 regularization mentioned at section A.5 As shown in Table 7 employing the 2 regularization can help to preserve the Mask Image Modeling features of the lower layers and yield consistent improvement across multiple benchmarks.
16


Reviews:
[{"review_id": "yegQba-ZBL1", "reviewer": "Reviewer_Yr9u", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to learn better visual representation by leveraging both contrastive learning (CL) and masked image modeling (MIM). The authors claim that simultaneously performing CL and MIM pertaining will lead to worse results. To mitigate this issue, the paper proposes Layer Grafted Pre-training, which finetunes a pretrained MIM model (e.g., MAE) using contrastive learning (e.g., MoCo-v3 and VICReg). The pretrained models are evaluated on ImageNet with linear and fine-tuning protocols.", "summary_of_the_review": "Overall, I think the paper is well-written and motivated. But it has limited technical contribution and provides few new insights and knowledge. The method is also not that effective. I tend to give a reject.", "strength_and_weaknesses": "Strength:\n- Combining contrastive learning and masked image modeling is a popular problem and has attracted much attention. This paper provides a very simple solution to benefit from contrastive learning and masked image modeling at the same time.\n- The paper shows high quality in writing. The presentation of both methods and experiments is intuitive. It is easy to follow and comprehend.\n\nWeakness:\n- The paper presents limited novelty and contribution. The method simply finetunes a strong MIM pretrained model using contrastive learning, and then transfer it to downstream classification task. It can be viewed as an intermediate fine-tuning, e.g.finetune a MIM model using ImageNet22k and transfer to ImageNet1k,  which is used in BEiT and adopted by some follow-up papers. The intermediate fine-tuning has been demonstrated to be very effective and can bring 1.5% Top-1 accuracy. \n- The experiments are insufficient to demonstrate the effectiveness. The method only improves 0.3% top-1 accuracy when fine-tuning on ImageNet. In addition, fine-tuning experiments are only conducted using ViT-B/16 backbone. The results of ViT-L/16 should be reported.\n- The experiments in Table 4 show that the method doesn't work when using VICReg. The fine-tuning result is the same as the original MAE, i.e., 83.6%. \n- What about conducting multi-task learning but performing masked image modeling at lower layers? This is an important baseline according to your motivation.\n- There are some papers (e.g., A, B, C, etc) that also combine contrastive learning and masked image modeling, including a few listed in related work. More comparison and discussion can be added.\n- The paper can also be viewed as using self-supervised pretraining (CL) improves self-supervised pertaining (MIM), which is similar to [D]. Discussion can be added.\n\n[A] Siamese image modeling for self-supervised vision representation learning.\n\n[B] MimCo: Masked Image Modeling Pre-training with Contrastive Teacher\n\n[C] Masked Image Modeling with Denoising Contrast\n\n[D] Self-Supervised Pretraining Improves Self-Supervised Pretraining", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lBYekkql8B", "reviewer": "Reviewer_z4dF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies an interesting layer-grafting technique to promote the combination of CL and MIM for representation learning. Specially, the authors find that directly combining such two paradigms shows the negative influence due to the gradient conflicts, and discovers that the sequential cascade in a certain order can address this problem and consistently improve the performance of representation learning. ", "summary_of_the_review": "The paper provides a new road for representation learning. The observed phenomenon is interesting and important. Then, a novel method is proposed. ", "strength_and_weaknesses": "The strength of this submission can be summarized as follows\n\n1)\tThe authors systematically studied the combination of CL and MIM and analyzes the conflict in the gradient direction that incurs the incompatibility of the two paradigms.\n\n2)\tDifferent from the straightforward combination, the authors proposed to reconcile the conflicts of two paradigms via layer grafting, namely, a cascade way. Specifically, the authors show the order the layer grafting make the different effect, where MIM contributes more in the lower layer and CL contributes more in the higher lower. Then, by means of their advantages of CL and MIM, the authors introduce the layer-grafted pretraining method to enhance representation learning.\n\n3)\tA range of experiments have demonstrated that the layer-grafted pretraining significantly improves the performance of either CL or MIM, and achieves the new state-of-the-art compared with the current strong baselines including MoCoV3 and SIM. More extensive analysis of LR search or ablation study as well as the finetuning, and segmentation further confirms the designing intuition and advantages of layer-grafted techniques.\n\n4)\tThe submission is well written with clear logic, and the topic of this work is well motivated based on sufficient evidence on the direct combination. \n\nMinor weakness\n\n1)\tLR search directly reflects the grafting degree among layers, which can be studied further in other different datasets to confirm whether it consumes much empirical experience.\n\n2)\tThe results of the straightforward combination of MIM and CL can be added to tables when compared with the state-of-the-art baselines. This will strengthen the advantages of the proposed layer-grafting techniques.\n\n3)\tLayer-grafting pretraining for more general scenarios can be discussed to motivate more explorations in the future.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ek6xjAR_T2T", "reviewer": "Reviewer_iQDY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a simple method to combine masked image modeling (MIM) and contrastive learning (CL) into a joint image pretraining framework. Specifically, MIM and CL are applied to the intermediate block and the end block respectively during training. Two steps are conducted: MIM is applied first and then its learning rate is decreased and CL is added into training. \n", "summary_of_the_review": "See my comments above.", "strength_and_weaknesses": "Strengths:\n1. The proposed method is simple and easy to follow. The writing is clear.\n2. Experimental results show improvement over models either trained by MIM or CL separately on the few shot learning setting.\n\nWeaknesses:\n1. While the simplicity of the method is desirable, the novelty of the proposed method is relatively limited. It is well known that combining CL and MIM cannot bring significant improvement, as found in [a,b]. In the paper, the authors only compare performance with [a] without a deep analysis of their differences. Reference [b] is missing. The authors are suggested to compare their method in more detail with existing methods to clarify their contributions, e.g. why their strategy is better than other methods.\n\n[a] C. Tao, et al. Siamese image modeling for self-supervised vision representation learning. arXiv preprint arXiv:2206.01204, 2022.\n\n[b] Z. Huang, et al. Contrastive masked autoencoders are stronger vision learners. arXiv preprint arXiv:2207.13532, 2022.\n\n2. The motivation for solving conflict gradients of MIM/CL may be problematic and need further verification. There are several factors ignored in current experiments which make the conclusion less convincing.\n- The gradient discrepancy is not severe according to Fig.1: most numbers are still positive, which may imply the gradients may not be the root cause.\n- Do those statistical results remain the same across the whole tranining process? Or it is only like this at specific training point (e.g. 100 epoch as in Fig. 1). In other words, MIM and CL may not always have gradient conflict, so the effect of gradient issue may be mistakenly enlarged.\n\n3. The performance is not competitive when compared with existing methods. The finetune result on ImageNet is the same as original MAE, and worse than [a,b] and others like iBot, showing the limits of applying this method in real use. Compared with finetune results, few shot results are less concerned since the paper mainly claims a better pipeline/training strategy over initial MAE and CL.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IYe3Yj-xjnn", "reviewer": "Reviewer_jbuW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new approach to combine both Contrastive learning (CL) and Mask Image Modeling (MIM) to take advantage of the best of both worlds. An analysis shows that a simple combination of the losses does not work well because there are some conflicts between the losses. To solve this problem, the paper introduces a two-step training approach strategy where the model is first trained with a MIM loss, and then fine-tuned with a CL loss. The proposed approach is evaluated on ImageNet in multiple scenarios: linear, 1% few-shot, 10% few-shot, and fine-tuning. The proposed approach improves performances in these scenarios. ", "summary_of_the_review": "Overall, I like the idea of combining both Contrastive learning (CL) and Mask Image Modeling (MIM) to take advantage of the best of both worlds. A problem to combine CL and MIM losses is well identified and a solution is proposed, but the proposed solution relies on existing techniques.  \n\n------------------------------------------\n\n[initial post rebuttal]\n\nAfter reading all the reviews, the rebuttal and the updated paper, I decrease my score to 5. Overall, I think the idea of combining MIM and CL is interesting and well motivated in the paper. However, as point out in multiple reviews, the performance gains on some benchmarks do not seem significant. In the fine-tuning setting, which is popular setting, the proposed approach is 0.8pt worst than C-MAE (Table A). The performance gain seems significant only in the few shots and linear settings. I think it is important to see a more consistent and significant gains on most of the benchmarks to validate the proposed approach. \n\n------------------------------------------\n\nI updated my review and rating to ignore the recent works and to take into account the linear and few-shot settings are the main targets of the paper. I would suggest to update the paper to reflect it. For example, the abstract is not clear about this point. ", "strength_and_weaknesses": "**Strengths**\n- The paper proposes a new approach to combine both Contrastive learning (CL) and Mask Image Modeling (MIM) to take advantage of the best of both worlds. \n- Combining CL and MIM losses does not seem straightforward. An analysis in the paper shows naive joint optimization of CL and MIM losses does not work well because there are some conflicts between the losses. Figure 1 shows the cosine similarity between MIM and CL gradients for each layer. For each layer, there are negative values that indicate the MIM and CL are optimized in opposite directions. The paper also gives a reason to justify these conflicts: the MIM loss is designed to reconstruct the input so it should not be invariant to data augmentations, whereas the CL loss is designed to ensure that the model remains invariant to some data augmentations.\n- The paper introduces a simple two-step training approach strategy to reduce the conflicts between MIM and CL losses. First, the model is trained with a MIM loss. Second, the model is fine-tuned with a CL loss. The proposed strategy is not specific to a Vision Transformer architecture and can be used with multiple Vision Transformer architectures. An empirical analysis shows that the order is important so the model should be trained with the MIM loss and it is not a good idea to start with the CL loss.\n- The proposed approach is evaluated on ImageNet in multiple scenarios: linear, 1% few-shot, 10% few-shot, and fine-tuning. The proposed approach improves performances in these scenarios for both ViT-B/16 and ViT-L/16 architectures.\n\n\n**Weaknesses**\n- The paper does not introduce new string technical contributions. The proposed approach relies on existing techniques, but the combination and context seem novel.\n- The proposed approach seems specific to Vision Transformer architectures and may not adapt to other model architectures like ConvNets. *[post-rebuttal] After the author answer, it does not seem to be a weakness because it can be used with ConvNets.*\n- I think adding studying the transfer learning performances will improve the quality of the paper. It is quite standard to study how the learned representations transfer to some downstream tasks. I think analysing the transfer learning performances will be more interesting than the visualization for feature space (Figure 3). *[post-rebuttal] The authors added new transfer learning experiments to answer this concern. Table C shows the performances of an UperNet with different pre-training methods.*\n\n\nMinor: there is an error at the end of the intro: “performing MIM->CL will considerably damage the performance.” It should be “CL->MIM”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cc0ilZ3RJf", "reviewer": "Reviewer_nkwf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes Layer Grafted Pre-training, a simple two-step approach for bridging two recent self-supervised learning methods, which are Masked Image Modeling (MIM) and Contrastive Learning. The key observation is that both objectives have conflict aspects, and separating them into lower and higher layers can enhance the stability of joint optimization of MIM and CL. Specifically, in a cascade fashion, the proposed method pre-trains early layers under MIM loss and then trains the other layers under CL loss. The extensive experiments showed the effectiveness of the proposed method on image classification tasks such as few-shot classification and linear evaluation.", "summary_of_the_review": "Overall, I have several concerns, as I mentioned above weaknesses section.\nFor now, I weakly recommend accepting this manuscript.", "strength_and_weaknesses": "Strengths\n- The writing is clear and easy to understand.\n- This manuscript tackles a recently popular and interesting problem that unifies two large paradigms (i.e., Contrastive Learning (CL) and Masked Image Modeling (MIM)) that divide recent self-supervised learning.\n- The proposed method outperforms the state-of-the-art self-supervised learning baselines on the image-classification tasks.\n\nWeaknesses\n- Overall, the authors validate the proposed method on CL-centric evaluation metrics (e.g., linear evaluation, few-shot classification). One of the main differences between CL and MIM is that \"MIM helps the model to learn rich local structures within the image\" (as mentioned in the manuscript), which can be beneficial to dense prediction-type downstream tasks such as object detection and semantic segmentation. For example, MAE [1] has shown superiorities of MIM in COCO object detection and segmentation, and on ADE20K semantic segmentation, while it shows poor performance on linear probing compared to fine-tuning on image classification tasks. However, the proposed method only highlights the benefits of image classification tasks, which are close to the benefits of CL. Moreover, improvements compared to CL are much smaller than MIM. In short, my question is, \" Is the proposed method a unified method of CL and MIM or a slightly better variant of CL?\" If so, does the proposed method inherit the benefits of MIM? (e.g., dense prediction tasks)\n- The key observation of the manuscript is that CL and MIM loss objectives are in conflict. However, I am questionable that the proposed MTL combination is the right way to conclude it. Specifically, the authors enforce the model to discriminate against all the augmentations under MIM loss, while invariant against the same augmentations under CL loss. I think combining MIM without augmentation (or minimal) and CL with augmentation is a more straightforward approach with less conflict. Furthermore, in the experiments section, the authors also adopt MAE models, which are pre-trained without augmentations. I would like to ask the authors to provide more explanations and valuable insights about this.\n- In the preliminary section 2.1, it is unclear which features are used for contrastive learning; which are they between outputs of the encoding function $f$ or the decoding function $d$? Does the decoding function $d$ be discarded after training MIM?\n\n[1] He et al., \"Masked Autoencoders Are Scalable Vision Learners,\" CVPR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster