CAE v2: Context Autoencoder with CLIP Latent Alignment
Authors: Xinyu Zhang
Jiahui Chen
Junkun Yuan
Qiang Chen
Jian Wang
Xiaodi Wang
Shumin Han
Xiaokang Chen
Jimin Pi
Kun Yao
Junyu Han
Errui Ding
Jingdong Wang
Venue: Accepted by TMLR ()

Abstract: Masked image modeling (MIM) learns visual representations by predicting the masked patches on a pre-defined target. Inspired by MVP(Wei et al., 2022b) that displays impressive gains with CLIP, in this work, we also employ the semantically rich CLIP latent as target and further tap its potential by introducing a new MIM pipeline, CAE v2, to learn a high-quality encoder and facilitate model convergence on the pre-training task. CAE v2 is an improved variant of CAE (Chen et al., 2023), applying the CLIP latent on two pretraining tasks, i.e., visible latent alignment and masked latent alignment. Visible latent alignment directly mimics the visible latent representations from the encoder to the corresponding CLIP latent, which is beneficial for facilitating model convergence and improving the representative ability of the encoder. Masked latent alignment predicts the representations of masked patches within the feature space of CLIP latent as standard MIM task does, effectively aligning the representations computed from the encoder and the regressor into the same domain. We pretrain CAE v2 on ImageNet-1K images and evaluate on various downstream vision tasks, including image classification, semantic segmentation, object detection and instance segmentation. Experiments show that our CAE v2 achieves competitive performance and even outperforms the CLIP vision encoder, demonstrating the effectiveness of our method. Code is available at https://github.com/Atten4Vis/CAE.

Published in Transactions on Machine Learning Research (09/2023)
CAE v2: Context Autoencoder with CLIP Latent Alignment
Xinyu Zhang"; Jiahui Chen Junkun Yuan Qiang Chen', Jian Wang', Xiaodi Wang', Shumin Han', Xiaokang Chen Jimin Pi', Kun Yao', Junyu Han', Errui Ding', Jingdong Wanglt
(zhangringul 4, ch wanggiian3,, wangziaodi0S, hankkmmng.g p.jimin01.yaoouuueee hanjunya dingerrui @baidu. com, wangjingdong @baidu.com com, jahuui Oben@ edu.cn, yuanjk@zju. edu.cn, pluczk@pku. edu.cn Baidu VIS School of Aulomation Science and Electrical Engineering, Beihang University
College of Compuler Science and Technology, Zheziang University" A Peking University
Reviewed on OpenReview: https: // openreview. net/ per//frmm? id= F36LOK7MOF
Abstract
Masked image modeling (MIM) learns visual representations by predicting the masked patches on a pre-defined target. Inspired by MVP (Wei et al. 2022b) that displays im- pressive gains with CLIP, in this work, we also employ the semantically rich CLIP latent as larget and further tap its potential by introducing a new MIM pipeline, CAE v2, to learn a high-quality encoder and facilitate model convergence on the pre-training task. CAE v2 is an improved variant of CAE (Chen et al., 2023). applying the CLIP latent on two protraining lasks, i.e., visible latent alignment and masked latent alignment. Visible latent aligmment directly mimics the visible latent representations from the encoder to the corresponding CLIP latent, which is beneficial for facilitating model convergence and improving the rep- resentative ability of the encoder. Masked latent alignment predicts the representations of masked patches within the feature space of CLIP latent as standard MIM task does, effec- tively aligning the representations computed from the encoder and the regressor into the same domain. We pretrain CAE on ImageNet-1KK images and evaluate on various down- stream vision tasks, including image classification, semantic segmentation, object detection and instance segmentation. Experiments show that our CAE v2 achieves competitive per- formance and even outperformns the CLIP vision encoder, demonstrating the effectiveness of our method. Code is available at https://github. www com//Atteoe///is/CC.
1 Introduction
Masked image modeling (MTM) (Bao el al. 2022) task has attracted numerous atfention in self-supervised representation learning, showing strong performance on a variety of downstream tasks. Previous MIM methods (Bao ct al. 2022 He et al. 2022 Xie et al.) 2022 Chen et al. 2023 usually mask out some image patches, and then predict these masked patches conditioned on representations of visible patches according to specific prediction targets. The architecture of these MTM methods can be unified with an encoder-decoder format, in which the encoder is used for the representation learning and the decoder is used for the prediction of the masked patches. When transferring to downstream tasks, MIM only maintains the learned encoder and discards other parts.
A high-quality pre-trained encoder can greatly benefit downstream tasks. To improve the encoding quality, previous works take efforts on two aspects, i.e., using better pretraining largets and decoupling the learning of encoder and decoder. As for the type of target, instead of low-level signals like RGB (He et al. 2022
"Equal contribution. 1 'Corresponding author.
Work was done when J. Chen, J. Yuan and X. Chen were interns in Baidu VTS.

---

Published in Transactions on Machine Learning Research (09/2023)
(a)
(b)
Zm
Encoder
Regressor
Visible latent alignment
Masked latent alignment
Z
CLIP
Zm
Encoder
Visible patches
Masked patches
Mask queries
Visible/Masked latent targets
The pipeline of CAE v2. (a) The encoder receives the visible patches and extracts the visible latent representations Zu, which is then put into the regressor to generate masked latent representations Zm conditioned on the mask queries. CAE v2 consists of two loss functions: visible latent alignment mimicking Zy the visible latent targets Zu, and masked latent alignment - regressing Zm with the masked latent targets Zm Both Zy and Zm are generated by the vision branch of CLIP model. After pretraining, the encoder (a) is applied to downstream tasks, while (b) is replaced with the downstream task part.
Figure 1:
Xie et al. 2022 and hand-crafted features like HOG (Wei al. level semantically rich targets to help the encoder learn more informative semantics, e.g., discrete visual
2022a some methods intuitively high-
tokens (Bao et al. 2022 Chen et al. 2023
El-Nouby et al. 2021 Peng al. 2022a Dong et al. 2021
features from momentum encoders Tao et al. 2022; Chen et al. 2022 Wu et al. 2023 and features from pretrained models [Wei et al. 2022b Fang et al. 2023b a Hou et al. 2022 Peng et al.
2022b Among these literature, MVP (Wei et al. 2022b) first utilizes CLIP latent as the masked prediction target in MIM, showing impressive gains on various downstream tasks. Another line like MAE (He al.] 2022) and CAE (Chen et al. 2023 attempts
partition the representation learning of the encoder and the reconstruction of the masked patches. For example, CAE introduces latent contextual regrossor to explicitly decouple the encoder learning and the decoder reconstruction, which effectively motivates encoder's power.
This paper also aims to acquire a high-quality vision encoder. Specifically, we introduce a new MIM pipeline, CAE v2, i.c., a context autocncoder with CLIP latent alignment. The pretraining target of CAE v2 is CLIP latent following MVP (Wei et al. 2022b). The architecture of CAE v2 is built upon the CAE (Chn et al., 2023) method, with the variation on retaining the encoder and the regressor yet discarding the decoder. This modification results in more lightweight model structure, enabling faster model pretraining and less computational cost.
CAE v2 contains two concurrent pretraining tasks, ie., visible latent alignment and masked latent alignment. Visible latent alignment is designed for an explicit optimization on the encoder, which imposes visible latent representations from the encoder to be close to those from the vision branch of CLIP model. encourages the encoder learn semantically rich information brought by supervision signals directly, which is beneficial for model convergence and performance improvement. Masked latent alignment responsible for the masked patch prediction. It directly regresses feature representations of the masked patches from the regressor to the corresponding CLIP latents standard MIM methods do. These two tasks are able to align the feature representations from the encoder with those from the regressor to be close to the same feature space of CLIP latents. In this way, both encoder and regressor can be fully learned, especially on the encoder that is transferred for downstream tasks. The two latent alignment serve as loss functions, in which they are robust to different loss types and the cosine distance is used by default.
We pretrain CAE on ImageNet-1K images without using specific ground-truth labels. Extensive experi- ments demonstrate that CAE v2 achieves competitive results across all scales of models, from the tiny
2

---

Published in Transactions on Machine Learning Research (09/2023)
size to the large size, various downstream tasks, including image classification, semantic segmentation, object detection, and instance segmentation.
In summary, our contributions are:
We develop a new MIM pipeline, CAE v2, which is an improved version of CAE with CLIP latent as the pretraining target for learning a high-quality encoder and facilitating model convergence.
We introduce visible latent alignment and masked latent alignment, aligning feature representations generated from the encoder and the regressor with CLIP latents.
Experiments show that CAE v2 effectively improves the representative ability of the encoder, achicving competitive performance across model sizes and various downstream vision lasks.
2
CAE v2
CAE v2 an improved variant CAE (Chen et al. 2023). Compared with CAE, CAE v2 only maintains the encoder and the regressor while discarding the decoder (Section 2.2p. Meanwhile, CAE v2 applies two latent alignment objectives that supervised by CLIP latents feature reprosentations extracted from the vision branch of CLIP (Section 2.3 which is different from CAE that uses one alignment loss and one reconstruction loss.
2.1 Preliminary: CAE
The network of CAE (Chen et al. 2023 is an training lasks: masked representation prediction and masked patch reconstruction. The key of CAE is to decouple learning of encoder from completing the pretraining tasks, and making predictions in the encoded representation space. We illustrate the computational graph of CAE in Figure 5 (b) in Appendix.
architecture, including two pre-
Let E D denote an input image. CAE first embeds x into N patches, which are then divided into two non-overlapped sets, i.e., visible patches X. and masked patches Xm. Here, N = -|o| |m]. The mask ratio is 7 |m] /N. The encoder takes the visible patches as input and outputs visible representations Zu; The regressor then predicts the latent representations of the masked patches Zm conditioned on the positions of masked patches, which are expected to be aligned with the representations Z,, computed from the encoder; The decoder then reconstructs the masked patches Y,, from the predicted encoded representations Zm to the form of target Y,.. CAE includes the reconstruction loss for the reconstruction of masked patches the expected targets (i.e., DALL-E), and the alignment. loss for the alignment of features from the encoder and the regressor.
2.2 Architecture
The overview of the pipeline of CAE v2 is shown in Figure and the computational graph is illustrated in Figure 5 (a) in Appendix. CAE v2 builds upon the foundation of CAE with two main structural modifica- tions: i) discarding the decoder and ii) applying two aligmment loss functions with CLIP latent as the target. Specifically, CAE v2 is an encoder-regressor architecture, in which visible latent alignment loss is directly applied on latent representations of visible patches computed from the encoder, and masked latent alignment loss acts on latent representations of masked patches predicted from the regressor. The supervision signals of these Lwo losses are both CLIP latents.
Encoder. The encoder F only receives visible patches X.. It maps the visible patches X, to the latent representations Z, across stack of transformer blocks that is based on self-attention. We employ series of ViI's (Dosovitskiy et al. 2021 as the encoder, including ViT-Tiny, -Small, -Base and -Large.
Regressor. Following Chen et al. (2023), the latent contextual regressor H. predicts the latent represen- tations of masked patches Zm, from Z, conditioned on the positions of masked patches. performs as the same cross-attention, which the queries are learnable mask tokens Qm, and the keys and values are both the concatenation of Z, and the output of previous layers (Q... for the first layer).
3

---

Published in Transactions on Machine Learning Research (09/2023)
Top acmurry of lineerr probing
78
Training Loss
78
74
MAE
72-
V2
73
IROT
CAE v2 my Rec
CAR
CAE v2 Rep
Figure 2: Analysis of the model convergence (left) training loss and (right) top-1 accuracy of lincar probing epochs. The x-axis denotes the pretraining epoch.
Decoder. The decoder is to map masked latent representations Zm to the format of target Ym- The decoder is based on a self-attention based transformer, different from the regressor based cross-attention. Different from CAE, CAE v2 does not have the decoder, since the regressor in CAE v2 can be responsible for prodicting the latent representations of masked patches.
2.3 Objective Function
Masking. We adopt the random block-wise masking strategy as in BETT [Bo et al. 2022). Instead of a unique mask ratio, we apply variable mask ratios for different sizes of models. In detail, we experimentally find that the optimal mask ratio is positively related with the model size, i.e., the larger the model, the higher the mask ratio. More analyses are provided Section 3.2
Targets. We utilize the vision branch of CLIP [Radford al. 2021 model to produce CLIP latent as the pretraining target. Specifically, the intact image fed into the target model 'T to get latent representations of patches, which are then divided into the visible latent targets Zn and the masked latent targets Zm according to the position of visible and masked patches, respectively.
Loss function. CAE v2 consists of two loss functions: visible latent aligmment and masked latent aligmment. Visible latent alignment loss (Z., Zw) is applied on the latent representations of visible patches Zu from the encodor, ensuring that the encoded representations lies the latent representation space of the visible latent targets Zy Masked latent alignment loss (Zms Zm) allows the latent representations of masked patches Zm predicted from the regressor to be close to the masked latent targets Zm
Overall, the whole loss function is:
8= E, Z,) + (z Zm).
(1)
By default, we use the cosine distance loss for both lo Zo)) and Ez (Zm, Zm), although different kinds of loss types have negligible influences on CAE v2. In this way, the Eq. [ can be refomulated as:
|m]
|m]
é = (Zo, Zo) + le (Zm, Zm)
1 10
Σ cos(z z())
Σ (1
cos (z zmm)
(2)
772
where 2 and z represent the latent representation from the encoder of the i-th visible patch and its corresponding CLIP latent. Similarly, z and z represent the latent representation from the regressor of the i-th masked patch and its corresponding CLIP latent. cos(u, v) T of two vectors.
u.v represents the cosine similarity
Differences between loss functions of CAE v2 and CAE (Chen et al.] 2023 CAE v2's loss functions (including visible latent alignment loss and masked latent alignment loss) differ from the original CAE's loss function (including reconstruction loss and alignment loss) on: i) Target. CAE v2's loss functions
4

---

Published in Transactions on Machine Learning Research (09/2023)
both use CLIP latent target, while the reconstruction loss of CAE uses DALL-E and the alignment loss uses latent representation from its encoder as target. ii) Loss type. CAF v2's loss functions both use the cosine distance loss by default, while CAE's reconstruction loss uses the cross-entropy loss and alignment loss uses the MSE loss. iii) Mode. CAE v2's loss functions directly optimize the encoder with visible latent alignment loss and the regressor with masked latent alignment loss, which encourages the encoder to be fully learned and fast. convergence. Differently, CAE's reconstruction loss applied on the decoder and alignment loss is used to align the representations from its regressor and its encoder, thus the encoder's learning is slow and implicit. These three differences on loss functions help our CAE v2 learn a better encoder than CAE. It can be verified on the linear probing downstream task that CAE v2 is superior than CAE by +10.3% (as shown in Table 6).
Discussion on the positioning of CAE v2 with respect to CLIP (Radford et al., 2021). CAE v2 is a MIM-based pre-training method, utilizing CLIP latent as the pre-training target. The reason of using CLIP latent that it can provide rich semantics (Wei et al. 2022b) since CLIP is pre-trained on large- scale image-lext pairs (ie., 400M image-pair privale data). CAE v2 is dedicated to learning a high-quality visual encoder with only pre-training on images without any ground-truth labels. In doing so, the CLIP training data is only implicitly utilized via the pre-trained CLIP latents, while is not explicitly Iearned for training. It is different from CLIP that pre-trains on image-text pairs, therefore, CAE v2 is expected to perform better than CLIP on image-based downstream vision tasks. Experiments in Section 3.3 validate the accuracy improvements of CAE v2 with respect to CLIP. It is note that the CLIP vision model only extracts feature representations (i.e., CLIP latents) as the pre-training targets, which is frozen during the pre-training of CAE v2. The encoder of CAE v2 is then transferred for the downstream task learning.
Study and discussion. Different from previous MIM works, CAE v2 directly optimizes the encoder with visible latent aligmment loss be on representations of the visible patches. Since supervision signals come from a semantically rich CLIP model, this loss can effectively facilitate the model convergence. It can be verified by Figure 2 that with 300-epoch pretraining schedule, the model with only using l, already achieves remarkable performance, which is largely superior than only using masked alignment loss lz. When expending the pretraining schodule to 800 epoch, the gap between only using ly and only using &g reduces, showing that & is good for the model convergence. Besides, the superior performance of only using & on the linear probing task indicates that the representative ability of the encoder improves. The underlying reason is that by successfully encourages the encoder to focus on the representation learning, instead of diverting some efforts to the masked patch prediction task. Consequently, the visible latent alignment plays an essential role in MIM pretraining task. Only using & also achieves comparable performance. Even on the high performance of only using bv, le can further improves the performance, showing that aligning the encoder and the decoder to the same feature space also effective for the encoder's learning.
3 Experiments
3.1 Settings
Baseline. The baseline of CAE v2 is replacing CAE's target to CLIP latent (as shown in No.1 in Table
Model structures. We study a scrics of vision transformer backbones (Dosovitskiy et al. 2021). includ- ing ViT-Tiny (12 layers with dim=192),, ViT-Small (12 layers with dim=384), ViT- (12 layers with dim=768), and ViT-Large (24 layers with d0m=1 Note that for ViT-Tiny, we follow Wang et al. (2022)
to increase the number of heads from 3 to 12, which gives better results on ImageNet-11 (Deng et al. 2009
For other models, strictly follow the model configurations as in Dosovitskiy et al. (021))
For the target model, we adopt the vision branch CLIP-Base/10 of CLIF for the pretraining experiments with WiT-Tiny/-Smalll-Bsse and CLIP-Large/14 for ViT-Large. The size of input images is 224x 224 for CLIP-Base/I1 16 and 196x 196 for CLIP CLip-Large/14.
Pretraining. Following most previous MIM methods (Bao el al. 2022 He el al. 2022 Chen et al. 2023 Wei et al.] 2022b Wang et al. 2022), we use ImageNet-IK 1K (IN- 1K) dataset (Deng et al. 2009 for all
pretraining experiments. The input images are with the size of 224 224 and are partitioned into 14 x 14
The official pretrained CL.P model is available at https////tttttccccccoa////////tt
5

---

Published in Transactions on Machine Learning Research (09/2023)
Table 1: Individual change from CAE to CAE v2. ly and la represent visible and masked latent alignment, respectively. The backbone is ViT-Base. All models are pro-trained for 300 epochs.
IN-IK
ADF20K
Object detection instance segmentation
No.
Method
LIN
FT
mloU
Mask R-CNN
Cascade Mask R-CNN
CAE
64.1
83.6
48.3
48.4 & 42.6
51.6 & 44.6
+ CLIP (baseline)
75.8
83.6
49.5
49.3 & 43.0
51.7 & 44.6
2
+ 11
78.4
85.0
52.7
51.8 & 44.3
53.8 & 45.8
+ to (CAE v2)
80.7
85.5
53.4
52.4 45.3
54.2 & 46.5
Table 2:
Ablation studies for visible latent align- ment loss f, and masked latent aligmment. loss & in CAE v2. All models are pre-trained for 300 epochs.
Table 3: Ablation studies for the type of loss function
in CAE v2. Both by and lz are used and all models are pre-trained for 300 epochs. We use the cosine distance by default (markod in gray
Default scttings are marked in gray
Loss function
IN-IK
ADE20K mloU
IN-IK
ADE20K
Model
Model
Type loss
by
LIN
FT
LIN
FT
mIoU
64.9
77.2
44.1
MSE
69.1
77.3
44.8
ViT-Tiny
68.8
77.4
44.2
ViT-Tiny
Smooth-/1
69.4
77.6
43.7
69.3
77.8
44.7
Cosine distance
69.3
77.8
44.7
73.9
82.4
49.6
MSE
77.3
82.7
49.8
ViT-Small
77.3
82.8
49.6
ViT-Small
Smootth-1
77.4
82.8
49.8
77.5
83.1
49.8
Cosine distance
77.5
83.1
49.8
78.4
85.0
52.7
MSE
80.4
85.3
52.9
ViT-Base
80.5
85.2
53.1
ViT-Base
Smooth-21
80.5
85.2
52.0
80.7
85.5
53.4
Cosine distance
80.7
85.5
53.4
patches with the patch size being 16 16 across all sizes of models. We apply random resized cropping and horizontal flipping during pretraining.
The default mask ratios in the pretraining stage are set to 15,, 25%, 50%, and 60% on ViT-Tiny, -Small. -Basc, and Large, respectively. Without clear specification, we use AdamW (Loshchilov & Hutter 2019 for oplimization and train CAE v2 for 300 epochs across all scales of ViTs (Dosovitskiy et al. 2021 More
detailed settings are listed in the appendix.
Evaluation. We evaluate our CAE v2 on various downstream tasks. For image classification, we conduct evaluations on ImageNet-IK (Deng al. 2009) with both linear probing (LIN) and fine-tuning (FT) proto-
cols. For semantic segmentation, we follow BETT (Bao al.) 2022) to use UperNet (Xiao et al. 2018 and report the mToU on ADE20K (Zhou et al. we use COCO (Lin et al.) 2014) as the evaluation dataset. We adopt both Mask R-CNN (He et al. 2017)
2017) dataset. For objection detection and instance segmentation.
and Cascade Mask R-CNN (Cai & Vasconcelos, 2018) frameworks and report AP and AP on the COCO val split. Please refer to the appendix for more training details on various downstream tasks.
3.2 Ablation Studies
In this subsection, we first illustrate individual change from CAE to CAF v2, and then analyse the effective- ness of visible latent alignment loss and masked latent alignment loss. After that we investigate mask ratio, type loss, layer number of the regressor, and masking sampling strategy. Details are provided below. Individual change from CAE to CAE v2. We experiment on each individual change from CAE to CAE v2 in Table 1 CLIP latent target. We first change CAE's target from Dall-E to CLIP. Compared with CAE, this setting improves the performance, e.g., +11.5%, 1.2% and 0.9% ImageNet-lk's 1k's linear probing, ADE20K and COCO object detection, showing that CLIP latent as target can improve the representation ability of the encoder. This selting can be seen as our baseline. ii) Masked latent alignment. We then change the loss function to the proposed masked latent alignment. The performance further improved by +2.6%, 1.4%, 3.2%, 2.5% on ImageNet-1k's lincar probing, ImageNet- fine-tuning, ADE20K, COCO object detection, respectively. It verifies the effectiveness of masked latent alignment. iii) Visible latent alignment. We further add the proposed visible latent alignment to directly optimize the encoder. The

---

Published in Transactions on Machine Learning Research (09/2023)
09.0
77.8
43.7
63.6
Litrar Preiiig
63
61.9
Fine tolotaaing
77.5
77.1
770
ADE2/K
15
30.9
76.8
52.0
40
41.8
7
15
25
50
75
50
15
25
75
15
25
50
75
90
77.0
62.7
83.1
49.0
Liter Probing
74.0
48.7
Finctziing
ADE2KK
10
46.5
VIT-Smell
65.0
73.0
15
50
75
15
25
75
15
25
50
75
80
00.3
80.7
85.3
05.5
52.5
53.4
054
794
Linertreiigg
84.3
512
Finc-tatig
ADE2/K
08
24
VIT-Base
71.8
88
82.1
45
66.5
15
50
75
90
25
75
15
25
50
75
Mosk mbo (5)
Mlac oeto (54)
Misk mio (%)
Figure 3: The optimal mask ratio is positive correlation with the model size in CAE v2. From top bottom, the table shows the linear probing (left column), fine-tuning (middle column) ImageNet-11K and semantic segmentation (right column) ADE20K with ViT-Tiny, -Small, and -Base model structure.
results are further increased by +2.3%, 0.5%, 0.7% and 0.6%, demonstrating the effectiveness visible latent alignment.
Visible latent alignment loss us. masked latent alignment loss. Previous MIM methods (Bao et al, 2022 He et al. 2022, Peng et al. 2022b Liu ct al., 2022b Chen et al., 2023 typically apply the loss function the predicted masked palches, illustrated in Figure Dilferently, CAE v2 applies two aligument losses on latent representations, i.e., visible latent alignment loss Eu on the representations of visible patches from the encoder, and masked latent alignment loss la on the representations atnenstaaions of predicted masked patches from the regressor. These two losses are independent that can be used either both of them or any individual one.
Table 2 shows the ablation results. One can see that the proposed t, and by can steadily improve the performance compared to the baseline (í.e., No.1 in Table D Specially, CAE v2 only with ly outperforms the baseline by large margins (e.g., +2.6% on linear probing), showing the important role of masked latent aligmment. Meanwhile, only using l, outperforms the strategy of only utilizing fg, which verifies that the direct optimization on the encoder is beneficial for enhancing the encoder's representative quality. Combining by with l, can achieve the best. performance. Note that although the encoder is subject to the implicit influences from the masked patch prediction task to some extent, CAE v2 can still ensure a good optimization on the encoder, since the encoder directly receives supervision signals from the scmantically rich CLIP model.
Mask ratio. We conduct experiments to analyse the influence of mask ratio, ranging from (15%, 25% 50% 75%, 90%], across different scales of models. The results are listed in Figure models prefer different values of mask ratios. The op- timal mask ratio exhibits a positive correlation with the model size, i.e., as the model size increases, a higher mask ratio performs better, and conversely, a smaller model benefits from lower mask ratio. We also observe that when the mask ratio exceeds a stable value, the performances declines rapidly. The under- lying reason might be that it is challenging for small- sized models to predict the masked patches from a limited subset of patches where most contextual in- formation is missing. Therefore, using more visible patches can reduce the difficulty of the masked pre- diction task, benefiting for the model convergence. In contrast, large-scale models are easily subject to over-
It shows that different scales of
7-1556
2556
5056
78%
-0%
Figure 4: Illsttaaionn of corrupted images with differ- ent mask ratios via the random sampling strategy (top row) and the block-wise sampling strategy (bot- tom row). We use the block-wisc sampling strategy by default.
7

---

Published in Transactions on Machine Learning Research (09/2023)
Table 4: The influence of the layer number of the regressor in CAE v2 and the decoder depth in MAE (He et al. 2022). Models are pre-trained for 300 epochs in CAE v2 and 800 epochs in MAE.
Table 5: Ablation studies for the mask sampling strategy in our CAE v2. All models are pre-trained for 300 epochs. We use the block-wise sampling by default (marked in gray
Gray represents the default setting.
Method
Model
# Layer
LIN
IN-IK
FT
Model
Mask strategy
LIN
IN-1K
FT
ADE20K
mloU
1
65.5
84.8
Random
69.1
77.4
43.8
MAE
Large
4
71.9
84.9
ViT-Tiny
Blockwise
69.3
77.8
44.7
8
73.5
84.9
Random
77.4
82.7
49.0
0
80.5
85.2
ViT-Small
Blockwise
77.5
83.1
49.8
1
80.6
85.3
CAE v2
ViT-Base
Random
80.6
85.4
52.4
4
80.7
85.5
ViT-Base
Blockwise 80.7
85.5
53.4
8
80.5
85.4
fitting when learning representations from overabundant visible patches, and thus a high mask ratio can make the masked prediction task more challenging to effectively mitigate the over-fitting problem. We set a mask ratio of 15%
for VVT-TinyySSmally-Bae /aa///BBae/aagggee in our work unless specified.
Type of loss. We use cosine distance loss as illustrated in Eq. for both visible and masked latent alignment loss functions. Instead of the cosine distance, it is possible to use other types of loss, like smooth-f1 and mean square error (MSE), which are studied and utilized in the previous works (Peng et al. 2022a Liu et al. 2022b Wei et al. 2022ch. We presents the results of these three kinds of loss types when using latent and masked alignment simultaneously in Table 3 The results show whatever the loss type is, our CAE v2 can achieve similar performance, while the cosine distance is slightly better than other types (e.g., 30.5% on linear probing and fine-tuning tasks). As previous works do (e.g., MASKDISTILL, (Peng et al. 2022b) we choose the optimal loss lype for CAE v2, i.e., cosine distance by default.
# layers in the regressor. We study four choices, f.e., {0-layer, 1-layer, 4-layer, 8-layer), for the number of layers in the latent contextual regressor. The results in linear probing and fine-tuning in Table 4 show that the effect of the layer number is minor. In contrast, the decoder depth in MAE (He et al. 2022) has a large impact on downstream tasks, especially lincar probing. We conjecture that our CAE v2 benefits from the explicit visible latent alignment on the visible patches, ensuring the representative quality of the encoder. When the regressor depth is O, CAE v2 reduces to only using visible latent alignment loss, which still achieves satisfactory performance, indicating the effectiveness of this loss. We empirically use 4-layer regressor for ViT-Base/-Laree and 1-layer regressor for TiT--aly/SSmal by default.
Mask sampling strategy. We also compare different mask sampling strategies in our CAE v2, i.e. the random sampling (He et al.) 2022) and the bise sampling (Bao et al. 2022 Chen et al. 2023 Wei et al. 2022b) (as illustrated in Figure
Table 5 shows that there are only approximately 0.1% gaps between these two sampling strategies on linear probing and fine-tuning, but the block-wise sampling performs better than the random sampling on semantic segmentation (e.g., 53.4% us. 52.4% on ViT-Base). We use the block-wise sampling strategy as the default option.
3.3
Main Results
Image classification on ImageNet-11K, Table D shows comparisons of different models using two evalu- ation methods: linear probing and fine-tuning.
In linear probing on ImageNel-IK,, CAE v2 demonstrates significant improvements over previous methods with other targets, e.g., BETT (Bao et al. 2022), MAE (He et al. 2022), CAE (Chen et al. 2023 and MaskFeat (Wei et al. 2022a). These gains are expected, as CLIP latents contain rich semantics than other targets. Compared to methods using CLIP latent as the target (MVP (Wei et al. 2022b) and MILAN (Hou et al., 2022)), ) CAE v2 also achieves superior performance (on ViT-Base with 300 epoch pretraining, CAE v2 us. MVP: 80.7% us. 75.4% and CAE v2 us. MILAN: 80.7% us. 78.9%).
In the fine-tuning task on ImageNet-1K, CAE v2 also achieves comparable performance across all scales of ViTs. Specifically, CAE v2 achieves 85.5% top-1 accuracy with ViT-Base surpassing all prior met
8

---

Published in Transactions on Machine Learning Research (09/2023)
Table 6: Pretraining evaluation the top-1 accuracy (%) linear probing (LIN) and fine-tuning FT
ImageNet-11 (Dnn et al. 2009); and mloU (%) on semantic segmentation on ADE20K Zhou et al.
2017 7
means our implementation using the officially released code. & means the results from Chen et al.
2023
and Wei al. ( All other results except for ours are from the original papers. The methods achieving
the best, second best and third place performance denoted in bold red, green and blue respectively.
IN-IK
ADE20K
Methods
# Epochs
Target
LIN
mloU
Methods using FYT-Tin:
MAE-Tiny (Wang et al.
2022
400
RGB
23.4
76.2
CAE (Chen et al.
2023
300
DALL-E
28.1
75.9
38.3
Distilled MAE-lite life
Wang et al.
2022
400
RGB
76.5
G2SD) (Wei et al.
2023
200
MAE-Bask
77.0
44.5
CAE v2
CLIP-Base
69.3
77.8
44.7
Methods using ViT-Small:
MoCo v3 Chen et al.
2021
300
Self-EMA
73.1
81.7
BETT
Bao et al. 2022
300
DALL-E
15.7
81.7
Mask
[EI-Nouby et
2021
300
DALL-E
81.5
CAE
Chen ct
2023
300
DALL-E
51.8
82.0
IBOT
al.
2112a
3200
Self-EMA
82.3
45.4
G2SD Wei al.
2023
MAE-Base
82.5
48.0
CAE v2
300
CLIP-Bass
77.5
83.1
49.8
Methods using ViT- Bose:
MoCo v3 [Chen et al. 2021
300
Self-EMA
76.5
83.2
47.2
DINO Caron
2021 M
Self EMA
77.3
47.2
iOOT Zhou al. 2022a
Self-EMA
50.0
BETT
Bao et
2022
DALL-E
56.7
83.2
45.6
SimMIM [ic ot al 2022
800
RGB
56.7
83.8
MAE
He et al. 2022
1600
RGB
68.0
83.6
48.1
CAE
hen et al. 2023
1600
DALL-E
70.4
83.9
50.2
SdAF
Chen et, al.
21/22
300
Self FMA
64.9
84.1
48.6
STM
Tao
2022
1600
SeIAEMA
76.4
83.8
MaskFeat
Wei et
2002a
1600
84.0
EI-Nou Maskt
et
2021
300
DALL-E
83.6
45.7
PeCo
Dong et al.
2021
800
VQGAN
84.5
48.5
data2vee [Bevvski ct
2022
800
Sclf-EMA
84.2
CMAE H Huang et al.
2022b
1600
RGB
84.7
50.1
ExtreMA (wu
2023
300
Sclf-EMA
73.3
83.7
47.9
CLIP (Radford
2021
Text.
80.2
84.9
51.1
MaskCLIP (Dong et
1600
Text
72.9
84.1
50.8
MVP Wei et al. 2022b
300
CLIP-Base
75.4
84.4
52.4
FD-CLIP [Wei et al. 22022
300
CLIP- Base
80.3
84.9
52.8
MILAN (Hou et al. 2022
400
CLIP-Bas Base
78.9
85.4
52.7
BETT V2 [Png
2022a
1600
VQ-CLIP-Bas
85.5
53.1
dBOT (Liu al
20226)
CLIP-Base
85.7
52.9
MASKDISTILL Peng et al.
2022b
300
CLIP-Bass Base
85.0
53.8
CAE v2
300
CLIP-Base
80.7
85.5
53.4
Methods using ViT Varge:
MoCo v3 Chen et al. 2021
300
Self-EMA
84.1
49.1
BEIT Bao
2022
1600
DALL-E
85.2
53.3
IBOT
Zhou et ml
2022a
1200
Self-EMA
81.0
84.8
MAE He el al. 2022
1600
RGB
75.8
85.9
53.6
CAE
Chen et a.
2023
1600
DALL-E
78.1
86.3
54.7
data2vec
(Beevski ct al.
2022
1600
Self-EMA
86.6
CLIP Radford ct
2021
Text
83.5
86.1
53.5
MVP Wei al.] 2002
CLIF-Baa
86.3
54.3
BEIT
[Png et. al.
20224
1600
VQ-CLIP Base
87.3
56.7
FD-CLIP
Wei ct al 2022c
300
CLip-Large
84.8
87.7
55.7
MILAN
Hou el. all 2022
400
Cl.IP-Large
84.3
87.8
57.9
dвOT [Luu al.
20226)
1600
CLIP-Large
87.8
56.2
MASKDISTTLL.
Peng al.
2022b
300
CLIIP-Large
87.6
57.9
CAE v2
CLIP-Large
84.4
87.6
57.9

---

Published in Transactions on Machine Learning Research (09/2023)
Table 7: Pretraining evaluation on object detection (DET) and instance segmentation (INS) on COCO Lin et al. 2014) with Mask R-CNN (He et al., 2017)) (left) and Cascade Mask R-CNN (Cai & Vasconcclos 2018 (right). All experiments are trained with the 1x schedule (12 epoch). Results except for CAE v2 are from Chen et al. (023) and Liu et al. (2022b) #Epochs refers to the pretraining epochs ImageNet-
denotes multi-crop pretraining augmentation. The methods achieving the best and second best performance
are denoted in bold red and green respectively.
Mask R-CNN
Cascade Mask R-CNN
Method
# Epochs
DET
INS
DET
INS
AP
APM
AP
APM
Methods using ViT-Small:
DefT
Touvron al. 2021
300
43.1
38.4
MoCo
v3
(Chen et al
2021
300
43.3
38.8
BETT
Bao et al.
2022
300
35.6
32.6
CAE Chen et al.
2023
300
44.1
39.2
BOT
Zhou
2022a
3200
19.4
42.6
CAE v2
300
49.0
42.2
51.5
43.9
Methods using VTT Base:
DeiT Touvron al. 2021
300
46.9
41.5
MoCo v3'
CChen et al.
202
300
45.5
40.5
DINO
Caron al. 2021
400
46.8
41.5
BETT Bao et al.
2022
800
42.1
37.8
MAE He
2022]
1600
48.4
42.6
51.3
41.3
data2vec (Baevski et a..
2022
800
41.1
37.0
iBoT
Zhou al.
2022a
1600
48.6
43.1
51.2
44.2
CAE Chen et al.
2023
1600
50.0
44.0
52.9
45.5
dBoT [Liu et 2022b
1600
53.6
CAE v2
300
52.4
45.3
54.2
46.5
Methods using ViT-Le
MAE (He
2022)
54.0
47.1
data2vec (Baevski et al.
2022
1600
46.1
41.0
BoT Zhou al.
2022a
1600
50.6
44.7
CAE CCen et al.
2023
1600
54.5
47.6
dBoT [Lu et al.
2022b
1600
56.8
CAE v2
300
55.2
47.3
56.9
48.6
except for dBOT LLu et al. 2022b). Note that dBOT only outperforms CAE v2 by 0.2% while requiring 1600 epochs for pretraining, demonstrating CAE v2's higher efficiency.
Semantic segmentation on ADE20K. Semantic segmentation is challenging task that needs classify
every pixel in image according to various semantie labels. CLIP (Radford al. 2021 latent serves
powerful target for this task, showing clear advantages. As shown in Table G CAE v2 significantly
outperforms methods pre-trained with other targets, e.g., 3.2% mloU improvement over CAE (Chen et al. 2023) when using ViT-Base. comparison to MVP (Wei et al. 2022b) and BEIT V2 (Peng et al. 2022a CAE v2 surpasses them with the same or less pretraining epochs. The superior performance persists when transitioning to ViT-Large, with which CAE v2 achieves 57.9% mloU on ADE20K (Zhou et al. 2017 outperforming previous methods.
COCO. We evaluate the pre-trained models COCO [Lin et al. 2014) with Mask R-CNN (He et al. 2017) and Cascade Mask R-CNN (Cai & Vasconcelos, 2018 He et al. (017]) in Table 7 We report the results for 1x (12 epochs) training schodule. Compared with other pretraining methods, CAE v2 excels at both two configurations. With Mask R-CNN, CAE v2 achieves a 4.9% increase with ViT-Small and 2.4% increase with ViT-Base on AP' compared to previous best method Chen et al. (2023). The superior performance maintained when employing Cascade Mask R-CNN as the fine-tuned model. For example, CAE v2 achieves 0.6% increase with ViT- and 0.1% increase with ViT-Targe Apt compared to Liu et al. 2022b).
Object detection and instance segmentation
10

---

Published in Transactions on Machine Learning Research (09/2023)
Compared with related works (e.g., MASKDISTILL Peng et al. 2022b) and MVP (Wei et al. 2022b] that also utilize CLIP latent as target, the performance gap may be caused by: i) architecture choices. CAE v2 is built upon CAE (Chen et al. 2023) with structural modifications, while MASKDISTILL and MVP are based on BEiT Bao et al. (2022). ii) method. CAE v2 introduces visible and masked latent alignment to directly optimize the encoder and the regressor, respectively. Differently, MASKDISTILL, and MVP supervise the overall model without separation. iii) loss type. Based on differences of the overall pipeline, different loss types perform inconsistent for achieving the best results. In detail, CAE v2 and MVP employ the cosine distance, while MASKDISTILL uses Smooth-E1.
4 Related Work
Masked image modeling (MIM) aims to learn transferable vision representations for various downstream tasks. It is inspired by the successful large-scale pretraining for transformers (Vaswani et al.
2017) with
masked language modeling (MLM) (Devlin et al.
2019 Chen et al. 2020a Brown et al., 2020
Dong et al.
2019 in NLP and can serve as a pretext task in self supervuue vision pretraining (Caron et al.
2018 Doersch
et al.
2015; Oord et al., 2018
Ermolov et al., 2021 Goyal et al.
2021: Li et al.) 2021 Zbontar et al. 2021:
He et al. 2020 Chen et al.
2020c bb Grill et al.
2020 Liu et al.) 2022a; Dong et al.
2022
Zhang et al.
2023
Huang et al. 2022a
Zhou et al. 2022b Assran et al. 2022
Yi et al. 2023
Li et al.
2022).
MIM
methods (Bao et al., 2022
He et al.) 2022; Xie et al., 2022 Chen et al. 2023
Baevski et al.
2022
Singh
ct al. 2022 Peng et al. 2022a Fang ct al. 2023b a Pong et al. 2022b Liu et al. 2022b Singh et al. 2023)
follow a mask-then-predich pipeline of (i) corrupting an image by masking several image patches based on
a pre-defined mask ratio and then (ii) learning to predict the missing content under specific targets as a
reconstruction task. Our CAE v2 follows this pipeline while using more semantic targets and incorporating
two alignment losses to effectively facilitate model convergence and improve the encoder's representative ability. In the following, we will discuss the related works.
Targets. Existing MIM methods explore different kinds of targels within their frameworks, including RGB
pixels (ee et al. 2022 Gao et al. 2022). , hand-crafted features like HOG descriptors @Wei et al.]
2022a
discrete visual tokens (Boo et al.] 2022 Chen et al. 2023 El-Nouby et al.] 2021
Peng et al.] 2022a Dong
et al.] 2021), and feature representations from momentum models (Tao et al. 2022 Chen et al. 2022
Wu et al. 2023) or from external pre-trained models (Wei et al. 2022a b Liu et al. 2022b Peng et al.
2022b; Fang et al.] 2023bb a Gao et al.) 2023 Ren et al. 2023). Among these methods, the features from
the vision branch of CLIP model play a significant. role as supervision signals, substantially improving the following downstream task performance (Wei et al.
2022b Peng et al., 2022b Fang et al. 2022b Liu et al.,
2023b Gao et al. 2023 Ren et al. 2023 Hou ct al. 2022 Ren et al. (023). The underlying reason is that
CLIP latents can provide multi-modality knowledge to aid vision-only MIM learning, which is beneficial for improving the ability of the encoder that is used for downstream tasks. In our CAE v2, we also use CLIP latents as the target for the model pretraining. Meanwhile, we go further one step to tap the potential of CLIP latent to further improve the quality of the encoder via the following introduced alignment losses.
Loss function. Most litcrature Wei et al.) 2022a He et al. 2022; Chen et al., 2023 Bao et al. 2022:
Peng et al.) 2022a b Fang et al. 2023b at Ren et al. 2023 Liu et al. 2022b apply the optimization to the predicted masked patches for the the masked patch prediction task. Following these methods, we also utilize a loss function for masked patch prediction task, which is called masked latent alignment since the optimization is applied to the representations of masked patches from the regressor. Furthermore, we find that directly applying supervision on the latent representations of visible patches from the encoder can constantly improve the model convergence and downstream transferring performance. We refer to this loss as visible latent aligument. Both visible and masked latent alignment use CLIP latent as the supervision target. In this way, the encoder can directly learn the semantic information from the powerful CLIP model, showing fast model convergence and high-quality encoder.
The most similar work to our approach is CAE (Chen et al., 2023 Our CAE v2 is an improved variant of CAE, yet there is no decoder. Meanwhile, both visible latent alignment and masked latent aligument are conducted on the latent representations that need to be similar to those from the feature space of CLIP model. Instead, CAE contains a decoder to recover the predicted masked representations to the discrete tokens, and align the predicted representations of the masked patches to the encoded representation space.
11

---

Published in Transactions on Machine Learning Research (09/2023)
In addition, several concurrent works also attempt to explore the potential of CLIP latents (Wei et al 2022b;
Peng et al. 2022a b. Fang et al. 2023b a Ren et al.) 2023 Liu et al. 2022b). For example, MILAN (Hou
et al. 2022) focuses on the architecture design, dBOT (Liu et al. 2022b) proposes training strategy, and MASKDISTILL (Pong et al. 2022b carefully studics optimization methods. FD-CLIP (Wei et al.) 2022c)
aims to improve the CLIP model with CLIP latent using feature distillation on intact image patches, which is not a MIM-based method. It worth noting that our CAE v2 orthogonal to these works. Specifically, we concentrate on tapping the potential of CLIP latent, and present CAE v2 with visible latent alignment and masked latent alignment that both use CLIP latent as the pretraining target, which is beneficial for learning high-quality encoder and facilitating model convergence. The proposed CAE v2 demonstrates that the explicit visible latent alignment is beneficial for the representation learning of the encoder. Moreover, we experimentally explore the relationship between the mask ratio and the model size. We hope our findings can provide valuable guidelines for the future MIM pretraining.
5 Conclusion and Limitation
This paper introduces CAE v2, context autoencoder with CLIP latent alignment, using the latent features from the vision branch of CLIP as the supervision target. CAE v2 consists of two aligmment. losses, i.e., visible latent alignment loss on the representations of visible patches from the encoder, and masked latent alignment loss on the predicted representations of masked patches from the regressor. Extensive analyses and experiments show that our CAE v2 effectively facilities model convergence and achieves high performance on various downstream vision tasks. CAE v2 also outperforms the CLIP vision encoder on these vision tasks, although CAE v2 is not designed for direct comparisons with CLIP.
Limitation. Limited by resources, we do not examine larger models, such as ViT-Huge and ViT- We plan to explore this in the future and hope that the findings in CAE v2 will provide valuable guidance.
References
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In ECCV, pp. 456-473, 2022.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, pp. 1298 1318- 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NIPS, 33:1877-1901, 2020.
Zhaowei Cai and Nuno Vasconcclos. Cascade r-cmn: Delving into high quality object detection. In CVPR, pp. 6154-6 0164,,66,2246 2018.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, pp. 132 149, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Tn ICCV, pp. 9650 9650-9660, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Nya Sutskever. Gener- ative pretraining from pixels. In ICML, pp. 1691 1703, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607, 2020b.
12

---

Published in Transactions on Machine Learning Research (09/2023)
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. IJV,, 2023.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. preprint arXiv:2003.04 04297, 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers. In ICCV, pp. 9640-9649, 2021.
Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In ECCV, pp. 108-124, 2022.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pro-training text encoders as discriminators rather than generators. In TCLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248 255, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirec- tional transformers for language understanding. In NAACI, pp. 4171-4186, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prodiction. In ICCV, pp. 1422-1430, 2015.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and IsiiooWuen Hon. Unified language model pre-training for natural language understanding and generation. NewrIPS, 32, 2019.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:2111.199 2021.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang. Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked antocncoders for vision bert pretraining. In ECCV, pp. 247-264, 2022.
Xiaoyi Dong, Yinglin Zheng, Jianmin Bao, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Wen Fang Chen, Dong, and Yu Nenghai. Maskelip: Masked self-distillation advances contrastive language-image pretraining. In CVPR, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112, 12. 10740, 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In ICML, pp. 3015-3024, 2021.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for ncon genesis. arXiv preprint 2XXi::208.11391, 2023a.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023b.
Peng Gao, Teli Ma, Hongsheng Li, Ziyi Lin, Jifeng Dai, and Yu Qiao. MCMAE: Masked convolution meets masked autoencoders. Tn NIPS, 2022.
13

---

Published in Transactions on Machine Learning Research (09/2023)
Peng Gao, Renrui Zhang, Rongyao Fang, Ziyi Lin, Hongyang Li, Hongsheng Li, and Qiao Yu. Mimic before reconstruct: Enhancing masked autoencoders with feature mimicking. arXiv preprint or orXis:2303.05, 054 2023.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. preprinl
02677, 2017.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features the wild. preprint (rXii:2103.01955 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self- supervised learning. In NeurIPS, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 16000-16009, 2022.
Zejiang Hou, Fei Sun, Yen- Chen, Yuan Xie, and S. Y. Kung. Milan: Masked image protraining language assisted representation. ArXiv, abs/2208.06049, 2022.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki. Green hierarchical vision transformer for masked image modeling. orXiv preprint arXio:2205.13333 2022a.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming- Ming Cheng, Dongmei Fu, Xiaohui Shen, and Ji- ashi Feng. Contrastive masked autoencoders stronger vision learners. arXiv preprint or S1::2277.13592 2022b.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
Xiaotong Li, Yixiao Ge, Kun Yi, Zixuan Hu, Ying Shan, and Ling yu Duan. mo-beit: Multi-choice dis- cretization for image bert pro-training. In ECCV, 2022.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740 755, 2014.
Hao Liu, Xinghua Jiang, Xin Li, Antai Guo, Deqiang Jiang, and Bo Ren. The devil is in the frequency: Geminated gestalt autoencoder for self-supervised visual pre-training. arXio preprint arXiv:2204.088 088227 2022a.
Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representations for masked autoencoders. arXiv preprint arXiv:2209.082 03917, 2022b.
Tly Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent. with warm restarts. In ICLR, 2017.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In JCLR, 2019.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. or arXiv preprint Xrii 1ft:8807.00 007.03748, 2018.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with voctor-quantized visual tokenizers. arXiv preprint or Xiw:2020 X6632208.06666, 2022a.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furn Wei. A unified view of masked image modeling. TMLR, 2022b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, pp. 1455- 2020.
Sucheng Ren, Fangyun Wei, Samuel Albanie, Zheng Zhang, and Han Hu. Deepmim: Deep supervision for masked image modeling. arXie preprint orXiv:2333.08 088117 2023.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Kumar Mahajan, Ross B. Girshick, Piotr Doll'an ar, and Laurens van der Maaten. Re- visiting weakly supervised pre-training of visual perception models. In CVPR, pp. 794 804, 2022.
Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, et al. The effectiveness of mae pro-prctraining for billion-scale pretraining. arXiv preprint arXiv:2303.13) 13496, 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXie preprint Xiw:210440996, 2021.
Christian Szogedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint 2Xxx:2206.01200 2022.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e Jeegou. Training data-efficient image transformers & distillation through attention. In ICML, pp. 10347 10357, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NewrIPS, 2017.
Shaoru Wang, Jin Gao, Zeming Li, Jian Sun, and Weiming Hu. A closer look self-supervised lightweight vision transformers. arXiv preprint 2xXio22255.14455 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yus Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-sueeriieed visual pre-training. In CVPR, pp. 14668-14678, 2022a.
Huang Wei, Zhiliang Peng, Li Dong, Furu Wei, Jianbin Jiao, and Qixiang Ye. Generic-to-specife distillation of masked autoencoders. arXiv preprint wxXi:2302.14774. 2023.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodality-guidel visual pre-training, ECCV, 2022b.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXin:2205.44141 14141, 2022c.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Zhirong Wu, Zihang Lai, Xiao Sun, and Stephen Lin. Extreme masking for learning instance and distributed visual representations. TMLR, 2023.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pp. 418 4344,4, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, pp. 9653 9663, 2022.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu,, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. In ICLR, 2023.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. preprint arXiv:1706 100: 700888888888 2017.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularivation strategy to train strong classifiers with localizable features. In ICCV, pp. 6023 6032, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, pp. 12310 -13000, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian. Hivit: simpler and more efficient design of hierarchical vision transformer. In ICLR, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through adc20k datasct. In CVPR, pp. 641, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image BERT pre-training with online tokenizer. In ICLR, 2022a.
Qiang Zhou, Chaohui Yu, Hao Luo, Zhibin Wang, and Hao Li. Mimco: Maskod image modeling pre-training with contrastive teacher. In ACMMM, pp. 4487-4495, 2022b.
16

---

Published in Transactions on Machine Learning Research (09/2023)
A Appendix
B Implementation Details
B.1 Model structures
Encoder. For ViT-Tiny, we follow Wang et al. (2022) to increase the number of heads from to 12. All others remain the same the standard ViT architecture (Dosovitskiy et al. 2021]. The end of encoder is fully-connected layer (FC) followed by layer normalization layer (LN) to map the target dimension to 512 for CLIP-Base and 768 for CI.PP-aare. Note that FC and LN are discarded during the fine-tuning on downstream tasks.
Regressor. The regressor in CAE v2 is a stack of cross-attention based transformer blocks. For ViT- we set the depth of regressor to 1/1/4/4, and the width to 96/384/768/ 1024. We also add a FC-LN after the regressor to map the target dimensson,, and these two layers share the same parameters to those in the encoder.
Bewy/-Small/-Bas/--aarrr
Targets. The targets in CAE v2 are derived from the last layer of the vision branch of CLIP model (Radford et al. 2021 i.e., the output of the projection head in the CLIP model with the dimension being 512 for CLIP Base and 768 for CLip-Large.
B.2 Pipeline Comparison
We compare the computational graphs for (a) our CAE v2, (b) CAE (Chen et al 2023 (c) BETT (Bao et al, 2022), and (d) MAF (He et al., 2022) in Figure 5 The key novelty of CAE v2 is visible latent alignment and masked latent alignment, which are directly optimize the feature space of the encoder and regressor to close to CLIP latents.
(a)
(b)
(c)
Figure 5: The computational graphs for (a) our CAE v2, (b) CAE (Chen et al. 2023 (c) BETT (Bao et al, 2022), and (d) MAE (He et al., 2022). The parts in cornflower blue are for loss function. The encoder in (a) and (b) receives the visible patches and outputs their latent representations. The regressor predicts masked latent representations. (a) Visible latent alignment is applied on the representation learning of the encoder, and masked latent alignment loss is for predicting representations of masked patches. (b) The decoder reconstructs the masked patches from masked latent representations to discrete tokens. (c) The input includes both visible patches and mask queries, and the representations for both of them are updated within the function R. (d) The encoder F only processes the visible patchos X,, and the decoder R. inputs both latent representations and mask queries. For simplicily, the positional embeddings are not included in computational graphs.
B.3 Training Details
pretraining. We present the default pretraining settings in Table The pretraining epoch for all exper- iments 300. For the CLIP model, we use base-size model for ViT- Tiaaiinnnaall-aaaaat and the large-size model for ViT-Large. The input size is 224x224 for CLIP-BA Base and 196x 196 for CLIP-Large. The input
images are the same as those in the backbone model. We only use ImageNet-1K (Deng et al. 2009 for pretraining.
Linear Probing on Iet (Deng et al., 2009). Following He et al. 2022
Doersch et al.
(2015), we adopt an extra batch normalization layer (Toffe & Szegedy 2015) without affine transformation
after the encoder. The default training details are shown Table 9
2The official pretrained CL.P model is available at http::////ttiaaceea//m/////s//aa
17

---

Published in Transactions on Machine Learning Research (09/2023)
Table 8: Pretraining setting for CAE v2 on ImageNet-IK.
Value
Config
ViT WiT-TinySmally /Base/ Bass//Lrgee
Optimizer
AdamW
Loshchilov & Hutter
2019
Peak learning rate
1.5e-
Minimal learning rate
le-5
Optimizer momentum
B1, 0.9, 0.98 (Chen et al.
2020a
Batch size
2048
Learning rate schedule
Cosine decay Loshchilov & Hutter 2017
Warmup epochs (Goyal et al. 2017
10
Max training epochs
300
Gradient Clipping
3.0
Weight decay
0.05
Drop path
0.1/0.1/0.100.2
Mask ratio
0.15/0.25/0.5/0..
Mask strategy
Random block-wise sampling
Input scale
min=0.4, max=1.0
Data Angmentation
Random resized crop & horizontal flip 0.4
Color jitter
Input size
224x224
Table 9: Linear probing setting for CAE v2 on ImageNet-IKK 1K.
Value
Config
VIT- Viny-Tnny/ /Small/ Base/ Large
Optimizer
LARS
You et al.
2017
Base learning rate
0.1
Weight decay
0
Optimizer momentum
0.9
Batch size
16384
Warmup epochs
10
Max training epochs
90
Data augmentation
Random resized crop & horizontal flip
Fine-tuning on ImageNet-1K (Deng et al. 2009). The default settings for ImageNet- 1K fine-tuning is 100/200/1000 100/50 for fair com- parison. The input size of all scales of models is 224x224. For ViT-Tiny, sweep the learning rate ranging from 1e-3 to 4e-3. For Se/SSaallBBaae//// Large, we select the learning rate from le-4 to 5e-4. By default, the learning rates and the layer-wise Ir decay are 4/2ee////2224//////// and 0.75/0.8/0.75/03. for ViT-Tiny/Smnn BBaae/Larrga respectively.
are in Table I0. The fine-tuning epoch for ViT-Tiny S-aasll--aaeee--at--
Semantic segmentation in ADE20K (Zhou et al., 2017). We follow the common setting in BETT (Bao 2022) use UporNet (Xiao et al.) 2018) as the task head and report the mIoU on ADE20K ZZoo ct al.
ee al.
2017p. We add relative position bias (Ralfel et al.) 2020 during fine-tuning. For different scales of models in CAE v2, we search for the optimal learning rate and layer-wise learning rate decay in Table 11 for Table
G in the main paper. Specifically, select the optimal learning rate from {8e-5, 1e-4, 1.5e-4, 2e-4] and from (1c-5, 2e-5, 3e-5, 4e-5) for ViT-Large.
for
Object detection and instance segmentation in COCO (Lin et al., 2014). We use COCO Lin el al. 2014 for the evaluation on object detection and instance segmentation. We adopt both Mask R-CNN (He ct al. 2017) and Cascade Mask R-CNN (Cai Vasconcelos 2018 frameworks and report AP and APm on the COCO val split. The input image is resized with the size of the short side between 480 and 800, while the size of the long side is no larger than 1333. Meanwhile, use the relative position embedding and rotary
position embedding (Su et al. 2021) during pretraining. Other training details are shown in Table 12
18

---

Published in Transactions on Machine Learning Research (09/2023)
Table 10: Fine-tuning setting for CAE v2 on ImageNet-1K:
Value
Config
ViT Vin-Tiny/ /aayy/Smal///a Bas///ase/aaaa
Optimizer
AdamW
Loshchilov & Hutter 2019
Peak learning rate
{1, 2, 3, 1je-3/ {1, 2, 3, 4, 5
Minimal learning rate
1e-6
Weight decay
0.05
Optimizer momentuum
, 0.9,, 0.999 (Chen et 2020a
Layer-wise Ir decay (Bao et al.
2022
Clark et al. 2020
(0.75, 0.8}
Batch size
1024
Learning rate schedule
Cosine decay ( Loshchilov & Hutter 2017
Warmup epochs [Goyal et al.
2017
Max training epochs
000/100/100/5500
Data Augmentation
RandAuug(00///9/99o0
Label smoothing (Szegedy al. 2016
0.0/0.1/0.1/0.1
Mixup (Zhang et al. 2018
0.2/0.8/0.8/0.3
Cutmix (Yun al. 2019
0.0/1.0/1.0/1.0
Color jitter
0.3/0.4/0.4/0.4
Drop path (Huang al. 2016
0.0/0.1/0.1/0.2
Input size
224x224
Table 11: Semantic segmentation setting for CAE v2 on ADE20K.
Value ViT-Tiny TiT-aal//maal/// Baaaaell/Baa///eee Barse//agee Large
Config
Optimizer
AdamW {0.8, 1, 1.5, 2Je-d/11, 2, 3, 4 4e--
(Lohcchllov & Hutter 2019
Peak learning rate
Minimal learning rate
1e-6
Weight decay
0.05
Optimizer momentum
81 0.0.9, 0.999 Chen et al. 2020a (0.65, 0.75, 0.8, 0.85, 0.95}
Layer-wise decay (Bao et al. Batch size
2022 Clark al. 2020
16
Learning rate schedule
Polynomial decay 1500
Warmup steps
Max training steps Drop path (Huang et al. 2016
160000
Input size
512x512
Table 12: Object detection and instance segmentation setting for CAE v2 on COCO.
Value
Config
VIT- VVT-l/mal/ Base/ Large
Optimizer
AdamW
Loshchilov & Hutter 2019
Peak learning rate
[1. 1.5, 2, 3, 4fo- 1e-6
Minimal learning rate
Weight decay
0.05
Optimizer momentum
31, 0.9, 0.999 CCen et 2020a {0.8, 0.85, 0.95)
Layer-wise Ir decay [ao et al.
2022 Clark et al. 2020
Batch size
16
Learning rate schedule
Step
Step epochs
8, 11
Max training epochs
12
Drop path (Huang et al. 2016
0.2
19


Reviews:
[{"review_id": "AHrbKqI22k", "reviewer": "Reviewer_U2yY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe authors clearly described the method through the text and the paradigm figure. They also presented many empirical results showing that the model achieves strong performance. The paper also presented ablation studies showing the effects of the loss components and the loss formulations. CAE v2 is also more computationally efficient and achieves good performance compared to other methods requiring far more computations.\n\nThis method also has a potential application point that may be omitted by the authors. Can the trained encoder serve as a computation-cheaper replacement for the full CLIP encoder? How similar are the predicted latents compared to the ground truth latent from CLIP for both the visible and invisible patches? Can the predicted results be combined together to serve as the latent to be used in the downstream task just as how CLIP latent will be used in the downstream task?\n\n\nWeaknesses:\n\nThe improvement from this method is really minor. The difference shown in the ablation studies is also small, making it unclear whether all the components or the loss formulation are critical. The authors need to show more studies supporting the importance of their designs.\n\nThere are also some methods that achieve very similar performance to CAE v2, but are not clearly introduced in the paper and differentiated from CAE v2, like FD-CLIP. \n\nAre CAE v2 encoders better than the CLIP encoder? I only see one comparison in ViT-Base, what about other architectures? The authors need to highlight this comparison, since it is critical that this method outperforms the CLIP encoder to be useful.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents the CAE v2 algorithm, which optimizes the visual encoder to project some of image patches similarly to the latent representations from CLIP as well as to recover the embeddings of the missing image patches. Through extensive empirical evaluations, the authors show that this method yield competitive performance compared to other methods also using CLIP latents.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness above.", "broader_impact_concerns": "no clear negative society impacts from this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5sJwXlWS0n", "reviewer": "Reviewer_4Cp8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper presents comprehensive comparisons across multiple datasets.\n- The method is presented in a clear and understandable manner.\n\nWeaknesses:\n\n- The paper lacks clarity in terms of its positioning with respect to CLIP. It is unclear whether the goal of CAE v2 is to outperform CLIP.  This clarification should be provided in the abstract and conclusions.\n- The changes made to CAE are not adequately explained:\n1. The distinction between a decoder and a regressor is unclear and requires further elaboration.\n2. The modifications to the loss function are not clearly described. It is necessary to provide the mathematical expressions of the new loss functions and explain how they differ from the original CAE's loss function, highlighting the expected improvements.\n3. Direct experimental comparisons between CAE and CAE v2 are ambiguous, especially when CAE utilizes DALE, making it difficult to assess the precise differences between the two approaches. The reported performance gap between CAE and CAE v2 is minimal in most experiments.\n4. An ablation study should be conducted, systematically demonstrating the improvements achieved by each individual change from CAE to CAE v2.\n\nIn conclusion, while the paper demonstrates strengths such as extensive dataset comparisons and clear presentation, it would greatly benefit from addressing the weaknesses highlighted above, providing clarity on the relationship with CLIP, explaining the changes to CAE more thoroughly, and conducting a comprehensive ablation study to showcase the incremental improvements of each modification in CAE v2.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Previous work introduced CAE, which utilizes CLIP latent representations as a target for masked image modeling, resulting in competitive performance across various downstream tasks. This paper introduces CAE v2, an updated version of CAE that replaces CAE’s regressor with a decoder and modifies the objective loss function. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address all the weaknesses that I commented in the previous box.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b1VCWfLEzH", "reviewer": "Reviewer_tWks", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The method is simple and effective. The needed for fewer training epochs is nice.\n+ Explanation of the method is generally clear, although some improvements can be made (see below).\n\nWeaknesses:\n+ The improvement is rather minor against baselines, even against the source model CLIP.\n+ The experiments show that the masked-prediction loss may not be necessary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript proposes a representation learning method that learns an image encoder from signals given by a pretrained CLIP image encoder. The method is similar to masked image models (e.g., masked autoencoders; MAE), which predicts the randomly-masked-out patch given the visible patches. However, instead of predicting raw patch pixels as in MAE, the method trains the encoder so that the encoding of visible patches is predictable of the CLIP encoding of *all* patches. The resulting method achieves higher performances on downstream tasks in less epochs, against both the base CLIP model and other baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Figure 3 and Table 1 show that L_m (the masked prediction loss that predicts the latents of masked patches) doesn't make much difference. Removing L_m leads to nearly identical performance. Admittedly, adding L_m increases performance by <0.5%, but this can be easily compensated by the ability to train longer because each iteration takes less compute. Longer training does improve performance according to Figure 3. If we care about total compute, we can train longer w/o L_m and get good performance. If we do not care, we can also just train as long as we want, and get good performance.Therefore, the introduction of L_m seems unnecessary to me. \n\n2. Section 2.1 talks about the preliminary on CAE, the prior work this paper is based on. It would be great to expand this section in a bit more details, e.g., with notations to make it clearer.\n\n3. Figure 2 talks about the computation graph of the proposed method and other baselines. The caption is really verbose and in my opinion the diagrams are not informative. I would suggest moving it to appendix, as Figure 1 is already quite clear..", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UN3KKik5NA", "reviewer": "Reviewer_rc6B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- Using the CLIP latents as targets for alignment makes sense (as shown by several other works like MVP). It is a natural step to try incorporate that into the CAE architecture.\n- The approach greatly simplifies the training paradigm compared to previous works. Compared to CAE v1, it doesn’t need a decoder. Additionally, the model is able to distill  the rich representations learned by multi-modal encoders like CLIP to a vision only encoder.\n- The experiments and ablations are exhaustive. Since the model architecture is simple, it has fewer moving pieces and all of them were ablated in the paper. \n\n## Weaknesses\n1. CLIP visual backbone matters much more than masked latent alignment. \n- It doesn't seem like the performance varies a lot as the number of layers in the regressors are increased. As pointed out by the authors, with 0 layers in the regressor, the model only relies on the visible alignment loss. And this performs very close to vision + mask alignment loss at least for the image classification task. '\n- This makes me wonder the role of masked latent alignment. Yes, it improves performance but only by a small margin. Most of the gains are coming from using a very rich visual backbone as the target (CLIP). \n\n2. Comparisons to other approaches like CAE might be a bit unfair. The current approach uses CLIP which has been trained on a large amount of image-text data which other approaches like CAE doesn't have access to. This is also evident from the fact that methods that use CLIP as target have very similar numbers across all tasks.\n\n3. The paper presents similar ideas as several recent works like MASKDISTILL, MVM, etc. I believe that the difference in performance is not due a fundamentally different idea, but subtle differences in architecture choices, losses (L1 vs Cosine, etc) etc. While the empirical improvements shouldn't be disregarded, I believe that the paper currently lacks in analyzing the performance gap with related works that explore the same idea of using a strong pre-trained visual encoder like CLIP as the target. \n \n\nMore clarifying questions:\n- In Section 3.3, Semantic Segmentation on ADE20K, the authors say that CAE v2 surpasses CLIP with same or less pertaining epochs. Can you please clarify this line, given that CAE uses CLIP during pre-training? \n\n\n## Final thoughts after reading the rebuttal: \nI thank the authors for the detailed rebuttal. I went through the paper and the rebuttal again, and I am glad to see new experiments. \nThe new experiments addresses many of my concerns:\n-- The authors show the contributions of each individual element in their approach (choice of loss function, masked latent alignment, visible latent alignment and demonstrate that each of them lead to improvement over baselines. \n-- The authors also modified the original manuscript to explicitly list the differences with existing work (specially MVP and MASKDISTILL). I think that makes the manuscript much more clear, and as a reader, I definitely understand the contributions better now. \n-- I am also happy to see the ablations table with other tasks as well. Thank you for providing such an exhaustive ablations table! \n\nOverall, after reading the rebuttal and revised manuscript, I think my concerns have been adequately addressed and I recommend accepting the paper! \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an improved masked image modeling (MIM) paradigm in which a vision encoder is learned by trying to align the encoded representations of both the visible and the masked targets with the correponding representations generated by a multimodal encoder like CLIP. Inspired from MVP, the approach modifies the CAE to use the representations from the CLIP visual encoder. The authors show that this simple adjustment leads to improved performance on several vision tasks — image classification, segmentation and detection. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would like to see ablations performance on all tasks (classification, segmentation, detection) for a thorough evaluation. It seems like number of layers in regressors don't matter much for image classification? Does this trend hold true for segmentation or detection? \n- The authors talk at several places about higher efficiency of their method. It would be nice if the paper includes training curves to back those claims. \n- The authors might want to tone down the contribution claim about visual latent alignment and masked latent alignment. Current set of experiments and comparisons to existing baselines doesn't make it clear that the gains are not coming from other choices in the architecture (loss functions, etc). ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FwbycO0Wsr", "reviewer": "Reviewer_dQLm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The method is clearly explained comparing related approaches and overall the paper is well-written.\n2. Extensive experiments conducted on various tasks and settings show consistent improvement of the proposed method over the compared approaches.\n\n## Weaknesses\n1. The benefit over CLIP is not clear. For masked and visible patches, CLIP latent is used as the target. Therefore, at best, the proposed model can be as good as CLIP (if I understand correctly). This raises the question that what is the purpose of this method and whether can it be better than CLIP. Please clarify this. For the second part of the question, I would like to see the CLIP method in comparisons wherever applicable.\n2. Related to the above, if CAEv2 can be better than CLIP in some cases, I would like to see a justification why and results to support this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an improvement over CAE (called CAE-v2) where masked image modelling (MIM) is trained with CLIP latent vectors as targets in an encoder-regressor fashion. The results on multiple datasets and tasks (classification, segmentation and object detection with finetuning/linear probing) show improvement of this approach over the compared methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by 5 expert reviewers. Initially, the authors did not submit the rebuttal on time which led to 2 reviewers giving a reject final rating and 1 accept final rating. After the authors submitted the rebuttal (2 weeks late), 2 other reviewers were able to take the rebuttal into account and they both gave accept final ratings. Reviewers found that most of the concerns are satisfactorily resolved in the rebuttal, and the paper, albeit a bit incremental over the authors' prior work CAE, had solid and strong experiments and achieved state-of-the-art results. Besides, the improvements proposed in this paper, especially the masked latent alignment loss, are proven to be significant in the ablation studies. The editor refrained from submitting the decision earlier, to enable authors and reviewers to have more discussion. Finally, the editor believes that discussions have get conclusive, and that most reviewers recommend acceptance of the paper or only have minor concerns. Hence, the editor recommends acceptance of the paper. The authors should take into account further reviewer concerns during the discussion after the rebuttal, including the clarifications about the positioning of the paper w.r.t. CLIP, and submit a minor revision of the paper.
Decision: Accept (poster)