Towards Sustainable Self-supervised Learning
Authors: Shang-Hua Gao
Pan Zhou
Ming-Ming Cheng
Shuicheng YAN
Venue: Submitted to ICLR 2023 (2023)
Keywords: sustainable
self-supervised learning
vision transformer

Abstract: Though increasingly training-expensive, most self-supervised learning (SSL) models have repeatedly been trained from scratch but not fully utilized since only a few SOTAs are adopted for downstream tasks. In this work, we explore a sustainable SSL framework with two major challenges: i) learning a stronger new SSL model based on the existing pretrained SSL model in a cost-friendly manner, ii) allowing the training of the new model to be compatible with various base models. We propose a Target-Enhanced Conditional (TEC) scheme, which introduces two components to existing mask-reconstruction based SSL. Firstly, we introduce patch-relation enhanced targets to encourage the new model to learn semantic-relation knowledge from the base model using incomplete inputs. This hardening and target-enhancing could help the new model surpass the base model, since they enforce additional patch relation modeling to handle incomplete input. Secondly, we introduce a conditional adapter that adaptively adjusts new model prediction to align with the target of each base model. Experimental results show that our TEC scheme can accelerate the learning speed and also improve SOTA SSL models, e.g., MAE and iBOT, taking an explorative step towards sustainable SSL.

Under review as a conference paper at ICLR 2023
ONE STEP TOWARDS SUPERVISED LEARNING
SUSTAINABLE
SELF-
Anonymous authors Paper under double-blind review
ABSTRACT
Although increasingly training-expensive, most self-supervised learning (SSL) models have repeatedly been trained from scratch but not fully utilized, since only few SOTAs are employed for downstream tasks. this work, we explore a sus- tainable SSL framework with two major challenges: learning a stronger new SSL model based on the existing pretrained SSL. model, also called " base" model, in cost-friendly manner, ii) allowing the training of the new model to Con-
be compatible with various base models. We propose ditional (TEC) scheme which introduces two components to the existing mask- reconstruction based SSL. Firstly, we propose patch-relation enhanced targets which enhances the target given by base model and encourages the new model to learn semantic-relation knowledge from the base model by using incomplete inputs. This hardening and target-enhancing help the new model surpass the base model, since they enforce additional patch relation modeling to handle incom- plete input. Secondly, we introduce conditional adapter that adaptively adjusts new model prediction to align with the target of different base models. Exten- sive experimental results show that our TEC scheme can accelerate the learning speed, and also improve SOTA SSL base models, e.g., MAE and iBOT, taking an explorative step towards sustainable SSL.
INTRODUCTION
Self-supervised learning (SSL) has achieved overwhelming success in unsupervised representation learning. with astonishingly high performance in many downstream tasks like classification (Zhou al., 2022a;b), object detection, and segmentation (Bao et al., 2021: He et al., 2022). pretext task is first built, e.g., instance discrimination task (He et al., 2020; Chen et al., 2021) or masked image modeling (MIM) (Bao al., 2021; He et al., 2022), and then pseudo labels are generated via the pretext task to train network model without requiring manual labels. Though successful, SSL is developing towards direction of requiring increasingly large training costs, e.g., 200 training epochs in MoCo (He et al., 2020) while 16,00 epochs in MAE (He et al., 2022) to release its potential. Unfortunately. most researchers only have limited computational budgets and often cannot afford train large SSL models. Moreover, the pretrained non-SOTA SSL. models are rarely used in practice, since SOTA is updated frequently and a previous one quickly becomes useless, wasting huge training resources. Thus, sustainable SSL framework much demanded.
SSL,
Just like how human experience is enriched and passed from one generation to the next in human society, try to let an SSL model inherit the knowledge from a pretrained SSL base model achieve superior rep- resentation learning ability for "sustainable" learning and also to improve learning efficiency than training new SSL model from scratch. Fig. 1 illustrates the sustainable SSL for more clarity, in which we call the new SSL model to be trained the new model and the pretrained SSL model as the base model. To surpass the base model, in sustainable SSL, the new model exploits not only the implicit base model knowledge but also the absent knowledge in the base model. Such a learning process follows a fully self-supervised manner and differs from the self-training schemes (Xie al., 2020; Yalniz et al., 2019) that require labels for supervised learn-
Pretrained hase
84.1%
Guide
Data
New SSL modell
84.8%
15.1
Guide
IROT Figure The concept sustainable SSL,
MAF
New surpasses

---

Under review as a conference paper at ICLR 2023
ing. This process can be regarded as special case of Knowledge Distillation (KD) (Hinton et al., 2015; Gou et al., 2021), which targets at learning a more powerful new model based on the base model in self-supervised setting.
In this work, we take an explorative step towards sus- tainable SSL by efficiently learning from existing pre- trained SSL models and surpassing them. In this work, achieve this challenging goal, we encourage the new model to learn not only knowledge of the base model but also more semantic-related new knowledge. We there-
TEC iBOT IMG
fore choose a reconstrcctttiii (He et al., 2022) SSL scheme to train the new model, in which the base model generates reconstruction targets from the full input im- ages and the new model tries to predict the generated targets from randomly masked image input. With this pretext task, the new model is forced to learn the se- mantics of the full input and its patch relations that the new model can reason the desired full information from an incomplete input. As illustrated by Fig. 2, the attentions of iBOT (Zhou et al., 2022a) miss some semantic regions, e.g., ears, while TEC with iBOT the base model captures all semantics and well distinguishes all different components of an input image. Because of its more powerful ability to capture comprchensive semantic, TEC helps achieve the challenging sustainable SSL, and actually can provide rich and flexible semantics for downstream tasks.
Self-attention visualization. Different colors denote attentions of dif- ferent heads. Black means no attention.
Figure 2:
However, different SSL base models could have various properties due to their various training tar- gets and training strategies, egg., iBOT models with more category semantics while MAE models with more image details (He al., 2022). So it important to build high-qualified and compatible reconstruction targets from the base model so that the new model learns these targets in a comple- mentary manner. A good model target should reveal the semantic relations among patches, e.g., the relation between car wheels and car body, so that new model can learn these general relation patterns and adapts to downstream tasks. To this end, we propose to enhance the target quality the base model by using two complementary reconstruction targets: a) the patch-dim normalization which normalizes base model targets along patch dimension to enhance the relations among input patches, and b) patch attention maps with rich semantics to filter out possible noise and establish the cor- relation between the whole image semantic and the patch semantic. For target compatibility, we introduce conditional adapters into the new model so that new model predictions can be adaptable to various base models with different properties. Given a base model target, adapters conditionally active and adjust mid-level features of the new model predict the target more effectively. These adapters are discarded after pretraining but can serve parameter-effice finetuning (Jia et al., 2022; Chen et al., 2022b) if kept.
We call the above method for sustainable SSL
as Target-Enhaneed Conditional (TEC) mask- reconstruction. As shown in Fig. 3. on ImageNet, TEC without any extra training data improves the SSL base model by a remarkable margin, egg, MAE (He et al., 2022) and iBOT (Zhou et al., 2022a). For instance, taking iBOT with 1600 epochs as base model, TEC with only 800 training epochs makes 1.0% improvement. Moreover, can significantly accelerate the SSL learning process and saves training cost. For example, training TEC for only 100 epochs with random initialization and a 300-epochs-traines MAE base model outperforms MAE trained with 1600 epochs. This work takes one step closer to sustainable SSL, and we hope our initial effort will inspire more works in the future to sustainably improve SSL in a friendle manner.
TEC:OT
ImageNNt Top-l Accuracy (%)
85.0
TECMAA
84.5
TEC M K300cp
BOT
84.0
MAE
83.5
also find that TEC
83.0
MAE
82.5
300
800
1600
Pretraining Epochs on VIT-B/16
Figure 3: Top1 accuracy on ImageNet- 1k. TEC models have the same color with their base model.
2

---

Under review as a conference paper at ICLR 2023
Conditional Adapter
Module
Base VIT encoder
Class token
Ingut atapler
Putch-dim
Encoder adapter ... Encoder adupter
Feslure mags
Prich selection
Patches
000
Multi-targets decoder
New VII encoder
Selection
Figure 4: The overall framework of the proposed TEC. The pretrained SSL base model in TEC generates patch-relation enhanced reconstruction targets, i.e., patch-dim normalized features and semantic attention maps. The new ViT encoder takes in masked image and the class token enhanced by the input adapter, and then sequentially passes the generated features into encoder adapters and the multi-target decoder predict the targets given by base model.
2
METHOD
2.1
OVERALL FRAMEWORK
An overall framework of the proposed target-enhanced conditional (TEC) raso-reccottttuttio method illustrated in Fig. TEC follows (He et al., 2022; Bao et al., 2021) to use Vision Transformer (ViT) (Dosovitskiy et al., 2020) for implementation. Under the mask-reconstruction framework (He et al., 2022), as shown in Fig. 4, TEC consists trained, conditional adapters for conditional pretraining, a multi-target decoder for reconstruction targets prediction, an SSL pretrained ViT encoder as the base model, and a turget-enhancing module to construct patch-relation enhanced reconstruction targets from the base model. Specifically, the base model is an SSr-pprttr ViT encoder (e.g.. in MAE (He et al., 2022)) and is used to generate the latent semantic of a full image. Then target-enhancing module enhances the latent semantic to construct two complementary reconstruction targets as the supervision of the new model. The new VIT encoder together with adapters takes in masked images, and generates adapted latent semantics that are then fed into the multi-target decoder to predict the base model targets. After pretraining, the new ViT encoder is kept for downstream tasks while other parts are removed. At below, we will introduce the conditional pretraining aided by adapters in Section 2.2 which helps the new model effectively predict base model targets, and elaborate on the target-enhancing module to generate high-qualified base model targets in Section 2.3.
new ViT encoder to pre-
2.2 CONDITIONAL PRETRAINING
As aforementioned, base models often have different properties, e.g., more global category semantic in iBOT while more local details in MAE. So the prediction of the new model should be compatible with any given base model. To resolve a similar issue on vanilla image pixel reconstruction, the works (Wang al., 2022a; Dong et al., 2022: Gao et al., 2022) manually select certain features from the mid-level layers of the encoder by trial and error to better align with the image pixel target. However, it is almost impossible manually select features from certain fixed layers that are compatible with different base models because of their possible different properties. So to better predict base model targets, the new model should have conditional adaptation ability regarding given SSL base model.
Given fixed pretrained model, the parameter-efffcient fine-tuning scheme introduces trainable extra modules with a small number of parameters into this pretrained model for adapting it to downstream tasks in both vision (Jia al., 2022; Chen et al., 2022b) and NLP (Houlsby al., 2019; Li & Liang, 2021; Liu et al., 2021) domains. For example, the prompting scheme (Li & Liang, 2021; Liu et al., 2021; Jia et al., 2022) concatenates learnable input tokens, e.g., class token, with patch tokens to activate certain semantic features of a fixed VIT model that are suitable for specific downstream tasks. Also, inserting lightweight adapter modules (e.g., MLP (Houlsby et al., 2019; Chen et al., 2022b) and residual blocks (Li et al., 2022b)) into a fixed model can modulate mid-level features of the model to predict features required by the downstream task. Inspired by these parameter-effie efficiet fine-tuning schemes, diversities of base models by equipping the new model with conditional adapters. Since our adapters only used for pretraining and will be removed during finetuning, they do not increase extra
introduce the adaptation scheme into the pretraining stage to handle the
3

---

Under review as a conference paper at ICLR 2023
inference costs. Actually, Tab. 4 shows that keeping these adapters in the inference phase enhances the parameter-efficien finetuning ability of the model. At below, we will introduce how to apply adapters, i.e., input and encoder adapters, into the new model encoder. Input adapter. For ViT networks, one often concatenates class token with the input patch tokens to learn the global semantics of the whole input. Since the prompting scheme shows the adaption ability of the class token, we propose further enhance the feature adaption ability of the class token by adding an input adapter. As shown in Fig. 5, the input adapter which is implemented by a small two-layer MLP layer enhances the representation ability of the class token so that the class token can better activate features in the new model according to the base model targets. Specifically, the class token T E Rº of the ViT is processed by the MLP layer to obtain an enhanced class token T' E &:O:
Input adapter
tolcon
MLP
Blocks
Encoder adapter
Figure 5: The input adapter and encoder adapter.
I' MT)(T).
where C' is the embedding dimension. During pretraining, T' is appended to the patch tokens. MLP enhances the representation ability of T and enables the new model to better predict base model targets. For inference, since (TP is shared by all input samples, one can compute it in advance to get T' the new class token, meaning extra cost is brought by MLP(T).
Encoder adapter. To modulate mid-level features the new model so that can adapt to the base model targets, we apply simple MLP with residual connection (Chen et al., 2022b) as encoder adapter in the pretraining phase. As we hope to remove adapters after pretraining for higher infer- ence efficiency, we need to keep the encoder network topology unchanged after removing adapters. So we put the input of adapters in the middle of the encoder and merge all adapter outputs at the end of the encoder. As shown in Fig. 5, given features X {X,, 1,..., D) from each encoder block where D is the encoder block number, we first uniformly divide them into N groups, in which
each group contains 3 blocks by default. Within the n4ah group, we merge features from all blocks:
Zn FC(Concat(X Xi....,X,). X,)).
Then we feed the feature Zn into adapter and obtain overall feature Ze:
z, = Un + MLP (Z,),
Ze = Z,1
(I)
where MLP a small MLP of two fully culne-coneected layers. The adapted features are then fed into the multi-target decoder to predict base model targets, which will be introduced in Section 2.3.
2.3 PATCH-RELATION ENHANCED RECONSTRUCTION TARGETS
To better exploit the knowledge of base models for sustainable SSL, our target-enhancing module constructs two complementary targets with enhanced patch relations: 1) feature-level targets with patch dimension normalization to strengthen the relations among patches; 2) semantic attention maps to learn relations between semantic patches and other patches. The feature- targets reveal the semantics of certain patches, while attention maps focus more on relations among patch feature.
Patch-dim normalized feature-level targets. Given a base model, we propose to normalize its target along the patch di- mension to enhance the spatial patch relations. Specifically, for input, assume its base model target is Y R where and C respectively denote patch number and channel di- mension. Then normalize Y along the patch dimension:
No Kim
Patch Marm
0.2
0.0
0.2
0.6
Patch Similarity
(Y-
(2) Figure 6: The patch similarity dis-
where flz and d are respectively mean and variance along the patch dimension. For MIM, this patch-dim normalization can better enhance the spatial rela- tions among tokens than the widely used feature normalization (Wei et al., 2022c;a; Baevski et al., 2022) along channel dimension. This is because as observed in Fig. 6, when using MAE for pre- training. the base model features of different patches actually have similar values and thus have high similarity score, since they all reveal the global semantic the image. This cannot well reveal the spatial relations among these patches. As result, the new model can easily reconstruct the feature
tribution of MAE.
4

---

Under review as a conference paper at ICLR 2023
target of masked patches due to their high similarity with visible patches, but does not well learn spatial relations among patches. Normalization along channel dimension can hardly enhance the patch relations it only consider the mean and variance within patch. Actually, channel-dim nor- malization even enlarges the similarity among patches as revealed in Fig. 6. In contrast, patch-dim normalization ensures values within each channel has clear difference and enhances the possi- ble inherent spatial relations among patches by obviously reducing the similarity among patches as shown in Fig. 6. Moreover, Tab. 6(c) shows that our normalization can significantly improves the new model performance. After normalization, following (He et al., 2022), the new model uses a fully-connected layer followed by the decoder to generate y for predicting the base model target Yy on masked patches:
Tea |M o (Y, 2
where M is the mask matrix and o denotes the glement-wise product.
Semantic attention-leval targets. Self-attention in pretrained ViT models has a powerful capability of capturing semantic relations among patch tokens (Caron et al., 2021; Li et al., 2022c; Ziegler & Asano, 2022). We then propose utilize the self-attention map for MIM to further enhance the semantic relation modeling capability of the new model. According to previous investigations on the effects of self-attention map in KD (Wu et al., 2022; Wang al., 2022b), not all attention map contains useful semantic relations, and severe noisy attentions even hinder student learning. Accordingly, it is necessary select parts of attention maps to reduce the possible severe noise and also help reduce training costs.
type of reconstruction target
Here we utilize the base model class token that contains sufficient global semantics select top similar patch tokens, which filters out the possible noises. Specifically, given the attention maps Ag E IR N between class token and patch tokens from the last ViT block in base model where I, and H respectively denote patch number and head number, we average the attention map Ag along head dimension to obtain A RIXL, Then, as shown in Fig. 4, we select top-k patches with the largest values in A and then compute the attention map Ap E F among the top-k patches and all patch tokens. Considering the importance of the class token, we further concatenate attention maps between itself and selected Ap obtain our final reconstruction targets, i.e., A, R4 x(*+1)xL Note, when we compute A,, temperature 7 is added before the Softmax operation to adjust the attention sharpness. For the new model, we respectively use two fully-comnected layers to project its decoder output into two predictions Z, R. l.C and Z& E R I.C. We select the same patches as in Ag from Zg to form Z,= E R Then we concatenate the class token cls in new model with ZÚ and compute the KQ attention map Za Softmax cls] Zx) E (x++1)) Finally, we compute the predicted entropy loss between the prediction Z, and the target A, as
Latt A, log Z.
Multi-target decoder. Due to the different properties of two reconstruction targets, namely feature target and attention target, one decoder in a new model for prediction is insufficient to handle them simultaneously and often results in prediction conflict. But using separate decoders for each target would increase the trainable parameters and thus slow down the training. To solve this issue, we use simple decoder adaptation scheme that constructs target-specific inputs and then feeds them into a shared decoder. Specifically, we feed the output feature Ze (see Eqn. 1) of the new model encoder into fully-connected layer and then fill the masked tokens with a learnable mask token to obtain Z,. Then similarly, given Ze, we also use fully-connected layer and a learnable mask token to obtain Zzm Next, we respectively feed Zj and Z into shared transformer-based decoder for predicting the feature and attention map targets of the base model. Unlike the large semantic gap between encoder output and vanilla image in MAE, the base model target has similar semantics to the new model predictions. So a shallow 2-layer decoder is enough and better than the 8-layer decoder used in MAE. This design also greatly reduces the training cost.
EXPERIMENTS
We evaluate our TEC on ImageNet-Ik (Deng et al., 2009) by pretraining randomly initialized VIT (Dosovitskiy et al., 2020) with a 16x 16 patch size and 224x224 image resolution for 300/800 epochs via AdamW (Loshchilov & Hutter, 2017) of 4,096 batchsize. To ensure the improvement is from TEC, we do not use any explicit/mplicit extra training data and the base models that are stronger than new model. Indeed, we respectively use iBOT (Zhou et al., 2022a) and MAE (He
5

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison with existing SSL. methods under ImageNet- Ik finetuning using ViT. t and gray color mean the usage of implicit /explicit extra data. The pretraining epoch number of TEC
denotes the one from randomly initialized weights under the guidance of base models, and does not
include that of the base model. Compared results are obtained from their reported results.
Model
Method
Epoch
Guidance Supervised
Topl acc.
Deit (Touvron et al., 2022)
800
83.8
DINO (Caron et al., 2021)
300
NA
82.8
MoCov3 (Chen* et al.. 2021)
300
NA
83.2
MixMIM (Liu et al., 2022b)
300
RGB
83.2
MFM (Xie al., 2022a)
300
Frequency
83.1
BETT (Bao al., 2021)
800
DALLE
83.2
SplitMask (El-Nouby et al., 2021)
300
NA
83.6
ConMIM (Yi et al., 2022)
800
Momentum
83.7
SimMIM (Xie et al., 2022b)
800
RGB
83.8
SIM (Tao et al., 2022)
1600
Momentum
83.8
CAE (Chen et al., 2022c)
1600
DALLEs
83,9
MaskFeat (Wei et al., 2022a)
1600
HOG
84,0
LoMaR (Chen et al., 2022a)
1600
RGB
84.1
ViT-Base
BootMAE (Dong et al., 2022)
800
84.2
data2vec (Baevski et al., 2022)
800
Momentum
84.2
Mugs (Zhou et al., 2022b)
1600
NA
84.3
MVP (Wei al., 2022b)
300
CLIP
84.4
PeCo (Dong al., 2021)
800
Perceptual codebook
84.5
CMAE (Huang et al., 2022)
1600
RGB
84.7
Ge2-AE (Liu el al., 2022a)
800
RGB+Frequency
84,8
FD-CLIP (Wei et al., 2022c)
300
CLIP;
84.9
MAE (Tl al., 2022]
1600
RGB
83.6
FD-MAE (Wei et al., 2022c)
300
MAE
83.8
TEC
300
MAE
84.7 41.1
TEC
800
MAE
84,8 +1.2
1501-ImageNNt--21
Momentum
84.4
iBOT (Zhou et al., 2022a)
1600
Momentum
84.1
SemMAE (Li et al., 2022a)
800
iBOT
84,5
TEC
300
iBOT
84,8
TEC
800
iBOT
85.1+1.0
MAE (Hc et al., 2022)
1600
RGB
85.9
ViT-Large
TEC
300
MAE
86.5 86.5
Table Semantic segmentation on ADE20k us- ing Upernet and ViT-B.
Table 3: Instance segmentation on COCO using Cascade MaskRCNN and ViT-B.
Method
Epoch
mloU
Method
APobox
AP mask
BETT
800
47.1
PeCo
800
48.5
Implementation from (Zhou et al., 2022a)
GE2-AE
800
48.9
iBOT
51.2
44.2
CAE
1600
50.2
CMAB
1600
50.1
ТECBоT
52.7
45.4
MAE
1600
48.1
Implementation from (Li et al., 2022b)
TECMAE
800
49.9
MAE
54.0
IBOT
1600
50.0
TEC:OT
800
51.0
TECMAF
54.6
47.2
et al., 2022) pretrained ViT model on ImageNet-Ik as our base model. Base models are obtained from their official publicly released versions. We the same masking strategy in MAE, e.g., 75% random masked ratio. See more training details Appendix.
3.1 PERFORMANCE COMPARISON
3.1.1 COMPARISON ON IMAGENET
Finetuning on ImageNet- 1k. Tab. 1 summarizes the finetuning performance on ImageNet- 11. One can observe that with iBOT as base model, TEC surpasses the base model by 0.7% under 300 training epochs from random initialization, and further makes 1.0% improvement for 800 epochs. Similarly, TEC respectively brings 1.1% and 1.2% improvement over the MAE base model under 300/800 training epochs. These results show that thanks to the proposed target-enhanced conditional MIM scheme, TEC actually further improve the strong MIM- based methods, e.g., MAE and iBOT used here. Besides, Tab. also shows that under similar or even less training cost, TEC outperforms other SOTA SSLs, including methods trained by implicitly extra data, egg., MVP (Wei al., 2022b)

---

Under review as a conference paper at ICLR 2023
Table 4: Top1 accuracy on the ImageNet-1k
Table 5: Semi-supervised semantic segmenta- tion on the ImageNet-S dataset.
dataset under parameter-efficient finetuning.
Method
Epuch
Settings
Top I acc.
Pretrain
Method
Epoch
mloU
MAE 1600
Linear probing
68.0
MAE
1600
38.3
SSL
MAC
800
42,9
Lineur probing
69.8
TECMAR
800
+Input adapler FT
72.6
MAE
1600+100
61.0
SSL+FT
+ Encoder adapter FT
79.9
TECMAE
800+100
62.0
and FD-CLIP (Wei al., 2022c). More surprisingly, TEC with only ImageNet-Ik data has an improvement of 0.7% over iBOT trained on ImageNet-22k, indicating more effectiveness of TEC pretraining over more training data. To the best of our knowledge, TEC sets a new SOTA 85.1% on ViT-B when solely using ImageNet-Ik, showing the potential of sustainable SSL learning. We also investigate the scaling ability of TEC by using ViT-Large, and observe that TEC surpasses the MAE pretrained base model by 0.6% with 300 epochs from random initialization.
Parameter-officiett finetuning on ImageNet-Ik, Parameter-cfficient finetuning methods, e-g., lin- ear probing, aims to finetune small amount of parameters for adapting to a downstream task. We test TEC under the linear probing setting which only finetunes a linear classifier the top of a frozen pretrained model. Tab. 4 reports the classification accuracy of ViT-B on ImageNet- Ik under linear probing. TEC improves the MAE base model by 1.8%, showing more rategory-related semantic information in the learned new model. Indeed, our input and encoder adapters used for pretraining can also be used for parameter-efficient finetuning. Finetuning input adapter via prompting brings remarkable improvement of 4.6%, and finetuning both input and encoder adapters makes 11.9% improvement over the MAE base model. This also shows the benefit of our proposed adapters.
Semantic segmentation on ImageNet-S. To test the pixel-level representation ability of TEC pre- trained models, we conduct semantic segmentation finetuning on ImageNet-S (Gao et al., 2021) which has pixel-level training labels on ImageNet. We use ViT-B the segmentation model with- out extra segmentation head, since the pretraining and finetuning data have no domain shift. Tab. shows that TECMAE improves MAE base model by 4.6% on mloU. When using supervised Ima- geNet fully-finetuun finetuned pretraining, TECMAE achieves a gain of 1.0% over MAE.
3.1.2 TRANSFER LEARNING ON DOWNSTREAM TASKS Here we investigate the transfer learning ability of TEC models on downstream tasks.
Semantic segmentation. For semantic segmentation on the ADE20k (Zhou et al., 2018) dataset, we use Upernet (Xiao et al., 2018) with VTT-B as the segmentation model. Tab. shows that ТEC surpasses the iBOT base model by 1.0% mloU, and TECMAE achieves 1.8% 38% improvement over its MAE base model. Thus, TEC pretrained models show greater transfer learning abilities on semantic segmentation compared to their base models. Also, TEC shows remarkable advantages over strong competitors with fewer pretraining epochs. For example, it outperforms MAE, CAE (Chen et al., 2022c), and CMAE (Huang et al., 2022) by 2.9%, 0.8%, and 0.9%, achieving new SOTA.
Instance segmentation. For instance segmentation on COCO (Lin et al., 2014), for faimess, we apply the Cascade MaskRCNN (Cai & Vasconcelos, 2019) implemented by iBOT (Zhou et al., 2022a) and ViTDet (Li et al., 2022b) for TEC with iBOT/MAE base models. Tab. shows that with the implementation from iBOT, TEC surpasses the iBOT base model by 1.5% on box AP and 1.2% on mask AP; and when using the implementation from ViTDet, TEC also achieves a gain of 0.6% on box AP and 0.5% mask AP, indicating stable improvements of TEC.
3.2 ABLATION AND ANALYSIS
We give the ablation study and analysis of our TEC. By de- fault, models are pretrained with 300 epochs and evaluated with the finetuning on ImageNet-lk.
Average Prepurtcet
0.40
BOT TECMAE
0.35
0.30
0.25
Conditional pretraining. The conditional adapters aid the SSL pretraining under different base models. Tab. 6(a) shows adapters stably improve the performance by 0.4% and 0.2% when using MAE and iBOT base models. To observe the adaptation difference to base models, we show the average proportion of encoder adapters contributing to the encoder output in Fig. 7, i.e., / Zee in Eqn. 1. iBOT base model requires adapters to provide more features
0.20
0.35
ViT Emeode
12
Figure 7: The average proportion of encoder adapters contributing to the encoder output Ze.
7

---

Under review as a conference paper at ICLR 2023
Table 6: Ablation study on ImageNet- 1K fully finetuning setting using ViT-B.
(a) Ablation study of proposed modules.
Patch-norm. feature
Semantic attention
Adapters
MAE base
iBOT base
Base model performance
83.6%
84.1%
84.2%
84.5%
84.3%
84.7%
84.6%
84.7%
84.7%
84.8%
(b) Effect of adapters.
(c) Patch-norm features.
(d) Init. with base model pretrain.
Topl
Topl acc.
Topl acc.
MAE base
83.6
MAE base
83.6
iBOT base
84.1
No adapter
84.2
NA
83.9
+ input adapter
84.3
Feature dim.
83.9
Load
84.4
encoder adapter
84.6
Patch dim.
84.2
Not load
84.8
(e) TEC accelerates MAE training.
(I) Effect of semantic-related patch attention.
Epoch
Topl acc.
Top1 acc.
MAE
1600
83.6
iBOT base
84. 1
TECMA 1GCM0E
84.7+1.1
No attention
84.5
MAE
300
82.9
Cls token only
84.5
TEC TECMAE300e
100
83.9
All attention
84.6
TECMAEE300p
300
-4.3-1.4
Attention select
84.7
from deeper layers, while MAE base model makes adapters focus more on shallow layers, which is constant with their properties, i.e., iBOT base model has more high-level category semantics while MAE model has more low-level image details.
Feature normalization on different dimensions. We normalize target features on the patch dimen- sion to stress the relative relations among patches, which differs from existing methods that nor- malize features on the channel dimension. In Tab. 6(c), normalizing on patch dimension achieves 0.3% gain than channel-dim normalization. In contrast, the channel-dim normalization has no effect compared to the unnormalized version. Channel-din normalization emphasizes the feature differ- ence in channels. Instead, our patch-dim normalization stresses the relations among patches, which is compatible with the patch prediction in the MIM scheme. Tab. 6(a) shows training with patch-dim normalized feature has the 0.6% gain over MAF/iBOT base models, showing its robustness over base models.
Semantic-relatee attention. The KQ attention maps naturally contain the semantic relations among patches and thus are used as the base model targets with enhanced patch-relation properties. Tab. 6(a) shows that using attention maps further improves the models trained with patch-dim normalization. Tab. 6(f) compares the effects of different types of attention maps. Only using the attention maps of the class token has no improvement, while the attention of semantic-related patches brings a 0.2% gain over the baseline. Therefore, it is the relation among patches that helps the MIM training. Compared to using all attention maps, using the selected semantic-related attention maps brings a larger gain by reducing the noise.
Accelerating the training process of base models. By default, we use the fully pretrained SSL. models as base models. To verify if TEC can improve an unconverged SSI. model, we use 300- epoch MAE pretrained ViT-B as the base model and train TEC with 100/300 epoch from random initialization. As shown in Tab. 6(c), the 300-epoch pretrained MAE gives 82.9% Top.1 accu- racy. In comparison, TECMAE300ep E300ep achieves 84.3%/83.999 with 300/100 epochs, surpassing the 300-epoch MAE base models with 1.4% Notably, TECMAE300e even outperforms the 1600- epoch pretrained MAE by 0.3% with only 100 epoch training, showing TEC can significantly ac- celerate the training process of the base model. Still, the TECMAF1600cg taught with 1600-epoch MAR base model further improves the TECMAE3000p by 0.4%, indicating our sustainable learning scheme relies on good base models to achieve better performance.
Initializing new model with base model weights or not. The new models in the TEC framework trained from random initialization. Tab. 6(d) compares the new model performance with/without loading the pretrained weights of the base model. The randomly initialized new model outperforms the model loaded with pretrained base model weights by 0.4%. We assume that randomly initialized
8

---

Under review as a conference paper at ICLR 2023
new models avoid the local minima of the base model, and the new model learns different weight distribution from the base model.
RELATED WORKS
Self-supervised learning. Self-supervised learning enables representation pretraining without hu- man annotation by training with pretext tasks, egg,, instance discrimination task (ID) and masked image modeling task (MIM). ID learn high category-related representations by pulling close rep- resentations from multiple views of one image Chen et al. (2020); Grill et al. (2020); Chen & He (2021); Zbontar et al. (2021); Caron et al. (2020). Extracting representations from multi-view re- quires larger training cost than supervised training. MIM learns semantics by reconstructing the information of masked patches from unmasked parts, which learns more spatial semantic details than ID. Due to the incomplete input, MIM usually requires longer training epochs than ID con- verge. (Huang et al., 2022: Wang et al., 2022a) explore combining the advantages of MIM and ID to further improve performance. Recently, (Kong & Zhang, 2022) reveals both MIM and ID learn occlusion-invariant features. We observe a trend that these SSL methods require increasingly large computing costs achieve SOTA, which hinders the development of new SSL methods. To remedy this, we explore sustainable SSL by learning from pretrained SSL models.
Masked image modeling on various targets. The reconstruction targets guide the MIM learning on different semantic spaces. MIM has explored various reconstruction targets, e.g., RGB pixels and to- kenizers. To make images similar the discretized language in NL.P (Devlin al., 2018), Beit (Bao et al., 2021) utilizes the DALLE pretrained tokenizer (Ramesh et al., 2021) as the prediction target. CAE (Chen al., 2022c) further decouples this pretext task prediction with the encoder. MAE and SimMIM (Xie et al., 2022b) show using RGB images as targets achieves competitive fully finetun- ing performance. MaskFeat (Wei et al., 2022a) reveals band-designed HOG feature (Dalal & Triggs, 2005) is effective target form. Ge2-AE (Liu et al., 2022a) and MFM (Xie et al., 2022a) find the image frequency domain can be complementary to RGB image targets. The perceptual codebook in PeCo (Dong et al., 2021) helps the model learn semantic information. The online momentum network (He et al., 2020) is used by iBOT and data2vec (Baevski et al., 2022) to provide updated prediction targets. BootMAE (Dong et al., 2022) takes advantage of RGB images and online net- work targets. (Yang et al., 2022) enhances the distillation from a large teacher model to a compact student model with the masking scheme. MVP (Wei et al., 2022b) introduces rich semantics learned from vision-language pretraining using the CLIP pretrained model (Ramesh et al., 2021) as the tar- get. Unlike these works that stress the unique properties of a specific reconstruction target, we show that all SSL pretrained models can serve as good base models with the help of tanget-enhancement in TEC. The adapters and sugeet-nnhancig scheme in TEC enables the good adaptability to various base model targets.
Self-supervised knowledge distillation. The sustainable SSL can be regarded as a special case of the self-supervised knowledge distillation as they both learn from SSL pretrained models. Reversed KD (Yuan al., 2020) shows weak teacher model can benefit the student in the supervised setting. ClusterFit (Yan et al., 2020) conducts training on the clustered pseudo-labels to reduce the overfitting to the pretext tasks. SEED (Fang et al., 2021) distillates knowledge from large SSL models to small models with contrastive loss. (Navaneet et al., 2022) uses MIP heads for feature regression to distill large SSL teachers to compact student models. (Xu et al., 2021) groups instances with teacher models and transfers the instance-relation knowledge to student models. As an exception, (Wei et al., 2022c) shows feature distillation improves contrastive-based SSL models but brings marginal gain over the SOTA MAE (He et al., 2022) model. Our sustainable SSL focuses on the new model outperforming the base model in a self- superviped manner. We show in Appendix that our TEC method is advantageous over several SOTA self-super distillation methods.
CONCLUSION
This work explores sustainable self-supervised learning by learning from pretrained SSL models. We propose target-enhanced conditional mask-reconstrruction learning scheme to learn from and surpass existing SSL models. The adapters help to adapt the new model to various base models during pretraining and can also serve as parameter-efficient finetuning modules. We utilize the mask- reconstruction scheme as the basis for surpassing base models, and we construct prediction targets with enhanced peteh-level relations to aid the MIM pretraining. Our method further improves the strong MIM pretrained methods, e.g., MAE and iBOT, proving the feasibility of sustainable learning. This work takes an initial step towards sustainable SSL, and we will explore a more general multi- round sustainable SSL. framework in the future.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for supf-super learning speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.00 082544, 2021.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: high quality object detection and instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43(5): 1483-1498, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in euf-ssueeeeeed vision transformers. In IEEE International Conference Computer Vision (ICCV), 2021.
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang. Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. arXiv preprint 0X.v:2//0000/// 2021.
Jun Chen, Ming Hu, Boyang Li, and Mohamed Elhoseiny. Efficient self-supervised vision pretrain- ing with local masked reconstruction. arXiv preprint arXiv:2206.00799 2022a.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang. and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205. 13535, 2022b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597-1607. PMLR, 2020.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning, arXiv preprint arXiv:220..03026 2022c.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2021.
Xinlei Chen* Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057) 2021.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox
and benchmark. https: /ttss:///iia /gttthttbbbb//. com wwwp///////////eeeeeeeeeeeeeeeee mm1 ab http/mmmegmettt aineenttttiiiin.i.. 2020.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR305), volume 1, pp. 1886-893, Icee, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255. leee, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre- pre- teeitraining of vision transform- ers. arXiv preprint arXiv:21 127 2021.
10

---

Under review as a conference paper at ICLR 2023
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. In European Conference Computer Vision (ECCV), 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprini arXiv:2112.10740, 2021.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. arXiv preprint arXiv:2101.04731 2021.
Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Qiao. Convmae: Masked convolution meets masked autoencoders. arXiv preprint wwXiv:2005.0892. 2022.
Shanghua Gao, Zhong-Yu Yu Li, Ming- Hsuan Yang. Ming-Ming Cheng, Junwei Han, and Philip Torr.
Large-scale unsupervised semantic segmentation. arXiv preprint arXiv:2106.03149, 2021.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A
survey. International Journal of Computer Vision (ICV), 12966):189-1819, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, 2022.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503 100xv:1503.02331,, 2(7), 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficcent transfer learning for nlp. In International Conference on Machine Learning (ICML). pp. 2790-2799, PMLR, 2019.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders stronger vision learners. arXiv preprint arXiv:2207.13522 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12119, 2022.
Xiangwen Kong and Xiangyu Zhang. Understanding masked image modeling via learning occlusion invariant feature. arXiv preprint arXiv:2208 04164, 2022.
Simon Korbblith, Mohammad Norouzi, Honglak Lee. and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning (ICML), pp. 3519-3529. PMLR, 2019.
Gang Li, Heliang Zheng. Daqing Liu, Bing Su, and Changwen Zheng. Semmae: Semantic-guides masking for learning masked autoencoders. arXiv preprint arXiv:2206. 10207, 2022a.
11

---

Under review as a conference paper at ICLR 2023
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 1XXi::2111000190 2021.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. arXiv preprint arXiv:2203.1633 2022b.
Zhong- Yu Li, Shanghua Gao, and Ming-Ming Cheng. Exploring feature self-relation for self- supervised transformer. arXiv preprint arXiv:2206. 05 184, 2022c.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pp. 740-755, 2014.
Hao Liu, Xinghua Jiang, Xin Li, Antai Guo, Deqiang Jiang, and Bo Ren. The devil is in the frequency: Geminated gestalt autoencoder for self-supervised visual pre-training. arXiv preprint arXiv:2204.08227, 2022a.
Jihao Liu, Xin Huang, Yu Liu, and Hongsheng Li. Mixmim: Mixed and masked image modeling for efficient visual representation learning. arXiv preprint arXiv:220S. 13137, 2022b.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:21033 10385, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1.05,000,, 2017.
KI. Navaneet, Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Simreg: Regression as simple yet effective tool for self-supervised knowledge distillation. arXiv preprint arXiv:2201.05131, 2022.
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, et al. Knowledge inheritance for pre-trained language models. arXiv preprint 1X5x:2105./3880 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning (ICML), pp. 8821-8831. PMLR, 2021.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint arXiv:2206.01204 2022.
Hugo Touvron, Matthicu Cord, and Herve Jegou. Deit ii:: Revenge of the vit. In European Confer- ence on Computer Vision (ECCV), 2022.
Luya Wang, Feng Liang, Yangguang Li, Wanli Ouyang, Honggang Zhang, and Jing Shao. Repre: Improving self-supervised vision transformer with reconstructive pre-training. arXiv preprint arXiv:2201.06857 2022a.
Shaoru Wang, Jin Gao, Zeming Li, Jian Sun, and Weiming Hu. A closer look at self-supervieed lightweight vision transformers. arXiv preprint TTt.:22055.44443 2022b.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 14668-14678, 2022a.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodality-uuide visual pre-training arXiv preprint 2rXiv:2203.05775, 2022b.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint 2r05v: 2022c.
12

---

Under review as a conference paper at ICLR 2023
Haiyan Wu, Yuting Gao, Yinqi Zhang, Shaohui Lin, Yuan Xie, Xing Sun, and Ke Li. Self- Self-supervised models are good teaching assistants for vision transformers. In International Conference on Ma- chine Learning (ICML), pp. 24031 122042. PMLR, 2022.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision (ECCV), pp. 418-434, 2018.
Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Masked frequency modeling for self-supervised visual pre-training. arXiv preprint arXiv:2206.07706, 2022a.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student im- proves imagenet classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10687-10698, 2020.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9653-96536 2022b.
Haohang Xu, Jiemin Fang, Xiaopeng Zhang. Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Bag of instances aggregation boosts sule-vuue distillation. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint arXiv: met.i 2019.
Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit: Improving generalization of visual representations. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6509-6518, 2020.
Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked gener- ative distillation. arXiv preprint arXiv:2205.01522, 2022.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang. Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. arXiv preprint urXiv:2205.09616. 2022.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. CoRR, abs/2103.03230 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision (ICCV,, 2018.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Leurning Representa- lions (ICLR), 2022a.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi- calti-graauuar self- supervieed learning framework. In arXiv preprint A.Xi:2203.1141.. 2022b.
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmen- tation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14502- 14511, 2022.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Towards general sustainable SSL. This work aims to take one step towards sustainable SSL based on existing pretrained SSL models. To make more steps towards sustainable SSL, we use the TEC pretrained models as the base model for a new round of TEC pretraining. Tab. shows that the second-round TEC trained with the first-round TEC base model achieves 85.2%. The possible reason for the smaller improvement in the second round is caused by the limited network capacity or two rounds of TEC pretraining learns similar knowledge.
Training cost comparison. Accelerating the training of larger language model with the help of a smaller pretrained model (Qin et al., 2021; Chen et al., 2021) has been proven possible in the NLP field. We follow them to add the FLOPs, training time, and parameters comparison as shown in Tab. 8. TEC requires shorter training time to achieve better performance than the base models. For example. TEC outperforms iBOT/MAE by 0.7% I 1% Top I accuracy with only 7%/20% train- ing time. TEC has similar number of parameters with MAE, because the shallow decoder saves parameters while adapters increase parameters. Both TEC and MAE have a larger number of param- eters compared to iBOT due to the extra decoder. But benefiting from the decoder, they only process the visible patches in the encoder, thus requiring a smaller training cost than iBOT. As only a part of the model requires gradients in some SSL methods, c.g. the base model in TEC and online model in iBOT, which requires no backward cost to compute gradients, we compare the FLOPs for network parts with/without gradients. Benefiting from only processing the unmasked patches in the encoder and the shallow two-layer decoder, TEC requires smaller training FLOPs with gradients than iBOT and MAE. The extra FLOPs (FLOPs without gradients) of the base model in TEC are smaller than the online network in iBOT because no extra head is needed for the base model in TEC. Compared to MAE, the extra FLOPs of the base model can be partly balanced by the smaller FLOPs with gradients in TEC. Therefore, TEC has a similar training time with MAE for each training iteration.
Comparison on parameter-efffiient finetuning. Tab. 9 reports the accuracy of 1) learning prob- ing and 2) adapter finetuning which only finetunes the adapter and the linear classifier for parameter- efficient finetuning. One can observe that 1) the linear probing performance of TEC relies on the base model, and 2) adapter finetuning significantly improves the performance. Indeed, most MIM- based models, e.g. BEIT and MAE, have much lower linear probing performance, since they do not use the global semantic learning loss, c.g. clustering loss or InforNCE instance discriminative loss. This also explains the lower performance of TEC compared with the global semantic learning methods, e.g. iBOT. But by finetuning the adaptors and also linear classifier, TEC improves iBOT with remarkable margin of 3.9%. This is because as shown in Fig. 2, iBOT focuses more on dis- tinguishing the patches related to global semantics and ignores the semantics of other patches, while TEC can group the patches into several semantic groups and further identify the semantics of each group. In this way, finetuning adapters help activate the semantic groups that are related to global semantics required by the downstream tasks, thus improving the model's discriminability on global semantics and showing good parameter-efficien finetuning performance.
Detailed schematic diagrams. We show more detailed schematic diagrams of conditional adapters in Fig. 10 and attention selection in Fig. 11 for better understanding of these modules.
Comparison with self-supervised distillation methods. We compare several recently proposed self-supervised distillation methods on the fully finetuning performance on ImageNet. Tab. 10 shows the remarkable improvement of TEC over other self- superviedd distillation methods. For using MAE VTT-B as the base model, TEC outperforms FD by a noticeable gain of 0.9%. When comparing with MaskFeat which also applies the MIM scheme, TEC has gain of 0.6% when using MoCov3 VIT-B as the base model.
Table 7: Towards general sustainable SSL using the TEC as the new base model.
Model
Base
Epoch
Topl acc.
iBOT
1600
84.1
TECIBOT
iBOT
800
85.1
TFC
TEC
800
85.2
14

---

Under review as a conference paper at ICLR 2023
Table 8: Training comparison.
Method
Epoch
Time (8xA100)
FLOPss (with grad)
FLOOPs grad)
Parameters
Top
VIT-B
17.6G
86.6M
IBOT
1600
361h
19.2G
19.2G
96.3M
84.1 1
TEC
25h
8.3G
17.6G
118.6M
MAR
1600
125h
9.8G
OG7
111.9M
83.6
TEC
300
25h
8.3G
17.6G
118.6M
84.7
Table 9: Topl accuracy on the ImageNet-Il Ik dataset under linear probing (LP), adapter finetuning (Adapter FT), and fully finetuning (Fully FT).
Method
Epoch
Settings
Top acc.
Fully FT Top
acc.
BETT
800
LP
56.7
83.2
SimMIM
800
LP
56.7
83,8
BooIMAE
800
LP
66.
84,
CAF
800
L.P
68.6
83.8
SemMAE
800
LP
68.7
84.5
CMAR
800
L.P
73.9
84.7
Ge2-AE
800
LP
75.3
84.8
MAE
1600
LP
68.0
83.6
TEC
800
LP
69.8
84.7
TEC
800
Adapter FT
79.9
84.7
iBOT
1600
LP
79.8
84.1
TECROT
800
LP
78.0
84.8
TEC:BO
800
Adapter PT
81.9
85.1
Visualization of patch-dim normalized feature-level targets and semantic attention-level tar- gets. We visualize the selected semantic attention-level targets from the iBOT base model in Fig. 8. The averaged attention maps of the class token (A.)) mostly focus on the high-semantic objects, thus making the selected patches belong to the high-semantic objects. The attention maps of selected patches contain the semantic relation between high-semantic objects and other regions. Different patches have some unique attention parts that differ from other patches. The attention maps of these selected patches focus on similar semantic objects but are complementary in some parts, which ex- plains why using attention maps of selected patches is better than only using class token attention maps as shown Tab. 6(f). We show the visualization of patch-dim normalized feature-level targets of iBOT base model in Fig. 9. Patch-dim normalized features more distinguishable compared to the original and channel-dim normalized features. The spatial relation among feature patches is more clearly shown by the patch-dim normalization.
Pretraining settings on ImageNet-Ik. We use the standard ViT network implemented in MAE. We give the pretraining settings in Tab. 11, which follows the pretraining settings in MAE. Due to the different properties SSL base models, we set different parameters of semantic-related attention for MAE and iBOT base models, shown in Tab. 12.
Inspired by (Baevski et al., 2022), we utilize the average output features of last two blocks of the base model the feature target. We measure the CKA similarity (Kornblith et al., 2019) of each mid-layer block to the output of the last block, and we observe the high feature similarity of last two blocks.
Table 10: Comparison with self supervipe distillation methods.
Method
Base
Arch
Epoch
Top
acc.
MAE
VYT-B
1600
83.6
FDMAE
MAE-VIT-B MAE-VIT-B
VIT-B
300
83,8
TECMAR
VTT-B
300
84.7
MoCov3
VYT-B
300
83.2
MaskFeatatoCowa
MoCov3-Vii VIT-B MoCov3-VIT
VIT-B
300
83.9
TRCMoCov3
VIT-B
300
84.5
15

---

Under review as a conference paper at ICLR 2023
Image
A
Multi-head attention maps of class token, top-3 patch, and top-9 patch.
Figure Visualization of the selected semantic attention level targets from the iBOT base model.
Fully finetuning settings on ImageNet-Ik. We give the fully finetuning settings on ImageNet- Ik in Tab. 13. We observe that TECs trained with different base models may have different properties. Since these base models have different finetuning layer decay values, we set different layer decay values for TECs trained with different base models.
Parameter-efficient finetuning settings on ImageNet-Ik. Following MAE, the linear probing settings are shown in Tab. 14. For parameter-efficien finetuning with the input adapter, same training settings as used by the liner probing in Tab. 14. When finetuning with the encoder adapters, we use the same training settings as used by the fully finetuning Tab. 13 due more parameters are contained in encoder adapters.
the
Semi-supervised semantic segmentation finetuning on ImageNet-S. We give the training set- tings of semi-supervised semantic segmentation finetuning on ImageNet-S in Tab. 15. We set different learning rates and layer decay weights for models initialized with pretrained weights with/without fully finetuning.
Downstream task settings. For semantic segmentation on ADE20K, we use the MMSegmenta- tion (Contributors, 2020) implementation of Upernet. The training configurations follow the MAE training configurations MMSegmennatiin. Specifically, the models are trained for 160k iterations with the batch-size of 16 8 GPUs. The AdamW optimizer is used with the initial learning rate of lc-4, weight decay 0.05, 1,500 warmup iterations, and poly learning rate decay schedule. The VIT-B with 16x 16 patch size is used the backbone. The image size set 512x512 and the
16

---

Under review as a conference paper at ICLR 2023
No
Patch
Channel
No
Patch
Channel
No
Patch
Channel
Figure 9: Visualization patch-dim normalized feature-level targets from the iBOT base model.
T (Cx1)
Input adapter
Encoder adapter Z (Cx(L + 1))
SUM
Class token
Z (Cx(L + 1))
FC
MLP
Encoder adapter
Class
(3Cx(L + 1))
T' (CX1)
(Cx(L 1))
2,(Cx(L+ 1))
Concat.
ViI blocks
New VIT encoder
(CXL)
Figure 10: The details of adapters for conditional pretraining.
default data augmentation in MMSegmennation is utilized, i.e., random crop. random flip, and photo metric distortion. A layer decay rate of 0.65 is utilized for the ViT backbone.
For instance segmentation COCO using Cascade MaskRCNN and ViT-B, follow the training configurations of iBOT and ViTDet for TЕCBOT and TECMAE- The ViT-B with 16x 16 patch size used the backbone. TEC:BTT follows the training strategy of iBOT using the MMDetection implementation. We train the model with the 3x schedule using the batch-size of 16 on 8 GPUs. The Adam W optimizer is used with the initial learning rate of le-4, weight decay of 0.05, and layer decay of 0.65. The learning rate is multiplied by 0.1 at the 27th and 33rd epoch. We train the model with 512x512 image size. The random flip and random resize crop augmentation is applied. For TECMAE, we use the ViTDet implementation of Cascade MaskRCNN, The model is trained with 100 epoch with the batch-size of 64 on 32 GPUs. The Adam W optimizer is used with the initial learning rate of lc-4, weight decay of 0.1, and layer decay of 0.6. The learning rate is multiplied by 0.1 at 89th and 96th epoch. The input image size is 1024 x 1024. The random flip and random resize crop augmentation is applied.
17

---

Under review as a conference paper at ICLR 2023
Base Model Attention Selection
My (fxxxi))
Select Tup-k pasches
Parch Allen
Cuncall
4, (Nx(k 1)xE)
Amtetion Maps
(HX(L 1)x(4+ 1))
A. (HXL)
Ae (1x6)
Pred.
Zg (LXC)
Select patches
(Zig as) (H + 1)xC))
New Model Maps ((1 + 1)xC)
Zg (Nxx( 1)x1)
Ze (LXC)
Zy (NXLXC,)
New Model Attention Selection
Figure 11: The details of the semantic attention map selection.
Table 11: Pretraining settings.
Configuration
Value
Optimizer
Adam W
Base learning rate
1.5c-4
Weight decay
0.05
Optimizer momentum
B1, 92 =0.9, 0.95 4096 Cosine decay 40
Batch size
Learning rate schedule
Warmup epochs
Augmentation
RandomResizedCrop
Table 12: Parameters semantic-related attention.
Base model
MAE (ViT-Base) MAE (ViT-Large)
1.8
15
1.4
iBOT (ViT-Base)
1.0
Table 13: Settings of fully finetuning and parameter-efficient finetuning with encoder adapters.
Configuration
Value
Optimizer
Adam W
Base learning rate
5c-4 (B), 1e-3(L.) 1c-6 (B), 1e-5(I.)
Min learning rate
Weight decay
0.05
Optimizer momentum
3,, =0.9, 0.999
Layer-wise Ir decay
0.55 (MAE-B), 0.65 (iBOT-B), 0.65 (MAE-L)
Batch size
1024
Learning rate schedule
Cosine decay 20 (B), (L)
Warmup epochs
Training epochs
100 (B), (L)
Augmentation
RandAug (9, 0.5)
Label smoothing
0.1
Mixup
0.8
Cutmix
1.0
Drop path
0.1
18

---

Under review as a conference paper at ICLR 2023
Table 14: Settings of linear probing and parameter-efficient finetuning with the input adapter.
Configuration
Value
Optimizer
LARS
Base learning rate
0.1
Weight decay
Optimizer momentum
0.9
Batch size
16384
Learning rate schedule
Cosine decay
Warmup epochs
10
Training epochs
90
Augmentation
RandomResizzdCrop
Table 15: Settings of semantic segmentation finetuning on ImageNet-S.
Configuration
Value
Optimizer
AdamW
Base learning rate
5e-4 (SSL), lc-4 (SSL+FT)
Weight decay
0.05
Optimizer momentum
31. -.9 0.999 0.60 (SSL). 0.45 (SSL+FT)
Layer-wise Ir decay
Batch size
256
Learning rate schedule
Cosine decay
Warmup epochs
Training epochs
100 RandomResizedCrop
Augmentation
Drop path
19


Reviews:
[{"review_id": "LgmFzGluMT", "reviewer": "Reviewer_nhwb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel method to train an SSL model by distilling knowledge from an existing SSL model (base model). By leveraging the base model, the method can help to learn a stronger SSL model in a cost-friendly way. Specifically, they propose two complementary reconstruction targets that enable knowledge distillation, 1)the patch-dim normalized features and 2) patch attention maps with rich semantics. They also have a conditional adapter module that adapts the new model's prediction to that of the base model. Exhaustive experimentation shows that their approach outperforms existing approaches on varied vision tasks and at a fraction of computation time.", "summary_of_the_review": "Overall, based on the strengths, weaknesses, and other factors like clarity, novelty, and reproducibility, I place this submission marginally above the acceptance threshold (6).", "strength_and_weaknesses": "1. The proposed method to distill knowledge from an existing SSL model to build a stronger new model is novel and addresses an important issue of training/improving SSL models in resource-constraint situations.\n2. Exhaustive experimentation shows that their approach outperforms existing approaches on varied vision tasks and at a fraction of computation time.\n3. The contribution of different parts of the model like patch-norm features, selective attention maps and others are well demonstrated by the ablation experiments performed by the authors.\n\nWeakness/Suggestions:\n1. It would be nice to see the computation comparisons in terms of FLOPs.\n2. I would recommend modifying the schematic diagram, especially the Attention Selection and Conditional Adapter part so that it is more intuitive from the schematic diagram itself. Currently, just from the schematic diagram, it is hard to understand these modules of the method.\n3. Maybe in the appendix, it would be nice to visualize some more reconstruction targets (i.e. the patch-dim normalized features and selected attention maps) so that it helps to develop an intuition for the readers for what knowledge the new model is trying to distill.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K4wy1K_c4Rl", "reviewer": "Reviewer_uYmT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an unsupervised method for improving already trained self-supervised models. The method, TEC, consists in reconstructing the masked patch representations of an image. The target, i.e. the feature representation of the dropped patches, are obtained from a pre-trained SSL model.", "summary_of_the_review": "For the reasons stated above (weaknesses section + problem of reproducibility), I lean towards rejection of the paper.\n\nThat being said, I still think the paper is promising and some reported results are strong. I will be willing to upgrade my initial assessment based on the authors’ rebuttal.\n\nIn any case (acceptance or rejection), I definitely think the paper should be re-written to tone down the “sustainability” aspect and present the contribution through the scope of “distillation” or “feature enhancing”.\n", "strength_and_weaknesses": "**Strengths**\n\n- Extensive comparisons with sota and thorough presentation of the related works.\n\n- The experimental results and absolute numbers reported in this paper are strong. This paper advances the state of the art in self-supervised representation learning.\n\n- In my opinion, the strongest result is in Table 7d, where the paper shows that MAE-300 epochs + TEC-100 epochs (i.e. 400 epochs)is competitive with MAE-1600 epochs pre-training. This convinces me that the method can speed up SSL pre-training.\n\n- Many ablations are presented.\n\n\n**Weaknesses**\n\n- Unconvincing “sustainability” ability of the paper.\nThis work starts from the observation that even though SSL is progressing in computer vision, models are not really used in practice for downstream tasks. The paper argues that this is because (i) pre-training SSL costs are too high and (ii) new and better SSL methods are appearing very frequently, preventing the field to stabilize on a framework.\nHowever this argumentation is flawed for the following reasons.\n(i) Models with extremely high pre-training budgets are being adopted in practice (GPTs, CLIP, models trained on JFT), meaning that expensive pretraining costs do no prevent the pretrained models to be used subsequently.\n(ii) The fact that new SSL methods arrive might just be a signal that SSL in vision is not fully mature yet and that further progress needs to be made to clearly beat supervised learning (which is still the most used pre-training in vision).\n\n- Table 6 is the weakest part of this paper. It shows that TEC does not achieve “sustainability” as advertised in Figure 1.\nUnlike the sustainability property advertised by the paper in Figure 1, Table 6 shows that a TEC model cannot be a based model for a new TEC-training round (marginal improvement of +0.1%).\n\n- Table 1 should include comparison with strong supervised baseline like DeiT-3. In addition, this is very misleading in Table 1 not to count the base model epochs into the total number of epochs. In other words, when training TEC-MAE for 300 epochs, the column should be 1900 epochs (1600 + 300) instead of 300.\n\n- Linear evaluation is 10 points below sota (iBOT for example). I find it weird that authors don’t report sota numbers in this Table 4.\n\n- The method is complex with many added components (input and encoder adapters, use of self-attention maps for feature selection). Complexity might not go towards sustainability.\n\n- This is not really a new SSL method but rather a feature distillation or enhancement one. Should compare with this literature (in last paragraph of related work) more and start from the same backbone as theirs for fair comparison.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8Qp6mmaIdTG", "reviewer": "Reviewer_2pYM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies a relatively new problem in self-supervised learning, which is called sustainable self-supervised learning. Based on a pretrained SSL model (or partially trained with certain epochs), the authors want to further improve it with the proposed target-enhanced conditional mask-reconstruction. The authors conduct experiments on benchmark datasets and shows the effectiveness of the proposed \"fine-tuning\" strategy.", "summary_of_the_review": "In summary, this paper studies an interesting paper and should have an impact on the computer vision field, while more clarification of motivation and experimental evaluation is needed.", "strength_and_weaknesses": "Strength: \n\n1. The problem setting is new or interesting. It should find applications in different areas, such as medical images. \n\n2. The proposed solution is reasonable and achieves good performance.\n\nWeaknesses:\n\n1. It seems that the authors should further clarify/motivate the proposed setting. Why do we need \"sustainable\" SSL. In the current introduction, the authors mention that researchers may have limited computational budgets to XXXXX, while it seems that the proposed solution cannot reduce the computation budget if I want to adopt a pretrained SSL model to downstream tasks. For my understanding, we still need to use a large ViT model and need to train many epochs. In other aspects, the previous SSL knowledge distillation can reduce the computational budget by re-training a small model. \n\n2. For the imagenet fine-tuning experiments, it is hard for me to understand the motivation of this new problem setting. This experiment looks like we want to continue \"pre-trained\" an SSL model with large-scale datasets. In this case, why do we not adopt a \"well-trained\" SSL model (MAE trained with 1600 epochs)? As it is only pretrained once, it can be done by a giant company and release to others for downstream tasks. In a word, the authors should reconsider the evaluation part to support the argument for adopting this new problem setting.\n\n3. It seems that the authors should compare with SSL knowledge distillation works. I understand the problem settings are different, while the method design should be similar. In some sense, the proposed method is also an advanced knowledge distillation method. \n\n4. For Table 1, it seems that the authors should clearly mention how the previous results () are obtained. For example, from released public models or re-implemented models? And how the base SSL model is acquired for the proposed framework. Did the authors re-train the base MAE model, or did the authors adopt the publicly released MAE model?\n\n5. The technical novelty may be incremental, as the key technical components are focused on designing new pretested tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5VAzpR_8G4x", "reviewer": "Reviewer_cZ41", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new reusable self-supervised learning framework by distilling knowledge from existing pretrained SSL models. Specifically, authors first introduce patch-relation enhanced targets to encourage the new model to learn semantic-relation knowledge and then introduce a conditional adapter that adaptively adjusts new model prediction to align with the target of each base model. Experiments using MAE and iBOT show the effectiveness of the proposed framework on ImageNet dataset.", "summary_of_the_review": "The experiments are limited and not convincing in the current version of the paper. Due to the limited novelty and lack of convincing experiments, my initial recommendation is to reject the paper (below the acceptance threshold).", "strength_and_weaknesses": "**Strengths**:\n\n- The paper is easy to read, and very well written with clear presentation of the figures/tables.\n- The overall problem of reusing pretrained SSL models is interesting and the solution proposed to solve the problem has clearly demonstrated advantage on ImageNet dataset. \n\n**Weaknesses**:\n\n- The main technical contribution of the paper is actually on distilling knowledge from an existing pretrained model to another new model. This is in fact very much similar to reversed KD presented in \"Revisiting knowledge distillation via label smoothing regularization\" (transferring knowledge from a small model to a large model) but there is no discussion about this in the paper. I don't fully agree with the authors that knowledge distillation only focuses on transferring knowledge from strong and large models to compact new models. Authors should clearly discuss about the similarity between the proposed framework and (reversed) KD.\n\n- Is there any restriction on the model size across base and new model? Since KD is agnostic to the model size, authors should perform experiments by transferring knowledge from a smaller SSL model for training a larger model, e.g., use of ViT-Small for training ViT-Base with MAE.\n\n- In many practical scenarios, multiple pretrained models are usually available that can be leveraged for training a new model? How this framework can be used to exploit multiple models with same and/or different architectures? Authors should perform experiments to verify this using multiple base models.\n\n- Can the proposed framework exploit SSL models trained using instance discrimination, e.g., SimCLR for training a new ViT MAE model? What about the performance in these scenarios?\n\n- Sustainability usually refers to savings in terms of training cost which is measured by FLOPs or GPU wall time. While the comparison includes number of epochs, it does not reflect real savings. I would encourage authors to include Accuracy vs FLOPs and Accuracy vs Wall time to demonstrate how use of pretrained SSL model can help in training efficiency.  See this paper: Knowledge Inheritance for Pre-trained Language Models for more details which uses a smaller pretrained model to accelerate training of a larger language model or bert2BERT: Towards Reusable Pretrained Language Models which solves a similar problem in NLP.\n\n- The proposed method uses a bunch tricks on top of the vanilla knowledge transfer strategy without analyzing other alternatives. What is the use of adapters at the input level and encoder level is not clear? Can authors qualitatively demonstrate how this helps in learning better features in the new model?\n\n- What about the number of trainable parameters? What about the increase in training time due to the increase in number of parameters? A more thorough analysis on this should be included in the paper.\n\n-  Besides empirical performance, how does semantic attention help learning diverse complementary features? Can authors perform visualization experiments to verify this?\n\n- What about the additional computation incurred by passing all the images through the base model? An analysis on how much data we need to pass through the base model for an effective knowledge transfer should also be included in the paper.\n\n- Did authors try considering this knowledge transfer as an additional regularizer on top of the standard MAE loss for the new model? What happens if we train the new model using a combination of the standard MAE loss that does depend on the base model and another loss that actually matches the prediction with the base model using the proposed framework?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject