SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders
Authors: Gang Li
Heliang Zheng
Daqing Liu
Chaoyue Wang
Bing Su
Changwen Zheng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Semantic-Guided Masking
Masked Autoencoders
Self-Supervised Learning
Semantic part learning

Abstract: Recently, significant progress has been made in masked image modeling to catch up to masked language modeling. However, unlike words in NLP, the lack of semantic decomposition of images still makes masked autoencoding (MAE) different between vision and language. In this paper, we explore a potential visual analogue of words, i.e., semantic parts, and we integrate semantic information into the training process of MAE by proposing a Semantic-Guided Masking strategy. Compared to widely adopted random masking, our masking strategy can gradually guide the network to learn various information, i.e., from intra-part patterns to inter-part relations. In particular, we achieve this in two steps. 1) Semantic part learning: we design a self-supervised part learning method to obtain semantic parts by leveraging and refining the multi-head attention of a ViT-based encoder. 2) Semantic-guided MAE (SemMAE) training: we design a masking strategy that varies from masking a portion of patches in each part to masking a portion of (whole) parts in an image. Extensive experiments on various vision tasks show that SemMAE can learn better image representation by integrating semantic information. In particular, SemMAE achieves 84.5% fine-tuning accuracy on ImageNet-1k, which outperforms the vanilla MAE by 1.4%. In the semantic segmentation and fine-grained recognition tasks, SemMAE also brings significant improvements and yields the state-of-the-art performance.

SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders
Gang Heliang Zheng'. Daqing Liu?, Chaoyue Wang', Bing Su', Changwen Zheng"
Institute of Software, Chinese Academy of Sciences*.
University Chinese Academy of Sciences? JD Explore Academy?, Renmin University of China
ucasligang@gmail. com, (zhengheliang, lindaqiing wangchaoyuus)@@j..com bingsuōruc. edu. en, changwen@iscan cn.cn
Abstract
Recently, significant progress has been made in masked image modeling to catch up to masked language modeling. However, unlike words in NLP, the lack semantic decomposition of images still makes masked autoencoding (MAE) different between vision and language. In this paper, we explore potential visual analogue words, i.e., semantic parts, and we integrate semantic information into the training process of MAE by proposing Semantic-Guiie Compared to widely adopted random masking, our masking strategy can gradually guide the network to learn various information, i.e., from intra-part patterns inter-part relations. In particular, we achieve this two steps. 1) Semantic part learning: design self-supervised part learning method obtain semantic parts by leveraging and refining the multi-head attention of a ViT-based encoder. 2) Semantic-guidee
Masking strategy.
MAE (SemMAE) training: we design masking strategy that varies from masking portion of patches in each part to masking a portion of (whole) parts in image. Extensive experiments on various vision tasks show that SemMAE can learn better image representation by integrating semantic information. In particular, SemMAE achieves 84.5% fine-tuning accuracy on ImageNet-Ik, which outperforms the vanilla MAE by 1.4%. In the semantic segmentation and fine- mne-grained recognition tasks, SemMAE also brings significant improvements and yields the state-of-the-ard performance. Our code is available at https://githann thub..th...... cww/cc////e//oo Semaan/////mMA
Introduction
Together with transformers, masked language modeling (MLM) has revolutionized the field of self-supervised learning (SSL) in natural language processing (NLP), which enables training of generalizable NLP models containing over hundred billion parameters [5]. The concept of MLM is quite intuitive, i.e., a portion of the data is removed and a model is trained to predict the removed content. Recently, significant progress has been made in masked image modeling catch up masked language modeling, where the masking mechanism is a key factor. Context encoder (28). an inpainting- baasdd masked image modeling (MIM) pioneer, proposes
random and fix-shaped mask; SiT 2 and BEiT (3] use random "blockwise" masks, where patches in the local neighbourhood masked together (also called GMML: group mask model learning); MAE [9 randomly masks out 75% patches of an image. Actually, masking mechanisms define the specific pretext task, i.c., what kind of information is be exploited and what kind of information is be predicted. Thus
"This work was performed when Gang was visiting JD Explore Academy as research intern. Cooresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

AttMask [2]] strategy to make informed decisions. ADIOS 30 takes one step further to "lear to mask" by adversarial training.
studies the problem of which tokens to mask and proposes an attention diiention-guided mask
Although promising performance has been achieved, there is still large gap for masked autoencoding (MAE) between vision and language due to different signal natures. A sentence can be semantically decomposed into words, while the semantic decomposition of an image not trivial to be obtained. To find a visual analogue of words, we investigate part-based image representation. Specifically, the real world is composed of objects, which consist of different parts. Therefore, part-based image representation is a fundamental image representation method that fits the inherent properties of objects 8 [5] [6] [8] 20]. For example, part-based Pictorial Stracture [16] dominated the image representation field for several years in the early days of computer vision, and Deformable Part Model (DPM) [5] was also a milestone in image recognition and detection. Moreover, GL.OM 20 argues that the hierarchical representation with five levels (i.e., the lowest level, sub-part level, part level, object level, and scene level) would be a powerful image representation method in the future. To this end, we argue that semantic parts would be potential visual analogue of words. With such visual analogue, more controllable hints can be built up to guide the learning of MAE, thus high-level visual representations can be well learned.
In this paper, we first propose a self- self--uuerrve supervised semantic part learning method to obtain semantic parts for each image. Our insight is that the spatial information to reconstruct an image is highly correlated to the position of semantic parts. In particular, our part learning model consists of ViT-based encoder together with an attention module that generates a class token and multiple attention maps, and JayleGAN-based decoder that reconstructs the original image. The attention maps are optimized to provide spatial information, and the class token is integrated into the decoder via AdalN to provide texture information. We find that the optimized attention maps can indicate part positions, and we conduct an argmax operation to obtain part segmentation maps. After that, we study how semantic parts can facilitate the learning of MAE. We design a masking strategy that varies from masking a portion of patches in each part masking a portion of (whole) parts in an image. Such design can gradually guide the network to learn various information, i.e., from intra-part patterns to inter-pan relations. Extensive experiments on various vision tasks (e.g., linear probing, fine- fine-tuning, semantic segmentation, and fine-gres grained recognition) show that SemMAE can learn better image representation by integrating semantics.
Our contributions include 1) designing a sulf-supe semantic part learning method that can generate promising semantic parts on multi-class datasets, i.e., ImageNet, and 2) verifying that semantic parts can facilitate the learning of MAE by proposing a semantic-guided masking strategy. While more importantly, we hope our attempts can provide insights for the community to study the visual analogue of words and unified vision and language modeling.
2 Related work
Semantic part learning. Part-based image representation is a fundamental image representation method that fits the inherent properties of objects 15 16 18 (0). However, due to the tremendous cost of labeling parts, there are still no large-scale datasets containing part labels. Thus previous works are mainly two-fold, i.e., mnsupervised/wwaklyyyeo -upprise part learning and few-shot part segmentation. part information by leveraging spatial priors, the semantics of convolutional channels, or designing
part learning methods [10] 22 39 propose to mine
contrastive proxy tasks. Few-shot part segmentation methods [4] 29 38) mainly learn an additional classifier over pre-trained features that are trained by GAN, self-supervised contrastive learning. or denoising diffuusin probabilistic modeling. Although promising results have been obtained, these models are designed to deal with fine-grained datasets, where all images belong to a single super-class (e.g., birds, cars, or human faces). It is much more challenging to solve the problem of unsupervised part learning on multi-class datasets such as ImageNet. With the development of ViT and self supervisd learning (SSL), some recent works show a potential solution. In particular, DINO 66) and iBOT 42 have observed intuitive semantics in the ViT trained by their SSL methods, where the multi-head attention maps can somehow indicate different semantic parts of an object. Inspired by these works, we design a reconstruction- method to further refine the attention maps learned by iBOT to obtain semantic parts on the ImageNet dataset.

---

(a) context encoder
(b) BEIT
(c) MAE
(d) AttMask
(e) MST
semantic parts
Iter1: 75% per part
iter #i
Iter #n: 75% parts
(f) ADIOS
(g) ours
Figure 1: Comparison of different masking strategies. Detailed information for each compared model can be found in Section 2 Masked Image Modeling.
Masked image modeling. Inspired by the success of Masked Language Modeling (MLM) (5] in pre-training of the NLP field, Masked Image Modeling (MIM) has been proposed recently and exhibits promising potential for visual pre-training 7 19 30 Existing works mainly study the problem of MIM from two directions, i.e., regression targets and masking strategies. In terms of regression
targets, BeiT [3] mc-BEiT 24, and PeCo [3] adopt tokens produced by VQ-VAL 32 or its variants. MaskFeat 35 studies broad spectrum of feature types and proposes to regress Histograms of Oriented Gradients (HOG) features of the masked content. MAE [99 and SimMIM
argue that predicting RGB values of raw pixels by direct regression performs no worse than the patch classification approaches with complex designs. In this paper, we follow MAE [9 to adopt the most pixels regression. In terms of masking strategies. SIT 22, MC-SSLO.0 [I] and BeiT 3 use block-wise masking strategy, where block of neighbouring tokens arranged spatially are masked. MAE [19] and SimMIM (37) use random masking with a large masked patch size or large proportion of masked patches. MST (25] and AttMask (21] propose to use attention maps guide the masking strategy. where the former proposes mask the nonessential regions to preserve crucial patches while the latter proposes to learn image representations with challenging tasks by masking the most attended tokens. Moreover, ADIOS (3) proposes learn optimal mask by adversarial training. Compared to these works, our SemMAE takes one step further and explicitly learn semantic parts build reasonable hints for masked image modeling. Figure I is illustration of different masking strategies.
simple and intuitive
3 Semantic-guided masked autoencoders
We propose a Semantic-guided Masked Autoencoder (SemMAE) for self-supervised image representation learning with mask image modeling. The framework of SemMAE is shown in Figure which consists of two key components, i.e., Semantic Part Learning (A) and Semantic- Guided Masinn (B). First, given image in Figure (a), we extract the class token in Figure) (b) and patch tokens in Figure 2 (c) by an iBOT-pretrained ViT. After that, we learn an embedding
(a),
the class token obtain part tokens in Figure 2 (d). We calculate the correlation of each part token to patch tokens to obtain attention maps in Figure 2 (c). whose texture information is further removed by a large-kernel kernel blur operation. The attention maps are optimized by diversity constraint and reconstruction task where the attention maps and the class token are fed into StyleGAN- based decoder control the spatial and texture information of the reconstructed image, respectively. Finally, we conduct argmax over the attention maps to obtain part segmentation maps in Figure and used them to guide the mask generation for MAE. Specifically, we design a masking strategy that varies from masking portion of patches in each part to masking a portion of (whole) parts in image. Such a design can gradually guide the network to learn various information, i.e., from intra-part patterns to inter-part part relations.

---

[A] Semantic Part Lemmnng
(0) part toiaan
diversity constraint
reconstrction task
embed
blur
comv
i801 pretrsined ViT
gir
[c] zatch takien
maps
(a] oriinal
-esed encocer
ortentinn
StyleG based decoder
[la] original image
argmax
(B) Semanti- G uided Masking
guide
VIT
decoder
Vi--oweed
partsoqmentaionn
Paration
An illustration of the proposed SemMAE, (A) Semantic Part Learning. A ViT-based encoder takes input an image in (a) and produces class token in (b) and patch tokens in (c). Our attention module first learns to embed the class token into part tokens in (d) and then generates an attention map for cach part token by calculating the correlation between the part token and patch tokens. As an objective function of the attention maps, our StyleGAN-barn information from the class token. (B) Semantic- Guideed Masking. We conduct argmax over the attention maps to obtain part segmentations in (f), which used guide the mask generation. During the training of the MAE, the masks vary from portion of patches each part to portion of (whole) parts an image.
Figure 2:
decoder learns to reconstruct the original image from attention maps with texture
3.1 Semantic part learning
In this subsection, we introduce our self-supervised semantic part learning method. Previous part learning methods mainly designed to deal with single- class datasets (i.e., fine tiee-graindd datasets where images belong to the same superclass). Few methods able to solve this problem under multi-class scenario (e.g., ImageNet). While some recent works (i.e.. DINO ( and iBOT [42) on ViT-based self-supervised learning show that the multi-head attention maps in their model somehow indicate different semantic parts of object. In this work, we take advantage of semantics learned in iBOT and design reconstruction task together with diversity constraint refine and obtain semantic parts.
In particular, given an image I, we first use an iBOT-pretrained VIT to extract its features, i.e., a class token F. RCx1 and patch tokens F E RCxHW Then, embed the class token into N part tokens F, E CxN The main idea of such embedding is to re- re-weight feature channels of the class token. As shown in previous methods (39), feature channels may be corresponding specific semantics and channel re-weighting can group channels with similar semantics together to obtain semantic part features. Thus
obtain part tokens by:
F ( = Fe sigmoid w c2 tanh(W F.)),
(1)
where E [1, 21, MJ. F E cx11 the 1th column vector of F, E RCNN, indicates hadamard product, ((() el and w (t) c2 are embedding weights, tanh(-) and sigmoid(-) are activation functions.
After that, we calculate the correlation of each part token to the patch token in each position, thus can obtain attention maps, i.e., the possibility of a semantic part to appear in each position:
M = F ② F = softmax F WF'WW WF).
where M NWW denotes N attention maps, & indicates correlation function, which is implemented by softmax(F; WT'Wr WF) in our work. Wp and W are embedding matrixes.
To learn such multi- attention maps (i.e., optimize the parameters in Equation Aand Equation 2 we propose reconstruction task. Our insight is that the spatial information to reconstruct an image highly correlated to the position of semantic parts. Thus, we adopt a StyleGAN to reconstruct the original image based on the spatial information from the attention maps and the texture information from the class token. ensure the attention maps learn spatial information, we 1) remove texture information from the attention maps by conducting a large-kernel blur operation
decoder

---

and 2) further feed the blurred attention maps to stacked convolutional layers. To integrate the texture information from the class token into the decoder, we use Adaptive Instance Normalization (AdaIN)
operation, which is widely used to integrate texture/style information:
Fal: AdaIN((conv(M)) F.) [.F.... (cov((M) p[[conv (M)J;) [W.F.]e,
(3)
((conv(M))).
where each feature channel [conv (M)]: is normalized separately, and then scaled and biased using the corresponding scalar components from the embedded class token Fe. Fa denotes the convolutional feature in the decoder. W, and Wb embedding weights, H]s denotes the ith feature channel, conv(-) denotes convolutional layers, (u-) and (.) calculate the mean and variance values, respectively. The reconstructed image I can be obtained by stacking convolutional and AdaIN layers:
= We use the Mean squared error (MSE) loss function to optimize such reconstruction task:
Fe)).
(4)
HW
Crec(I.1) = HW
((i.))
(5)
2
Moreover, to obtain diverse multiple attention maps, we follow previous work 40) and add a diversity constraint over attention maps:
m.
+ Σa m
Laio (M)- = N2
(0-
(6)
m
where attention maps are optimized to be different from each other, m, and m, denotes the 7th and jth attention map, respectively. The overall objective function can be denoted by:
L = Lree (I, + ALdio(M)
where À is the loss weight.
3.2 Semantic-guided masking
After finished semantic part learning, we move to the next stage, i.e., semantic-guided MAE training.
Our informed masking strategy is based on the part information learned in Subsection 3.1 Specifically, we can obtain multiple attention maps by Equation 2 where each attention map m E 7 xxx 1 indicates
the possibility of the corresponding semantic part appearing in H x W positions. Thus we conduct argmax(-) operation over attention maps to obtain part segmentation, where each patch classified into a particular semantic part. The patches in the same semantic part compose visual analogue of words, which are semantically meaningful. To leverage such visual analogue of words for MAE training. most intuitive way is to mask portion of semantic parts and learn to predict the masked semantic parts by other parts. However, due to the learned semantic parts being coarse- coaree-griined (c.g.. 6 parts for each image), experimentally find that such masking strategy makes the task too hard to effectively learn image representations.
To this end, we propose an easy-to- to-hard reconstruction task, which can provide reasonable hints (i.e.. visible patches) for the model predict the masked patches during the training process of the MAE Specifically, at the beginning of the training process, we mask portion of patches in each part, thus the masked patches can be predicted based on the visual patches that belong to the same semantic part. Such a design can facilitate the models to learn intra-part patterns. After that, we gradually mask all patches belonging to some parts and a portion of patches belong to the remaining parts. Finally, mask all patches belonging to portion of parts and predict the remaining patches belong to the other parts, where inter-part relations or visual reasoning ability can be learned.
Algorithm I shows the details to obtain the number of masked patches for each semantic part. First, we define two masking settings, i.e., mask a portion of patches in each part and 2) random select some parts to mask (the whole part). The number of masked patches for each semantic part can be calculated for these two settings. After that, we introduce an interpolation hyper-parameter a. A small means the first setting dominates the masking strategy, and vice versa. is adjusted based on training iterations and keeps increasing during the training process. Finally, we random mask certain number of patches based on the calculated masking number.
5

---

Algorithm Algorithm of Semantic-Guided Masking in a PyTorch-like style.
Input: L, 2, neun_palches, mask_rabio, tolal_epoches epoch
L: the number of patches per image.
# x E RLxx; the token embeddings of image patches.
# num hum_patches IN the patch number of each part, where N the number of parts.
# mask_ratio: the ratio of masked patches.
total eeachee: the number of pre-training epochs.
# epoch: current epoch number.
# mask portion of patches in each part
1: nun mun_mask1 = quask_ratio * num sum_palchess
# randomly select some parts mask (with tricks to ensure a fixed mask ratio)
2: shuf fae_num nwm sat___mtmmttthheeeeaaae = shuffle fle_parts(num__.. (mum_patchee))
3: marks = L mask ratio-cumsum(shettt fle wtt.://feeemmmaatcceheeeeeeeeeee pam__atchhss)sshuf (inicdes))ssufffeeeee.tttttd num
4: murks_remainns = where(marks O, O, marks)
5: num_ num_mask2 = saatremmmrrsseemiin <
< www_mmtthsss,,, marks_remains
6: num_maski unshuf fle_parletrte www.ppatttsummmmampa mnmm_makk2)
# adaptive masking by interpolating between num mask1 and num mask2
7: a = Total epothe_ 7
epoch
8: num_mask = (1 a.) * nuum_aski + a mum__ask2
9: return num_mask # reum_mask E IR Nx;; the number of patches to be masked in each part.
4 Experiments
4.1 Experiment setup
Semantic part learning. As introduced in Section 3.1 we use ViT-small 14 as our part learning encoder, which is pre-trained by self-superviser method iBOT 42 We follow iBOT [42 to learn semantic parts for each image, as the head number of the multi-head attention in ViT-Small The size of the blur kernel is experimentally set to be 7, and the loss weight À in Equation 7 is set to be 0.03. The experiment is performed on ImageNet-Ik [ dataset. The parameters of the ViT-based encoder are fixed, and we only optimize the attention module and the StyleGAN-basee decoder. Our model converges fast, which only takes hours on one A100 GPU card.
Semantic-guided MAE training. We follow MAE 19 and adopt an mucoder-decodee structure to perform MIM. Our method general for VIT backbones, while most experiments are conducted with relatively small version, i.e., the original ViT-Base [14), due to the limitation of computation resources. We follow the most comment setting to optimize our model by AdamW [27] with a learning rate of 2.4e-3. The batch size is set to be 4096, and the weight decay is set to be 0.05. We use a cosine learning rate strategy (26] with warmup 17]. The warmup number set to be 40 epochs, and we pre-train our model for 800 epochs. For data augmentation, we only employ random horizontal flipping in our pre-training stage. The hyper-parameter 7 in Algorithm i experimentally set to be 2. Our model is trained on 16 A- GPUs for 3 days, and more details can be found in our code, which is in the supplemental material and will be made publicly released.
Table 1: Quantitative evaluation of the effectiveness of integrating semantic information for MAE.
16x 16 patch size
8 x patch size
Setting
MAE [19
SemMAE
MAE [19
SemMAE
Linear probing
63.7
65.0
66.8
68.7
4.2 Semantic-guided MAE
The effectiveness of integrating semantic information. We conduct experiments under two different settings (i.e., with patch size of 16x 16 and 8 8x8) to verify the effectiveness of integrating semantic information for training MAE, The results in Table I show that integrating semantic information can bring 1.3% and 1.9% accuracy gains for lincar probing, respectively. As use 8x8 patch size to

---

Table 2: The optimal patch size for different models.
Model
Patch size
Fine-tuning Acc.(%)
Model
Patch size
Fine-tuning Acc.(%)
32x32
82.8
16x16
83.26
MAE 19
16x16
82.7
8x8
83. 10
SimMIM 37
8x8
82.1
16x
83.34
SemMAE
4x4
82.0
8x8
84.50
learn semantic parts, the coarse-grained patch (i.e., large patch size) in the pre-training stage would cause imprecise part segment and suppresses the benefits semantic parts. To further study the impact of patch size for masked image modeling,
conduct fine-tuning experiments in Table 2 can be observed that in SimMIM and original MAE, a larger patch size performs better; while in our SemMAE, more precise semantic parts with 8x8 patch size significantly improve the performance. Thus in the following experiments, we adopt 8 8x8 patch size for SemMAE. is notable that although using smaller patch size, our parameters and computational cost do not increase during pre-training and linear probing as only 1/4 patches each image are used. Specifically, we leverage the learned attentions maps remove 3/4 patches that most likely be the background.
detailed study on masking strategies. Once obtained semantic parts, a most intuitive way to leverage such visual analogue of words for MAE training is to mask portion of semantic parts and make the model to predict the removed content. However, due to the learned semantic parts being coarse-graing (e.g.. 6 parts for each image), we experimentally find that such masking strategy makes the task too hard to effectively learn image representations. The results can be found in Table where masking 75% parts cause 13.9% performance drops compared random masking. Moreover, can be observed that masking 75% patches per part achieves comparable results with random masking. The self-supervised learning task of masking 75% patches per part would encourage the model to learn local contexts/intra-put patterns, and masking 75% parts would encourage the model to learn inter-part relations. Interestingly, find that the former task enable the model to further learn better image representation in the latter task. The results in Table 3 show that our proposed adaptive masking strategy (i.e., varying from masking 75% patches per part masking 75% parts gradually) with y = 2 yields the best performance.
Mask strategy Random masking
Y
Linear probing
66.8
Mask 75% patches
0
66.5
3
Mask 75% parts
52.9
1
0
1
63.3
x=2
1/3
66.2
==
1/2
67.3
Adaptive masking
0
1
67.9
2
68.7
training progress
3
68.6
Figure 3: The curves of and
Table 3: Quantitative evaluation of different masking strategies.
4.3 Semantic part learning
We first evaluate the effectiveness of our proposed semantic part learning method. Both qualitative and quantitative experiments are conducted. Note that most of the previous part learning models are designed for single-class datasets and cannot be effectively applied to ImageNet. iBOT [42] is not proposed for part learning, while the multi-head attention in their model achieves the state-of-the -te-art part learning performance on ImageNet. Figure showss the qualitative comparison of our model and iBOT [4). and it can be observed that our model can generate more complete semantic part segmentation maps where different parts and the background are better separated with less noise. Moreover, as there is no part segmentation ground truth, we conduct quantitatively evaluation in an indirect way by training SemMAE and analyzing ImageNet classification performance. The results in Table 4 show that the semantic parts obtained by iBOT are not able to benefit the learning of MAE, while our semantic part learning methods can generate more precise part segmentation maps, which vital to learning a better image representation.

---

image
IBOT
ours
image
iBOT
ours
Figure Qualitative comparison of semantic part learning. Different color indicates different semantic parts, and it can be observed that our model can better separate different parts and the background with less noise.
Table 4: Quantitative evaluation of semantic part learning terms classification accuracy($)).
Semantic parts for masking Baseline (w/o parts) iBOT-initialized parts Our learned parts
Linear probing Acc. (%)
63.7
63.6
65.0
Table 5: System-level comparison ImageNet- k in terms of classification accuracy using ViT-Base as the encoder. Note that list the best performance in previous papers with 224 224 inputs, and some experiment settings (e.g., training epochs and patch size) may different.
Method
Pre-train dataset Pre-train epoches
Linear probing
Fintuning
Traning from scratch
ViT 14
77.9
DeiT 31
81.8
ViT 19
82.3
Contrastive-bused SSL Pre-Training
AttMask 42
ImageNet-Il 1K
100
75.7
DINO 6
ImageNet- IK
300
78.2
82.8
MoCo
ImageNet-1K
300
76.5
83.2
iBOT ImageNet-IK
1600
79.5
84.0
MIM-based SSL Pre-Training
BeiT 3
ImageNet-1K
800
56.7
83.2
MAE 19
ImageNet-1K
1600
68.0
83.6
SimMIM 37
ImageNet-IK
800
56.7
83.8
SemMAE
ImageNet-IK
800
68.7
84.5
4.4 Compared with other methods on ImageNet
Linear probing and fine-tuning on ImageNet- IK classification dataset is the most common setting to evaluate SSL methods. We collect all competitive methods that report their results on ImageNet-1K
dataset. For example, we do not include the related work MST (25] and ADIOS (30] as they evaluate their model other benchmarks. Table 5hoho the comparison of our model and previous models terms linear probing and fine-tuning. For fair comparison, all experiments adopt the same input size, i.e., 224 x 224 unless specified otherwise. Compared with "training from scratch", our SemMAE can significantly improve the performance for both linear probing and fine-tuning. For lincar probing, our SemMAE outperforms the most competitive MIM-based methods by 0.8%

---

even with fewer training epochs. For fine-tuning, our SemMAE achieves 84.5% top-1 classification accuracy, outperforming SimMIM(37] and MΛ by 0.9% and 0.7% respectively. Moreover, our SemMAR can also surpass previous contrastive learning-hasse hased methods 6 for fine-tuning.
4.5 Downstream tasks
Fine-grained image classification. Table 6 shows results of transfer learning fine-graines datasets. Our model can surpass the most competitive MAE [99] with a clear margin, i.c., 0.3%, 0.6%, and 0.2% on the [Naturalists(33), CUB-Bic Beee-BirdBi [34] and Stanford-Cart [3] dataset, respectively. These results show the promising transfer ability of our SemMAE for downstream classification tasks.
Table 6: Fine-tuning results fine-grained datasets.
Method
iNa19
CUB
Cars
BeiT
K
79.2
94.2
DINO 3
78.6
93.0
iBoT 19
79.6
94.3
MAE 19
81.8
86.5
94.2
SemMAE
82.1
87.1
94.4
Semantic segmentation. Semantic segmentation aims to assign label to each pixel of the input image. We evaluate our SemMAE on the widely used semantic segmentation dataset ADE20K [4]], which contains 25K images and 150 semantic categories. We follow the most common setting to use the task layer in UPerNet (36] and fine-tune the pre- trained ViT-Base model. We use the standard setting that pre-train ViT-Base model with patch size of 16 x 16 and fine-tunes 160K iterations with a batch size of 16. Such an experiment can validate the transfer ability of our SemMAE for semantic segmentation. As shown in Table 7 SeMAE surpasses MAE by 0.2 (46.3 46.1) mloU and outperforms the supervised pre-train model by 1.0 mloU. These results show the promising transfer ability of our SemMAE for dense prediction visual tasks.
Table 7: Semantic segmentation results on ADE-20K.
Method
mloU
Supervised Pre- Tre-Training
45.3
Self-Supervised Tre-Trinnng
BeiT
45.8
MAE(800 epochs)
46.1
SemMAE (Ours)
46.3
5 Conclusion
In this paper, we study the visual analogue of words and propose semantic-guided masked autoencoder model to reduce the gap between masked language modeling and masked image modeling. Our proposed self-supervised semantic part learning method can generate promising semantic parts on ImageNet and we show that the learned semantic parts can facilitate the lcarning of MAE. Unlike the main-stream random masking strategy, semantic-guide mask strategy can effectively integrate semantic information in the pre-training process. Extensive experiments with superior results show the effectiveness of our SemMAE.
Limitations: due the lack of part segmentation labels, the semantic part in our work is kind of coarse (e.g., 6 parts per image), making it not an ideal visual analogue of words yet. Moreover, using a small patch size increases the computational cost in the fine-tuning stage. the future, we will 1) investigate finer-grained semantic parts (e.g., 20-30 parts per image) by few-shot part segmentation and 2) replace the widely obtained patch-based tokenization with part-based tokenization further reduce the gap between vision and language modeling.

---

References
[1] Atito, S., Awais, M., Farooq, A., Feng, Z., Kittler, J.: Me-sslO. O: Towards multi-concept self-supervised learning, arXiv preprint arXiv:2111.15340 (2021)
[2] Atito, S., Awais, M., Kittler, J.: Sit: Self-supervised vision transformer. arXiv preprint
arXiv:2104.03600 (2021)
[3] Bao, H., Dong, L.,, Piao, S., Wei, E: BEit: BERT pre-training of image transformers. In: ICLR (2022)
[4] Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., Babenko, A.: Label-efficient semantic segmentation with diffusion models. In: ICLR (2022)
[5] Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. NeurlPS 33, 1877-1901 (2020)
[6] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV, pp. 9650-9660 (2021)
[7] Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., Sutskever, I.: Generative pretraining from pixels. In: ICML.. pp. 1691-1703 (2020)
[8] Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., Yuille, A.L.: Detect what you can: Detecting and representing objects using holistic models and body parts. In: CVPR. pp. 1979- 1986 (2014)
[9] Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: ICCV. pp.
(2021)
[10] Choudhury, S., Laina, I., Rupprecht, C., Vedaldi, A.: Unsupervised part discovery from contrastive reconstruction. NeurIPS 34 (2021)
[11] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. CVPR. pp. 248-255 (2009)
[12] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
(2018)
[13] Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F. Yu, N.: Peco: Perceptual codebook for bert pre-training of vision transformers, arXiv preprint arXiv:2111.12710 (2021)
[14] Dosovitskiy, A.. Beyer, L.. Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
[15] Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. TPAMI 32(9), 1627-1645 (2009)
[16] Fischler, M.A., Elschlager, R.A.: The representation and matching of pictorial structures. TC 100(1), 67-92 (1973)
[17] Goyal, P., Dollár, P., Girshick, R.. Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K.: Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 1rXiv:1706.02666 706.02677 (2017)
[18] He, J., Yang. S., Yang, S., Kortylewski, A., Yuan, X., Chen, J.N., Liu, S., Yang, C., Yuille, A.: Partimagenet: A large, high-quality dataset of parts. arXiv preprint arXiv:2112.00933 (2021)
[19] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.063 (2021)
[20] Hinton, G.: How to represent part-whole hierarchies in a neural network, arXiv preprint 1rXiv22102.12621 (2021)
10

---

[21] Kakogeorgiou, I., Gidaris, S., Psomas, B., Avrithis, Y.. Bursuc, A., Karantzalos, K., Komodakis, N.: What hide from your students: Attention ceattioonguieed masked image modeling. arXiv preprint arXiv:2203.122 12719 (2022)
[22] Krause, Jin, H., Yang, J., Fei-Fei, L.: Fine-grained recognition without part annotations. In: CVPR. pp. 5546-5555 (2015)
[23] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: Proceedings of the IEEE international conference on computer vision workshops. pp. 554-561 (2013)
[24] Li, X., Ge, Y., Yi, K., Hu, Z., Shan, Y., Duan, L.Y.: mc-beit: Multi-choice discretization for image bert pre-training. arXiv preprint arXiv:2203.15371 (2022)
[25] Li, Z., Chen, Z., Yang, F., Li, W., Zhu, Y., Zhao, C.,, Deng, R., Wu, L., Zhao, R., Tang, M., et al.: Mst: Masked supervid transformer for visual representation. NeurIPS 34 (2021)
[26] Loshchilov, I., Hutter, F: Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv:1608.088 039983 (2016)
[27] Loshchilov, I. Hutter, F: Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101 (2017)
[28] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: CVPR. pp. 2536-25666222 2544 (2016)
[29] Saha, O., Cheng, Z., Maji, S.: Ganorcon: Are generative models useful for few-shot segmentation? arXiv preprint urXiv:21 (2021)
[30] Shi, Y., Siddharth, N., Torr, P.H., Kosiorek, A.R.: Adversarial masking for self-supervised learning. arXiv preprint trXiv:2201.13100 (2022)
[31] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers distillation through attention. In: ICML. pp. 10347-10357 (2021)
[32] Van Den Oord, A., Vinyals, O., et al:: Neural discrete representation learning. NeurIPS 30 (2017)
[33] Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: Proceedings the IEEE conference computer vision and pattern recognition. pp. 8769-8778 (2018)
[34] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-uesd dataset (2011)
[35] Wei, C., Fan, H., Xie, S., Wu, C.Y., Yuille, A., Feichtenhofer, C.: Masked feature prediction for self-supervised visual pre-training. arXiv preprint 1rXiv:2112.091333 12.09133 (2021)
[36] Xiao, Liu, Y., Zhou, B., Jiang. Y., Sun, J.: Unified perceptual parsing for scene understanding. In: ECCV, pp. 418-434 (2018)
[37] Xie, 7.,, Zhang, 7.,, Cao, Y., Lin, Y., Bao, J., Yao, 7.,, Dai, Q., Hu, H.: Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:21 .09886 (2021)
[38] Zhang, Y., Ling, H., Gao, J., Yin, K., Lafleche, J.F., Barriuso, A., Torralba, A., Fidler, S.: Datasetgan: Efficient labeled data factory with minimal human effort. In: CVPR. pp. 10145- 10155 (2021)
[39] Zheng, H., Fu, J., Mei, T., Luo, J.: Learning multi-attention convolutional neural network for fine-grained image recognition. In: CVPR. pp. 5209-5217 (2017)
[40] Zheng, H., Fu, J., Zha, 7.J., Luo, J.: Learning deep bilinear transformation for fine fine-grained image representation. NeurlPS 32 (2019)

---

[41] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: CVPR. pp. 633 641 (2017)
[42] Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A.. Kong, T:: ibot: Image bert pre-training with online tokenizer. arXiv preprint vrXiv:2111.000 1.78222 (2021)
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] You strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or a URL)? [Yes] In the supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section Experiment Setup.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section E Experiment Setup.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In our code.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
12

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "Yhdmh_xC1WH", "reviewer": "Reviewer_KRLr", "summary": "A semantic-guided masking modeling method is proposed in this paper. To achieve this, an iBOT-pretrained ViT is used to extract the features, and then StyleGAN-based decoder is trained to learn semantic parts. Finally, the learned semantic parts attention maps are used to guide the mask modeling per-training. Extensive experiments are performed on classification and semantic segmentation downstream tasks. ", "questions": "- Please see *Strengths And Weaknesses*\n- In the end of the line 155: convolutional **can** AdaIN layers --> convolutional **and** AdaIN layers\n- How about the total training cost including the iBOT pre-training, part learning, semantic guide pre-training, compared to the other pre-training methods. ", "limitations": "The proposed method assumes that the images are object-centric. It's true for ImageNet dataset. How about the proposed method on none-object images, e.g. texture images, scene images, and et.al? ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- Semantic-guide MAE training is an interesting idea. But it does not convice me that the reason is just to have a potential visual analogue of words.  I encourage the authors to make it clearer why we need semantic-guide MAE training, and more efficient. \n\n# Weaknesses\n- The proposed methods involved too many components, iBOT feature extractor, including StyleGAN-based decoder, an intermediate reconstruction tasks, gradually masking strategy, and et.al.  to make it work. The complicated whole framework makes it hard to analyze how the proposed method works. \n- It lacks the intuitive explanation why the proposed intermediate reconstruction tasks can help to learn the part attention maps. \n- The improvement on downstream tasks is marginal as shown in Table 6 and Table 7. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WzS1vJHIS-s", "reviewer": "Reviewer_YRge", "summary": "This paper proposed a new semantic-guided masking strategy for unsupervised vision transformer training. The paper claims that semantic parts of objects are the visual analogue of words, masking semantic parts from intra-part patterns to inter-part would help the vision transformer learn better representations. For such purpose, the paper first designs a self-supervised semantic part learning method that aims to generate semantic parts of images from the ImageNet dataset. Then, the proposed semantic-guided masking strategy performs the unsupervised model training. Most of the experiments were performed on the ViT-base backbone and showed improvements on unsupervised representation learning.", "questions": "1.For the proposed SemMAE, decreasing patch size largely improved its performance. Please explain the reason behind this and compare it to the opposite behavior in both SimMIM and the original MAE.\n\n2.Why six parts per image were selected in the paper? For different kinds of objects in the ImageNet dataset, what are the six parts corresponded to?", "limitations": "Yes, the authors adequately discussed the limitations of this paper. Meanwhile, potential negative societal impacts are not found yet.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n1.This paper is well organized. It raises an interesting and reasonable assumption, i.e., semantic parts in an image carry important semantic information and may benefit unsupervised masked training strategy. To explore the difference between language tokens and image tokens seems a meaningful topic and this paper seems to find a new way to define semantic tokens for vision transformers.\n\n2.If the ground-truth semantic part segmentations are given, the comparisons with existing MAE methods should be unfair. Thus, the paper designs a self-supervised semantic parts prediction model. We can see that the semantic part segmentation results would largely influence the masked learning results. It seems the proposed semantic part learning method works well in exploring parts' information that is contained in the representations extracted from a pre-trained transformer.\n\n3.The experimental settings are reasonable and related results are promising. Most claims in this paper can be supported by their experiments.\n\n[Weaknesses]\n1.Besides objects (or semantic parts), the background is an important and large part contained in images. This paper does not discuss how to deal with the background. If we regard the semantic parts as the visual analog of words, what is the background for? How should we treat it?\n\n2.I am wondering why this paper chooses the masking training strategy to validate its idea (i.e., semantic parts of objects are the visual analogue of words). Are there any other tasks related to this idea? In addition, what is the relationship between the idea and the masking strategy (i.e., masking parts from intra-part patterns to inter-part)? These questions were not well explained?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2L8gcfeIiTI", "reviewer": "Reviewer_NYYu", "summary": "In the submission, the authors hope to explore a more efficient/reasonable masking strategy for training masked autoencoders (MAEs). Their intuition is that current masking strategies (e.g., random masking) cannot utilize semantic information contained in images, as what have been done in masked language modeling. Therefore, they proposed that masking semantic parts in the image would help training MAEs, named Semantic-Guided Masking. Based on this idea, the authors devised (i) a semantic part learning strategy for predicting semantic parts from the image; (ii) a MAE masking curriculum called semantic-guided MAE for gradually masking semantic parts in the image. Finally, experiments show the proposed semantic-guided MAE bring significant improvements compared to existing masking strategies.", "questions": "1. In Line 147, I am wondering why the authors attempt to remove texture information, and the reason for choosing a large blur kernel. \n\n2. How do the authors decide the part segmentation results of stage-I being good enough? According to my understanding, for a fair comparison, no labels should be introduced.\n", "limitations": "Yes, related limitations have been discussed and there is no obvious negative societal impact.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. In my opinion, the idea of this paper is interesting. Compared to randomly masking images, masking semantic parts seems reasonable, and experiments show it indeed brings improvements. Meanwhile, the idea of recovering semantic parts of an image giving other parts (or information) is an important research topic of image understanding, this paper should bring some new thoughts to the community.\n\n2. Learning to predict semantic parts from images is a challenging task, especially under the unsupervised training setting. To execute the proposed semantic-guided MAE, the authors proposed a new semantic part learning strategy which is novel and useful. According to my understanding, the performance of following semantic-guided MAE is highly relying on semantic part segmentation results. It is impressive that using only a style-based generator and reconstruction loss can further refine the semantic part information extracted from a pre-trained ViT encoder.\n\n3. This paper is well-written. The motivation and idea of this paper are clear and are well-explained. Although visual transformer (or MAE) made great progress in the past year, it is still chasing language modeling. I am glad to see papers that explore the differences and connections between language tokens and image tokens.\n\nWeakness:\n\n1. The whole pipeline of the proposed method seems a little bit complicated. It first learns to predict part segmentations from a pre-trained iBOT, then retrains an MAE from the scratch. I am curious how about training a random MAE first, and then iteratively performing part learning and semantic-MAE learning. I understand the authors listing it in their future work, yet it still can be discussed in this paper.\n\n2. In table 4, `iBOT-initialized part' performs worse than the baseline model, the authors should have more discussion on it. How will the part learning influence the Semantic-MAE results?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept