MCMAE: Masked Convolution Meets Masked Autoencoders
Authors: Peng Gao
Teli Ma
Hongsheng Li
Ziyi Lin
Jifeng Dai
Yu Qiao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Masked auto-encoders
Convolution Neural Networks
Vision Transformer

Abstract: Vision Transformers (ViT) become widely-adopted architectures for various vision tasks. Masked auto-encoding for feature pretraining and multi-scale hybrid convolution-transformer architectures can further unleash the potentials of ViT, leading to state-of-the-art performances on image classification, detection and semantic segmentation. In this paper, our MCMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computational cost and pretraining-finetuning discrepancy. To tackle the issue, we adopt the masked convolution to prevent information leakage in the convolution blocks. A simple block-wise masking strategy is proposed to ensure computational efficiency. We also propose to more directly supervise the multi-scale features of the encoder to boost multi-scale features. Based on our pretrained MCMAE models, MCMAE-Base improves ImageNet-1K finetuning accuracy by 1.4% compared with MAE-Base. On object detection, MCMAE-Base finetuned for only 25 epochs surpasses MAE-Base fined-tuned for 100 epochs by 2.9% box AP and 2.2% mask AP respectively. Code and pretrained models are available at \url{https://github.com/Alpha-VL/ConvMAE}. 

MCMAE: Masked Convolution Meets Masked Autoencoders
Peng Gao'
Teli Ma Hongsheng Li Ziyi Lin? Jifeng Dai Yu Qiao'
1 Shanghai AI Laboratory, Shanghai, China
2 MMLab, CUHK
Sense'Time Research
Abstract
Vision Transformers (VIT) become widely-adopted architectures for various vision tasks. Masked auto-encoding [2, 1, 28, 55] for feature pretraining and multi- scale hybrid convolution trrtioonntrasffmmer architectures [12, 21, 49, 34, 57] can further unleash the potentials of ViT, leading to oftte-of-thh-art performances on image classification, detection and semantic segmentation. In this paper, MCMAE framework demonstrates that multi-scale hybrid convolution-transoonnn discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computational cost and cetrraniing-fietuning discrepancy. To tackle the issue, adopt the masked convolution to prevent information leakage in the convolution blocks. A simple block-wise masking strategy is proposed ensure computational efficiency, We also propose to more directly supervise the multi-scale features of the encoder boost multi-scale features. MCMAE-Basd Base improves ImageNet- 1K finetuning accuracy by 1.4% compared with
can learn
On object detection, MCMAE-Bass finetuned for only 25 epochs surpasses MAE-Base fined-tuned for 100 epochs by 2.9% Abbox and 2.2% A pmask respectively. Code and pretrained models are available https: //gi gighu........ com/ haop/lpha-aaa VL/ConvMAE.
1 Introduction
Self supervuee learning frameworks, such as DINO [6], MOCO-V3 [10], MAE [28], unleash the potential of Vision Transformers (ViT) and achieve high performance on various downstream vision tasks [33, 30, 58]. Among them, Mask Autoencoders (MAE) [28] demonstrate superior learning ability and scalability. Motivated by BERT [15, 46, 4] in natural language processing, MAE utilizes an asymmetric encoder and decoder architecture, in which masked tokens of the encoder are reconstructed by the decoder. Experiments show that MAE can learn discriminative and scalable representations from ImageNet-IK [14] without relying large-scale datasets, such as ImageNet-22K.
Local inductive bias [49, 21, 34, 12, 19, 57] and hierarchical representations [42, 53] are explored for boosting the performance of VIT. The combination local convolution and global transformer operations leads to clear improvements on image classification [33], object detection [30], and semantic segmentation [58]. contrast to MAE [28], well pelll-peffrmingg multi-scale backbones built upon local and global operations are mainly trained in supervised manner. A natural question is whether multi-scale backbone with local and global operations, which show promising performance on supervised learning be exploited to enhance the masked auto-encoding paradigm [28, 15, 2, 65].
In this paper, simple and effective self-supervised learning framework, dubbed MCMAE, is architectures
proposed to train scalable representations by introducing hybrid and masked convolution into the masked nuto-enccders. Although the modifications the original MAE are minimal, MCMAE shows great success pretraining visual representations for boosting the performances of various tasks.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Different from MAE [28], the encoder of MCMAE progressively abstracts the input image into multi-scale token embedding. while the decoder reconstructs the pixels corresponding to masked tokens. For high-resolution token embedding at early stages, convolutions blocks are adopted to encode local content. For low-resolution token embedding at late stage, transformer blocks are used to aggregate global context. The encoder therefore obtains both local and global FOV at different stages and generates discriminative multi-scale features. Note that the MCMAE encoder is partly motivated by the strong hybrid convolution and transformer backbones, including Co-AtNet [12], Early Convolution [57]. Container [21] and Uniformer [34]. However, previous hybrid convolution- transformer networks were either not explored for masked auto-encoding [21, 34, 20] or show very similar performance to MAE [52, 59]. Instead of designing novel architectures, we focus on making basic hybrid convolution tranttooomtta architectures work for mask auto-encoding and conduct extensive experiments to demonstrate its effectiveness on various downstream tasks.
The efficient and effective training of MCMAE is enabled by block-wise masking strategy with masked convolution [60, 25, 31, 48, 24, 40]. The masking strategy adopted in current mask- autoencoding frameworks, such as BEiT [2], MAE [28]. SimMIM [59], cannot be naively used for MCMAE as all tokens need to be kept in the later transformer stages. This leads to unaffordable computation cost for pretraining large and huge models, losing MAE's efficiency advantage of omitting masked tokens transformer encoder. In addition, directly pretraining with the convolution- transformer encoder causes pretraing veeraing-finetuning discrepancy only visible tokens processed during finetuning stages.
To tackle the issues, we focus on designing hybrid wwwvollttoo-transfoonn architectures suitable for mask auto-encoding. Specifically, our MCMAE adopts a block-wise masking strategy to first obtain a mask for the late stage in transformer and then progressively upsamples the mask to larger resolutions in early convolutional stages. In this way, tokens processed by late stages can be completely separated into masked tokens and visible tokens and inherit the computation efficiency of MAE. To prevent information leakage, the convolution blocks at early stages are equipped with masked convolutions, which avoid mixing up features of masked and visible regions in late stages to ensue the training effectiveness. Masked convolution has been well explored in sparse feature extraction [25, 48, 24, 60] and image inpainting [40]. It can be naturally integrated into the hybrid convolution-transfooo architecture to enable masked auto-encoding.
Our MCMAE can naturally provide multi-scale features for object detection and semantic segmenta- tion, which are required by modern detection [30] and segmentation frameworks [58]. Multi-scale features from the pretrained MCMAE can significantly improve the performances of object detection and semantic segmentation compared with MAE. MCMAE with masked-based autoencoding can even surpass the super-suee pretraining of Swin and MViT [42, 36].
In summary, our contributions can be summarized below: (1) We present the strong and efficient self- supervised framework MCMAE, which is easy to implement but show outstanding performances on different tasks. (2) The proposed MCMAE naturally generates hierarchical representations and exhibit promising performances on object detection. (3) MCMAF-Baa improves the ImageNet finetuning accuracy by 1. 1.4% compared with MAF-Base. On COCO 2017 with Mask-RCNN, MCMAE-BA achieves 53.2% Pbox and 47. % A Pimask with a 25-epoch training schedule while MAE-Base attains 50.3% A Abbox and 44.9 44.9% A Pmask with 100 training epochs. On ADE20K with UperNet, MCMAE-Base surpasses MAE-Base by 3.6 mloU (48.1 1 1% vs. 51.7%).
2
Approach
2.1 A Brief Revisit of MAE
Masked Autoencoders (MAE) [28] is self-supervised method for pretraining ViT by reconstructing masked RGB patches from visible patches. Although MAE has simple design, it has been proven to be a strong and scalable pretraining framework for learning visual presentations. MAE consists of transformer-based encoder and decoder, where only visible patches are fed into the encoder and learnable mask tokens are processed by the decoder for image reconstruction to learn visual representations. As the encoder only needs to process small portion of visible tokens, it alleviates the scalability problem to pretrain large vision models.
2.2 MCMAE
MCMAE is simple and effective derivative of the popular MAR [28] with minimal but effective modifications on the encoder design and the masking strategy. The goal of MCMAE is to learn
2

---

EXWW'3
W/14
WJB
Blodewwse Masking
UpSenople
UnSammlle
BV4
B/G
RV16
Palzl Livlokttng
Potch Limbekiingg
Coneclation
Cunnciltioon
Moddod
Bluck
Mekoei
blook
Fllok
Block
Stage
Stags2
Stage3
HIEEW/EHC
H/0KW/0442
Encoder
(НЛ65*//6)
Decoder
HBBK
Maeked Convolution
Reetan Endooklings
Mask
SaidkCozr Fllaton
Mack
Mink
Value
Multi-Scale Fusion
Figure 1: The pipeline of proposed MCMAE which consists of a hybrid convolutiontransffommet encoder, block-wise masking strategy with masked convolution and multi-scale decoder.
discriminative multi-scale visual representations and to prevent pmetranning-fnnttunig discrepancy when applies MAE [28] on convolution cratttionnttanffommer networks.
Directly applying the original masking strategy on the feature maps of the convolation-transffrmee encoder would make transformer layers keeping all tokens during the pretraining. jeopardizing the training efficiency. We introduce hierarchical masking strategy coupled with masked convolution for the convolution stages to ensure only a small number visible tokens are input into the transformer layers. The overall pipeline of MCMAE is shown in Figure 1.
The Hybrid Convolution-transffrmeet Encoder. There are previous strong hybrid convolution- transformer architectures, such as Co-AtNet [12], Container [21]. BoTNet [49], Uniformer [34] and Early Conv [57]. Without using such complicated architectures, we show that a simple design of multi-scale convolution-transffomner encoder can already learn powerful representations for various downstream tasks. As shown in Figure 1, our encoder consists of 3 stages with output spatial resolutions of 4
H W respectively, where H W is the input image resolution. The first two convolutional stages use convolution blocks to transform the inputs to token embeddings
E1 E # xC: and Eg E R W. xC2 Our convolution blocks follow the design principle of the transformer block by only replacing the self-attention operation with the 5 depthwise convolution The third transformer stage uses commonly used self-attention blocks to obtain token embeddings
Es E R 1 W. xCa, Between every stage, stride-2 convolutions are used to downsample the tokens to half its previous spatial resolution. The local convolutions in stages and have relatively small field-of-view, the transformer blocks in stage 3 aggregate and fuse features from the coarse- grained features and extend the field of view to the whole image. Different from other ViTs, such as CPT [1]], Container [21], Uniformer [34], CMT [26]. Swin [42], which replace absolute position embedding [42] with relative position embedding or zero zero-padded convolution at the inputs of the first stage [11, 21, 34, 26], find that adding absolute position embeddings the inputs of the transformer stage-3 leads to the optimal performance. The class token is also removed from our encoder which shows limited influence.
Block-wise Masking with Masked Convolutions. Mask auto-encoders, such as MAE [28] and BEiT [2], adopt random mask on the input tokens. However, the same strategy cannot be directly W. feature applied to our MCMAE encoder. Uniformly masking stage-1 input tokens from the 4 x 4 maps would cause all tokens of stage-3 have partially visible information and requires keeping all stage-3 tokens. Therefore, we propose to first generate the random mask to mask out p% (e.g., 75%)

---

Slagel
Starg
Slage:
Transformer Block
Convolutton
Convolution
Down Sample
Block
Blockk
Local Global ttention
X2
X11
E3
E4
Figure 2: Overview of finetuning MCMAE for object detection and semantic segmentation. The intermediate features of different stages multi-scale inputs for an FPN [38] module.
of stage-3 input tokens and upsample the mask by 2 times and 4 times to obtain the corresponding block-wise masks for masking stage-2 and stage-1 inputs, respectively. The corresponding masked tokens in the three stages are dropped in the encoding process and are reconstructed by the decoder for feature learning. In this way, MCMAE only needs keep as few as 25% tokens the coneuminn transformer blocks for training and the efficiency of MCMAE is not compromised.
However, the 5 depthwise convolutions in the first stages naturally lead receptive fields larger than the masked patches and cause information leakage when reconstructing masked tokens. To avoid such information leakage and ensure the quality of pretraining, we adopt masked convolution [25, 48] in the first two stages, so that the masked regions would never be involved in the encoding process. The use of masked convolution is crucial to the superior performance of MCMAE and the pretraining-tesse -tetting discrepancy is prevented by removing partially masked tokens from stage.
The Multi-scale Decoder and Loss. The decoder of the original MAE [28] takes input both visible tokens Eg from the encoder and the mask tokens [Mask], and transform them in stacked transformer blocks for image reconstruction. Our MCMAE encoder obtains multi-scale features E₁, Eg. Eg,, captures both fine- and coarse-grained image information. To better supervise the pretraining of such multi-grained representations, we downsample E₁ and Eg to the same size of E3 with stride-4 and stride-2 convolutions and fuse multi-grained tokens via a linear layer to obtain visible tokens Eg.
E. Linear(StrideCConv((S 4) + StrideConv(E2.. Es),
(I)
where StrideConv((- k) represents stride-k convolution. The multi-scale decoder is illustrated in the bottom-left part of Figure 1. The same losses from MAE [28] are used for reconstructing masked image patches and only the reconstruction of masked patches considered in the objective function.
2.3 MCMAF for Object Detection and Semantic Segmentation
After pretraining, the proposed MCMAE can naturally generate multi-scale feature maps, which can be processed by existing object detection and semantic segmentation heads.
As shown in Figure 2, finetune MCMAE for object detection, FA feature map of 1 1/32 input resolution is first obtained by 2 x max pooling Eg. However, as the MCMAE stage-3 has 11 global self-attention layers (in our MCMAE-base model) with excessive computational cost, we follow Benchmarking ViT [37] to replace all but 1st, 4th, 7th, 1 11th global self-attention layers in stage-3 windows. The modified local self-attention layers are still initialized by the pretrained global self-attention layers. A global relative position bias [2, 42, 28, 37] shared between global transformer blocks. Similarly, local relative position bias [2, 42, 28, 37] is shared by local transformer blocks. In this way, the heavy computational and GPU memory costs of the stage-3 are much mitigated. The multi-scale features E,, Ez, Eg, F, are then fed into the MaskRCNN [30] head for object detection. To finetune MCMAE semantic segmentation, its stage-3 architecture is kept as the images in segmentation datasets have relatively smaller resolutions. The multi-scale features are feed into UperNet [58].
thifted-windors local self-attention layers [42] with alternatively shifted

---

2.4 MCMAE for Video Understanding
Attention based models [54, 63, 50, 3, 42] have demonstrated superior performance on video un- derstanding. Our MCMAE can also be extended to serve as a strong video pretraining framework, dubbbed as VideoMCMAE, with simple modifications. Specifically, VideoMCMAE replaces image patch embedding with cube embedding, after which stage I and stage 2 perform local spatial-temporal feature fussin with masked 3D convolutions. Stage 3 still adopts stacked transformer blocks for spatial-temporal fusion. The spatial position embedding extended to spatial-temporal embedding. Similar to the multi-scale decoder proposed in Section 2.2, outputs from stages 1, 2 and 3 are fused before feeding into about VideoMCMAE pretraining are in appendix B. Note that unlike previous approaches, which initialize models pretrained on images [34, 36, 7]. our VideoMCMAE pretrained from scratch on pure video datasets.
temporal transformer decoder for masked pixel reconstruction. Details
3 Experiments
To validate our proposed MCMAE, we conduct experiments of image classification on ImageNet- IK [14] dataset. The pretrained MCMAE is also extensively tested on object detection and semantic segmentation. By default, we report performance of our the MCMAE- base model with multi-scale decoder, which has similar parameters and FLOPs the MAE-base.
3.1 ImageNet-1K Pretraining and Finetuning
Experimental Setup. ImageNet- 1K [14] consists of 1.3M images of 1 categories for image classification and is split to the training and validation sets. We pretrain our MCMAE on ImageNet- 1K training set. By default, we fix the mask ratio to 25% following the original MAE [28]. The decoder is designed to have 8 transformer layers with 512 feature dimensions and 12 attention heads. We adopt a 1600-epoch cosine learning rate schedule with the first 40 epochs for warming up. The AdamW W optimizer is utilized with a base learning rate of 1.5 x 10-4, a weight decay of 0.05 and batch size of 1024. Random cropping is employed as data augmentation during pretraining. After pretraining, the MCMAE encoder is used for supervised finetuning on ImageNet-I1 1 training set for 100 epochs using the cosine learning rate schedule. We follow the default finetuning parameters of the original MAE [28] except for the layer-wise learning-rate decay parameters (0.65, 0.75, 0.85). For finetuning, we report the classification accuracy on the ImageNet validation set of the finetuned and pretrained (linear probe) MCMAE encoders.
Results on ImageNet-1K Finetuning. We report the accuracy of MCMAE on Table and conduct
Methods
Backbone
Params. (M) Supervision Encoder P-Epochs
FT
(%)
LIN (%)
BETT [2]
ViT-B
88
DALLE
100%
300
83.0
37.6
MAE [28]
VIT-B
88
RGB
25%
1600
83.6
67.8
SimMIM [59]
Swin-B
88
RGB
100%
800
84.0
N/A
MaskFeat [55]
VIT-B
88
HOG
100%
300
83.6
N/A
data2vec [1]
ViT-B
88
Momentum
100%
800
84.2
N/A
MCMAE
CVIT-B
88
RGB 25%
1600
85.0
70.9
Table 1: Comparison with state-of-the art mask auto-encoding schemes with similar model size. FT and LIN denotes ImageNet-IK finetuning and linear probe accuracy respectively.
comparisons with state-of-the-ar mask autoencoding methods. BEiT [2] pretrains ViT-B through the prediction of visual tokens tokenized by the DALL--F encoder. With 300-epoch pretraining, BEiT can reach a finetuning accuracy of 83.0% and linear-probe accuracy of 37.6%. Compared with BETT, MCMAF processes only 25% visible tokens in the encoder and has lightweight decoder for reconstruction. MCMAE can surpass its finetuning accuracy and linear-probe accuracy by large margins (+2.0% Compared with the original MAE pretrained for 1,600 epochs, our MCMAE surpasses its finetuning accuracy by 1.4% with same number pretraining epochs. SimMIM [59] adopts Swin-B [42] to generate hierarchical representations. MCMAE achieves improvement over its finetuning accuracy (+I (+1.0%). MaskFeat [55] uses HOG [13] features as prediction targets. Data2vee [1] incorporates momentum encoder [29] to generate predictions in an online manner. Both MaskFeat and Data2vec have higher computational costs than our MCMAE. They can be considered complementary directions for improving the mask auto-encoding scheme.
3.2 Object Detection
Experimental Setup. COCO dataset [39] has been widely adopted for benchmarking object detection frameworks. MCsN-RCNN [30] is one of the most popular frameworks for object detection. We
5

---

Methods
Pretraining
P-Epochs
F-Epoohs
A Aboox
A pmask
Params (M) FLOPs (T)
Benmarking [37]
INTK w/o labels
1600
100
50.3
44.9
118
0.9
ViTDet [35]
INIK w/o labels
1600
100
51.2
45.5
111
0.8
MIMDET [20]
IN1K w/o labels
1600
36
51.5
46.0
127
1.1
Swin+ [42]
INTK w/ labels
300
36
49.2
43.5
107
0.7
MViTv2 [36]
K w/ labels
300
36
51.0
45.7
71
0.6
MCMAE
INIK w/o labels
1600
25
53.2
47.1
104
0.9
Table 2: Performances of different pretrained backbones on object detection with
[30].
Models
Pretrain Data
P-Epochs mloU
Params (M) FLOPs (T)
DeïT-B [51]
INIK w/ labels
300
45.6
163
0.6
Swin-B [42]
INIK w/ labels
300
48.1
121
0.3
MoCo V3 [29]
INTK
300
47.3
163
0.6
DINO [6]
INIK
400
47.2
163
0.6
BETT [2]
INIK+DALLE
1600
47.1
163
0.6
PeCo [17]
INIK
300
46.7
163
0.6
CAE [9]
INIK+DALLE
800
48.8
163
0.6
MAE [28]
INIK
1600
48.1
163
0.6
MCMAE
INIK
1600
51.7
153
0.6
Table 3: Comparison with different pretrained backbones on ADE20k with UperNet.
employ the encoder of pretraied MCMAE as the backbone for MasK-RCNN. We finetune Mask- RCNN on COCO train2017 7 split and report A A and A pmask on val2017 split. We follow most setups of Benchmarking VIT [37]. We report the model performance on object detection under 25 epochs cosine schedule with base learning rate of 8.0 x 10-5, a weight decay of 0.1.
Results on COCO 2017. We compare the performances of state-of-the-art visual backbones in Table 2. Benchmarking ViT [37] extensively explores using plain ViT with Mask-RCNN. Compared with Benchmarkin ViT [37] finetuned for 100 epochs COCO, MCMAE can significantly improve Apboz and Apmosk by 2.9% and 2.2% with 25 finetuning epochs. ViTDet [35] improves Bench-
marking ViT [37] by introducing simple feature pyramid module. MIMDet [20] adds randomly initialized convolution stem and randomly drops input tokens to increase the training efficiency of Mask-RCNN [30]. Note that MIMDet [20] introduces extra parameters due the incorporation of
MAE decoder. Compared with improved version of Benchmarking ViT [37], such as ViTDet [35] and MIMDet [20], MCMAE surpass them by 2.0% and 1.7% with a shorter finetuning schedule (25 epochs vs 100/36 epochs), fewer parameters (104M vs 111M//27M) and similar FLOPs (0.9T). This validates the effectiveness of our proposed MCMAE framework. Swin [42] and MViTv2 [36] are state-of-the-art hierarchical visual backbones. Although adopting a simpler multi-stage architecture, MCMAE outperforms Swin annd M by 4.0% and 1..2%%1.445 in terms A / pmask Note that Swin [42] and MVIT [36] v2 are pretrained for 300 epochs with 100% tokens in supervised manner while MCMAE is only pretrained using masked autoencoder with 25% visible tokens, which is efficient for object detection.
3.3 Semantic Segmentation
Experimental Setup. ADE20K [64] is a widely-used semantic segmentation dataset which contains 25,562 images of 150 fine une-grained categories. The dataset split into training, validation, and testing sets. We leverage the UperNet [58], a hierarchical segmentation network head to compare MCMAE with other backbones. Our MCMAE with UperNet [58] is finetuned on ADE20K training set and tested on validation split. In the training phase, the backbone is initialized with the weights pretrained for 1600 epochs on ImageNet- IK and other modules are initialized with Xavier initialization. We adopt a 16k-iteration polynomial learning rate schedule with the first 1500 iterations for warming up. The Adam W [44] optimizer is adopted with an initial learning rate of 10 -4 a weight decay of 0.05 and batch size of 16. We follow the default finetuning configurations of MAE on ADE20K except for the feature dimensions for the decoder head and the layer- wise learning rate decay is set as 0.75.
Results on ADE-20K. We report the Mean Intersection over Union (mloU) performance of MCMAE
and other state-of-the-arn backbones Table 3. With the 300-epoch pretraining, MoCo V3 [10] can reach 47.2 mloU when finetuned on semantic segmentation. BEIT [2]. PeCo [17] and CAE [9]
utilize discrete VAE as visual tokenizer to create the targets. Both BETT and CAE adopt the
DALL-E [47] codebook trained on 250M images, while PeCo trains codebook only on ImageNet-

---

82.7
20.7
Accusaro 150 Semeninggeeee
90.3
Top AcCURC (5) dinc
81.7
70
66.9
96.0
81.3
169.3
69-
79.4.
67.2
-67.6
ViceMMA WiceMMMAE
VideONAAE
ViGeMMCHA
78-
77.8g
multi-erate
666
966.1
mull-scate fuston
8000
2400
#pocts, scwee)
apactio (lng sooie]
Figure 3: Finetuning accuracy Kinetics-400 and Comething-SSmmething
1K. Compared with these methods, our 1600-epoch pretrained MCMAE achieves much higher performance (51.7%). Compared with MAE pretrained 1600 epochs, our MCMAE outperforms by 3.6% mloU, demonstrating the hierarchical representations of MCMAE largely diminishes the transfer gap between pretrained backbones and downstream networks.
3.4 Video Understanding
Experimental Setup. To validate the video understanding ability of VideoMCMAE, pretrain on Kinetics-400 (K400) [32] and Something-somettiee
V2 (SSV2) [23] independently and report the finetuning accuracy K400 and SSV2. The video pretraining and finetuning protocol closely follow the image protocol explained in Section 3.1. For SSV2, we finetune 50 epochs and turn off the flip augmentation. Unlike random masking in image pretraining, tube masking [50] with 90% mask ratio proposed by VideoMAE is adopted as the default masking strategy. For testing, we use the same number of views VideoMAE for fair comparison, i.e., spatial temporal views for Kinetics-400 and 3 spatial x temporal views for Something- Something-Something Something-v? All results are reported using only the finetuning dataset without extra image video data.
Results on K400 and SSV2. We compare the finetuned accuracy on K400 and SSV2 with Video- MAE [50] for different pretraining epochs. As shown in Figure 3, VideoMCMAE outperforms VidcoMAE by clear margin at 200 and 800 pretraining epochs. Notably, on Kinetics-400, VideoM- CMAE pretrained for 200 epochs slightly outperforms VideoMAE at 1600 epochs, and 800-epoch pretrained VideoMCMAE with multi-scale decoder outperforms VideoMAE at 1600 epochs by more than 2.9%. On Something Soaeetiinn-Somthhingggge
800-epoch model with multi-scale decoder slightly outperforms VideoMAE at 2400 epochs, which indicates 3x reduction in pretraining epochs.
ImageNet
COCO
ADE20K
Pretrain Epochs
FT
LIN
Abbor
A Amask
mloU
200
84.1
62.5
50.2
44.8
48.1
400
84.4
66.9
51.4
45.7
49.5
800
84.6
68.4
52.0
46.3
50.2
1600
84.6
69.4
52.5
46.5
50.7
Table 4: The influence of increasing pretraining epochs on various downstream tasks.
3.5
Ablation Study of MCMAE
conduct extensive ablation studies on MCMAE to analyze different components of MCMAE (see Table 5 and 6). By default, we report the performance of MCMAE without multi-scale decoder during ablation studies.
Pretraining epochs. For MAE, longer pretraining epochs significantly improve the learned representations learned. We pretrain MCMAE-Base with 200, 400, 800 and 1600 epochs to test the influences MCMAE. We report the ImageNet-1K finetuning (FT) and linear probe (LIN) accuracies, Abbox and A pmusk of COCO, mloU of ADE20K on Table 4. We observe improved performances most downstream tasks with longer pretraining epochs.
Input token random masking, As shown in Table 5, we replace the proposed block-wise mask strategy with MAE's input token random masking- Compared with our MCMAB-base, the ImageNet- IK finetuning accuracy drops from 84.6% to 84.2% which validates that the proposed simple

---

block-wis masking strategy can alleviate pretraining weerrännin-fmmttuinng discrepancy. Input-token random masking results in all tokens in stage-3 being processed by computationally intensive transformer blocks and causes FLOPs to increase by 1.7x.
Influence of masked convolution. Masked Convolution can prevent information leakage due to the overlapping window in convolution. Removing masked convolution decreases the ImageNet- 1K finetuning accuracy from 84.6% to 81.5% which demonstrates that information leakage in convolution stages hinders feature learning in mask autoencoding.
Convolution kernel sizes in stages 1 and 2. Enlarging the kernel size in convolution is shown to be effective for semantic segmentation [45] and visual backbone designs [16, 43]. We also with enlarging the 5 x 5 kernel size in stages I and to 7 and 9 9. As shown by Table 5, we observe that larger kernel sizes barely influence the performance of MCMAE on ImageNet- IK accuracy. We hypothesize that the transformer blocks in stage-3 already provide global FOV which can cancel out the gains introduced from large kernels.
Multi-scale Decoder. In Table 6, incorporate multi-scale decoder into MCMAE-base and pretrain for 200 and 1600 epochs. Comparared with MCMAE pretrained 200 epochs, multi-scale decoder improve classification accuracy, detection A detection A Apmaak and segmentation mioU by 0.3%, 0.6% 0.6% and 0.4%, respectively. Given longer pretraining. multi-scale decoder can improve classification accuracy, linear probe accuracy, detection A pbox, detection A pmask and segmentation mIoU by 0.4%, 1.6%, 0.7%, 0.6%. 1.0%, respectively. This indicates that fusing multi- multi-grained tokens for mask reconstruction can lead to powerful representations. We will explore more advanced multi-scale decoder architectures such as UNet in the future.
84
Acourar
50-
Momp
45-
MAE 36CMAE
46
Benchmarking VIT
MCMAE
45
25
45
58
65
25
45
05
25
45
d5 75 d5
Finetuiin Eoochs
Lincar Probing Epocis
Ricturiin Epochs
ImageNet Finetuning
(b) ImageNet Linear Probing
(c) COCO Detection
Figure 4: Convergence of MAE and MCMAE on various tasks.
Masked Conv
Block Masking
5 5
7
9
P-Epochs
FT (%)
FLOPs
Conv
Conv
Conv
84.6
1x
X
X
84.2
1.7x
X
81.5
1x
800
/
X
84.5
0.997 X
V
X
84.4
1.003 x
84.6
1.007
Table 5: Ablation study the influence of the masked conv, block masking, kernel size in stages 1 and of MCMAE on ImageNet- IK finetuning accuracy.
P-Epochs
Method
FT (%)
LIN (%)
Apboz
A Ammask
mloU
MCMAE-
84.1
N/A
50.2
44.8
48.1
200
w/ multi-scale decoder
84.4
N/A
50.8
45.4
48.5
MCMAE-BR --ase
84.6
69.4
52.5
46.5
50.7
1600
w/ multi-scale decoder
85.0
70.9
53.2
47.1
51.7
Table 6: For a base MCMAE pretrained for 200 epochs and 1600 epochs, we ablate the multi-scale decoder on ImageNet finetuning and object detection on COCO.
Convergence speed. We compare the convergence of MCMAE and MAE terms of ImageNet-1K finetuning, linear probing accuracy and COCO Abbo in Figure 4. For fair comparison, MCMAE

---

and MAE are both pretrained for 1600 epochs. MCMAE not only attains strong final results but also significantly increases convergence speed on various tasks, Specifically, MCMAE can surpass the final performance of MAF at 58 epochs on ImageNet-1k IK finetuning. On COCO object detection, MCMAF surpasses MAF 16 epochs, indicating 6.6x faster convergence speed.
Related Work
Vision Transformer. Vision Transformer (ViT) [18. 5] achieved oonte-of-the-eee art results on various vision tasks. To increase the convergence speed and improve accuracy, well-explored locality
inductive bias have been reintroduced into vision transformer [66, 22, 62, 41, 27, 61, 51, 19, 56, 26], among which, hybrid architecture of convolution and transformer design [49, 57, 12, 21, 34] can achieve state-of-the-art performance of a wide range of tasks. Our MCMAF is highly motivated by the hybrid architecture design [21, 34, 12, 57] in vision backbones. Instead of designing new architectures, MCMAE aim to unleash the powerful representation induced by hybrid architectures through MAE-style pretraining with several insightful modifications.
Self-supervised Representation Learning. Contrastive learning [8, 29, 6. 10] learn invariances by comparing augmented views of un-labeled images. Recently Mask-Autoencodingg
motivated by BERT [15] raised be promising methodology. Autoencutoni can learn strong representation through masked patch reconstruction with simple data augmentation. BEiT [2] introduced Mask- Autoencoding into Vision Community. MAF [28] introduced an asymmetric encoder and decoder architecture where masked tokens is skipped in computation-heay encoder and only pass all tokens through a light-weight decoder. iBoT [65] and Data2Vec [1], PeCo [17] and MaskFeat [55] explore different reconstruction targets. Different from previous improvements of Mask-autoencoding.
MCMAE introduce hierarchical representations architectures into MAE.
Conclusion
We propose simple self-supervised learning framework named as MCMAE which demonstrate the hybrid local-global blocks [21, 34, 26, 19, 57, 49] can boost the performance of MAE [28] to generate discriminative multi-scale features [38, 53, 42]. The computational efficiency and low -ineiinngnnnninggggg gap of original MAE can be well maintained under our MCMAE. MCMAE exhibits significantly improved performances on various vision tasks and can be easily implemented. We will study combining improved reconstruction targets with MCMAE in the future.
Negative societal impact: We do not foresee nagative social impact from the proposed work.
Acknowledgement: This work was supported in part by the National Natural Science Foundation of China (Grant No. 62206272) and Shanghai Committee of Science and Technology (Grant No.
21001100100).
References
[1] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. DataZvec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint
arXiv:200..03555 2022. 1, 5, 9
[2] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint
urXiv:2106.0820 08254. 2021. 1, 2, 5, 6,
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding, arXiv preprint arXiv:2002.0509 2(3):44 2021. 5
[4] Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leareer..
Advances in neural information processing systems, 13:1877-1901, 2020. 1
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9650-9660, 2021. 1, 6,
[7] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2299-6308,
2017. 5
[8] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607 PMLR, 2020. 9

---

[9] Xiaokang Chen, Mingyu Ding. Xiaodi Wang, Ying Xin, Shentong Mo. Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03026 2022. 6
[10] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages
9640-9649, 2021. 1.6,9
[11] XiangXiang Chu. Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.
Conditional positional encodings for vision transformers. arXiv preprint or arXiv:2 102022008880 10882, 2021. 3
[12] Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34, 2021. 1, 2,
[13] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR 005) volume 1. pages 886-893. Icee, 2005. 5
[14] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Icce, 2009. 1,5
[15] Jacob Devlin, Ming-Was Mein-WeiCCC Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec-
tional transformers for language understanding. arXiv preprint 18X0vv:81000480000 04,000 2018. 1.9
[16] Xiaohan Ding. Xiangyu Zhang. Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in cans. arXiv preprint arXiv:2203 067177 2022. 8 (17] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:2111.122
2021. 6,9
[18] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020. 9 [19] Stéphane Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pages 2286-2296. PMLR, 2021. 1,9
Convit:
[20] Yuxin Fang, Shusheng Yang, Shijie Wang. Yixiao Ge, Ying Shan, and Xinggang Wang. Unleashing vanilla vision transformer with masked image modeling for object detection. arXiv preprint arXiv:2204.02964
2022. 2, 6
[21] Peng Gao, Jiasen Lu, Hongsheng Li. Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context
aggregation network. arXiv preprint arXi::2/06.01401, 2021. 1, 3,
[22] Peng Gao, Minghang Zheng. Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3621-3630, 2021. 9
[23] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video datahase for learning and evaluating visual common sense. In Proceedings of the IEEE international conference computer vision, pages 3842-5850. 2017. 7
[24] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9224-9232, 2018. 2
[25] Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. arXiv
preprint arx: (ee61000600001 01307, 2017. 2,4
[26] Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Ychui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolu- tional neural networks meet vision transformers. arXiv preprint
2021. 3,9 [27] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer.
Advances in Neural Information Processing Systems, 34, 2021. 9
[28] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders
are scalable vision learners. arXiv preprint ar arxiv: (tt:://1.00 1106377, 2021. 1, 3, 4, 5, 6, 9, 12
[29] Kaiming He. Haoqi Fan. Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020. 5, 6,
[30] Kaiming He. Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE
international conference on computer vision, pages 2961-2969. 2017. 2, 4, 5, 6, 12
[31] Ajay Jain, Pieter Abbeel, and Deepak Pathak. Locally masked convolution for autoregressive models. In Conference on Uncertainty in Artificial Intelligence, pages 1358-1367. PMI.R 2020.
[32] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vijaynnarasimhan
Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv
preprint arXiv: 705, /05.060000005000 2017. 7
[33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012. 1
[34] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint
arXiv:2201 2022. 1, 2,2,3,55 5,9
10

---

[35] Yanghao Li, Hanzi Mao. Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection, arXiv preprint arXiv:2203. 22522222...62.7.... 2022.
[36] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong. Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint
12.05:211..01526, 2021. 5,
[37] Yanghao Li. Saining Xie, Xinlei Chen. Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking
detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11225 2021. 4, 6 [38] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21 2117-2125, 2017. 4,
[39] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. S 2014. 5
[40] Guilin Liu, Fitsum A Reda. Kevin J Shih. Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European conference on computer vision (ECCV), pages 85- 155100,200 2018. 2
[41] Shilong Liu, Feng Li, Hao Zhang. Xiao Yang. Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint 2201 2022. 9
[42] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference
Computer Vision, pages 10012-10022, 2021. 1, 2, 3, 4,5, 6, 9
[43] Zhuang Liu, Hanzi Mao. Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.0 03344, 2022. 8
[44] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[45] Chao Peng. Xiangyu Zhang, Gang Yu. Guiming Luo, and Jian Sun. Large kemel matters matters-improve semantic segmentation by global convolutional network. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4353-4361, 2017. 8
[46] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019. 1
[47] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh. Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-iman -ot-at-- generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021. 6
[48] Mengye Ren. Andrei Pokrovsky, Bin Yang, and Raquel Urtasun. Sbnet: Sparse blocks network for fast inference. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8711-8720, 2018. 2.4
[49] Aravind Srinivas. Tsung-Yi Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE/CVF conference on computer
vision and pattern recognition, pages 16519-16529, 2021. 3,
[50] Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203, /2602, 2022. 5,7
[51] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021. 6.9
[52] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. arXiv preprint arXiv:2203.09700 2022. 2
[53] Wenhai Wang. Enze Xie, Xiang Li. Deng- Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 568-578, 2021. 1.9
(54) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
[55] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuu Wu, Alan Yuille. and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training, arXiv preprint arXiv:2112.0911 2021. 5. [56] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai. Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 22-31, 2021.
[57] Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. Early convolutions help transformers see better. Advances in Neural Information Processing Systems, 34, 2021. 1, 2, 3, 9
[58] Tete Xiao, Yingcheng Liu, Bolei Zhou. Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418-434,
2018. 1, 2. 4, 6
[59] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:21 111. 1.0886. 2021. 2, 5 [60] Zhenda Xie, Zheng Zhang, Xizhou Zhu, Gao Huang, and Stephen Lin. Spatially adaptive inference with stochastic feature sampling and interpolation. In European conference on computer vision, pages 531-548. Springer, 2020.
11

---

[61] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi. Zi-Hang Jiang, Francis EH Tay. Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021. 9 Gongjie Zhang. Zhipeng Luo, Yingehen Yu, Kaiwen Cui, and Shijian Lu. Accelerating detr convergence via semantic-aligned matching. arXiv preprint arXiv:2203.066 00883, 2022.
[62]
[63] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European conference on computer vision (ECCV), pages 803-818, 2018. Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision,
[64]
127(3))302-321. 2019. 6
[65] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie. Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 1XXv:2211.078333 2021. 1.9
[66]
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 2020. 2020. 9
6 Appendix
Architecture Details of MCMAE Encoder. The details of our hybrid convolution-transfomoe encoder is explained below. Given an input image I xx stage 1 of MCMAE encoder generates a high-resolution token embeddings E, E x 4 using non-overlapping 4 x 4 strided convolution firstly. Then E₁ is feed into stacked convolutional blocks which is repeated L1 times, where L1 stands for the number of layers in stage 1. Similar as stage 1, stage 2 further downsamples feature map into token embeddings E2 E RC2x x using non-overlapping x 2 strided convolution. E2 is processed by L2 layers of convolutional blocks again. After local information fusion utilized in stage I and stage 2, stage 3 perform global feature fusion using transformer block. E2 is projected into tokens embeddings F3 E IR. f xCxxs using non-overlapping 2 x 2 strided convolution. Es mixing with Intermediate Positional Embedding (IPL. is feed into a pure transformer block with L3 layers. We denote the number of attention heads in stage 3 as H., The mip-ratios in FFN for different stages is denoted as Pl, P2 and P3 in respectively. Stage and stage 2 is designed to capture fine-grained details on high resolution feature map. Stage 3 can perform dynamically global reasoning efficiently on rather low-resolution feature map. At the same time, stage 3 can enlarge the filed-of-viers (FOV) of backbone which benefits a wide range of downstream tasks. The encoder of MCMAE can seamlessly inherits the merits of convolution and transformer block. The architecture details for small, base and large model is listed in Table 7. MCMAE small, base, large and huge share similar parameter scale with the encoder of MAE-small, MAE-base, MAE-large and MAE-huge.
Model MCMAE-S
[C1, C3]
[1, L2 L3
E1, Eg. Ex
P1, Pg, Ps
H
#Params (M)
[128, 256, 384]
[2, 2, 11]
[56, 28, 14]
[4, 4, 4]
6
22
MCMAE-B
[256, 384, 768]
[2, 11]
[56, 28.14]
[4, 4,44 4]
12
84
MCMAE-B B
[256, 384, 768]
[2, 11]
[56, 28.14]
[8, 8, 4]
12
88
MCMAE-L
[384, 768, 1024]
2,
[56. 28, 14]
[8, 8, 4]
16
322
MCMAF-H
[768, 1024, 1280]
2,3
[56, 28, 14]
88, 4]
16
666
Table 7: Architecture details of MCMAE small, base, large and huge. MCMAE-B* represents multi- scale encoder with large mip-ratios in stage I and stage 2. [C1, C2, Cz). [L1, L2, Ls], [Er, E2, Eg] and [P, Ps. Pa] represents channel dimension, number of layer, spatial resolution and mlp-ratios for each stage 1, stage 2 and stage 3. H, stands for the number of attention heads in stage 3.
Model Scaling up and down. We design MCMAE of different parameters scales to match those of MAE-small, MAE-base, MAE-large and MAE-huge. Detailed network architectures are in appendix. The finetuning performances are shown in Table 8. Compared with the original MAE [28] of different scales, our MCMAE of different scales consistently outperform its MAE counterparts on Imagenet finetuning. This suggests that MCMAE can be an efficient learner for different paramter scales.
Feature Map Visualization. We provide some visualization of multi scale feature maps generated by MAE and MCMAE backbone with the Mask R-CNN [30] method in Fig. 5. The masked convolution reveals much more fine-gren fine-grained features compared with the pure vision transformer architecture of MAE, especially in feature maps with a stride of 4.
12

---

Model size
Method
P-Epochs
Small
Base
Base*
Large
Huge
MAE MCMAE
1600
79.5
83.6
N/A 84.9
85.9
86.9
800
82.6
84.6
86.2
N/A
Table Ablation study of model scales.
MCMAE
MA
MCMAE TE
MAE
MCMAE
MAI
MCMAE
MAE
MCMAE
MAE
Results
Stride4 Feebure Map
Stride88 Feature Map Stride 16 Feeture Map Stride332 Resture Map
Figure 5: Visualization of feature maps with different strides.
13


Reviews:
[{"review_id": "tpXLzQK9bca", "reviewer": "Reviewer_mqzA", "summary": "This paper proposed a new self-supervised learning framework by integrating hybrid convolution-transformer architectures and masked convolution into the masked auto-encoders. The proposed method can achieve computational efficiency and low pretraining-finetuning gap at the same. Extensive experiments on several computer vision tasks demonstrate the effectiveness of the proposed method.", "questions": "See \"Weaknesses\".", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n\n- The paper is well written and easy to follow. Sufficient technique details are provided.\n- The proposed method is well motivated and simple. Several key components are proposed to address heavy computational cost and pretraining-finetuning discrepancy.\n- The proposed method is flexible and can be applied in both image classification and object detection.\n\n__Weaknesses__\n\n- It seems hybrid convolution-transformer architectures have been explored in previous works but show how very similar performance to MAE (Lines 45-47). Why the proposed method can make them work for MAE? The differences from previous work and the contribution of the paper remains vague.\n- Some parts of the method are not clearly illustrated. For example, in “Block-wise Masking with Masked Convolutions”, the authors state that “Uniformly masking stage-1 input tokens would cause all tokens of stage-3 to have partially visible information and requires keeping all stage-3 tokens”. Why the proposed method can address this issue? What is the key idea of the proposed method?\n- The required training epochs vary from different methods. I wonder whether the proposed method can still outperform others under the same training epochs.\n\n__Post Rebuttal__\n\nI thank the authors for their response. Most of my concerns have been addressed. I increased my rating and recommend acceptance for this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_ljYpO-7La0", "reviewer": "Reviewer_KmDT", "summary": "This paper proposes a self-supervised framework using a hybrid convolution-transformer architecture, to obtain multi-scale, hierarchical representations. Masked convolution is introduced to prevent information leakage in convolution blocks and block-wise masking strategy is applied to improve computational efficiency. The resulting model achieves competitive performances in image classification and dense prediction tasks such as object detection. ", "questions": "How would the duration of the pre-training epochs impact the backbone performances? How much performance drop if changing to shorter training epochs, such as reducing it from 1600 to 800 epochs?", "limitations": "adequate ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths: \n1. This work effectively extends the self-supervised MAE framework to the hierarchical, convolution-transformer hybrid architecture. \n2. The resulting model outperforms existing self-supervised models in classification and dense prediction tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SCw1JSaqqO", "reviewer": "Reviewer_fnaM", "summary": "The paper starts with the hypothesis that a multiscale hybrid convolution-transformer can learn better representations using masked inputs than vanilla ViTs. The original masking scheme proposed in the MAE paper can be computationally prohibitive when directly applied to hybrid models. This paper presents a multiscale block-wise masking strategy with masked convolutions to efficiently train a hybrid transformer-convolutional model for representation learning. The paper shared a broad range of empirical results on classification, detection, segmentation, and video understanding tasks to show the effectiveness of the proposed technique.", "questions": "1. Did you experiment with larger models of the order of ViT-L and ViT-H? How do these results scale?\n2. Why did you generate limited data points for VideoConvMAE and VideoConvMAE-multiscale in figure 3?\n3. Why did you not include the CoAtNet results in the experiments?", "limitations": "Yes, authors adequately addressed the limitations.\n", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The novelty of the paper lies in its proposed multi-scale hybrid convolution-transformer encoder, which can generate hierarchical representations and possibly exceed the performance of vanilla ViTs. The idea of hybrid models already exists in multiple pieces of literature (CoAtNet, Early Convolutions etc.). Masked convolutions were introduced in the PixelRNN paper (https://arxiv.org/pdf/1601.06759.pdf). The strength of this paper is in its novel combination of existing ideas to produce a very simple hybrid framework that effectively combines the strength of convolutions and transformers.\n\nI also like the idea of performing masking at the late stage and then progressively upsampling the mask to larger resolutions to avoid the requirement of keeping all tokens in stage 3.\n\nThe proposed setup naturally generates hierarchical representations and fits nicely with Feature Pyramid Networks. It is a nice way to generate a feature pyramid with local context via convolutions and global context using transformers.\n\nQuality: The paper primarily describes experiments using ViT-B scale networks. It covers a broad set of vision tasks but it does not cover scale. It would be nice to see whether the proposed scheme continues to outperform existing masking techniques for larger models. There is also limited runtime comparison with existing techniques.\n\nThe paper shares very informative results of ablation experiments comparing random masking, regular convolutions, multi-scale decoders etc.\n\nClarity: The paper is very well written, with a nice flow, and explains the concepts with ease. \nNit: line 56 pretraing -> pretraining \n\nSignificance: The paper proposes a simple and effective hybrid convolution-transformer encoder, which naturally generates hierarchical representations from an image and outperforms a number of existing techniques. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mt_r_atVyQT", "reviewer": "Reviewer_qdV5", "summary": "This paper addresses the difficulty of applying MAE training with convolutional layers. The proposed ConvMAE adopts masked convolutions in the early stage of convolutional layers by applying convolution on the masked featuremaps. In this way, the information leak is prevented. With the proposed ConvMAE training, the ViT with early convolutional layers can benefit from the MAE training and achieved better transfer learning results comparing to standard ViT.  It achieves superior performance on ImageNet & MS-COCO.", "questions": "Would the masked featuremap introduce some bias or artifacts to the training along the mask edges? It may be helpful to find a way to alleviate the boundary issue.", "limitations": "The comparison with the MAE training with standard ViT backbones may be unfair, due to introducing extra computation cost with the convolutional layers. It would be helpful to further break down the improvements.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Novel training strategy to enable MAE training for models with convolutional layers.\n2. Strong performance on various transfer learning tasks. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "2nxBGoV8-N8", "reviewer": "Reviewer_Nkn5", "summary": "The paper proposes ConvMAE, a hybrid convolution-transformer architecture that is friendly to MAE-like pre-training. MAE was originally proposed with ViT, and due to omitted mask tokens in the backbone encoder, MAE is not trivially extensible to convolutional networks. The work extends MAE by resorting to the hybrid design of first using convolutions, and then using transformers. The masking is done block-wise (at the resolution of the transformer); and masked convolutions are used to avoid potential cheating. Extensive experiments are done on ImageNet classification, object detection, semantic segmentation, video classification. Various ablation analysis is also provided. ", "questions": "- Please address the concerns mentioned above, especially the first point. ", "limitations": "I do not see potential negative societal impact concern. The paper also points this out at the end, which is adequate to me.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) Self-supervised learning, especially masked auto-encoding for images is an emerging topic in computer vision. A breakthrough in this direction can bear huge significance. The work aims at fixing the limitation of MAE by introducing an hybrid architecture of convolutions and transformers, which is definitely important and relevant to the NeurIPS audience.\n\n(+) The paper is well written, and is clear enough for readers to follow through with good illustrations. \n\n(+) The experiments are extensive and conclusive. The downstream transfers include image classification, object detection, semantic segmentation, and even video understanding is involved (which by itself could be an independent investigation). The ablations and the conclusions are also covering most of the things I can think of -- a solid paper clearly with a lot of hard work behind the scene.\n\n(-) I think the \"Conv\" part of \"ConvMAE\" is an over-emphasis. The architecture only has 4 conv layers in the bottom of the network, while it has 11 transformer blocks for the base model (ViT-B has 12 blocks in total). So my current understanding is that ConvMAE has a similar architecture as in:\n\nXiao, Tete, et al. \"Early convolutions help transformers see better.\" Advances in Neural Information Processing Systems 34 (2021): 30392-30400.\n\nThis means the majority of the architecture is still transformers, and in this regard, the difference/significance over original MAE is not that salient. This is the biggest concern about the paper -- it has a risk of over-sell with the term \"Conv\" in it.\n\n(-) One minor concern is about the scalability of ConvMAE. The paper is highly focused on the model size of the base model. It is unclear the benefit of ConvMAE can still hold when the model size further scales up, as shown in pure ViT-based MAE.\n\n(-) Some minor typos need to be fixed with proof-reading:, e.g., should define what is FOV at page 2, and the mask ratio should be 75% instead of 25% for MAE if I recall correctly.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept