Green Hierarchical Vision Transformer for Masked Image Modeling
Authors: Lang Huang
Shan You
Mingkai Zheng
Fei Wang
Chen Qian
Toshihiko Yamasaki
Venue: NeurIPS 2022 Accept (2022)
Keywords: Self-Supervised Learning
Masked Image Modeling
Vision Transformers

Abstract: We present an efficient approach for Masked Image Modeling (MIM) with hierarchical Vision Transformers (ViTs), allowing the hierarchical ViTs to discard masked patches and operate only on the visible ones. Our approach consists of three key designs. First, for window attention, we propose a Group Window Attention scheme following the Divide-and-Conquer strategy. To mitigate the quadratic complexity of the self-attention w.r.t. the number of patches, group attention encourages a uniform partition that visible patches within each local window of arbitrary size can be grouped with equal size, where masked self-attention is then performed within each group. Second, we further improve the grouping strategy via the Dynamic Programming algorithm to minimize the overall computation cost of the attention on the grouped patches. Third, as for the convolution layers, we convert them to the Sparse Convolution that works seamlessly with the sparse data, i.e., the visible patches in MIM. As a result, MIM can now work on most, if not all, hierarchical ViTs in a green and efficient way. For example, we can train the hierarchical ViTs, e.g., Swin Transformer and Twins Transformer, about 2.7$\times$ faster and reduce the GPU memory usage by 70%, while still enjoying competitive performance on ImageNet classification and the superiority on downstream COCO object detection benchmarks.

Green Hierarchical Vision Transformer for Masked Image Modeling
Lang Huang', Shan You"; Mingkai Zheng", Fei Wang', Chen Qian', Toshihiko Yamasaki
'The University of Tokyo; "SenseTime Research; 'The University of Sydney
(langhuang, yamasakki@@cvm wtt.:mm...uttkee. ac-tokyo.ac.j
{youshan, wangfei, giannchenn/seensttiine
Abstract
We present efficient approach for Masked Image Modeling (MIM) with hi- erarchical Vision Transformers (ViTs), allowing the hierarchical ViTs to discard masked patches and operate only on the visible ones. Our approach consists of three key designs. First, for window attention, we propose a Group Window Attention scheme following the Sevide-and--e and-Conquer strategy. To mitigate the quadratic com- plexity of the self-attention w.r.t. the number of patches, group attention encourages a uniform partition that visible patches within each local window of arbitrary size can be grouped with equal size, where masked self-attention is then performed within each group. Second, we further improve the grouping strategy via the Dynamic Programming algorithm to minimize the overall computation cost of the attention on the grouped patches. Third, as for the convolution layers, we convert them to the Sparse Convolution [25, 13] that works seamlessly with the sparse data, i.e. the visible patches MIM. As a result, MIM can now work on most, if not all, hierarchical ViTs in green and efficient way. For example,
train the hierar- chical ViTs, e.g., Swin Transformer [49] and Twins Transformer [14], about 2.7 faster and reduce the GPU memory usage by 70%, while still enjoying competitive performance on ImageNet classification and the superiority on downstream COCO object detection benchmarks.
Introduction
Driven by the great success of Masked Language Modeling (MLM) [56, 57, 15, 6) in natural language processing (NLP) and the advancement Vision Transformers (ViTs) [17, 49, 66, 77], Masked Image Modeling (MIM) emerged as a promising sele-supervieed pre-training paradigm for computer vision (CV). MIM learns representations from unlabelled data by masked prediction, e.g., predicting
the discrete tokens [3], the latent features [81, 68, 2]. the pixels [28, 73] of the randomly masked input image patches. Among them, the representative work Masked Autoencoder (MAF) [28] exhibited competitive performance well impressive efficiency. In essence, MAE proposed an asymmetric encoder-decoder architecture for MIM, where the encoder (e.g., a standard VIT model [17]) operates only on visible patches, and the light-weight decoder recovers all patches for mask prediction.
On the one hand, the asymmetric encoder-decoder architecture significantly reduces the computation burden pre-training On the other hand, MAE only supports the isotropic ViT [17] architecture as the encoder, while most of the modern vision models adopt hierarchical structure [43, 31, 49], in part due to the need of handling the ccale-variations of visual elements. In fact, the hierarchical structure and local inductive bias are crucial in various CV tasks that require representations of different levels
Corresponding author.
Code and pre-trained models: https: /ttp///gttttttttt com/Layyee//GreemMMM
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

GPU HOURS / EPOCH
GPU MEMORY (GB)
IN-1K ACCURACY (%)
Ours SimMIM (192) SIMMIM (224)
Ours (224) SimMIM (1922) SImMIM (224)
(224)
SimMIM (192) SimMIM (224)
5,6
1200
1123
86
85.1
85.4
1000
800
727.0
85
84
84.0
84.1
2.8
600
2.0
398.3
1:3
284.8
233.6
121.6
Swin-B
Swin-L
Swin-B
Swin-L
Swin-B
Swin-L
Figure 1: Comparison with SimMIM in terms of efficiency. All methods use a Swin-B/Swing backbone and batch size of 2,048. The experiments of our method are conducted on a single machine with eight V100 GPUs, CUDA 10, and PyTorch 1.8, while those of SimMIM require or 4 machines.
or scales to make predictions, including image classification [31], and object detection [22]. Yet it is still not straightforward how the hierarchical vision transformers, e.g.Swin Transformer [49], can be integrated into the MAE framework. Moreover, though the work SimMIM [73] has explored Swin Transformer for MIM, it operates computation costs compared with MAE, As a concrete example, we find that even the base size model of SimMIM cannot be trained on single machine with eight 32GB GPUs, let alone the larger ones. The computation burden makes it difficult for a wider range of researchers to dive into this field of research, not even mention the amount of carbon emission during the model development process.
both visible and masked patches and suffers from heavy
To this end, we strive to devise a new and green approach for MIM with hierarchical models, in the spirit of Green AI [61, 74]. Our work focuses on extending the asymmetric encoder-decoder architecture of MAE to hierarchical vision transformers, particularly the representative model Swin visible patches only. We identify that the major obstacle is the inductive bias of hierarchical ViTs, i..., the locality induced by i) window attention [36, 49, 77] with non-overlapped window partition, and ii) convolution/pooling [66, 19, 14, 27] with overlapped window partition. These operators are incompatible with random masking as it creates various-sized local windows that are infeasible for computing in parallel.
Transformer [49], for the sake of efficient pre- pre-training
This paper provides the first attempt to address this drawback. We present a Green Hicrarchical Vision Transformer for Masked Image Modeling that advocates more practical method with drastically improved efficiency. Our methodology is conceptually simple and consists of three key designs.
1. Guided by the Divide-and- Lan--Coqueer principle, we present a Group Window Attention scheme first partitioning the local windows with uneven numbers of visible patches into several equal-sized groups and then applying the masked attention within each group.
2. We formulate the aforementioned group partition as a constrained optimization problem, where the objective is to find a group partition that minimizes the computation cost of the attention on the grouped tokens. Inspired by the concept of Dynamic Programming [5] and the greedy principle, we propose an Optimal Grouping algorithm that adaptively selects the optimal group size and partitions the local windows into a minimum number of groups.
3. We convert the convolution layers in the hierarchical ViTs to the Sparse Convolution [25, 13, I]. which was originally designed for handling sparse point cloud data and works seamlessly with the masked inputs in MIM.
Our methodology is generic and does not make any modification to the architecture of the backbone models, such that we can make apple-to-apple comparisons with the bascline operating on both visible and masked patches. In our experimental evaluations, we observed that our method requires substantially less training time and consumes much less GPU memory while performing on par with the baseline on both ImageNet I [60] classification and MS-COCO [47] object detection. Concretely, using Twins-L [1 14 [49]/Swin-L [49], our method achieves up to 2.7x speedup and consumes as few as 30% of GPU memory compared with the baseline SimMIM at the pre-training stage, while achieving 83.9 83.9%%/8..8. 83..8% % top-1 fine-tuning accuracy on the ImageNet- K that is on par with
those of SIMMIM.
2

---

2 Related Works
Self-Supervised Learning. Representation learning is a long-standing and fundamental question in CV. For a long time, representation learning had been dominated by supervised learning. Until recent three years, self-supervised learning (SSL.) exhibited impressive performance and attained significant attention. Generally, SSL solves proxy task without the actual interest to learn good representations. According to the proxy tasks, SSL methods can be categorized into generative approaches and discriminative approaches. Generative approaches predict the original data based on the partially observed inputs [65, 55, 44], predict the transformation applied to the input [52, 21J, or model pixels in the input space [42, 23, 39, 33]. Masked Image Modeling also falls into this category. Discriminative approaches received more interest during the past few years, especially the contrastive learning methods. Contrastive learning creates multiple views of images with set of random data distortions and encourages the representations to be invariant to the distortions. A large number of contrastive learning approaches [70, 53, 29, 10, 79, 80] drive the training by maximizing the similarities between positive samples (i.e., views from the same image) while minimizing those between the negative samples, and some works simply get rid of the negative pairs [26, 7, 11, 37, 78, 8). Beyond contrastive learning on global features, several methods proposed to maintain the spatial information of representations and use region/mmaskpiixllleeed contrastive learning [67, 72, 71, 32, 353.
Masked Language/Image Modeling. Self-supervised pre-training has revolutionized the NLP. Among them, the Masked Language Modeling (MLM) proposed in BERT [15] and its variants [6] are the most dominant methods, which learn representations by predicting the tokens that randomly masked in the input sentence. Masked Image Modeling has a similar idea of predicting corrupted images, and some of these methods [65, 55] were proposed Even preceded to BERT. These methods were, however, unable to perform on par with other pre-training paradigms at that time. Until recently, aided by the significant advancement of Vision Transformers [17], several MIM methods presented promising results [17, 3, 28, 73, 68] and became the state-of-the-art of self-supervieed learning in CV. These methods can be roughly differentiated in terms of the prediction target, e.g. color bins [17]. discrete tokens [3, 16] from pre- pre-trained VAEs [63, 58], raw pixels [28, 73], and handcrafted features [68]. Among these approaches, MAE [28] exhibited competitive performance as well as impressive efficiency as it discards the masked tokens and operates only on the visible ones.
Isotropic and Hierarchical Vision Transformers. The seminal work Vision Transformer (ViT) [17] revolutionized the conventional view of images. ViT, and its variant [62], treat an image as sequence of patches and adopt pure Transformer [64] backbone model the patches for image classification, achieving an impressive performance even compared with the Convolutional Neural Networks. Nevertheless, while the results of ViT are promising in classification, its performance on dense prediction tasks is less favorable, which is largely due to its low-resolution feature maps inherited from its isotropic structure and the quadratic complexity of self-attention [64]. To this end, a strand of works proposed hierarchical structure [66, 49, 19, 77], efficient attention [36, 49, 14, 38], and locality bias [18. 69, 27, 76] for ViTs, unleashing the potential of ViTs as general-purpose vision backbones. Our work performs studies upon the representative hicrarchical ViTs, Swin Trans- former [49] and Twins Transformer [14] but generalizes to any other ViT's with window local attention or convolution/pooling..
Green AL Witnessing the exponential growth of computations of big Al models [15, 6, 48], the concept of Green Al attains mounting attention in recent years [61, 74]. Rather than being merely obsessed with accuracy, Green Al advocates making efficiency an important measure of Al models, championing the greener approaches that are more inclusive to the research community. This work follows the path of Green Al and presents a greener approach for MIM with hierarchical ViTs.
3 Approach
3.1 Preliminary
Notations. Let X E RC denote the input feature where C, H. and W are the numbers of channels, height, and width of M, E {0, 1) Hx denotes the (spatial) mask generated randomly during training where 0 indicates patch is invisible for the encoder, and vice versa.
3

---

X
8
X
Optimal Grouping
Masked Attention
Mask
Figure Illustration of the Group Window Attention scheme. In Masked Image Modeling (MIM), the input X, where different colors indicate the tokens belong to different local windows, is randomly masked, producing X of which most tokens invisible. Our Group Window Attention first performs an optimal grouping to group the visible tokens the local windows into several equal-sized groups, forming X. Finally, we perform the Masked Attention within each group to ensure no inter-window information leakage.
Masked Image Modeling. MIM learns representations by predicting the masked portion of input X from partial observation X Mask X, M). Existing MIM methods fall into two categories regarding the Mask(,, operation. Most methods [3, 73, 68] use the Hadamard product for masking and retain the masked patches, i.e., X Mask M) X M with M broadcasted for C times along the channel dimension. In sharp contrast to these methods, Masked Autoencoders (MAE) [28] proposes throwing the masked patches at the masking stage, i.e.,
X
f- Mask(X, M) [Xi, M, 1
(I)
MAE designs asymmetric and isotropic encoder-decoder architecture to take advantage of partial inputs: the encoder operates only the visible patches X without mask tokens; the decoder reconstructs the original images from representations of visible patches and masked tokens. This design allows MAE to achieve competitive performance 3x training speedup compared with the ones operating on all patches. Nevertheless, the MAE works only with the isotropic ViTs and it is unclear how to translate the efficiency of MAE to hierarchical ViTs, which exhibited nearly unanimous superiority over the isotropic ones most vision tasks [66, 49, 19, 14, 77]. In this paper, we attempt answer this question and propose much greener approach for MIM with Hierarchical ViTs.
well impressive efficiency, e.g.,
3.2 Green Hierarchical Vision Transformer for Masked Image Modeling
Adapting existing hierarchical VITs for MIM. A typical hierarchical VIT mainly consists of Feed-Forward Networks (FFNs) and efficient attentions, which fail to operate only on the visible patches. We identify that the major obstacle the inductive bias of hierarchical ViTs, i.e., the locality induced by i) window attention [36, 49, 77] with non-overlapped window partition, and ii) convolution/pooling [66, 19, 14, 27] in FFNs attentions with overlapped window partition. These operators are incompatible with random masking as it creates various-sized local windows that are infeasible for computing parallel. To this end, we present two key insights to make most, if not all, hicrarchical ViTs be able to operate only on visible patches in green manner.
Group Window Attention. Regarding the non-overlapped window partition, propose Group Window Attention scheme that significantly improves the computation efficiency window attention on masked features. Given the masked feature X = Mask X, M) following Equation (1), we collect of uneven local windows X ) of which each element contains the visible tokens only, with sizes (w;] ) accordingly. As shown in Figure 2, our Group Window Attention first uses an Optimal Grouping algorithm partition the uneven windows into several equal-sized groups and then performs Masked Attention within each group to avoid information leakage. In the next two subsections, will elaborate these components, respectively.
Incorporating with Sparse Convolution. Our intuition is the features of mask inputs can be viewed sparse tensors where only features of (a small number of) visible patches retained while

---

Algorithm Optimal Grouping
Require: The number of visible patches within each local window {wo (wo)
1: Minimum computational cost c" <- +000
2: for Ds max; (w;) ==1 to E W; do
3: Remaining windows - { W4 J= partition II 4- 0; the number of group ng f
4: repeat
5: TH, f Knapsack((p) (Js, 4). as in Equation (7)
6: II - U nngs < & An,
7:
Bgg 5 11g +1
8:
until =
9:
c < Cgg., IT), as in Equation (8)
10:
if C* then
11:
c II + 11
12:
end if
13:
end for
14:
return Optimal group partition
the others omitted. From this perspective, propose directly replace all the convolution/pool- ing operations in hierarchical ViTs with the highly optimized Sparse Convolution [24, 25, 13, 1], originally designed for the sparse 3D point cloud data.
3.3 Optimal Grouping with Dynamic Programming
General formulation. The first step of the optimal grouping is to find an indexes partition with
respect to the group size 9s:
I= {
(2)
s.t. Uj T -= - Σ wi), Σ 1m j
Σ
Wi and Vj
Σ Wk S Is. RER;
(3)
where ng is the number of resulting groups. The conditions in Equation (3) constrain the partition to contain all local windows with no duplicate and enforce the actual size each group to smaller than Sz. Based on the partition II, we obtain a set of grouped tokens (X,) that
X, = Concatt (1 : Ak E Ty)), X, E RC
(4)
upon which the Masked Attention performed. Finally, we apply the inverse operation of the partition II to recover the positions of output tokens.
With the formulation above, there remain two questions unresolved: 1) how choose the optimal group size g and 2) how to obtain the optimal partition IT" given g² To this end, we formulate our objectives the below min-min optimization problem,
g. = argmin, C(g., Partition(g, (wi)t))
(5)
Partition (g., (w.)...) = argming [IT], s.t. Equation (3),
(6)
where C(-) is function measuring the computation cost of the attention with the grouped tokens. Intuitively, Equation (5) aims to find the optimal group size g that the computation cost of the optimal partition w.r.t. g is the minimum. Equation (6) searches for the optimal partition, with the constraints of Equation (3). Having the optimal group size, we can directly obtain the optimal partition II Partition (g", nn..). Next, we will present in detail how we solve the above optimization problem.
Group partition with Dynamic Programming. We find that the optimization problem in Equa- tions (6) and (3) is a special case of the multiple subset sum problem with identical capacities (MSSP-D, variant of the well-known 0-1 multiple knapsack problem with identical capacities (MKP-I) [40, Chapter 10]. In our case, the group size is analogous to the capacity of knapsacks, the
Here we assume the number of tokens E, w; is divisible by for simplicity. In practice, we pad the group My when |ry] is smaller than g..
5

---

Query
Mask
Key
Value
Figure 3: Illustration of the Masked Attention scheme. Given group of tokens, first compute their pairwise attention weights and then set the attention weights between tokens from different local windows to -oo (indicated by the gray cells). The final attention output is then computed with the masked attention map.
numbers of visible tokens {we Jn.. are analogous to the values of goods, the weights goods are the same their values, and the number of knapsacks is unbounded. Although MSSP-I is strictly -o-complete, there exist multiple polynomial time approximation schemes for it, e.g. using the dynamic programming (DP) algorithms [5]. Specifically, make use of the DP algorithm for the single knapsack problem (or the subset sum problem):
π f Kuapsack(gs> 4)
which selects a subset that ΣnER |u| S Is out the full set & (the pseudo-code of this algorithm given in Appendix). We alternatively apply this algorithm to the remaining full set & and exclude the selected subset ı from o until & is empty. In practice, we found that our algorithm is very fast and the time cost is negligible because the number of local windows is often small, e.g. less than 100 our pre-training stage.
Cost function. Because we mainly care about the efficiency, we use the FLOPs to measure the computation complexity of the multi-heas attention on grouped tokens, i.e.,
C(g., 11) [1I| (4g.C² 2) = ng (4g. Cg²
where C is the number of channels. Although the complexity is quadratic w.r.t. the group size Is. using smaller J. might produce more groups (and more padding) and suffers from suboptimal efficiency. Therefore the optimal group size is determined adaptively during training.
Putting everything together. We sweep over the possible values of group size, from max; wy to E, Wi, to find the optimal group size. For each selected group size, firstly the DP algorithm in Equation (7) to partition the windows and then calculate the computation cost of the attention this partition. The with minimum cost is chosen the optimal group size. The pseudo-code of the optimal grouping summarized in Algorithm 1.
3.4 Masked Attention
Because non-adjacent local windows partitioned into the same groups, masking the attention weights is needed to avoid the information exchange between these local windows. As illustrated in Figure 3, having computed the attention map, we retain only the intra-window attention weights (i.e., the block-diagonal elements) and discard the inter-window ones. A similar masking scheme is also applied the retrieving of relative position bias [49], where we the original absolute position of each token and compute the relative positions on-the-fly to retrieve the corresponding biases.
3.5 Batch-wise Random Masking
observed that the per-sample random masking strategy would deteriorate the efficiency of our method: 1) it might produce different numbers of groups of local windows for each sample, which intractable for the parallel computation the Masked Attention; 2) when the mask patch size
smaller than the largest patch size of the hierarchical models, some patches might contain both visible

---

168
Stage1
1e8
Stage2
168
Stage3
1.2
1LOP5
1.0
0.8
0.8
0.6
50
100
150
200
250
50
100
150
200
30
40
Group size
Group size
Group size
Figure 4: The optimal group size at each stage. The figure of the fourth stage is omitted here because there only one local window in this stage, so the grouping is not necessary. The simulation repeated 100 times, of which the mean and standard deviation (the shaded regions) are reported.
and masked inputs. In this case, we can not discard such patches during training and fail to fully take advantage of the sparsification. Therefore, we propose to set the mask patch size to the same value the largest patch size of the encoder (e.g., 32 for most hierarchical models, which is also the default choice of [73] and use the same random mask for all samples in the same GPU device (a.k.a., micro-batch).
Experiments
4.1
Implementation Details and Experimental Setups
We conduct experiments on the ImageNet- IK [60] (BSD 3-Clause License) image classification dataset and MS-COCO [47] (CC BY 4.0 License) object detection/instance segmentation dataset. The Swin-Base, Swin-Large [49] and Twins-Large [14] models, which consist of four stages with features of stride 4/8/16/32, are used as the encoder throughout this paper, allowing for direct comparisons with the baseline SimMIM [73] with Swin Transformer. The models are first pre-trained on the ImageNet-IK dataset without label and then fine-tuned on downstream tasks. All the experiments of our method are performed on single machine with eight 32G Tesla V100 GPUs, CUDA 10.1, PyTorch [54] 1.8, and automatic mixed- -exeddprecision training [51]. For the experiments involving convolution, replace all the standard convolutions with the sparse convolution implemented in [1] as discussed in Sec. 3.2.
Pre-training setup. We patchify images of size x24x224 with patch size of 4x4 and randomly mask the patches with a ratio r (r 0.75 by default) following the scheme in Section 3.5. The input images are first transformed by set of simple data augmentations, including random cropping and horizontal flip, and standardization. Following the prior work MAE [28], we adopt a lightweight decoder that consists of ny (by default na 1) standard transformer blocks with embedding dimension of 512. The decoder takes the representations of visible patches and the mask token as input and is appended after the final stage of the encoder to learn representations for the masked patches. It is followed by linear layer to predict the normalized pixel values of the masked patches. The models
trained for 100/200/400/80X epochs with total batch size of 2,048. We the AdamW optimizer [41] with the cosine annealing schedule [50]. We set the base learning rate to 1.5e -4 the weight decay 0.05, the typer-parameters of Adam B1 0.9, 82 0.999, the number of warmup epochs to 40 with an initial base learning rate 17 The effective learning rate is scaled linearly by batch_size aaac_ssiee/256
Fine-tuning on the ImageNet-1K dataset. For fine-tuning, we drop the decoder and directly append a 1,000-wa way fully-connected layer to the average- poorled-pooeed output of the encoder the classifier. The models are also optimized by the AdamW optimizer [41] with 100 training epochs total, 20 warmup epochs, base/warmup learning rate of I 1.25e 4/2.5e-7 the cosine annealing schedule [50], a weight decay of 0.05. A layerwise learning rate decay [3] of 0.9/0.8/0.9, 9, and stochastic depth [34] used for Swin Swin-B/Swin. B/Swin-LLTwins-L respectively. The data augmentations
ratio of 0.2/0.3/0.2 the same as [3, 73].
Fine-tuning on the MS-COCO dataset. We adopt the Mask R-CNN [30] architecture with the FPN [46] as the detector. models are fine-tuned on the MS-COCO [47] 2017 train split 1~118k 18k
images) and finally evaluated on the val split (~55k images). We use batch size of 16, AdamW
7

---

Imagetet top1 AOc, (%)
Imageher Top3. Acc. 186)
Imagehet Top1 AcC (50
83.7
63.2
83.6
83.5
83.1
83.1
83.4
63.3
03.0
83,0
83.2
[b] The
The Number of Training Epochs
Figure 5: Ablation studies of (a) the choice of the mask ratio r, (b) the number of transformer blocks nd in the decoder, and (c) the number of pre-training epochs.
optimizer [41] with a learning rate of le-1, 1, a weight decay of 0.05. The 1 xx/3x schedule in the mmdetection [9] adopted, which uses 12/36 training epochs in total and decays the learning rate at the & and 11 of the total epochs by factor of 10. The standard COCO metrics, including AP, APso. 12 and APrs for both object detection and instance segmentation are used for evaluation.
4.2 Ablations studies
Efficiency comparison with SimMIM. We compare the efficiency of our method to the baseline SimMIM in Figure 1. The evaluations are performed on a single machine with eight 32GB V100 GPUs for our method and on or 4 machines for the SimMIM because it fails to fit into a single machine with the default batch size of the original paper [73] (i.e., 2,048). As we see from the figures, the training of SimMIM with images of size 224 very slow and memory-hungry. Although training with smaller images considerably reduces the training time and memory consumption, it still far lags behind our method with images of size 2242. Concretely, with the same amount of training epochs, method performs par with baseline with ~2x speedup and ~60% of memory reduction using Swin-B. We also observe that the efficiency improvements become larger with the larger Swin-L., e.g., 2.7x speedup compared with SimMIM highlighting the efficiency of method with larger models. In addition, the efficiency comparison using Twins Transformer is presented in Appendix A. 1.1. for completeness, which exhibits similar superiority of our approach.
The optimal group size Is on each stage. Because hierarchical models have multiple stages with different scales of features, the optimal group size of each stage might also be different. Regarding this, design simulation experiment to analyze the optimal 9s at different stages. In the simulation, we randomly generate 100 masks following Section 3.5, compute the costs w.r.t. different choices of gs. and report the mean/standard deviation of the costs in Figure 4. Note that the analysis of the 4th stage is omitted here because it only has one local window. In general, we observed that the cost increases quadratically w.r.t. the group size, except for some cases where the group size is exactly equal to the sum subset of local windows. Another intriguing observation is that the cost at each stage seems to be the minimum around the point that 9. 49, which is equal to the window size of the window attention. This observation indicates that we may not need to sweep over all possible group sizes but simply set 9s p X p in practice. Moreover, analyze the influence of the optimal grouping and the cost of each part of it in Appendix A.2, showing that optimal grouping is indeed crucial for the efficiency of our method.
The influence of the mask ratio, the number of decoder blocks, and the pre-training epochs. From Figure 5(a), we can see that the performance of our method is quite stable with the mask ratio r varying from 0.5 to 0.85, which conforms with the observation of [28]. In Figure 5(b), we also study the influence of the depth of the decoder. Intriguingly, the results suggest that fewer decoder blocks produce better results. This study favors the simple prediction head design of SimMIM [73] with hierarchical models and is in contrast to the observation of MAE [28] with isotropic ones. For simplicity and efficiency, we fix r = 0.75 and the number of decoder blocks to throughout the paper. Furthermore, we study the impact of pre-training budgets the fine-tuning accuracy increases steadily w.r.t. the number of training epochs and does not seem to stagnate, suggesting its potential for further performance boost.
method. As shown Figure 5(c),
Pre-training with larger window size. The work of [48] puts forth that using a larger window size is beneficial for fine-tuning. In practice, however, it might be less practical because of the

---

Table Top1 accuracy on the ImageNet- 1K validation with the Swin-B or ViT-B models. All methods are trained with images of size 224 x 224 in both the pre-training and fine-tuning except for SimMIM 192 using
x 192 the pre-training.
Method
Model
#Params
PT Ep. Ep. Hours
Total Hours
FT Ep.
Acc.
Training from scratch
Scratch, DeiT [62]
VIT-B
86M
81.8
Scratch, MAE [28]
VIT-B
86M
300
82.3
Scratch, Swin [49]
Swin-B
88M
0
300
83.5
Scratch, Twins [14]
Twins-L
99M
300
83.7
Supervised Pre-truining
Supervised, SimMIM [73]
Swin-B
88M
300
100
83.3
Supervised, SimMIM [73]
Swin-L
197M
300
100
83.5
Pre-training with Contrastive Learning
MoCov3 [12]
VIT-B
86M
800
100
83.2
DINO [8]
ViT-B
86M
800
100
82.8
Pre-training with Masked Image Modeling
BETT [3]
VIT-B
86M
800
100
83.2
MaskFeat [68]
VIT-B
86M
800
100
84.0
MAE [28]
ViT-B
86M
1600
1.3
2069
100
83.6
SimMIM224 [73]
ViT-B
86M
4.1
3307
100
83.8
SimMIM192 [73]
Swin-B
88M
800
2.0
1609
100
84.0
SimMIM [73]
Swin-L
197M
800
3.5
2821
100
85.4
Ours
Swin-B
88M
800
1.1
887
100
83.8
Ours
Twins-L
99M
0.8
83,9
Ours
Swin-L
197M
800
1.3
1067
100
85.1
quadratic complexity of self-attention w.r.t. the window size. Fortumately, operating only on the visible patches permits the training with a larger window size with little extra cost. As displayed Table I, pre-training with doubled window size only marginally increases the training time/GPU memory by less than 10%/20% yet brings moderate performance improvement despite p 7 in the fine-tuning stage.
Table 1: A larger window size p
P X
Time
Mem.
Acc.
7 14 x 14
1.1h
121.6G
83.2%
1.2h
148.9G
83.4%
4.3 ImageNet-1K Classification
We fine-tune the pre-trained models on the ImageNet-IK dataset and report the results the validation set Table Here, we make direct comparisons with the models 1) trained from scratch with longer training schedule, 2) trained with contrastive learning, and 3) trained with other MIM approaches. Our approach achieves $3.8%//3.99%/S. 1.% top-1 fine-tuning accuracy with the Swin- L/TWins-L/Swin-L backbone, which is clearly superior to the supervised leaminn/coonrrativee learning methods and on par with other MIM methods using backbones with similar capacities. The results demonstrate the effectiveness of our method, in addition the substantial efficiency improvements over both MAE and SimMIM.
4.4 MS-COCO Object Detection and Instance Segmentation
Finally, we evaluate the transfer learning performance of our pre-trained models to the MS-COCO object detection and instance segmentation dataset. Here, directly use the code base of the super- vised Swin Transformer without any modification to the fine-tuning strategy. For direct comparisons, we reran the experiments for the supervised Swin-B and SimMIM using their public checkpoints. The experiment results are summarized in Table 3. Compared with the supervised pre-trained Swin-B, our approach performs prominently better terms of all metrics, e.g., 1.5% absolute improvement in AP', In addition, we also observed that our approach still performs comparably to the SimMIM on dense prediction tasks. More significantly, our approach outperforms most of the baselines from [45] using 3x 10x more fine-tuning epochs and advanced data augmentations [20]. These experiments,

---

Table 3: MS-COCO object detection and instance segmentation. All methods are based on the Mask R-CNN [30] architecture with the FPN [46] neck. The methods in gray are cited from [45]. Most of them use much longer training schedules and advanced data augmentations.
Method
Backbone
PT Ep. PT Hours
FT Ep.
AP
AP AP
APM
AP
AP75
Training from scratch Benchmarking [45]
VIT-B
0
400
48.9
43.6
Supervised Pretraining
Benchmarking [45]
VIT-B
300
992
100
47.9
42.9
PVT [66]
PVT-L
300
36
44.5
66.0
48.3
40.7
63.4
43.7
Swin [49]
Swin-B
300
840
36
48.5
69.8
53.2
43.2
66.9
46.7
Self-Supervised Pre-training
MoCov3 [12]
VIT-B
800
47.9
42.7
BETT [3]
VIT-B
800
100
49.8
44.4
MAE [28]
VIT-B
1600
2069
25
48.1
MAE [28]
VIT-B
1600
2069
100
50.3
44.9
SimMIM [73]
Swin-B
800
1609
50.4
55.5
68.2
47.9
Ours
Swin-B
800
887
36
50.0
70.7
55.4
44.1
67.9
47.5
combined with the results in Table 2, verify that our approach achieve outstanding performance with impressive pre-training efficiency.
5 Conclusion
In this paper, we present green approach for Masked Image Modeling (MIM) with hierarchical Vision Transformers, e.g., Swin Transformer [49] and Twins Transformer [14], allowing the hi- erarchical models to discard masked patches and operate only on the visible ones. Coupling the efficient Group Window Attention scheme, the DP-aloorithm-base based Optimal Grouping strategy, and the Sparse Convolution, our approach can train the hierarchical models ~2.7 faster and reduce the GPU memory consumption by up to 70%, while still enjoying competitive performance on ImageNet classification and the superiority of downstream MS-COCO object detection benchmarks. We hope that this work will facilitate future self-supervised learning methods targeting effectiveness as well efficiency.
Limitations. One of the limitations of our algorithm that requires the batch-wise masking scheme (as in Section 3.5) to achieve the best efficiency. Although this limitation only has little impact on the MIM pre-training, restrains the application settings, e.g., training ViTs with token sparification [59, 75] that requires instance-wise sparsification. These applications are beyond the scope this work and we will leave them for future study.
our method on broader range of
Broader Impact. This work proposed a green approach for MIM with hierarchical ViTs, promi- nently alleviating the heavy computation burden of MIM. On the one hand, this work provokes the efficiency as well as the effectiveness of MIM, which may inspire new algorithms and investigations in this direction. On the other hand, since the pre-training datasets might contain biases, our approach, in the same way as other replicating these biases. This concern can be mitigated by combining the FairML methods [4].
weterrvseeeeseesserriiiiiiiiieie learning methods, might also be susceptible to
References
[1] spconv. https: com com/braavlleer///pccon
A. Bacvski, W.-N. Hsu, Q Xu, A. Babu, and M. Auli. Data2vee: A general framework for self-supervised learning speech, vision and language. arXiv preprint arXiv:2202.0355 2022.
H. Bao, Dong, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0 08254, 2021.
[4] S. Barocas, Hardt, and A. Narayanan. Faimess in machine learning. Nips tutorial, 1:2, 2017.
[5] R. Bellman. Dynamic programming. Science, 153(3731):34 1966.
10

---

[6] T. B. Brown, B. Mann, N. Ryder. M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry.
A. Askell, al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, 2020.
[7] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, 2020.
[8] Caron, H. Touvron, Misra, H. Jégou, J. Mairal, Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. (tti::/04..42944, /4294, 2021.
[9] K. Chen, J. Wang. J. Pang, Y. Cao. Y. Xiong, X. Li, S. Sun, W. Feng. Z. Liu, J. Xu, et al. Mmdetection: Open mmlah detection toolhox and benchmark. arXiv preprint arXiv: 1996000055, 2019.
[10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
[11] X. Chen and K. He. Exploring simple Siamese representation learning. arXiv preprint arXiv:2011.10566.
2020.
[12] X. Chen, S. Xie, and K. He. An empirical study of training self-supervieed vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9640-9699, 2021.
[13] C. Choy, J. Gwak, and S. Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1775-3084,77 2019.
[14] X. Chu, Z. Tian, Y. Wang, Zhang, H. Ren, X. Wei, H. Xia, and C. Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
[15] J. Devlin, M.-W. Chang, K.
Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171 4186, 2019.
[16] X. Dong, J. Bao, T. Zhang, D. Chen, W. Zhang, I.. Yuan, D. Chen, F. Wen, and N. Yu. Peco: Perceptual codebook for hert pre-training of vision transformers. arXiv preprint arXiv:21 I. 12710, 2021.
[17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbor,, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Hcigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 19999, 2020.
[18] S. d' Ascoli, H. Touvron, M. L. Leavitt, A. S. Morcos, G. Biroli, and L. Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pages 22862228626000222 2296. PMLR, 2021.
[19] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages
6824-6835, 2021.
[20] G. Ghiasi, Y. Cui, A. Srinivas. R. Qian, T.-Y. Lin, E. D. Cubuk, Q. Le, and B. Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IRFF/CVE Conference on Computer Vision and Pattern Recognition, pages 2918-2928, 2021.
[21] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference Learning Repreventation, 2018.
[22] R. Girshick, J. Donahue, T. Darrell, and Malik. Rich feature hicrarchies for accurate object detection and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 580-587, 2014.
[23] I. Goodfellow, J. Douget-Abadic, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680. 2014.
[24] B. Graham. Sparse 3d convolutional neural networks. arXiv preprint arXiv: 15X5.02000 2015.
[25] B. Graham, M. Engelcke, and Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018
pages
[26] J.-B Grill, F. Struh, F. Altché, C. Tallec, P. Richemond, F. Buchatskaya, C. Doersch, Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33, 2020.
[27] J. Guo, K. Han, H. Wu, Y. Tang, X. Chen, Y. Wang, and C. Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12175-12185 2022.
[28] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
11

---

[29] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020. [30] K. Hc, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961 22692299,,440 2017.
[31] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[32] O. J. Hénaff. S. Koppula, J.-B. Alayrac, A. v. d. Oord, O. Vinyals, and J. Carreira. Efficient visual pretraining with contrastive detection. arXiv preprint arXiv:2103. 10957. 2021.
[33] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Newral Information Processing Systems, 33:6840-666 6801, 2020.
[34] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, pages 646-661. Springer, 2016.
[35] I.. Huang, S. You, M. Zheng. Wang, C. Qian, and T. Yamasaki. Learning where to learn in cross-view self-superrised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14451-14460, 2022.
[36] L. Huang, Y. Yuan, J. Guo, C. Zhang, X. Chen, and J. Wang. Interlaced sparse self-attention for semantic segmentation. arXiv preprint 1907. 12273, 2019.
[37] L. Huang, C. Zhang, and H. Zhang. Self-adaptive training: Bridging supervised and suff-supervieed learning. arXiv preprint arXiv:2101.08732 2021.
[38] Z. Huang, Ben, G. Luo, P. Cheng. G. Yu, and B. Fu. Shuffle transformer: Rethinking spatial shuffle for vision transformer. arXiv preprint arv::r2066...... 03650, 2021.
[39] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4401 -4410, 2019.
[40] H. Kellerer, U. Pferschy, and Pisinger. Knapsack problems. Springer, 2004.
[41] D. Kingma and Ba. Adam: A method for stochastic optimization. In International Conference on Learning Reprexentations, 2015.
[42] D. P. Kingma and M. Welling, Auto-encoding variational Bayes. In International Conference on Learning
Representations, 2014.
[43] A. Krizhevsky, I. Sutskever, and G. F.. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, volume 25, pages 1097-1105, 2012.
[44] G. Larsson, M. Maire, and G. Shakhnarovich. Colorization a proxy task for visual understanding. In Proceedingy of the IEEE conference computer vision and pattern recognition, pages 6874-6883, 2017.
[45] Y. Li, S. Xie, X. Chen, P. Dollar, K. Hc, and R. Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.
2021.
[46] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2117-2125, 2017.
[47] T.-Y. Lin. M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects context. In European conference on computer vision, pages 740-755. Springer. 2014.
[48] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint 2rXiv:21111 11.22111..988 1.09883, 2021.
[49] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10023 2021.
[50] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
[51] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston,
O. Kuchaiev, G. Venkatesh, al. Mixed precision training. arXiv preprint 17Xiv:1710.03740, 2017.
[52] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69 84. Springer, 2016.
[53] A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 807.0037487.347 2018
12

---

[54] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan. T. Killeen, Z. Lin, N. Gimelshein. L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035, 2019.
[55] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In JEEE Conference on Computer Vision and Pattern Recognition, pages 2536-2544 2016.
[56] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018.
[57] A. Radford. J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[58] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pages 8821-8831. PMLR, 2021.
[59] Y. Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34, 2021.
[60] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, 7. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
[61] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni. Green ai. Communications of the ACM, 63(12):54-33, 2020.
[62] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[63] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[64] A. Vaswani. N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[65] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning, pages 1096-1103, 2008
[66] W. Wang, E. Xie, X. Li, D.-. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 368-578, 2021.
[67] X. Wang, R. Zhang, C. Shen, T. Kong, and L. Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[68] C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint a1X.v221 2021.
[69] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang. Cvt: Introducing convolutions to vision transformers, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22-31, 2021.
[70] Z. Wa, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[71] T. Xiao, C. J. Reed, X. Wang, K. Keutzer, and T. Darrell. Region similarity representation learning, arXiv preprint arXiv: 12902, 2021.
(72) Z. Xie, Y. Lin, Z. Zhang, Y. Cao, S. Lin, and H. Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16684-16693, 2021.
[73] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao. Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:211 S.di:2111.008888 2021.
[74] J. Xu, W. Zhou, Z. Fu, H. Zhou, and L. Li. A survey on green deep learning. arXiv preprint 1 2021.
[75] H. Yin, A. Vahdat, J. Alvarez, A. Mallya, J. Kautz, and P. Molchanov. Adavit: Adaptive tokens for efficient vision transformer. arXiv preprint arXiv:21 1tf::22/2..07658 2021.
[76] W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang. J. Feng, and S. Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 10819-10829, 2022.
13

---

[77] Y. Yuan, R. Fu, L. Huang. W. Lin, C. Zhang. X. Chen, and J. Wang. Hrformer: High-resolution vision transformer for dense predict. Advances in Neural Information Processing Systems, 34, 2021.
[78] J. Zbontar, L.. Jing, Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy 2021.
reduction. arXiv preprint arXiv:2103,00
[79] Zheng, Wang. S. You. C. Qian, C. Zhang, X. Wang, and C. Xu. Weakly supervised contrastive
learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
10042-10051, October 2021.
[80] Zheng. S. You, Wang, C. Qian, Zhaan, X. Wang, and C. Xu. Ressl: Relational self- supervisee learning with weak augmentation. Advances in Neural Information Processing Systems, 34, 2021.
[81] J. Zhou, C. Wei, H. Wang. W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint (xxx:2111.0088.. 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the supplemental materials.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In the supplemental materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xgcy7J31I-", "reviewer": "Reviewer_Yf6s", "summary": "In this paper, the authors propose a method to perform MAE style pre-training on Swin transformers. The main idea is to dynamically put the visible patches into groups and then perform attention within each group. To ensure that information doesn't \"leak\" out of each window, attention masking is performed in each group. This method can be efficient because of the Swin structure -- attention can only happen within a window, so the size of each group can be small and well-packed. The grouping is solved by dynamic programming to optimize FLOPs. The authors evaluate this approach on ImageNet and COCO and show that this method is efficient while being competitive in accuracy.", "questions": "1. I wonder if the authors have any intuition on why the accuracy of SimMIM is slightly higher than the proposed method.\n2. I wonder if MIM on Swin works well with a larger backbone (e.g., Swin-L, Swin-H, etc.). I know that this might not be the main focus of this paper, but if MIM doesn't look promising on Swin, then speeding this up is less important. Conversely, if MIM on large Swin models works very well, then the importance of this work will be large.", "limitations": "I think the main limitation is that the method focuses on Swin Transformers only, but not other hierarchical models.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The 2x speedup looks promising and useful. Given the wide use of Swin transformers, an efficient way to pre-train with SSL can be quite impactful. \n+ The idea to perform grouping for fast MAE on Swin is interesting and novel. \n+ The idea to optimize grouping based on FLOPs with DP is simple but seems to work well. \n+ The accuracy on COCO suggests MIM+Swin is likely a good idea. \n\n\nWeaknesses:\n- The accuracy on ImageNet is a bit disappointing: Swin+MIM trained for 900 epochs obtains 83.7, while Swin-from-scratch trained for 300 epochs already obtains 83.5. Looking at ImageNet results alone, it's not clear if we need MIM on Swin.\n- The title and writing overall suggests that the method works on general hierarchical ViT models, but in reality, the authors only show that the method works on Swin transformers. It's not just lack of experiments, it's not even clear how one can apply this approach to other hierarchical models (e.g., MViT). \n- The experiments are limited to <=800 epoch training. Given that MIM is usually more useful in long-training regime, missing >800 epoch experiments make it harder to compare to MAE. \n- How much time does the DP solver take? How much time does the grouping, masking, or ungrouping take? Some more detailed analysis would be helpful. \n- (minor:) In figure 1, without accuracy, it's hard to appreciate the method by speed only.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "wqbox9KFCB", "reviewer": "Reviewer_RDo2", "summary": "This paper presents a cost-effective approach for masked image modeling (MIM) pretraining with hierarchical vision transformers. MIM has been proved effective for vision transformer pretraining, but hierarchical ViT faces a challenge in MIM pretraining that both visible and masked patches have to be involved, which greatly decreases the pretraining efficiency. This paper presents a solution which reduces the pretraining cost of Swin Transformer by around half while achieving on par performance. ", "questions": "Please refer to the previous section. ", "limitations": "No potential negative societal impact is identified. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper addresses a relevant problem with a technically sound solution. The cost of MIM pretraining for hierarchical ViTs is indeed a painpoint in vision backbone learning. The proposed group window attention scheme is reasonable, and the evaluation on Swin Transformer is c. \n\nThe paper claims that the proposed approach is for hierarchical ViTs, but the experiments are carried out only for Swin Transformer. It would be more convincing if the authors could show the results for at least another ViT. Or, the authors should consider not to claim the proposed approach to be generally applicable to hierarchical ViTs. \n\nThe group window attention algorithm solves an optimization problem to determine the group size and the number of groups. The sole optimization target is the computation cost, leaving the pretrain quality completely unconsidered. Does the selection of group size affect the pretrain quality?\n\nThe authors use dynamic programming to solve the group partition problem, but does a simple and heuristic algorithm, such as g_s=max{w_i}, already suffices? The complexity of the dynamic programming algorithm is not well justified/ablated. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Nw8YQrjs_Y", "reviewer": "Reviewer_dAsW", "summary": "The paper first introduces to properly apply the MAE pre-training method on Swin Transformer backbone architecture. To adapt the visible-only encoder with local window attention in Swin, the paper proposes a new Group Window attention mechanism with Dynamic Programming to automatically tune the group size and local windows. The introduced method significantly reduces the pre-training hours of SimMIM on Swin Transformer with marginal performance drops.", "questions": "I like the motivation and story of this work but would like to see more experiments that would make it more solid. Please see Weaknesses for details.", "limitations": "The authors discussed the limitation of the batch-wise masking scheme in the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Writing: The paper is generally well written and easy to follow. The story of green AI is interesting.\n+ Contribution: It is significant to well apply the MAE-style pretraining method on hierarchical Vision Transformer architectures, especially benefitting dense prediction downstream tasks, e.g., detection and segmentation. It is the first work to investigate this problem so far as I know.\n\nWeaknesses:\n\nI am generally satisfied with this work but have several concerns about the experiments:\n1. **Marginal improvements:** The improvements over training from scratch on ImageNet is marginal, i.e., 83.5 vs. 83.7. The pretraining hours are comparable (840 for training from scratch and 887 for this work), it may raise a question that if 83.7 can also be achieved by training from scratch for 887 hours. By the way, why not compare to SimMIM_224 on Swin-Base?\n2. **Lack of ablation studies:** (i) I think it is important to implement this method on the vanilla MAE with ViT-Base to show the performance gaps caused by group window attention. (ii) I would like to see if the performance can be further improved with more training epochs. For example, for 800 epochs, SimMIM_192 takes double the training hours than this work but +0.3% performance gains. I want to know if this method can also achieve 84.0% with double training hours (800ep -> 1600ep).\n3. **More model scales:** It is better to show the power of this method on other sizes of Swin Transformer, e.g., small, large, etc.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept