HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer
Authors: Xiaosong Zhang
Yunjie Tian
Lingxi Xie
Wei Huang
Qi Dai
Qixiang Ye
Qi Tian
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Hierarchical vision transformers
self-supervised learning
masked image modeling

Abstract: There has been a debate on the choice of plain vs. hierarchical vision transformers, where researchers often believe that the former (e.g., ViT) has a simpler design but the latter (e.g., Swin) enjoys higher recognition accuracy. Recently, the emerge of masked image modeling (MIM), a self-supervised visual pre-training method, raised a new challenge to vision transformers in terms of flexibility, i.e., part of image patches or tokens are to be discarded, which seems to claim the advantages of plain vision transformers. In this paper, we delve deep into the comparison between ViT and Swin, revealing that (i) the performance gain of Swin is mainly brought by a deepened backbone and relative positional encoding, (ii) the hierarchical design of Swin can be simplified into hierarchical patch embedding (proposed in this work), and (iii) other designs such as shifted-window attentions can be removed. By removing the unnecessary operations, we come up with a new architecture named HiViT (short for hierarchical ViT), which is simpler and more efficient than Swin yet further improves its performance on fully-supervised and self-supervised visual representation learning. In particular, after pre-trained using masked autoencoder (MAE) on ImageNet-1K, HiViT-B reports a 84.6% accuracy on ImageNet-1K classification, a 53.3% box AP on COCO detection, and a 52.8% mIoU on ADE20K segmentation, significantly surpassing the baseline. Code is available at https://github.com/zhangxiaosong18/hivit.

Published as a conference paper at ICLR 2023
HIVIT: A SIMPLER AND MORE EFFICIENT DESIGN OF HIERARCHICAL VISION TRANSFORMER
Xiaosong Zhang
Yunjie Tian
Lingxi Xie?
Wei Huang'
Qi Dai
Qixiang Yel.3
Qi Tian
'UCAS
Huawei Inc.
3 Peng Cheng Lab
(zhangxiacsong18; tianyunjiel9, huangweil www.:eeii1emmmmllemmmemmmmmammme www.uccacccooo ac {198808xc, baigi1989)@ggail. qxyelucas ac. on tian. well@hhuwwe.coom com
ABSTRACT
There has been debate on the choice of plain vs. hierarchical vision transform- ers, where researchers often believe that the former (e.g., ViT) has simpler de- sign but the latter (e.g., Swin) enjoys higher recognition accuracy. Recently, the emerge masked image modeling (MIM), a self-suuer pre-training method, raised new challenge to vision transformers in terms of flexibility, i.e., part of image patches or tokens are to be discarded, which seems to claim the advan- tages of plain vision transformers. In this paper, delve deep into the compar- ison between ViT and Swin, revealing that (i) the performance gain Swin is mainly brought by deepened backbone and relative positional encoding, (ii) the hierarchical design of Swin can be simplified into hierarchical patch embedding (proposed in this work), and (iii) other designs such as shifted-window attentions can be removed. By removing the unnecessary operations, we come up with a new architecture named HiVIT (short for hicrarchical ViT), which simpler and more efficient than Swin yet further improves its performance on fully-supervised and self-supervised visual representation learning. particular, after pre-trained using masked autoencoder (MAE) on ImageNet-I K, Hi HiVIT-B reports 84.6 44.6% ac- curacy on ImageNet- IK classification, a 53.3% box AP on COCO detection, and 52.8% mloU ADE20K segmentation, significantly surpassing the baseline. Code is available at https::///// httpp://github. com/ wttp.//aaaaaiaaoooog.oaaaaa 8/hiiii
INTRODUCTION
Deep neural networks (LeCun et al., 2015) have advanced the research fields of computer vision, natural language processing, etc., in the past decade. Since 2020, the computer vision community has adapted the transformer module from natural language processing (Vaswani et al. 2017 Devlin ctt al. 2019) to visual recognition, leading to a large family of vision transformers Dosovitskiy al.)
2021 Liu et al. 202T Wang et al. 202 Zhou et al. 2021 Dai et al., [021 Li et al. 2021a that replaced the dominance of convolutional neural networks (Krizhevsky et al. 2012 He et al. 2016
Tan & Le 2019). They have the ability of formulating long-range feature dependencies, which naturally benefits visual recognition especially when long-range relationship is important.
There mainly two families of vision transformers, namely, the plain vision transformers Doso- vitskly et al. 2021 Touvron et al., (021) and the hierarchical vision transformers (Liu et al. 2021
Wang al.) 2 Dong al.] 2021a Chen al., 2021a). differing from each other in whether multi-resolution features are used. Intuitively, visual recognition requires hierarchical information and the hicrarchical vision transformers indeed show superior performance. However, the hierar- chical vision transformers have introduced complicated and asymmetric operations, e.g., Swin (Liu al. 202T) used regional self-attentions with shifted windows, hence, they encounter difficulties
"Equal Contribution.
1 "Correponndig Author.

---

Published as a conference paper at ICLR 2023
when the tokens need to be flexibly manipulated. A typical example lies in masked image modeling (MIM), a recent methodology of visual pre-raining (Bao et al.) 2021] He et al. 2021 Xie et al, 2021b), in which a random subset of image patches are masked from imput and the model learns by reconstructing the masked contents. In such a circumstance, the plain transformers (e.g.. ViT) can directly discard the masked tokens, while the hierarchical vision transformers (e.g.. Swin) must feed the entire image (with the masked patches filled with dummy contents) into the encoder (Xie et al.) 2021b), slowing down the training procedure and contaminating the original data distribution.
This paper tries to answer the following question: is it possible to design an alternative vision trans- former that enjoys both the flexibility of plain models and the representation ability of hierarchical models? We start with ViT and Swin, the most popular plain and hierarchical models. We design a path that connects them, with each step only changing single design factor. The modifications include (a) increasing network depth, (b) adding relative positional encoding, (c) adding hierarchi- cal patch embedding, (c') adding shifted-window attentions (an alternative to (c)), and (d) adding the stage 4S We find that (a)( are the main factors that contribute to visual recognition, while (c") shall be replaced by (c) and (d) can be discarded. In particular, the window attentions were designed to reduce the computation of self-attentions in the high-resolution (i.e. low-level) feature maps, but we find that, under a sufficient network depth (e.g., Swin-B B used 24 transformer blocks), the low-level self-attentions only have marginal contribution and can be removed.
Based on the analysis, we present a hierarchical version of VIT named HiViT. Following (a)())C) discussed above, the modification beyond the original VIT is minimal. At the base level, the archi- tecture has 24 transformer blocks (the number of channels is reduced) where the first 4 appear as hierarchical patch embedding that replaces the plain counterpart and the others are equipped with relative positional encoding- - one needs only a few lines of code to replace ViT with HiViT.
The superiority of HiViT is validated using two sets of experiments. We first perform fully-supervised image classification on 1K (Deng jet al.j 2009), where HiViT enjoys consistent accuracy gains over both ViT and Swin, e.g., at the base level, HiViT-B reports a 83.8% top-1 accu- racy, which is +200% over ViT-B and +0.3% over Swin-B, and the number of learnable parameters is about 1/4 fewer than both competitors. Con- tinuing to MIM, the advantages of HiViT persist. Table [ shows the comparison on model size, training speed, and recognition accuracy. Under the MAE framework (He et al.) (021), with 1600 epochs of pre-training and 100 epochs of fine-tuning, HiViT-B reports 84.6% top-1 accuracy on ImageNet- IK, which is 1.0% over ViT-B (trained with MAE) and +-066% over Swin-B (trained with Sim- MIM (Xie et al. 202 More importantly, HiViT enjoys the efficient implementation that discards all masked patches (or tokens) at the input stage, and hence the training procedure is as simple and efficient applying MAE on VIT. The pre-trained models also show advantages on downstream tasks, including linear probing (a 71.3% top-1 accuracy on ImageNet-IK) semantic segmentation (52.8% mIoU on ADE20K (Zhou al., 2017) ()). and object detection and instance segmentation (a 53.3% box AP and a 47.0% mask AP) on COCO (Lin et al. 2014a) under the 3x training schedule).
Table 1: Compared ViT and Swin, HiViT is faster in pre- training, needs fewer parameters, and achieves higher ac- curacy, All numbers in % are reported by pre-training the model using MIM (ViT-B and HiViT-B by MAE and Swin- B by SimMIM) and tie-ttnnng it to the downstream data. Please refer to experiments for detailed descriptions.
Architecture
VIT-B
Swin-B
HiVIT-B
Params (M)
86.6
88.0
66.4
FLOPPs (G) Pre-training time
17.5
15.4
15.9
8.8
14.3
10.3
per epoch (mins)
ImageNet-IKK acc. (%)
83.6
84.0
84.6
COCO Ap (%)
51.2
52.3
53.3
ADE20K mloU (%)
48.1
52.8
52.8
Overall, the core contribution of this paper is HiViT, hierarchical vision transformer architecture that is off-the-shelf for a wide range of vision tasks. In particular, with MIM being a generalized paradigm for self- superviede visual representation learning, HiViT has the potential of being directly plugged into many existing algorithms to improve their effectiveness and efficiency.
'Stage refers to components processing the same resolution in hierarchical models. In this paper, stage 2, and 4 respectively refer components processing 562, 28 14?, and 72 resolutions in image classification.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Vision transformers (Dosovitskiy et al. 2021) were adapted from the natural language process- ing (NLP) transformers (Vaswani et al. 2017] Devlin et al. (019)). opening a direction of designing vision models with weak induction bias (Tolstikhin al. 2021]. Early vision transformers (Dosovit- skiy et al. 2021) mainly adopted the plain configuration. and efficient training methods are strongly required Touvron et al. 2021}. To cater for fision-rrenndy priors, Swin Transformer (Liu et al, 2021] proposed a hierarchical architecture that contains multi-level feature maps and validated good performance in many vision tasks. Since then, various efforts emerged in improving hicrarchical vision transformers, including borrowing design experiences from CNNs (Wang et al., |022] Wu 2021; Vaswani et al.] 202T), adjusting the design of self-attention geometry (Dong et al.
ct al.
202 la
Yang al. 2021 and integrating convolution and transformer modules (Srinivas et al.
2021
Gao et al. 2021 Dai et al.) 2021: Peng et al. 2021 Xiao et al. 2021
Guo et al. 2022 Ali
al.
2021
Pan et al. 2022), etc.
In the context of computer vision, self-supervised pre-training aims to learn compact visual rep- resentations from unlabeled data. The key is to design a pretext task that sets a natural constraint for the target model to achieve by tuning its weights. Existing pretext tasks are roughly partitioned into three categories, namely, geometry-based proxies that were built upon the spatial relationship of image contents (Wei et al., 2019; Noroozi & Favaro 2016 Gidaris et al. 2018),
contrast-based proxies that assumed that different views of an image shall produce related visual features (He al.,
2020 Chen et al., [2000) Grill et al. 2020 Caron et al. [02T]] 2020 Xie et al. 2021a Tian et al.
proxies that required visual representations to capable of recovering the original image contents (Zhang et al. [2016] Pathak et al. 2016 He et al. 202T |ao et al.) 2021 Tian et al. 2022). After the self-supervised learning (a.ka. pre-training) stage, the target model often evaluated by fine-tuning in downstream recognition tasks the popular examples include image classification (Deng et al. 2009), semantic segmentation (Zhou et al. 2017). object detection and instance segmentation (Lin et al. 20144), etc.
2021, J. and generation-base
We are interested in a particular visual pre-trat mne-trining method named masked image modeling (MIM) (Bao et al. 2 He al. 2021). The flowchart is straightforward: some image patches (corresponding to tokens) are discarded, the target model receives the incomplete input and the goal is to recover the discarded patches. MIM is strongly related to the masked language modeling (MLM) task in NLP. BEiT (Bao et al.) 2021) transferred the task to computer vision by masking the image patches and recovering the tokens produced by pre-trained model (knwon as the tokenizer).
MAE (He et al. 2021 improved MIM by only taking the visible tokens as input and computing loss at the pixel level the former change largely accelerated the training procedure as encoder's computational costs went down. The follow-up works explored different recovery targets (Wei et al.) 202T), complicated model designs (Fang et al. 2022 Chen et al. 2020), and other pretext tasks.
Our insights. It is worth noting that MIM matches plain vision transformers very well because each token is an individual unit and only the unmasked tokens are necessary during the pre-training process. The properties does not hold for hierarchical vision transformers, making them difficult to inherit the good properties (e.g., training efficiency). Although SimMIM (Xie et al. 2021b) tried to combine Swin Transformer with MIM, it uses all tokens, including those corresponding to the masked patches, shall be preserved during the encoder stage, incurring much heavier computational costs. In this paper, we design a hierarchical vision transformer that (i) integrates the advantages of VIT and Swin series and (ii) better fits MIM in terms of accuracy and speed.
3 HIVIT: SIMPLE AND EFFICIENT HIERARCHICAL VISION TRANSFORMER
3.1 PRELIMINARIES: VISION TRANSFORMERS AND MASKED IMAGE MODELING
We start with the vanilla plain vision transformer, abbreviated as VIT (Dosovitskiy et al. 2021 Mathematically, let the target model be fxx 0) where denotes the learnable parameters. A training image is first partitioned to a few patches, X' {xi, X . }M). where M is the number of patches. In ViT, each image patch is transferred into a token (i.e., a feature vector), and the tokens are propagated through a few transformer blocks for visual feature extraction. Let there be T. blocks, where the 1-th block takes the token set of (((1-)) as input and outputs 2/(), and
3

---

Published as a conference paper at ICLR 2023
Embedding Visible Patcles
MLP
Mersing
Marging
x
VIT-B Acc. 81 8%
HIVIT-B (Ours)
Swin-B stages) Acc. 83 83.4%
Swin-B Acc, 83.5%
Acc. 83.8% Impul
Input Enbedding. 16 16
Inpul
Ingut
Embeskingg
Embeedingg x
Embeddingg, 4 x
Window
Window
56
x
MLP. nitoo33
MLP nationi
MLP. rations
Merging,
Merging,
Merging,
Window Atteation
28 x 28
x
x 2
x
MLP. miioo33
MIP: national
rationd
Merging, 2x
Merging, 2 2
Margin, 2
Global Amention
Global Artention
Global Attenionn
Amenttinn
14
x 12
x 20
x 20
18
Meginn, 2
7 x
x
MLP rationl
Output
Outpus
Output
Output
Figure 1: Top: hicrarchical patch embedding, the way that HiVIT uses for low-level feature ma- nipulation (see the dashed box in the HiViT-B architecture). Note that the first step (i.e. patch embedding) involves embedding small (4 4) patches unlike ViT that directly embeds 16 x 16 patches and thus loses rich spatial information. Bottom: the transition from VIT to Swin, with HiViT appearing in the midst. Please refer to Table for details and results.
4 X. The main part of each block is self-attention, for which three intermediate features computed upon um ((-1), namely, the query, key, and value, denoted as q (!-1), k((l-)), and v-((-1) respectively. Based on these quantities, the self-attention of Zm (1-1) is computed by SA Zm ((-1)) =
where 1/ 1/ Dayy is a scaling vector. Auxiliary operations, including layer normalization, multi-layer perceptron, skip- layer connection, are applied after the self-attention computation.
softmax (1-1) ((-1)
qm
q
(1-1). (1-1) /VDiey
Dayy key
VM (1-1)
VIT is simple yet effective on vision tasks, but lacks the ability of extracting hierarchical visual features as most CNNs (Simonyan & Zisserman 2013 He al. 2016 Tan & Le 2019) have. Swin Transformer (Swin for short) is a popular variant over VIT that allows the spatial resolution to reduce gradually throughout the backbone, and avoids high computational costs with shifted- window attentions. Since then, there has been competition between ViT and Swin Swin showed initial advantages on detection and segmentation, yet ViT-based models managed to catch up with better implementation details (Li al. 2022a).
The emerge of masked image modeling (MIM) becomes a new and important decider for the com- paradigm of self-supervised visual representation learning that involves
petition. MIM is feeding a partially masked image to the target model f(x; 0) and training the model to recover it. Following the above notations, MIM randomly chooses subset M' (1,2...., M). feeds the
4

---

Published as a conference paper at ICLR 2023
patches with IDs in M (denoted as X') into f(; 0) (a.k.a,, the encoder), and appends auxiliary decoder to it, aiming at recovering the masked image contents. The objective is either computed on tokenized features (Bao et al., 2021) pixels (He et al. 202T In the context of MIM, ViT enjoys an efficient implementation that the tokens not in X" can be discarded at the beginning of encoder, while Swin must feed the entire input image, with the masked patches filled with dummy contents, into the encoder (Xie et al., 202 In comparison, Swin not only requires heavier computation of encoder, but also contaminates the original image data distribution with the dummy tokens.
3.2 TRANSITION FROM VIT TO SWIN: SEEKING FOR REALLY USEFUL OPERATIONS
We look forward to a variant of vision transformers that keeps the ability of learning hierarchical visual representations yet gets rid of the burdens in manipulating the MIM task. For this purpose, we inherit the methodology used in Chen et al. (202 ld) to gradually transit a ViT-B model into a Swin-B model. We allow only one modification at each step that can distinguish useful operations. The intermediate models are illustrated in Figure [
A hierarchical transformer calls for efficient function to deal with high-resolution feature maps. We investigate two possibilities here. One the solution offered by Swin that computes shifted- window attentions note that these operations actually banned the efficient implementation of MIM as the masked patches can no longer be discarded from input. Our solution termed hierarchical patch embedding, Figure (upper), solely involves MLP and patch merging operations while does not involve either window attention or overlapped convolution. In patch merging operations, we combine 2x2 features along the channel dimension and introduce a full-connect layer (with learn- able parameters) to convert them to a proper number of channels. As shown below, our solution achieves higher accuracy yet requires lower computational costs. More importantly, avoids infor- mation exchange across Stage 3 tokens, which brings the efficient implementation of MIM back if the masking operation is performed on Stage 3.
Table 2:
ImageNet-IK classification and ADE20K segmentation results of the models along the transition process, from VIT-B to Swin-B. "RPE" indicates the relative position embedding, "Low Att." " indicates whether Window Attention is used in Stage and Stage 2, "Mid Att." indicates whether Window Attention is used in Stage 3, and "T.P." is short for throughput.
Module #
Low Mid Params FLOPs Att.
INIK Acc.
ADE
Model
Dims RPE
T.P.
562
28
14a
7
Att.
mloU
(o) ViT-B
12
768
X
X
X
86.6M
17.5G
292
81.8
45.5
(a)
(o) Depth
24
512
X
76.7M
15.8G
317
82.9
48.1
(b)
(a) RPE
24
512
76.8M
15.8G
310
83.2
48.6
(c)
(b) Hierarchical
2
20
512
66.4M
15.9G
286
83,8
49.5
(c') (b) Low Att.
2
20
512
v
V
X
66.3M
16.0G
275
83.4
47.9
(d) (c') Stage 4
2
18
2
512
V
V
87.8M
15.5G
278
83.5
48.0
We evaluate the models on ImageNet-IK IK classification and ADE20K segmentation, where the de- tailed settings are elaborated Sections 4.1 and 4.3] respectively. For these models, the classifica- tion accuracy, the number of parameters and FLOPs are listed in Table We summarize the impact of each step as follows, where (%, %) stand for the accuracy gains compared to the prior step on ImageNet- 1K and ADE20K, respectively.
Step (a) (+1.1%, +2.6%) by using deeper architecture with fewer channels. This factor tributes the largest advantage of Swin-B.
Step (b) (+0.3%, +0.5%) by replacing vanilla positional encoding with relative positional encoding (RPE). This small but consistent gain validates the effectiveness of RPE.
Step (c)
(+0.6%, +0.9%) by introducing hierarchical patch embedding to replace the first blocks.
The considerable gains validate the effectiveness of hierarchical patch embedding.
Step (c') (+0.2%, -0.7%) by introducing high-resolution (56x56 56 and 28x28) token maps and adding 7x7 shifted-window attentions. As an alternative solution (used by Swin) to Step (c), it causes significant ADE20K segmentation accuracy and Compared to Step (c), conclude that hierarchical patch embedding is more effective and efficient than lowelevel transformer blocks (with hifteed-winoww attentions).
5

---

Published as a conference paper at ICLR 2023
Attention Varfance
66.5
4
2e-5
36 Layers
Layers
(a) Vanilla VIT with 12 layers
(h) Deepened ViT with 24 layers
Figure 2:
As measure of instability of self-attention, compute the variance of attention between models trained from scratch, starting from N = 3 different random seeds.
Step (d) (+0.1%, +0.1%) by adding Stage (2 blocks with 7x7 token maps), which involves (i) reducing the number of blocks at Stage 3 by 2 (moving them to Stage 4) and (ii) modifying Stage 3 from global attentions to shifted-window attentions. This operation increases the parameter count by 33% because Stage 4 contains 8x-dim channels while Stage 3 only has 4x-dim channels. Expect for the little gain, this operation this friendly to MIM since we take Stage (with 14 x 14 tokens) as the playground of masking patches.
From the above, learn the lesson that, besides RPE, Swin mainly benefits from using deeper architecture and introducing hierarchical patch embedding. Other operations, including window attentions and the Stage 4 with 7 x 7 tokens, do help visual recognition. Before entering the next part, we would like to offer some side notes. If Step (a) is not performed before Step (b), i.e., the first 4 layers of a 12-layer ViT-B (not 24-layer deepened one) is replaced by hierarchical patch embedding. the performance drops. This fact shows that early atfentions can be replaced by hierarchical patch embedding only when the model deep enough.
To diagnose the above statement, we design an experiment on two ViTs. The first is a vanilla ViT with 12 layers and a channel number of 768, and the second is a deepened ViT with layers and 512 channels. For each architecture, we train N models individually from scratch (fully-supervised on ImageNet-IK). After that, we randomly sample 1,000 images from the ImageNet- IK test set, and compute the intermediate output at each transformer block for each image each block, it contains 12 (the head number) attention weight maps sized 196 x 196 (196 = 14 x 14 the token number). We first average the 12 maps into one (for each individual model), and then compute the variance of weights at each position, and finally average the 196 x 196 variances into a single number, indicating whether the attentions computed at this image and this block stable in other words, both the vanilla and deepened ViT have 1,000 variances (for 1,000 images) at each block.
In Figure 2 one attention map at each block is small, indicating that the models have learned stable attention patterns. Some outliers show large variance in deep layers, which imply that inter-patch information is no longer needed in deep layers. In the deeper ViT, for almost all samples, attention maps of the shallow layers have much higher v indicating that these layers do not learn reliable attention maps. These observations show that early layers cannot learn effective attentions in deeper architecture, hence removing the shallow window attentions in HiViT has little negative impacts.
that, in the vanilla VIT, for most samples, the variance of the average
3.3 HIVIT AND ITS APPLICATION ON MIM
Summarizing the above analysis, we obtain HiVIT, the halfway architecture when Steps (a)-(c) are performed while others are discarded. An illustration of HiViT-B is shown Figure [ In fully- supervised ImageNet-1K classification, it achieves higher accuracy than other variants using fewer learnalee parameters.
"Exceptions happen in the first two layers, where the 16 x 16 patch emhedding just brings in initial patch features, and find that the attentions less meaningful though the variances small.

---

Published as a conference paper at ICLR 2023
Table 3: Fully-supervised results ImageNet-IK. The throughput of models reported
single Tesla-VI00 GPU, indicates a stronger baseline of VIT proposed in He et al. (202I).
Throughput (image
ImageNet-IK Top-1 Acc.
Model Type
Model
image size
Params
FLOPs
EffNet-B3
300
12M
1.8G
732
81.6
EffNet-B4
380
19M
4.2G
349
82.9
ConvNets
EftNet-B5
456
30M
9.9G
169
83.6
EliNet-B6
528
43M
19.0G
97
84.0
EffNet-B7
600
66M
37.0G
55
84.3
DeïT-S
224
22M
4.5G
940
79.8
Plain
DeïT-B
224
86M
17.5G
292
81.8
Transformers
VīT-B
384
86M
55.4G
86
77.9
ViT-Bi
224
86M
17.5G
292
82.3
CoAtNet-0
224
25M
4.2G
81.6
CoAtNet-1
224
42M
8.4G
83.3
CoAtNet-2
224
75M
15.7G
84.1
Transformers w/ Convolution
MViTv2-T
224
24M
4.7G
82.3
MViTv2-S
224
35M
7.0G
83.6
MViTv2-B
224
52M
10.2G
84.4
CSWin-T
224
23M
4.3G
701
82.7
CSWin-S
224
35M
6.9G
437
83.6
CSWin-B
224
78M
15.0G
250
84.2
Swin-T
224
28M
4.5G
755
81.2
Swin-S
224
50M
8.7G
437
83.1
Hierarchical
Swin-B
224
88M
15.4G
278
83.5
Transformers
HiViT-T (ours)
224
19M
4.6G
850
82.1
HiViT-S (ours)
224
38M
9.1G
436
83.5
HiVIT-B (ours)
224
66M
15.9G
286
83.8
More importantly, find that HiViT fits MIM very well. To reveal this, show example by following the convention to partition the entire image into 14 x 14 patches each patch corresponds to token Stage 3. Since (i) Stages and 2 modified that they do not contain inter-token operations and (ii) Stage 4 is removed so that all the tokens in Stage 3 are symmetric, we can use simple implementation of MIM where all masked patches directly discarded from input. As we shall see in experiments, this implementation not only accelerates the self-supervised learning process by nearly 100%, but also brings higher accuracy various downstream tasks. We conjecture that part of the improvement comes from that we have cleaned input data to avoid dummy patches so that the gap between upstream and downstream image data becomes smaller.
EXPERIMENTS
We build our base model, called HiViT-B, to have computation complexity similar Swin-B but slightly smaller than ViT-B. We also introduce HiViT-T, HiViT-S and HiViT-L, which are versions of about 0.25x, 0.5x and 2x the computational complexity of HiViT-B, respectively. We first conduct fully supervised experiments with labels using the proposed HiViT on ImageNet- 1K dataset (Deng al., 2209). Then, HiViT models tested using masked image modeling supervised methods (MIM) (ee et al., 2021). The pre-trained models are also transferred downstream tasks including object detection on COCO (Lin et al. 20144) and semantic segmentation on ADE20K (Zhou al, 2017). The detailed experimental settings elaborated in Appendix
4.1 FULLLY-SUPERVISED IMAGE CLASSIHCCATIN
ImageNet Results. The fully supervised training results are shown in Table 3 Compared to the vanilla ViT models, the HiVIT variants report much better results in terms of classification accuracy

---

Published as a conference paper at ICLR 2023
and computational complexity. In particular, HiViT-T/B surpass DeiT-S/B (ViT equipped with better training strategy) by 2.3% and 2.0%, respectively, with similar computational costs. Com- pared to the hierarchical baseline (i.e. Swin), HiViT still shows competitive performance, with HiViT-T/S/B beating Swin-T/S/B by 0.9%, 0.4%, and 0.3%, respectively, with similar computa- tional costs. In addition, by removing shallow attentions and Stage (with large channel numher), HiViT-T/S/B requires 32.2%, 24.4% and 24.4% fewer parameters compared to Swin-T/S/B models. Hence, HiViT is simple architecture that does not require further optimization tricks beyond Swin. We will see its generalization ability in later experiments.
We note that some recent variants of vision transformers (e.g. CSWin (Dong et al, 202 and MViTv2 (Li et al. (022b)) achieved higher accuracy on ImageNet-IK, I.. It should be noted that these models involved convolution operations throughout the network. The convolution operations hinder serializing the input, making them difficult to be applied to masked image modeling. We classify these models into the category of mixed models (transformers w/ convolution) in Table 3 Although HiVIT is inferior to these architectures in fully-supervised ImageNet classification, that HiVIT better benefits from pre-training. For example, CSWin-B (Dong et al. 202 la, with a comparable complexity to HiVīT-B, reports 84.2% top-1 accuracy ImageNet- IK under fully- supervised learning, higher than 83.8% reported by HiViT--,, and a 50.8% A.hoo on COCO object detection. However, after pre-training using MIM and fine-tuning. HiViT-B improves these numbers to 84.6% and 53.3% (Sec. 4.3). showing its advantages self-supervised learning.
will see later
Table 4: The fine-tuning accuracy ImageNet- 1K when the model is pre-trained without labels.
Method
Network
Params
FLOPs
Epochs
Top-1 Acc.
BEiT Bao et al.
2021)
ViT-B
86M
17.4G
400
83.2
CAE (Chen el al.
2022
ViT-B
86M
17.4G
800
83.6
MaskFeat Wei et al. 2021
ViT-B
86M
17.4G
800
84.0
SimMIM Xie et al.
2021b
ViT-B
86M
17.4G
800
83.8
SimMIM Xie et al.
2021b
Swin-B
88M
15.4G
800
84.0
MixMIM (Liu et al.
2022
Swin-B
88M
15.4G
600
84.4
MAE (He al. 202
ViT-B
86M
17.4G
1600
83.6
Ours
HiViT-B
66M
15.9G
1600
84.6
SimMIM Xie et al. 2021b)
Swin-I
197M
35.8G
800
85.4
SimMIM Xie et al.
20216
SwinV2-H
658M
118.1G
800
85.7
MixMIM Liu et al. MAE (He et al. 202
2022)
Swin-L
197M
35.8G
600
85.7
ViT-L
304M
61.6G
1600
85.9
Ours
HiViT-L
267M
63.8G
1600
86.4
4.2 SELF-SUPPRVISED VISUAL REPRESENTATION LEARNING
Fine-tuning. The fine-tuning results are provided in Table 4 and only the encoder part is used to test. As shown in Table 4 under the MAE pre-training pipeline, HiViT-B achieves 1.0% per- formance gain over ViT-B, validating the structural advantage of our hierarchical transformer over plain vision transformers. Our results are still competitive when compared hierarchical trans- former Swin-B pre-trained using SimMIM (Xie et al.) 2021b) and MixMIM (Liu et al.] (2022). When scaling the large scale, HiViT-L maintains the advantage over the plain transformer (ViT-L) with similar computational costs and fewer parameters, and claims more significant advantage (86.4% vs. 85.7%) compared to Swin-L (using MixMIM) and V2-H (using SimMIM).
Training Efficiency. HiViT only requires the active (unmasked) tokens as input so that it enjoys high efficiency during the MIM pre-training. Running on eight NVIDIA Tesla V-100 GPUs, each pre-training epoch on ImageNet- takes 10.3 minutes, which is almost 2x as fast as the version that we feed all patches into the encoder (the case if we use Swin-like backbone). We conjecture that this is part of the reasons why SimMIM (using Swin) reduced the input resolution to 192 x 192, which may result performance drop in downstream visual recognition tasks.
4.3 TRANSFERRING SELF-SUPERVISED MODELS TO DENSE PREDICTION
Objection Detection on COCO. For objection detection, we build a Mask RCNN detector (He) al. 2017) beyond the pre-trained models. our hierarchical model provide feature maps
8

---

Published as a conference paper at ICLR 2023
Table COCO detection fine-tuning results transferred from self-supervised pre-training.
COCO
Method
Network
Detector
Epochs
Aphox
Apmask
MoCo v3 Chen al.
102
VIT-B
Mask RCNN
12
45.5
40.5
BEiT
Bao al. 2021
VIT-B
Mask RCNN
12
42.1
37.8
PeCo
Dong
202 1b)
VIT-B
Mask RCNN
12
44.9
40.4
MAE
He et al.
2021
VIT-B
Mask RCNN
12
48.4
42,6
CAE
Chen al.
2022
VIT-B
Mask RCNN
12
49.2
43.3
Ours
HiViT-B
Mask RCNN
12
51.3
44.6
Supervised Li ct al.
202
ViT-B
Mask RCNN
100
47.9
42.9
Supervised Dong et
2021
CSWin-B
Mask RCNN
36
50.8
44.9
Supervised [Li et al.
2022b)
MViTv2-B
Mask RCNN
36
51.0
45.7
MAE (Li et al. 2022a
ViT-B
ViTDet
100
51.2
45.5
SimMIM (Xie et al.
2021b)
Swin-B
Mask RCNN
36
52.3
MixMIM (Liu et al.
2022)
MixMIM-B
Mask RCNN
36
52.2
46.5
Ours
HiVIT-B
Mask RCNN
36
53.3
47.0
multiple resolutions as inputs for the FPN head, leading to significant advantage of HiVIT over the plain transformers. As shown in Tab. 5 compare the performance with state-of-the-ard meth- ods. When fine-tuned for 12 epochs (1x), ), HiViT-B significantly outperforms ViT-B (pre-trained by different methods), reached an Aphux of 51.3% and Apmus 44.6%. When fine-tuned for 36 epochs (3x), HiViT-B still outperforms Swin-B pre-trained by other methods by a ~ 1.00% AP.
ADE20K. The results on ADE20K are shown in Ta- ble 6 We report the mean intersection over union (mloU) values. Similar to the situation on COCO, HiViT-B bene- fits from hierarchical features and reports a 52.8% mloU, surpassing ViT-B pre- trained by MoCo-v3 (Chen et al. 2021c). BEiT (Bao et al. 2021). 2022). and MAE (He al. 2021 by sig- nificant margin of at least 4.0%. In terms of hierarchical vision transformers, HiViT- outperforms MixMIM-B and reports the same performance SimMIM-B. On these dense prediction benchmarks, the advantage of HiViT (over ViT) becomes much more significant, validating the usefulness of hierarchical visual repre- sentations.
Semantic Segmentation
Table 6: Semantic segmentation results on ADE20K.
Method
Network
mlaU
Supervised He et
2021
ViT-B
47.0
MoCo v3 (Chen ct
2021
ViT-B
47.3
BETT (Bao ct all 2021
ViT-B
47.1
PeCo (ong a 202
VIT-B
48.5
MAE (Tle
202T
VIT-B
48.1
CAE (Chen al. 2022
ViT-B
48.8
l). CAE (Chnn et al.
SimMIM (Xee et all
2021bb
Swin-B
52.8
MixMIM (Lu et al.
2022
MixMIM-B B
50.3
Ours
HiViT-B
52.8
5 CONCLUSIONS
This paper presents hierarchical vision transformer named HiVIT. Starting with Swin Transform- ers, we remove redundant operations that cross the border tokens the main stage, and show that such modifications do harm, but slightly improve the model's performance in both fully- supervised and self-supervised visual representation learning. HiViT shows clear advantage in integrating with masked image modeling, on which the efficient implementation of MAE can be directly transplanted from VIT to HiVIT, accelerating the training speed significantly. We expect shelf replacement of ViT and Swin in the future research, especially
that HiT becomes an when the models need to be pre-trained with masked image modeling.
Limitations. Despite the improvement observed the experiments, our method has some limita- tions. The most important one lies in that the masking unit size is fixed this implies that we need to choose a single 'main stage'. Fortunately, the 3rd stage of Swin Transformers contribute most parameters and computations, hence it is naturally chosen, however, the method may encounter dif- ficulties in the scenarios that no dominant stages exist. In addition, we look forward to advanced architecture designs that go beyond the constraints. A possible solution lies in modifying lowelevel code (e.g., in CUDA) to support arbitrary and variable grouping tokens, but, more essentially, we expect flexible learning framework beyond MIM that supports variable sizes of masking units.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Our research focuses on two fundamental topics in computer vision, namely, (i) designing an ef- ficient architecture of deep neural networks and (ii) self-supervised visual representation learning- The datasets used for evaluation are public benchmarks that were widely used in the community. To best our knowledge, this research does not have ethical concerns to report.
REPRODUCIBILITY STATEMENT
We have included the code in the supplementary material. After the review process, we will also open-source the pre-trained models (which can also be obtained by running our code on public datasets, e.g. ImageNet- to case the community to reproduce our results.
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China (NSFC) under Grant 62225208, 62171431 and 61836012.
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. NeurIPS, 34, 2021.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2/06.08254 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 19-494 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 142944 2021.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In ICCV, 2021a.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu. Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 9906: 190.001550. 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo. Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. arXiv preprint arXiv:220..03026, 2022.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised visual transformers. arXiv preprint TxXi:22044.0207, 2021b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self self- supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021c.
10

---

Published as a conference paper at ICLR 2023
Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In ICCV, pp. 589-589990 189-598,20 2021d.
Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: large-scale hierarchical image database. In IEEE CVPR, pp. 248 258.. IEEE CS, 2009.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HLT (1), 2019.
Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows, arXiv preprint arXiv:2107.00652 2021a.
Xiaoyi Dong Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre- tooiirriiing of vision transform- ers. arXiv preprint arXiv:2111.12711 2021b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Yuxin Fang. Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. CoRR, abs/2202.03382, 2022.
Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation networks. In NeurIPS, pp. 19160-19171, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271 22221227122284 2020.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 12175-12185, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. JEEE CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In IEEE ICCV, 2017.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint 1rX.v:2111.06377,, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, pp. 1106-1114, 2012.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553) 2015.
Chunyuan Li, Jianwei Yang. Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-super vision transformers for representation learning. arXiv preprint arXiv: (et:2//06.099 10Xiv:2106.0974 09785, 2021a.
11

---

Published as a conference paper at ICLR 2023
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429, 2021b.
Yanghao Li, Mao Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. arXiv preprint arXiv:2203. 16527, 2022a.
Yanghao Li, Chao- Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 4804 4814, 2022b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 20.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In ECCV, pp- 740-755, 2014b.
Jihao Liu, Xin Huang, Yu Liu, and Hongsheng Li. Mixmim: Mixed and masked image modeling for efficient visual representation learning. arXiv preprint arXiv:2205.13133 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pp. 69-84. Springer, 2016.
Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2035, 2022.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, pp.
2016.
Zhiliang Peng, Wei Huang. Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. arXiv preprint arXiv:2105.03889, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In CVPR, pp. 16519-16529, 2021.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pp. 6105-6114, 2019.
Yunjie Tian, Lingxi Xie, Xiaopeng Zhang, Jiemin Fang, Haohang Xu, Wei Huang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Semantic-aware generation for self-supervised visual representation learning. CORR, abs/2111.13163, 2021.
Yunjie Tian, Lingxi Xie, Jiemin Fang, Mengnan Shi, Junran Peng, Xiaopeng Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Beyond masking: Demystifying token-based pre-training for vision transformers. CoRR, abs/2203. 14313, 2022.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. NewrIPS, 34, 2021.
12

---

Published as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, volume 139, pp. 10347-10357, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In CVPR, pp. 12894-12904, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. arXiv preprint wrXiv2210613399, 2021.
Chen Wei, Lingxi Xie, Xutong Ren, Yingda Xia, Chi Su, Jiaying Liu, Qi Tian, and Alan L Yuille. Iterative reorganization with weak spatial constraints: Solving arbitrary jigsaw puzzles for unsu- pervised representation learning. In CVPR, pp. 1910-1919, 2019.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:21 2021.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint S0d:::2003.18000 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pp. 418-434, 2018
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in Neural Information Processing Systems, 2021.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16684-16693, 2021a.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:211 211.20986,, 2021b.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641 2021.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, pp. 649- 666. Springer, 2016.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba Scene parsing through ade20k dataset. In CVPR, pp. 633-641, 2017.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2 1186. 2021.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
Table 7: Configurations for HiViT variants.
Depth 28
Dim
Heads
Params (M)
FLOPPs
Model
562
14a
562
28²
14a
562
28²
142
(G)
HiViT-T (tiny)
1
10
192
384
19.2
4.6
HiViT-S (small)
2
20
192
384
37.5
9.1
HiViT-B (base)
2
2
20
128
256
512
66.4
15.9
HiViT-I. (large)
4
4
36
192
384
768
12
266.8
63.8
Model Configurations Three models HiViT-T/S/B/L are trained through supervised learning or self-supervised learning with configurations included in Tab. 7 The "Depth" represents the block number on different stages (562. 28², and 14a are the 1st, 2nd, and 3rd stages respectively). "Dim" and "Heads" respectively denote the dimension and attention head numbers. We align the mod- els according to FL.OPs while using fewer parameters than the compared methods. We report the inference throughput speed in Tab. by testing the 2242 images using a V100 GPU.
Experimental Details for Supervised Learning We first evaluate HiV with supervised learning on ImageNet- 1K (Deng et al. 2009) which contains 1.28M training images and 50K validation ones divided into 1,000 categories. We follow Swin (Liu et al., (02T) and use the same training settings. Specifically,
the AdamW optimizer (Loshchilov & Hutter 2017) with initial learning rate 0.001, a weight decay 0.05, batch size 1024, a cosine decay learning rate scheduler, and linearly warm-up for 20 epochs. All the models are trained for 300 epochs with augmentation and regularization strategies (Liu et al.] [202T] and exponential moving average (EMA). The input size is 224 224 by default. The output feature of 3rd stage followed by an average pooling layer and then classifier layer. The drop path rates 0.05, 0.3, and 0.5 are respectively used for HiViT-T/S/B.
Experimental Details for Self-supervised Learning For self-supervised pre-training, the ImageNet-IK 1K training dataset without labels is used. The pre-trained models is fine-tuned and tested on the validation dataset. The pre-training settings follow those of MAE (He et al. (021). Specif- ically, the mask ratio is set 75% in default. The normalized target trick also adopted. In base model, we use the AdamW optimizer (Loshchilov & Hutter 2017) with an initial learning rate of 1.0 10-1, weight decay of 0.05, and learning rate follows the cosine decay learning schedule with warm-up for 40 epochs. The batch size is set to 4096 and the input size is 224 x 224. The over- all pipeline is an encoder-decoder framework where the decoder is designed to have transformer layers followed by a reshape operation to cast the feature to 3 x 224 x 224. Random cropping and random horizontal flip are used for data augmentation. When fine-tuning, we follow the settings from (He et al.] 2021) where the models are trained for 100 epochs using the AdamW optimizer with a warm-up for 5 epochs, a weight decay 0.05, and the input size 224 x 224. We use the layer- wise learning rate decay 0.85. The initial learning rate is set to 5 10 and batch size is set to 1024.
Experimental Details for Down-stream Tasks We transfer the self self-supervised pre-trained mod- els to object detection on MS COCO and semantic segmentation on ADE20K. We use the Mask R-CNN (He al., 2017) head implemented by the MMDetection library (Chen al. 2019). On the MS COCO dataset, we adopt the AdamW optimizer (Loshchilov & Hutter 2017) with an mitial learning rate of x 10 which decays by 10x after the 9-th and 11-th epochs when 1x training schedule (12 epochs) is adopted. The layer-wise decay rate is set to 0.9 and the multi-scale train- ing and single-scale testing strategies are used. For ADE20K, we employ the UperNet (Xiao al. 2018) head following BEiT (Bao et al. 2021). The AdamW optimizer with a learning rate x 10 to train the model for 80k iterations with a batch size 32.
14

---

Published as a conference paper at ICLR 2023
Table 8: Detailed comparison of the pre-training time and GPU memory cost.
Pre-training time (mins pre epoch)
Pre-train
Model
Input Size
GPU Memory
MAE (He et al. 2021)
ViT-B
224x224
15282M
8.8
SimMIM Xie al.
2021bb
Swin-B
192x192
18245M
14.3
SimMIM Xie et al.
2021b
Swin-B
224x224
25710M
18.7
Ours
HiViT-B
224x224
17906M
10.3
Table 9: Ablation studies on shallow stage depth.
Module #
Params
FLOPs
ImageNet
562
28
142
(M)
(G)
Acc.
0
0
24
77.1
16.0
84.1
22
71.8
15.9
84.5
2
2
20
66.4
15.9
84.5
3
61.1
15.8
84.4
4
4
16
55.8
15.7
84.1
Comparison of the Pre-training Time and GPU Memory Cost We test on an 8x V100-SXM2 platform with PyTorch 1.8.0 and CUDA 10.2. All the methods use batch size of 1024. As shown in Tab. 8 compared to MAE, SimMIM requires more memory and pre-training time because all the image patches (including the visible and the mask patches) need be computed. With the same input resolution, the proposed HiViT models requires significantly shorter pre-training time per epoch and smaller GPU memory than SimMIM's.
Depth of Shallow Stages We ablate the effect of shallow stage depth, Tab. 9 by setting different block numbers for stage 1, 2, and 3 (562, 282, and 14 respectively). The best setting (2-2-20 block) achieves 84.5% performance. Changing the depth of shallow stages to be smaller or larger leads to worse performance.
15


Reviews:
[{"review_id": "sqsHPUKPQW", "reviewer": "Reviewer_mVGH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a hierarchical ViT, by analyzing the effective design in Swin compared with the plain ViT. The model contains a hierarchical patch embedding with only MLP used in early stages, using transformer blocks with global attention in the 14x14 stage, and removing the last stage. Experiments on ImageNet, COCO and ADE20K have shown quite promising results. The architecture is also compatible with MAE, which can achieve higher accuracy with the mask pretraining.", "summary_of_the_review": "Overall, I think this paper is well organized. It studies an important task, where the hierarchical models are not well compatible with the powerful MAE. The analysis of the plain and hierarchical ViTs are quite impressive. \n\nHowever, this proposed model is a simple plain transformer with hierarchical patch stem, which has been used in several prior works. Hence, the technical novelty is limited. The key contribution of using MAE for hierarchical model pretraining, does not give impressive results. The experiments do not present the actual training cost saved with HiViT, meanwhile the improvement with 1600epoch MAE pretraining only bring limited performance improvement compared with ViT.\n\nTherefore, the reviewer votes for the borderline in the first round. \n", "strength_and_weaknesses": "Strength:\n- The paper is well written, the contributions are clearly stated and supported by empirical experimental results and analysis.\n- The paper gives a step-by-step analysis on the differences between ViT and Swin, which \n- The proposed model addresses the problem of incompatibility between MAE and hierarchical vision transformers, by simply removing the 7x7 stage and the self-attention in early stages.\n- The authors performed rich experiments to validate the effectiveness of the proposed HiViT. The training and design details are provided, and the overall method seems reproducible.\n\nWeaknesses:\n- This paper claims that early attentions can be replaced by hierarchical patch embedding only when the model is deep enough, with an attention variance comparison on the deepened plain ViT. However, as the proposed model removed the self-attention on hierarchical ViT (Swin) instead of plain ViT, does the statement still hold on Swin?\n- In the analysis section, the authors proposed a variant of Swin without stage4 and does not hurt performance. However, the model without stage4 also used global window in stage 3 (as illustrated in Fig1, bottom, 3rd & 4th figures), which may also have a significant impact on the performance and computational cost of the model. Therefore, the experiment may not well support the claim that stage4 is not helpful.\n- The overall architecture of HiViT comprises an overlap patch embedding that embeds the image to 14x14 and use continuous transformer blocks like plain ViTs. Yet, using hierarchical patch embedding have been proposed by several papers, e.g., papers use overlapping patch embed [1-4], and another similar work that also removed the self-attention in early stages, and only use MLP in 56x56 and 28x28 [4]. \n[1] Xiao, Tete, et al. \"Early convolutions help transformers see better.\" Advances in Neural Information Processing Systems 34 (2021): 30392-30400.\n[2] Guo, Jianyuan, et al. \"Cmt: Convolutional neural networks meet vision transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[3] Ali, Alaaeldin, et al. \"Xcit: Cross-covariance image transformers.\" Advances in neural information processing systems 34 (2021): 20014-20027.\n[4] Pan, Zizheng, et al. \"Less is more: Pay less attention in vision transformers.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 2. 2022.\n- Since an important claim of this paper is enabling hierarchical architectures to trained with MAE and save pretraining cost, a more detailed comparison of the training time or GPU usage, shall be given for a better reflection of the capacity of the architecture for pretraining, as an example, the Table2 in [5].\n[5] Huang, Lang, et al. \"Green Hierarchical Vision Transformer for Masked Image Modeling.\" arXiv preprint arXiv:2205.13515 (2022).\n- The performance gain in Table 4 may not well reflect whether HiViT has “structural advantage over plain vision transformers”. ViT-B has 77.9% acc while HiViT has 83.8% acc in the supervised training, while MAE+ViT-B has 83.6%(+5.7%) acc and MAE+HiViT has 84.6%(+0.8%) in the pretraining setting. Therefore, it seems HiViT do not gain much from the pretraining, even with 1600 epochs. This phenomenon should be further explained.\n- There are some other minor issues, such as mistakes in writing. For example, “As an alternative solution (used by Swin) to Step (c), it causes a significant ADE20K segmentation accuracy and Compared to Step (c)”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pJZkYfV0zt", "reviewer": "Reviewer_5NXC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study what makes hierarchical ViTs (in particular SwinTransformers) work, and based on the insights, propose a new vision transformer architecture called HiViT. The authors show that HiViT works better than Swin and vanilla ViT on ImageNet and ADE20K. Moreover, HiViT doesn't use window attention, makes it compatible to MAE style efficient SSL pre-training. The authors show that masked pre-training on HiViT obtains better accuracy than MAE on vanilla ViTs.", "summary_of_the_review": "While the technical novelty of this paper is somewhat limited, I still think this paper can be a useful reference for practitioners and future researchers on architecture design. Overall, I rate this paper as marginally above the acceptance threshold.", "strength_and_weaknesses": "Strengths:\n+ Swins, ViTs, and MAE are state-of-the-art and widely-used tools in computer vision. The three main contributions, (1) understanding what makes Swin works well, (2) advancing architecture design, and (3) showing that HiViT works with masked-pretraining, are practically useful. I can see that many computer vision practitioners will be interested in this paper.\n+ The ablations in Table 2 are clear and convincing. \n\nWeaknesses:\n- While HiViT works well with masked-pretraining, without masked pre-training it is not very strong; For example, MViTv2 is more accurate and uses only 2/3 of FLOPs. This makes HiVIT's usefulness in practice limited.\n- Many ablations are not entirely new. For example, the effectiveness of RPE and hierarchical resolutions have been shown in the original Swin paper. The finding that I find novel is that C512 x 24L works quite a lot better than C768 x 12L. However, this is technically a small change. I thus overall would not rate the technical contributions high.\n- Adding runtime, flops, and params into Table 2 would makes the trade-offs clearer.\n\n- minor points on writing:\n1) why use the word \"merging\" instead of \"pooling\", which is more concretely defined?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6LoWnzqisK", "reviewer": "Reviewer_vrWv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the improvement from plain ViT to hierarchical Swin-Transformer step-by-step and proposes a new hierarchical vision transformer that only uses hierarchical patch embedding at the beginning and keeps the global attention at 14x14 level instead of window attention. It shows similar or better performance than Swin transformers on Imagenet classification, segmentation and detection. ", "summary_of_the_review": "\nOverall, the paper presents a good study of plain ViT and hierarchical ViT, and provides a simpler hierarchical ViT with similar or better vision task performance and is friendly to many follow-up techniques of plain ViT. The proposed method can be widely adapted with its simple design and good performance. It would be much better if authors can provides further studies regarding the proposed patch embedding module. \n", "strength_and_weaknesses": "Strength:\t\n1.\tThe paper provides a detailed study regarding prevail Swin-Transformer architecture, identifying the key parts for its superior performance v.s. plain ViT, e.g increasing depth and decrease channels, add relative position embedding. \n2.\tThe paper proposes an alternative way to enjoy the performance gain from hierarchical ViT which is simpler and friendly for techniques that are designed for plain ViT, i.e. Mask Image Modeling.\nWeaknesses:\n1.\tThere are several part of descriptions not clear enough. \na.\tTable 2, no explanation for RPE, Low Att, Mid Att, etc in table caption. \nb.\tStage definition missing, though can be referred, but better have clear definition. \nc.\tCan authors also add Param and Flops to Table 2, as changing components will change computation as well.\nd.\tIncomplete sentences: Page 5, Step (c’) ‘causes a significant accuracy’;  Sec 3.3 ‘all tokens in Stage ? are symmetric’\ne.\tMerging operation details? One can only infer it from figure 1, can authors add some descriptions about it? \n2.\tRegarding the proposed hierarchical patch embedding, can authors give more explanation about the specifical design as 2 consecutive MLP with ratio 3, how authors reach this design. As this is a critical module proposed, can authors provide some ablation studies about it, e.g. why uses a different MLP ratio of 3 as latter stages uses 4 instead? Will allocate computations differently to 56x56, 28x28 influence the final performance? What computation percentage should be used in patch embedding v.s. latter stage (14x14) for better performance? \n \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%