HRFormer: High-Resolution Vision Transformer for Dense Predict
Authors: Yuhui Yuan
Rao Fu
Lang Huang
Weihong Lin
Chao Zhang
Xilin CHEN
Jingdong Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformer
High-Resolution
Pose Estimation
Semantic Segmentation

Abstract: We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet [45]), along with local-window self-attention that performs self-attention over small non-overlapping image windows [21], for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the HighResolution Transformer on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin transformer [27] by 1.3 AP on COCO pose estimation with 50% fewer parameters and 30% fewer FLOPs. Code is available at: https://github.com/HRNet/HRFormer

HRFormer: High-Resolution Transformer for Dense Prediction
Yuhui Yuan 12
Rao Fu'
Lang Huang'
Weihong Lin
Chao Zhang*
Xilin Chen
Jingdong Wang"
University of Chinese Academy of Sciences Peking University
IInttitute of Computing Technology, CAS
*Microsoft Research Asia BBaidu
Abstract
Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast the original Vision Trans- former that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet [46)), along with local-window self-attention that performs self-attention over small non-overlapping image windows [21]. for improving the memory and computation efficiency. In addition, we introduce convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High- Resolution Transformer on both human pose estimation and semantic segmentation tasks, c.g., HRFormer outperforms Swin transformer (27) by 1.3 AP on COCO pose estimation with 50% fewer parameters and 30% fewer FLOPs. Code is available at: https:: /tttt:://gttub... com http//RRt//////t //HRFrrmer
We present
Introduction
Vision Transformer (ViT) 13 shows promising performance on ImageNet classification tasks. Many follow-up works boost the classification accuracy through knowledge distillation [42]. adopting deeper architecture [3], directly introducing convolution operations [66 48 redesigning input image tokens [54), and etc. Besides, some studies attempt to extend the transformer to address broader vision tasks such as object detection [4, semantic segmentation [63 37], pose estimation [51] video understanding [2 30), and
This work focuses the transformer for dense prediction tasks, including pose estimation and semantic segmentation.
Vision Transformer splits an image into sequence of image patches of size 16 x 16, and extracts the feature representation of each image patch. Thus, the output representations of Vision Transformer lose the fine-grained spatial details that are essential for accurate dense predictions. The Vision Transformer only outputs a single-scale feature representation, and thus lacks the capability to handle multi-scale variation. To mitigate the loss of feature granularity and model the multi-scale variation, we present High-Resolution Transformer (HRFormer) that contains richer spatial information and constructs multi-resolution representations for dense predictions.
The High-Resolution Transformer is built by following the multi-resolution parallel design that is adopted in HRNet 46]. First, HRFormer adopts convolution in both the stem and the first stage as several concurrent studies
50 also suggest that convolution performs better in the early stages. Second, HRFormer maintains high-resolution stream through the entire process with parallel medium- and low-resolution streams helping boost high-resolution representations. With feature maps of different resolutions, thus HRFormer is capable to model the multi-scale variation. Third,
"Corresponiigg author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

MIISA
MIISA
(1)
Figure 1: Illustrating the HRFormer block. The HRFormer block is composed of (a) local-window self- attentionm and (b) feed-forward network (FFN) with depth-wise convolution. The local-window self-attention scheme is inspired by the interlaced sparse self-attention [56
HRFormer mixes the short-range and long-range attention via exchanging multi-resolution feature information with the multi-scale fusion module.
At each resolution, the wondoww self-attention mechanism is adopted to reduce the memory and computation complexity. We partition the representation maps into set of non-overlapping small image windows and perform self-attention in each image window separately. This reduces the memory and computation complexity from quadratic to linear with respect spatial size. We further introduce a 3 x 3 depth-wise convolution into the feed-forward network (FFN) that follows self-attention, to exchange information between the image windows which are disconnected in the local-window self-attention process. This helps to expand the receptive field and essential for dense prediction tasks. Figure T the details of HRFormer block.
the
We conduct experiments on image classification, pose estimation, and semantic segmentation tasks, and achieve competitive performance various benchmarks. For example, HRFormer-B gains + +1.0% top-1 accuracy on ImageNet classification over DeiT-B 42] with 40% fewer parameters and 20% fewer FLOPs. HRFormer-B gains 0.9% AP over HRNet- W48 41 COCO val set with with 32% fewer parameters and 19% fewer FLOPs. HRFormer-B B + OCR gains +1.2% and +2.0% mIoU over HRNet-WAR OCR [55] with 25% fewer parameters and slightly more FLOPs PASCAL-Context test and COCO- Stuff test, respectively.
Related work
Vision Transformers. With the success of Vision Transformer (ViT) I3 and the data-efficient image transformer (DeiT) [42), various techniques are proposed to improve the ImageNet classifica- tion accuracy of Vision Transformer [12 advancements, the community has verified several effective improvements such as multi-scale feature hierarchies and incorporating convolutions.
43
48
16
[4
17
27, 22 40). Among the very recent
For example, the concurrent works MVIT 14], PVT 47 and Swin 27 introduce the multi-scale feature hierarchies into transformer following the spatial configuration of a typical convolutional architecture such as ResNet-50. Different from them, our HRFormer incorporates the multi-scale fea- ture hierarchies through exploiting the rulti-rresoluion parallel design inspired by HRNet. CvT (48), CeiT 133, and ViT (25) propose to enhance the locality of transformer via inserting depth-wise convolutions into either the self-attention the FFN. The purpose of the inserted convolution within our HRFormer different, apart from enhancing the locality, it also ensures information exchange across the non-overlapping windows.
Several previous studies B6 19 have proposed similar local self-attention schemes for image classification. They construct the overlapped local windows following the strided convolution, resulting in heavy computation cost. Similar to 21 27], we propose to apply the local-window self-attention scheme to divide the input feature map into non-overlapping windows. Then apply the self-atte aff-atttenion within each window independently so as to improve the efficiency significantly.
There are several concurrently-deveooee works 63 37 use the Vision Transformer to address the dense predict tasks such as semantic segmentation. They have shown that increasing the spatial resolution of the representations output by the Vision Transformer important for semantic segmen-

---

cons:
book
arriee
cone
Figure 2: Illustratin the High- Nigh-Resolution Transformer architecture. The multi-resolution parallel trans- former modules are marked with light blue color areas, Each module consists of multiple successive multi- resolution parallel transformer blocks. The first stage is constructed with convolution block and the remained three stages are constructed with transformer block.
lation. Our HRFormer provides different path to address the low-resolution problem of the Vision Transformer via exploiting the multi-resolution parallel transformer scheme.
High-Resolution CNN for Dense Prediction. The high-resolution convolutional schemes have achieved great success on both pose estimation and semantic segmentation tasks. In the development of high-resolution convolutional neural networks, the community has developed three main paths in- cluding: (i) applying dilated convolutions to remove some down-sample layers [6 [52]. (ii) recovering
high-resolution representations from low-resolution representations with decoders 38 [
32 and
(iii) maintaining high-resolution representations throughout the network [46] 15
45 59 20
Our HRFormer belongs to the third path, and retains the advantages of both vision transformer and
HRNet [46]
High-Resolution Transformer
parallel transformer. We follow the HRNet [46] design and start from high- resolution convolution stem as the first stage, gradually adding high-to-low resolution streams one by one as new stages. The multi-resolution streams connected in parallel. The main body consists of a sequence of stages. In each stage, the feature representation of each resolution stream is updated with multiple transformer blocks independently and the information across resolutions is exchanged repeatedly with the convolutional multi-scale fusion modules.
Figure 2 jillustrates the overall HRFormer architecture. The design of convolutional multi-scale fusion modules exactly follows HRNet. We illustrate the details of the transformer block in the following discussion and more details are presented in Figure
Local-window self-attention. We divide the feature maps X N into set of non-overlapping small windows: X >> [X,,X2.'. X.,... Xr), where each window is of size x K. We perform multi-head self-attention (MHSA) within each window independently. The formulation of multi-head self-attention on the p-th window is given as:
MultiHead(Xp)) Concat hhad (X, (p),,-... "eeii...,,e,,,,,,,. headd(p)H) E RX*"D,
(1)
(X, w ) (X)
(ead(X,))) = Softmax
X #
(2)
VDHH
8, X, MultiHead W. x
where W. E RDDD Wh И represents the number of heads, D represents the number of channels, N represents the input resolutions, and X, represents the output representation of MHSA. We also apply the relative position embedding scheme introduced in the T5 model 35 to incorporate the relative position information into the local-window self-attention.
9 D Wh D and Wh E R for h (1.... H}.
With MHSA aggregates information within each window, we merge them compute the output X MHSA.
Xp) Merge X MHSA
The left part of Figure Iillustrates how local-window self-attention updates the input representa- tions, where the multi-heal self-attention operates within each window independently.
3

---

Table 1: The architecture configuration of HRFormer. LSA: local-window self-attention, FFN-DW: feed-forward network with 3 x depth-wise convolution, (M,, M2, Ma. M4): number of modules. (Bi, B2, Bs, Ba): the number of blocks. (W1, i, W2, Wa, W4): the size of windows, (H1, H2, H3, H4): the number of heads, (Ri, R2, Ra. Ra): the MLP expansion ratios.
Stage
Stage
Stage
Stage 1
x 1,64
LSA. I...W1.. .i.M; PFN-DW,R, R₁
ISA,W1. FFN-DW. Ri
ISA,W; Hi FFN-DW Ra
4x
x 3,64
XB1 X1
XBax
X Ma
XB4xM4
x 1,256
LSA.Wa, H2 FFN-DW,R,
LSA, HFN-DW, R22
LS,,W2 ,2 FEN-DW RR
8x
x Ma
B3 Ma
XB4X
Ls. FFN.DW,
a FFN DW, Ra
16x
XB
x XB4XM4
LSA L,A,W, H. FFN-DW RR
XB4
Table 2: HRFormer instances. HRFormer-,, HRFormer-S, and HRFormer-B represents tiny, small, and base
HRFormer model, respectively.
#modules
#blocks
#channels (C1, C2, Cs, C4)
#heads
Model
(M1, M2, Ma, Ma)
(B,, B2, Bs, Ba)
(H1, H2, Ha,
HRFormer-T
1,,,)
(2, 2,
(18, 36, 72, 144) (32, 64, 128, 256)
(1,2,4,8)
HRFormer-S
(1, 1, 2,2)
(2, 2, 2,2)
(1, 4,8)
HRFormer-B
(1,1,4,2)
(2, 2, 2,2)
(78, 156, 312, 624)
(2,488, 16)
FFN with depth-wise convolution. Local-window self-attention performs self-attention over the non-overlapping windows separately. There is no information exchange across the windows. handle this issue, we add 3 x 3 depth-wise convolution between the two point-wise MLPs that form (OODW--Coo..MMPP() (M)P()). The right part of Figure shows an example of how FFN with 3 x depth-wise convolution updates the 2D input representations.
the FFN Vision transformer: MT.((DW-Com) Conyy.CCo..do
Representation head designs. As shown in Figure the output of HRFormer consists of four feature maps of different resolutions. We illustrate the details of the representation head designs for different tasks following: (i) ImageNet classification, bottleneck and the output channels are changed to 128, 256, 512, and 1024 respectively. Then, we apply the strided convolutions to fuse them and output a feature map of the lowest resolution with 2048 channels. Last, we apply global average pooling operation followed by the final classifier. (ii) pose estimation, we only apply the regression head over the highest resolution feature map. (iii) semantic segmentation, we apply the semantic segmentation head over the concatenated representations, which are computed by first upsampling all the low-resolution representations the highest resolution and then concatenate them together.
send the four-resolution feature maps into
Instantiation. We illustrate the overall architec- ture configuration of HRFormer in Table []] We use (M,,M2, Ma, Ma) and (B1, B2, B3, Ba) to represent the number of modules and the number of blocks of (statel, stage2, stage3, stage4), respectively. We use (C1, C2 C3 CA). (H1, H2, H3. H4) and (R1, R2, R3, R&) rep- resent the number of channels, the number of heads and the MLP expansion ratios in trans- former block associated with different resolu- tions. We keep the first stage unchanged fol- lowing the original HRNet and use the bottle- neck as the basic building block. We apply the transformer blocks in the other stages and each transformer block consists of locdl-window self-attention followed by FFN with 3 x
Figure 3: Illustrating that FEN with 3 3 depth-wisc convolution connects overllaaag windows.

---

Figure 4: Example results of HRFormer-B COCO pose estimation val: containing occlusion, multiple persons, viewpoint and appearance change.
Figure 5: Example results of HRFormer-B + OCR Cityscapes val (left one), COCO-Stuff test (middle two), and PASCAL-Context test (right two).
depth-wise convolution. We have not included the convolutional multi-scale fusion modules in Table ! for simplicity. In our implementation, we the size of the windows on four resolution streams as (7, 7, 7, 7) by default. Table 2 illustrates the configuration details of three different HRFormer instances with increasing complexities, where the MLP expansion ratios (R1, R2, R3, Ra) are set as 444,4,44 for all models and are not shown.
Analysis. The benefits of 3 3 depth-wise convolution twofold: enhancing the locality and the other one is enabling the interactions across windows. We illustrate how the FFN with dept convolution is capable to expand the interactions beyond the non-overlapping local windows and model the relations between them in Figure 3 Therefore, based on the combination of the local- window self-attention and the FFN with 3 depth-wise convolution, can build the HRFormer block that improves the memory and computation efficiency significantly.
4 Experiments
4.1 Human Pose Estimation
Training setting. We study the performance of HRFormer on the COCO 26 human pose estimation
benchmark, which contains more than 200K images and 250K person instances labeled with 17 keypoints. We train our model on COCO train 2017 dataset, including 57K images and 150K person instances. We evaluate our approach on the val 2017 set and test-dev 2017, containing 5K images and 20K images, respectively.
We follow most of the default training and evaluation settings of mmpose and change the optimizer from Adam to AdamW. For the training batch size, we choose 256 for HRFormer- T and HRFormer-S and 128 for HRFormer-B due limited GPU memory. Each HRFormer experiment on COCO pose estimation task takes 8x 32G-V 100 GPUs.
Results. Table 3 reports the comparisons COCO val set. We compare HRFormer to the representative convolutional method such as HRNet [41 and several recent transformer methods, including PRTR (23). TransPose- -H-A [5T), and TokenPose-L/D24 244. HRFormer-B gains 0.9% with 32% fewer parameters and 19% fewer FLOPs when compared to HRNet-Was with an input size of 384 x 288. Therefore, our HRFormer-B already achieves 77.2% w/o using any advanced techniques such as UDP (00 and DARK (59). We believe that our HRFormer-B could achieve better results by exploiting either UDP or DARK scheme. We also report the comparisons on COCO test-de dev set in Table Our HRFormer- B outperforms HRNet-W W48 by around 0.7% with fewer parameters and FLOPs. Figure 4 shows some example results of human pose estimation on COCO val set.
huuns///ghhibbcommoggssmmmmmmmmp Apache License 2.0
5

---

Table 3: Comparison on the COCO pose estimation val set. The number of parameters and
FLOPs for the pose estimation network are measured w/o considering neither human detection nor keypoint grouping. All results are based on ImageNet pretraining. means the numbers are not provided in the original paper.
Method
input size
#param.
FLOPs
AP
A
AP
APM
AP
AR
HRNet-Wa3
41
256 x 192
28.5M
7.1G
74.4
90.5
81.9
70.8
81.0
78.9
HRNet-W33 41
384 x 288
28.5M
16.0G
75.8
90.6
82.7
71.9
82.8
81.0
HRNet-W-18 41
256 192
63.6M
14.6G
75.1
90.6
82.2
71.5
81.8
80.4
HRNet-W448
384 x 288
63.6M
32.9G
76.3
82.9
83.4
81.2
PRTR 23
512 x 384
57.2M
37.8G
73.3
89.2
79.9
69.0
80.9
80.2
TransPose-H-Ad 51
256 x 192
17.5M
21.8G
75.8
80.8
TokenPose-L//22 24
256 x 192
27.5M
11.0G
75.8
90.3
82.5
72.3
82.7
80.9
HRFormer-T
256 192
2.5M
1.3G
70.9
89.0
78.4
67.2
77.8
76.6
HRFormer-T
384 x 288
2.5M
1.8G
72.4
89.3
79.0
68.2
79.7
77.9
HRFormer-S
256 x 192
7.8M
2.8G
74.0
90.2
81.2
70.4
80.7
79.4
HRFormer-S
384 x 288
7.8M
6.2G
75.6
90.3
82.2
71.6
82.5
80.7
HRFormer-
256 192
43.2M
12.2G
75.6
90.8
82.8
71.7
82.6
80.8
HRFormer-B
384 288
43.2M
26.8G
77.2
91.0
83.6
73.2
8/122
82.0
Table 4: Comparison on the COCO pose estimation test-dev set. The number of parameters and FLOPs for the pose estimation network are measured w/o considering neither human detection keypoint grouping. All results are based ImageNet pretraining.
Method
input size
#param.
FL.OPs
AP
A
Ap
ApM
AP
AR
HRNet-W332 41
384 x 288
28.5M
16.0G
74.9
92.5
82.8
71.3
80.9
80.1
HRNet-WAR 41
384 288
63.6M
32.9G
75.5
92.5
83.3
71.9
81.5
80.5
PRTR
512 384
57.2M
37.8G
72.1
90.4
79.6
68.1
79.0
79.4
TransPose-H-AG 51
256 x 192
17.5M
21.8G
75.0
92.2
82.3
71.3
81.1
TokenPose-L/D2. 24
384 x 288
29.8M
22.1G
75.9
92.3
83.4
72.2
82.1
80.8
HRFormer-S
384 x 288
7.8M
6.2G
74.5
92.3
82.1
70.7
80.6
79.8
IRRForer-B
384 288
43.2M
26.8G
76.2
92.7
83.8
72.5
82.3
81.2
4.2 Semantic Segmentation
Cityscapes. The Cityscapes dataset 99] is for urban scene understanding. There are a total of 30 classes and only classes are used for parsing evaluation. The dataset contains 5K high-quality pixel-level finely annotated images and 20K coarsely annotated images. The finely annotated 5K images are divided into 2, 975 train images, 500 val images and 1, 5.5 test images. We set the initial learning rate 0.0001, weight decay as 0.01, crop size as 1024 512, batch size as 8, and training iterations as 80K by default. Each HRFormer + OCR experiment on Cityscapes takes 8x 32G-V100 GPUs.
Table reports the results on Cityscapes val. We choose use HRFormer + OCR our semantic segmentation architecture. We compare our method with several well- Vision Transformer based methods [63] 37] and CNN based methods 6 62 55). Specifically, SETR-PUP and SETR- MLA use the ViT-Large [3] as the backbone. DPT-Hybrid uses the ViT-Hybrid [3] that consists of a ResNet-50 followed by 12 transformer layers. Both ViT-Large and VYT-Hybrid are initialized with the weights pre-trained on 1mageNet-21K. where both of them achieve around 85.1% top1 accuracy on ImageNet. DeepLabv3 (6) and PSPNet
are based on dilated ResNet-101 with output stride According to the fourth column of Table 5 HRFormer + OCR achieves competitive performance overall. For example, HRFormer-B + OCR achieves comparable performance with SETR-PUP while saving 70% parameters and 50% FLOPs.

---

Table 5: Comparison with the recent SOTA on semantic segmentation tasks. We report the mloUs on Cityscapes val. PASCAL-Comiext test, COCO-Stulf test, and ADE20K val. The number of parameters and FLOOss are measured on the image size of 1024 1024, and the output label map size of 19 x 1024 1024. All results are evaluated with multi-scale testing. 8: the results are obtained with extra pre-training on ADE20K.
Mcthod
#params.
FLOPs
Cityscapes PASCAL-Context COCO-Stuff ADE20K
Transformer backbone
SETR-PUP 63
317.8M
2326.70 7G
82.2
55.3
50.1
SETR-MLA 63
309.5M
2138.6G
55.8
50.3
Swin-S + UperNet
27
81.16M
1036.50G
49.5
Swin-B UperNet
2
121.18M
1187.90G
49.7
PVT-Large Semantic FPN 47
65.1M
-G
43.5
CNN backbone
Deeplabv3 7
87. 1M
1394.0G
80.7
54.1
PSPNet 62
68.0M
1028.8G
80.0
54.0
43.3
HRNet- OCR 55
74.5M
924.7G
56.2
45.7
CNN+ Transfommer backbone
DPT-Hybrid (37
124.0M
1231.5G
60.5
49.0
HRFormer-B OCR
56.2M
1119.9G
82.6
58.5
43.3
50.0
HRFormer-B + OCR + SegFix [57
56.2M
1119.9G
83.2
PASCAL-Contex The PASCAL-Context dataset (2 challenging scene parsing dataset that contains 59 semantic classes and background class. The train set and test set consist of 4, 998 and 5, 105 images respectively. We set the initial learning rate as 0.0001, weight decay as 0.01, crop size 520 520, batch size 16, and training iterations 60K by default. We report the comparisons on the fifth column of Table 5 Accordingly, HRFormer-B + OCR gains 1.1%, 1.5% over HRNet-W445 OCR, SETR-MLA with fewer parameters and FLOPs, respectively. Notably, DPT-Hybrid achieves the best performance through extra pre-training the models on ADE20K in advance. Each HRFormer OCR experiment PASCAL-Context takes 8x 32G- V100 GPUs.
COCO-Stuff. The COCO-Stuff dataset is a challenging scene parsing dataset that contains 171 semantic classes. The train set and test set consist of 9K and 1K images respectively. We set the initial learning rate as 0.0001, weight decay as 0.01, crop size as 520 x 520, batch size 16, and training iterations as 60K by default. We report the comparisons on the last column of Table and and HRFormer-B OCR outperforms the previous best-performing HRNet-Wal 448 + OCR by nearly 2%. Fach HRFormer + OCR experiment on COCO-Stuff takes 8x 32G-V100 GPUs. Figure shows some example results on Cityscapes, PASCAL-Contex,, and COCO-Stuff.
4.3 ImageNet Classification
Training setting. We conduct the comparisons ImageNet-1K, which consists of 1.28M train images and 50K val images with 1000 classes. We train all models with batch size 1024 for 300 epochs with AdamW [28] optimizer, cosine decay learning rate schedule, weight decay 0.05, and a bag of augmentation policies, including rand augmentation [TO] mixup [60,, cutmix (58), and on. HRFormer-T and HRFormer-S require 8 x 32G-V100 GPUs and HRFormer-B requires 32 x 32G-V100 GPUs.
Results. We compare HRFormer to some representative CNN methods and vision transformer methods Table where all methods are trained on ImageNet-1K only. The results of ViT-Large with larger dataset such as ImageNet-21 K not included for fairness. According to Table HRFormer achieves competitive performance. For example, HRFormer-B gains 1. 1.0% over DeiT-B while saving nearly 40% parameters and 20% FLOPs.
7

---

Table 6: Comparisons on ImageNet-IK: val.
Method
image size
#param.
FLOPs
Top-1 acc.
ResNet-18 18 [8
224 x 224
11M
1.8G
69.8
ResNet-50
224 x 224
26M
4.1G
78.5
ResNet- 101
224 X 224
45M
7.9G
79.8
HRNet- W18
46
224 x 224
21.3M
4.0G
76.8
JIRNet-W33
46
224 x 224
41.2M
8.3G
78.5
HRNet- W48 46
224 x 224
77.5M
16.1G
79.3
RegNetY-4G 34
224 x 224
21M
4.0G
80.0
RegNetY-SG [4
224 x 224
39M
8.0G
81.7
RegGetY-16G 34
224 x 221
84M
16.0G
82.9
ViT-B/16 [3
224 x 224
86M
55.4G
77.9
ViT-L/16 [3
224 x 224
307M
190.7G
76.5
DeïT-T
224 x 224
5M
1.3G
72.2
DeïT-S
224 x 224
22M
4.6G
DeïT-B
224 x 224
86M
17.5G
DeiT-B'm
42
384 x 384
86M
55.4G
83.4
Conformer-T
33
224 x 224
23.5M
5.2G
81.3
Conformer-S
33
224 x 224
37.7M
10.6G
83.4
Conformer-B B
33
224 x 224
83.3M
23.3G
84.1
PVT-T
224 x 224
13.2M
1.9G
75.1
PVT-S
224 x 224
24.5M
3.8G
79.8
PVT-M
224 x 224
44.2M
6.7G
81.2
PVT-L
224 x 224
61.4M
Swin-T
224 x 224
29M
4.5G
81.3
Swin-S
27
224 x 224
50M
8.7G
83.0
Swin-B 27
224 x 224
88M
15.4G
83.5
Swin-B 2
384 x 384
88M
47G
84.5
HRFormer-T
224 x 224
8.0M
1.8G
78.5
IRRorrer-S
224 x 224
13.5M
3.6G
81.2
HRFormer-B
224 x 224
50.3M
13.7G
82.8
Table 7: Study of the 3x3 depth-wise convolution in FFN. We report the topl acc., mloU, and AP on ImageNet val, PASCAL--Cottes test, and COCO pose estimation val, respectively. Results on PASCAL-Contex are evaluated with single-scale testing. The number of parameters and FLOPs measured on ImageNet.
Method
#param.
FLOPs
ImageNet
PASCAL-Context
COCO
FFN 3x3 DW-Conv.
7.9M
1.76G
77.83
46.84
66.88
FFN w/ x 3 DW-Conv.
8.0M
1.83G
78.48
49.74
70.92
4.4 Ablation Experiments
Influence x depth-wise convolution within FFN We study the influence of the depth- wise convolution within FFN based on HRFormer-T in Table 7 We observe that applying 3 x 3 depth-wise convolution FFN significantly improves the performance on multiple tasks, including ImageNet classification, PASCAL-Conter segmentation, and COCO pose estimation. For example, HRFormer-T FFN w/ 3x 3 depth-wise convolution outperforms HRFormer-T FFN w/o 3x depth-wise convolution by 0.65%, 2.9% and 4.04% on ImageNet, PASCAL-Context and COCO, respectively.
Influence of shifted window scheme & 3x3 depth-wise convolution within FFN based on Swin-
We compare our method with the shifted windows scheme of Swin transformer 27 Table For fair comparisons, we construct Intra-Winn Window transformer architecture following the same

---

Table Influence of shifted window scheme & 3x3 depth-wise convolution within FFN based on
Sw
Method
3x 3 depth-wise convolution in FFN
#param.
FLOP%s
ImageNet top1 acc.
Swin-T
X
28.3M
4.5G
81.3
Swin-T
28.5M
4.6G
82.2
IntraWin-T
X /
28.3M
4.5G
80.2
IntraWin-T
28.5M
4.6G
82.3
Table 9: Shifted window scheme v.s. 3x depth-wise convolution within FFN based on HRFormer- T.
shifted
3x3 depth-wise
ImageNet topl acc.
COCO PASCAL-Context mloU
#param.
FLOPs
window scheme
convolution within FFN
AP
8.0M
1.8G
78.5
49.7
70.9
X
7.9M
1.6G
76.6
43.3
67.3
architecture configurations of Swin-T (27 except that we do not apply shifted windows scheme. We see that applying 3x3 depth-wise convolution within FFN improves both Swin-T and Intrawin- T. Surprisingly. when equipped with 3x 3 depth-wise convolution within FFN, Intrawin-T even outperforms Swin-T.
Shifted window scheme v.s. 3x3 depth-wise convolution within FEN based on HRFormer- T. In Table [ we compare the 3 depth-wise convolution within FFN scheme to the shifted window scheme based on HRFormer-T. According to the results, we see that applying 3x3 depth- wise convolution within FFN significantly outperforms applying shifted window scheme across all different tasks.
Comparison to VIT, DefT & Swin on pose estimation. We report the COCO pose estimation results based the two well-known transformer models, including ViT-Large [33, DeiT-BA 42 and Swin-B [27] in Table 10. Notably, both ViT-Large and Swin-B are pre-trained on ImageNet21K in advance and then finetuned ImageNet1K and achieve 85.1% and 86.4% top-1 accuracy respectively. DeiT-BBR is trained on ImageNet1K for 1000 epochs and achieves 85.2% top-1 accuracy. We apply deconvolution modules upsample the output representations of the encoder following the SimpleBaseling 49 for three methods. The number of parameters and FL.OPs are listed on the fourth and fifth columns of Table I0] According the results in Table [0] we see that our HRFormer-B achieves better performance than all three methods with fewer parameters and FLOPs.
Comparison to HRNet. We compare our HRFormer to the convolutional HRNet with almost the same architecture configurations via replacing all the transformer blocks with the conventional basic block consisting of two 3 x 3 convolutions. Table IT shows the comparison results on ImageNet, PASCAL-Context, and COCO. We observe that HRFormer significantly outperforms HRNet under various configurations with much less model and computation complexity. For example, HRFormer-T outperforms HRNet-T by 2.0%, 1.5%, and 1.6% on three tasks while requiring only around 50% parameters and FLOPs, respectively. In summary, HRFormer achieves better performance via exploiting the benefits of transformers such as content-dependent dynamic interactions.
Conclusion
In this work, we present the High-Resolution Transformer (HRFormer), a simple yet effective trans- former architecture, for dense prediction tasks, including pose estimation and semantic segmentation. The key insight is to integrate the HRFormer block, which combines local-window self-attention and FFN with depth-wise convolution to improve the memory and computation efficiency, with multi-resolution parallel design of the convolutional HRNet. Besides, HRFormer also benefits from adopting convolution in the early stages and mixing short-range and long-range attention with multi-scale fusion scheme. We empirically verify the effectiveness our Former on both pose estimation and semantic segmentation tasks.

---

Table 10: Comparisons to VIT & DeÎT on COCO pose estimation val. marks the methods
pretrained on ImageNet-22K.
Method
image size
#param.
FLOOPs
COCO
ViT-Large!
256 192
308.5M
60.1C
69.2
DeiT-Bm
256 192
90.0M
17.9G
69.0
Swin-B#
256 192
93.2M
17.6G
74.3
IRRoomer-B
256 192
43.2M
12.2G
75.6
Table 11: Comparisons to HRNet. We report the top1 acc., mloU, and AP on ImageNet val, PASCAL-Context test, and COCO pose estimation val, respectively, Results on PASCAL-Context are based on single-scale testing. The number of parameters and FLOPs are measured on ImageNet.
Method
Sparam.
FLOPs
ImageNet
PASCAL-Context
COCO
HRNee-T
15.6M
2.7G
76.5
47.8
69.3
HRFormer-T
8.0M
1.8G
78.5
49.3
70.9
IIRNet-S
24.5M
5.0G
78.7
52.3
73.1
HRFormer-S
13.5M
3.6G
81.2
53.8
74.0
HRNet-B
85.3M
20.3G
81.4
55.2
75.1
HRFormer-B
50.3M
13.7G
82.8
58.5
75.6
References
[1] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. PAMI, (9(1))2241112222 2112):2221229955 12455, 2017.
[2] Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, July 2021.
[3] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[5] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scake vision transformer for image classification. arXiv preprint arXiv: 14899, 2021.
[6] Liang- Kann-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions pattern analysis and machine intelligence, 40(4):834-848, 2017.
[7] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706. 05587, 2017.
[8] MMPose Contributors. Openmmlab pose estimation toolbox and benchmark.
https://
github. com/ owwn- wwwpeeeemmammmmmneseeeeeeeeeeeee 2020.
[9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, In CVPR, 2016.
[10] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPRW, pages 702-703, 2020.
[11] Zihang Dai. Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. arXiv preprint
2021.
[12] Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint a1Xiv:2103.10697, 2021.
10

---

[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[14] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2104 /227, 2021.
[15] Damien Fourure, Rémi Emonet, Élisa Fromont, Damien Muselet, Alain Trémeau, and Christian Wolf. Residual conv-deconv grid network for semantic segmentation. In BMVC, 2017.
[16] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herve Jegou, and Matthijs Douze. Levit: A vision transformer in convnet's clothing for faster inference. In ICCV, pages 12259-12269, October 2021.
[17] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In NeurlPS, 2021.
[18] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pages 770-778, 2016.
[19] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, pages 3464-3473, 2019.
[20] Junjie Huang, Zheng Zhu, Feng Guo, and Guan Huang. The devil in the details: Delving into unbiased data processing for human pose estimation. In CVPR, June 2020.
[21] Lang Huang, Yuhui Yuan, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Interlaced sparse self-attention for semantic segmentation. arXiv preprint arXiv:1907.122: 2019.
[22] Zihang Jiang. Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng. Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet. arXiv preprint arXiv:2104. 10858, 2021.
[23] Ke Li, Shijie Wang. Xiang Zhang, Yifan Xu, Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. In CVPR, pages 1944-1953, 2021.
[24] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. arXiv preprins
axXiv:2/04.03516, 2021.
[25] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.00 0.7077 2021.
[26] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 1xXi:21133144030 2021.
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint
arXiv:1711.05101, 2017.
[29] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation the wild. In CVPR, 2014.
[30] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. arXiv preprint arXiv:2102.007 19, 2021.
[31] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, pages 483-499, 2016.
[32] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, pages 483-499, 2016.
[33] Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. ICCV, pages 367-376, October 2021.

---

[34] Iija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In CVPR, pages 10428-10436, 2020.
[35] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1 1-67, 2020.
[36] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv:1906.05909, 2019.
[37] Rene Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pages 12179-12188, October 2021.
[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241, 2015.
[39] Shreyas Saxena and Jakob Verbeek, Convolutional neural fabrics. NeurIPS, 29:4053-4061. 2016.
[40] Aravind Srinivas, Tsung-Y Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In CVPR, pages 16519-16529, 2021.
[41] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, pages 6693-5703 2019.
[42] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 22877, 2020.
[43] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2 103. /7239, 2021.
[44] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In CVPR,
pages 12894-12904, 2021.
[45] Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang. et al. Deep high-resolution representation learning for visual recognition. PAMI, 2020.
[46] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. PAMI, 2019.
[47] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.122 12122, 2021.
[48] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang, Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 158.58880,,88 2021.
[49] Bin Xiao, Haiping Wu, and Yichen Wei. Simple basclines for human pose estimation and tracking. In ECCV, 2018
[50] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. arXiv preprint arXiv:2106. 14881, 2021.
[51] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Towards explainable human pose estimation by transformer. arXiv preprint urXiv:2012.14214 2020.
[52] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv: 2015.
[53] Kun Yuan, Shaopeng Guo. Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2103. 1116, 2021.
[54] Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Francis EH Tay. Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. arXiv preprint arXiv:2 1986, 2021.
[55] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation, arXiv preprint 1166, 2019.
12

---

[56] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet: Object context network for scene parsing. arXiv preprint arXiv:1809.009/ 009100999000.0..000 2018.
[57] Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refinement for segmentation. In ECCV, pages 489-506. Springer, 2020.
[58] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon
Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pages
2019.
[59] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-aware coordinate representation for human pose estimation. In CVPR, pages 7093-7102, 2020.
[60] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
[61] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: new vision transformer for nigh-resolution image encoding. ICCV 2021, 2021.
[62] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, pages 2881 1-290, 2017.
[63] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to-sequene
perspective with transformers. In CVPR, 2021.
[64] Yisu Zhou, Xiaolin Hu, and Bo Zhang. Interlinked convolutional neural networks for face parsing. In ISNN, pages 222-231, 2015.
13


Reviews:
[{"review_id": "m8pIjfPbivl", "reviewer": "Reviewer_143m", "summary": "This paper proposes a way to replace the main convolution layers in the HRNet with self-attention layers. A feedforward network with 3x3 depth-wise convolution is also proposed to replace the MLP in transformer. The proposed High-Resolution Transformer achieves performance improvements for dense prediction tasks such as pose estimation and semantic segmentation while saving memory and computation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method presented in this paper is not very novel, but technically sound.\nStrong points:\n- The proposed FFN completes the task of information exchange between different windows well.\n- Experimental results on pose estimation and semantic segmentation show that the proposed High-Resolution Transformer is better than HRNet in terms of performance, memory cost and computation cost.\nWeak points:\n- Not very novel. The main innovation of this paper is to design a FFN with 3x3 depth-wise convolution to perform information exchange across different windows. The rest is simply replacing the main structure of HRNet with the local attention layer.\n- There is no description of limitation in this paper. Considering that DETR has the disadvantage of slow convergence, I wonder how the convergence speed of the architecture proposed in this paper compares with HRNet?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no description of limitation in this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "laxxfNd5kCd", "reviewer": "Reviewer_Jvmd", "summary": "This paper modifies HRNet replacing standard convolutional layers with a transformer block which consists of a local multi-head attention operation (on non-overlapping windows) followed by a residual block with a 3x3 depth-wise convolution. The proposed model is benchmarked on ImageNet, COCO pose estimation, and a number of segmentation datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- combining the advances in dense architecture design from HRNet with transformer blocks is a good step to take, taking advantage of approaches that work well for ConvNets makes a lot of sense as we see this wave of new transformer-based models.\n- thorough experiments show that the proposed block is a good upgrade to HRNet, this is especially clear from the comparison made in Table 10\n- experiments across many benchmarks including classification, pose, and segmentation are comprehensive showing the proposed model is competitive with recent work\n- the proposed block is clear and straightforward\n\nWeaknesses:\n- it seems that a number of relevant results are left out of the benchmark comparisons - notably results that outperform the proposed method. For example, Swin [26] is absent from the ImageNet-1K results (Table 3). I also notice that [19] and [54] are cited but not included in the results in Table 5. While getting state-of-the-art results is nice, it is important to acknowledge recent work even if it has better performance\n- use of a 3x3 depth-wise convolution is a straightforward enough change that it feels as though it gets an unnecessary amount of attention in the paper (for example, I appreciate what Figure 3 is communicating, but I am not sure it is needed). Given the simplicity of the module, it would have been nice to see ablations with other strategies to pass information across windows - like, how does 3x3 conv compare to consecutive local window attention blocks with shifted windows?\n\nOverall I am currently borderline with the paper, I think it is a nice, simple idea that clearly improves a common architecture in the literature. That said, given the simplicity I think it would be nice to see more exploration of the design decisions of the model (e.g. different window sizes at different resolutions, alternatives to pass info across windows besides a conv layer). Also, I think it would be better to not leave out relevant recent work from the benchmark results.\n\n**Update**\n\nI was borderline on the paper, but the authors have put a good amount of work into their rebuttal to address my concerns. While not the most technically novel, it is well motivated and executed. Overall the results are in the ballpark of other work in this space, I personally find Table 10 the most compelling case for the paper as there is plenty of work building off of HRNet, and this proposed version improves it across all tasks while using less params and compute. I raise my score from a 5 to a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "currently no discussion is included in the submission", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "g5RyjZRpZc7", "reviewer": "Reviewer_tHJt", "summary": "In this paper, the authors proposed a new model framework (HRT) using transformers for dense prediction in computer vision tasks.\n\nComparing to conventional patch-based vision transformers for dense prediction (such as segmentation and human pose estimation), HRT has three key components: 1) pixel-based local-windowed multiheaded attention with reduced compute overhead; 2) Depth-wise convolution (inside FFN modules) to aggregate info across local-windowed attention 3) Multi-resolution framework to enhance the representation quality for dense predictions (instead of using deconv after patch-based  VIT transformer encoders).\n\nEmpirically on image classification, human pose estimation and segmentation tasks, the authors shows that HRT can attain higher accuracy at similar or lower param count / FLOPs than 1) conv / deconv baselines and 2) conventional patch-based ViT encoder + deconv baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: I think the techniques (including pixel-based attention, depth-with convolution in FFN and Multi-resolution backbone) are novel to be used in transformer based models for dense prediction in vision tasks. I don't think the fact that depth-width conv and multi-resolution backbone have been used in the convolutional models affects the originality of this paper. By leveraging these techniques in the paper for transformer models, it leads to a new design paradigm for attention based models for dense predictions. \n\nQuality: Given this is empirical-study-oriented work, I think the full model accuracy demonstration is impressive across tasks using various baselines but 1) the ablation study needs to be more systematic to fully discussing the merits of the techniques and 2) Additional results/info needs to be provided to achieve a full evaluation and reproducibility. Specifically:\n\n1. Ablation: In order to ablate out the contribution of each of the proposed technique, it should demonstrate results when each of the technique is ablated out or added (in a technique-individual fashion or an incrementally building-up fashion). In the current ablation study, when comparing to patch-based VIT, it is unclear whether the major benefits is from the pixel-based-attention or from the multi-resolution design. These information are relevant to readers who are looking for the most critical technique in a trade-off between system and statistical performance.\n\n2. Additional results: I suggest reporting of results over multiple random seed with both averaged accuracy and error bar. This will help fully evaluate whether the results are statistically meaningful or reliable. Besides this, key hyperparameters / tuning protocol needs to be revealed to get full reproducibility.\n\nI am willing to raise the score if the above experiment concerns are resolved.\n\n\nClarity: I think the paper is meaningfully well written. Here are a few minor suggestions on further improvement:\n\n1. Figure 1,2 and their captions does not very explicitly cover the fact that it is pixel based attention. This potentially triggers some confusion when people step in with the typical feeling that ViT is patch-based.\n\n2. It might be helpful to visualize the accuracy as a function of param count or flops to compare the different models / methods; this could help better demonstrate the advantage across the spectrum. The current results are organized in tables which is okey but less straight-forward to fully extract the information.\n\nSignificance: Using transformer / attention based model is an active topic for computer vision, the techniques proposed in this paper (such as pixel based attention instead of patch-based) could create very positive impact on the model design trend for various type of vision tasks. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the attached checklist answers the questions adequately. The only thing that worth mentioning is there is no technical limitation discussion (the author give N/A for the questions but I do not see this being a factor that downgrades the score for the review.) I suggest some technical discussion on the situations where the proposed methods might not be preferred etc.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see anything improper or having ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "aEgRGdU7Nh", "reviewer": "Reviewer_G26c", "summary": "The paper proposes a vision transformer like HRNet (a CNN-based visual architecture) for dense prediction. The architecture is tested on classification/segmentation and pose estimation.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Totally, transferring an idea (HRNet) from CNN to vision transformer is ok, but the authors should provide the rationality or effectiveness. Though the architecture is different from some recent vision transformer, the improvement is marginal.\n\nOriginality: The idea of this paper is highly like HRNet, and the attention mechanism in Figure 1 is not a new thing. Actually, most new vision transformers reduce the cost by divide-and-conquer. For example, SwinTransformer etc. all resort to self-attention in a local region.\n\nClarity: The paper is easy to follow.\n\nOthers (to be improved):\n1. The authors cite some papers: PVT/Swin/CvT etc. in related works. So, I think the authors have read these papers before finishing their own work. Since there have been very few published vision transformers, I think it is neccessary to compare with these arxiv-ones. I think the design of PVT/Swin/CvT are simpler than this paper and they show better performance, which makes me doubt the effectiveness of HRNet in vision  transformers.\n\n2. I think the results in Table 4 show the significance of vision transformer over CNN, but does not prove that HRT is a good choice for vision transformer.\n\n3. A little advice: In table 3, all models take a 224x224 image as input, it is also ok to explain it in the caption rather than repeat it again and again in the table.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a "high-resolution Transformer" architecture for dense prediction tasks using a multiresolution hierarchy and self-attention over local windows. All reviewers agreed that the ideas were sensible and technically sound, and that the experiments demonstrated improvement. There were concerns about missing comparisons (reviewers Jvmd, G26c) and missing ablations (tHJt, Jvmd), but the authors seem to have addressed all of these in rebuttal. There were also concerns of limited novelty (reviewers 143m, G26c), but I don't think this alone should disqualify the paper. Therefore I recommend acceptance.
Decision: Accept (Poster)