GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagation
Authors: Chenhongyi Yang
Jiarui Xu
Shalini De Mello
Elliot J. Crowley
Xiaolong Wang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Visual Recognition
Vision transformer architecture

Abstract: We present the Group Propagation Vision Transformer (GPViT): a novel non- hierarchical (i.e. non-pyramidal) transformer model designed for general visual recognition with high-resolution features. High-resolution features (or tokens) are a natural fit for tasks that involve perceiving fine-grained details such as detection and segmentation, but exchanging global information between these features is expensive in memory and computation because of the way self-attention scales. We provide a highly efficient alternative Group Propagation Block (GP Block) to exchange global information. In each GP Block, features are first grouped to- gether by a fixed number of learnable group tokens; we then perform Group Propagation where global information is exchanged between the grouped fea- tures; finally, global information in the updated grouped features is returned back to the image features through a transformer decoder. We evaluate GPViT on a variety of visual recognition tasks including image classification, semantic seg- mentation, object detection, and instance segmentation. Our method achieves significant performance gains over previous works across all tasks, especially on tasks that require high-resolution outputs, for example, our GPViT-L3 out- performs Swin Transformer-B by 2.0 mIoU on ADE20K semantic segmentation with only half as many parameters. Code and pre-trained models are available at https://github.com/ChenhongyiYang/GPViT.

Published as a conference paper at ICLR 2023
A HIGH RESOLUTION NON-HIERARCHICAL VISION TRANSFORMER WITH GROUP PROPAGATION
GPVIT:
Chenhongyi Yang"! Jiarui Xu* Shalini De Mello Elliot J. Crowley'
Xiaolong Wang?
'School of Engineering, University of Edinburgh
UC San Diego 'NVIDIA
ABSTRACT
We present the Group Propagation Vision Transformer (GPViT): a novel non- hierarchical (i.e. non-pyramidal) transformer model designed for general visual recognition with high-resolution features. High-resolution features (or tokens) are natural fit for tasks that involve perceiving fine-grained details such as detection and segmentation, but exchanging global information between these features is expensive in memory and computation because of the way self-attention scales. We provide a highly efficient alternative Group Propagation Block (GP Block) to exchange global information. In each GP Block, features first grouped to- gether by a fixed number of learnable group tokens; we then perform Group Propagation where global information is exchanged between the grouped fea- tures; finally, global information in the updated grouped features is returned back the image features through a transformer decoder. We evaluate GPVIT on a variety of visual recognition tasks including image classification, semantic seg- mentation, object detection, and instance segmentation. Our method achieves significant performance gains over previous works across all tasks, especially on tasks that require high-resolution outputs, for example, our GPVIT-L3 out- performs Swin Transformer-B by 2.0 mloU on ADE20K semantic segmentation with only half as many parameters. Code and pre-trained models are available https: / /gtttub...///ggg/// com/ Chwethhngyyaangggeeeoooa GGVi.
INTRODUCTION
Vision Transformer (ViT) architectures have achieved excellent results in general visual recognition tasks, outperforming ConvNets in many instances. In the orig- inal VIT architecture, image patches are passed through transformer encoder lay- ers, each containing self-attention and MLP blocks. The spatial resolution of the im- age patches is constant throughout the net- work. Self-attention allows for information to be exchanged between patches across the whole image i.e. globally, however it is computationally expensive and does not place an emphasis on local information ex- change between nearby patches, as a con- volution would. Recent work has sought to build convolutional properties back into vision transformers (Liu et al., 2021; Wu al., 2021; Wang et al., 2021) through a hierarchical (pyramidal) architecture. This design reduces computational cost, and improves ViT performance tasks such as detection and segmentation.
High-resolution features
Feature Grouping
Group Propagation
Feature Ungrouping
An illustration of our GP Block. It groups image features into fixed-size feature set. Then, global information is efficiently propagated between the grouped features. Finally, the grouped features are queried by the image features to transfer this global information into them.
Figure I:
Is this design necessary for structured prediction? It incorporates additional inductive biases c.g. the assumption that nearby image tokens contains similar information, which contrasts with the
"Equal Contribution

---

Published as a conference paper at ICLR 2023
motivation for ViTs in the first place. A recent study (Li et al., 2022a) demonstrates that plain non-hierarchical ViT. a model that maintains the same feature resolution in all layers (non-pyramida), can achieve comparable performance counterpart. How do we go one step further and surpass this? One path would be to increase feature resolution (i.e. the number of image tokens). A plain VIT with more tokens would maintain high-resolution features throughout the network as there is no downsampling. This would facilitate fine-grained, detailed outputs ideal for tasks such as object detection and segmentation. It also simplifies the design for downstream applications, removing the need to find a way to combine different scales of features in hierarchical ViT. However, this brings new challenges in terms of computation. Self-attention has quadratic complexity in the number of image tokens. Doubling feature resolution (i.e. quadrupling the number of tokens) would lead 16x increase in compute. How do we maintain global information exchange between image tokens without this huge increase in computational cost?
object detection and segmentation tasks hierarchical
In this paper, we propose the Group Propagation Vision Transformer (GPVIT): non-hierarchical ViT which uses high resolution features throughout, and allows for efficient global information exchange between image tokens. We design novel Group Propagation Block (GP Block) for use in plain ViTs. Figure provides a high-level illustration of how this block works. In detail, we use learnable group tokens and the cross-attention operation to group a large number of high-resolution image features into a fixed number of grouped features. Intuitively, we can view each group as a cluster of patches representing the same semantic concept. We then use an MLPMixer (Tolstikhin et al., 2021) module to update the grouped features and propagate global information among them. This process allows information exchange a low computational cost, as the number of groups is much smaller than the number of image tokens. Finally, we ungroup the grouped features using another rrossaattention operation where the updated grouped features act as key and value pairs, and are queried by the image token features. This updates the high resolution image token features with the group-proyy proppppropagtte information. The GP Block only has a linear complexity in the number of image tokens, which allows it to scale better than ordinary self-attention. This block is the foundation of our simple non-hierare non-hierarchical vision transformer architecture for general visual recognition.
We conduct experiments multiple vi- sual recognition tasks including image clas- sification, object detection, instance seg- mentation, and semantic segmentation. We show significant improvements over pre- vious approaches, including hierarchical vision transformers, under the same model size in all tasks. The performance gain is especially large for object detection and segmentation. For example, in Figure 2, we show GPVIT's advantage over the non- hierarchical DeiT (Touvron et al., 2021a) and hierarchical Swin Transformer (Liu et al., 2021) those recognition tasks. In addition, our smallest model GPViT-LI can outperform the Swin Transformer-B (Liu et al., 2021) by 2.6 Apb and 1.4 in COCO Mask R-CNN (He et al., 2017) ob- ject detection and instance segmentation with only 30% as many parameters, and GPViT-L2 outperforms Swin Transformer-B B by 0.5 mloU on UperNet (Xiao et al., 2018) ADE20K semantic segmentation also with only 40% as many parameters.
DeiT-B
Swin Transformer-Base
GPVIT-L4
Accuracy
88
AP
Image Classification
Object Detection
Mask AP
Instance Segmentation
Semantic Segmentation
Figure 2: A comparison on four visual recognition tasks between GPVIT and the non-bierarchical DeiT (Touvron et al., 2021a) and the hicrarchical Swin Transformer (Liu et al., 2021).
2 RELATED WORK
Vision Transformers. Vision Transformers have shown great success in visual recognition. They have fewer inductive biases, e.g. translation invariance, inlee-ivarriance, and feature locality (Xu et al., 2021b) than ConvNets and can better capture long-range relationships between image pixels. In the original ViT architecture (Dosovitskiy et al., 2021; Touvron et al., 2021a), images are split into patches and are transformed into tokens that passed through the encoder of transformer (Vaswani et al.,
2

---

Published as a conference paper at ICLR 2023
2017). Based this framework, LeViT (Graham et al., 2021) achieves significant performance improvement over ViT by combining convolutional and transformer encoder layers. An important development in VIT architectures is the incorporation of hierarchical feature pyramid structure, as typically seen in ConvNets (Wang et al., 2021; Liu et al., 2021; Xu et al., 2021a; Wu et al., 2021; Fan et al., 2021). For example, Liu et al. (2021) propose shifted windowing scheme to efficiently propagate feature information in the hierarchical ViT. Such pyramid architecture provides multi-scale features for a wide range of visual recognition tasks. Following this line of research, recent work has studied the use of hierarchical features in ViT's (Ren et al., 2022b; Guo et al., 2022; Li al., 2022b; Dong et al., 2022; Hatamizadeh al., 2022; Chen et al., 2022a; d' d'Ascoli al., 2021; Lee et al., 2022). For example, Ren et al. (2022b) introduce using multi-resolution features as attention keys and values to make the model learn better multi-scale information. While this is encouraging, it introduces extra complexity in the downstream model's design on how to utilize the multi-scale features effectively. Recently, Li et al. (2022a) revisited the plain non non-hierarchical ViT for visual recognition; using such model simplifies the use of features and better decouples the pre-training and downstream stages of model design. Our work extends on this as we examine how to efficiently increase the feature resolution in non-hierarchical VIT.
Attention Mechanisms in ViTs. A bottleneck when using high resolution features in ViTs is the quadratic complexity in the computation of the self-attention layer. To tackle this challenge, several local attention mechanisms have been proposed (Liu et al., 2021; Huang et al., 2019; Dong et al., 2022; Xu et al., 10; Zhang al., 2022; Han al., 2021) allow each image token to attend to local region instead of the whole image. However, using only local attention hinders model's ability to exchange information globally. To counter this problem, Region ViT (Chen et al., 2022a) and GCVIT (Hatamizadeh et al., 2022) first down- their feature maps and exchange global information between the down- samp-s features, before using elffaatentionn to transfer information between the original image features and the down-sampled features. This is similar in spirit to GP Block the grouped features are
our GP Block. However, unlike Region ViT and GCVIT, not constrained to particular rectangular region, but can correspond to any shape even entirely disconnected image parts. There is recent work using transformer decoder layers with cross-attentinn between visual tokens and learnable tokens (Carion et al., 2020; Cheng et al., 2022; Jaegle et al., 2022; Hudson & Zitnick, 2021), however, there are three fundamental differences between these and ours: (i) Each of our GP blocks operates
'encoder-decoder architecture with two rounds of cross-attention between visual tokens and group tokens: the first round groups the visual tokens for group propagation, and the second round ungroups the updated groups back into visual tokens; (ii) The underlying functionality is different: GP blocks facilitate more efficient global information propagation throughout the ViT, while previous work applies the decoder obtain the final results for inference (e.g bounding boxes, or masks in Carion et al. (2020); Cheng et al. (2022)); (iii) The GP block is a general module that can be insert into any layer of the VIT, while previous work utilizes the decoder only in the end of the network.
High- High-Resolution Visual Recognition. Previous work (Wang et al., 2020; Cheng et al., 2020) has shown that high-resolution images and features are beneficial to visual recognition tasks, especially to those requiring the perception of fine-grained image details, for example, semantic segmenta- tion (Wang et al., 2020), pose-estimation (Sun et al., 2019), and small object detection (Yang et al., 2022). For example, HRNet (Wang et al., 2020) introduces a high-resolution ConvNet backbone. It maintains high-resolution branch and exchanges information between different resolutions of features with interpolation and strided convolutions. Inspired by this work, HRFormer (Yuan et al., 2021) and HRViT (Gu et al., 2022) replace the convolutions in HRNet with self-attention blocks. GPVIT is even simpler: it maintains single-scale and high-resolution feature maps without requiring any cross-resolution information be maintained.
Object-Centrie Representation. Our idea of performing information propagation among grouped regions is related to object-centric representation learning (Wang Gupta, 2018; Kipf & Welling, 2017; Watters et al., 2017; Qi et al., 2021; Locatello et al., 2020; Kipf et al., 2022: Elsayed et al., 2022; Xu et al., 2022). For example, Locatello et al. (2020) proposes slot-attention, which allows automatic discovery of object segments via a seff-supervised reconstruction objective. Instead of using reconstruction, Xu et al. (2022) utilizes language as an alternative signal for object segmentation discovery and shows it can be directly transferred to semantic segmentation in a zero-shot manner. All the above work extract object-centric features for downstream applications, while our work inserts this object-centric information propagation mechanism as a building block inside ViTs to compute
3

---

Published as a conference paper at ICLR 2023
Local
GP Blogg
Local Attm
Local Ath
Locd Atto
Sanulae Alti
(a) Overall Architecture of Group Propagation Vision Transformer (GPVIT)
W.
Murtipy
Softmax
W
Group
Toikons
St
20000
Nill
RUDO AG
Grouped
W
L.
Image
Feaiture
Residual
Connection
Feature Grouping
Group Propsgation
Feature Ungrouping Projection
(b) Architecture of Group Propagation Block (GP Block)
Figure 3: (a), GPVIT architecture: The input image first fed convolutional stem that downsamples by a factor 8. Each pixel of this downsampled image is treated as high resolution image token or feature, and positional embeddings are added (not shown in figure). These features are fed into our transformer, which consists of 12 encoder layers. of these use local attention, and 4 use proposed GP Block to propagate global information between image features. (b). GP Block: Image features grouped using fixed number learnable group tokens. MLPMixer module then used to exchange global information grouped features. Next, the grouped features are queried by, and concatenated with the image features to transfer global information to every image feature. Finally the updated image features are transformed by feed-forwart forward network to produce the output.
update
high-resolution representations more efficiently and improve high-resolution features. In this respect, our work is related to Li & Gupta (2018) where the graph convolution operations are inserted into ConvNets for better spatial reasoning.
3
METHOD
We present the overall architecture of our Group Propagation Vision Transformer (GPViT) in Figure (a). GPVIT is designed for general high-resolution visual recognition. For stable training, we first feed the input image into a down- wown-sampling convolutional stem generate image features (also known as image tokens), as in Dosovitskiy et al. (2021); Liu et al. (2021). In GPVIT we downsample by factor by default. The features are therefore higher resolution than in the original VIT where the factor is 16. Unlike most recently proposed methods (Liu et al., 2021; Li et al., 2022b) that adopt a pyramid structure to generate features in multiple resolutions, we keep the features a high resolution without any down-sampling.
After combining the initial image features with positional embeddings (Vaswani et al., 2017), we feed them into the core GPVIT architecture. We replace the original self-attention block in ViT with local attention to avoid the quadratic complexity of self-attention. However, stacking local attention blocks alone does allow for long-range information exchange between patches and therefore is harmful to performance. To counter this problem, we propose the Group Propagation Block (GP Block)-which we describe in full in Section 3. 3.1 efficiently propagate global information across the whole image. In our implementation, we use mixture of GP Blocks and local attention layers to form our GPVIT and keep the overall depth unchanged. Lastly, we average the final features get the model's output.
3.1 GROUP PROPAGATION BLOCK
Our key technical contribution the GP block, which efficiently exchanges global information between each image patch with linear complexity. We visualize the structure of the GP block in Figure (b). It has a bottleneck structure and comprises of three stages, namely, Feature Grouping, Group Propagation, and Feature Ungrouping. In the first stage the image features are grouped, then in the second stage global information propagated between the grouped features, and in the last stage, this global information is transferred back to the image features.
4

---

Published as a conference paper at ICLR 2023
Feature Grouping. The imput GP Block is matrix of image features X E RNCC (The blue tokens in Figure 3 (b)) where N is the total number of image features (or image tokens) and C is the dimensionality of each feature vector. We use M learnable group tokens stored in matrix G E TRMCC (the multi-colored tokens Figure (b)) where the group numher M is a model hyper-parameter. Grouping is performed using simplified multi-head attention operation (Vaswani et al., 2017), which gives us grouped features Y E I M XC (the half-and-hale tokens in Figure (b)):
QKT Vd
Attention(Q,4s V) = Softmax
V
(1)
Y = Concatqay ( Gn, W XA, W Xn)),
where d is the channel number, à is the head index, and W (Q. are projection matrices for the query, key, and values, respectively in the attention operation. We remove the feature projection layers after the concatenation operation and set W and W be identity matrix. Therefore, the grouped features are simply the weighted sum of image features at each head where the weights are computed by the attention operation.
Group Propagation. After acquiring the grouped features, we can update and propagate global information between them. We use an MLPMixer (Tolstikhin et al., 2021) (Equation 3; the red box in Figure 3 (b)) to achieve this, MLPMixer provides good trade-off between model parameters, FLOPs, and model accuracy. MLPMixer requires a fixed-sized input, which compatible with our fixed number of groups. Specifically, our MLPMixer contains two consecutive MLPs. Recall that Y E MxC contains the grouped features from the first Feature Grouping stage. We can update these features to E BMCC with the MLPMixer by computing:
Y' Y + MLP, (LayerNorm (Y)
ÿ Y" + (LayerrLa)o (Y)))
(4)
where the first MLP is used for mixing information between each group, and the second is used to mix channel-wise information.
Feature Ungrouping, After updating the grouped features, we return global information the image features through a Feature Ungrouping process. Specifically, the features are ungrouped using transformer decoder layer where grouped features are queried by the image features.
U Concatqay (Attention(W:) ( Xn, W Yn, W Yn),,
Z' W proj Concat(U,
=' FFN(Z
Z -DWConvZ")).
(6)
where W (.(Q.KV)) are the projection matrices in the attention operation, W Wrojj is a linear matrix that projects concatenated features Z' to the same dimension as image features X, FFN is forw-ror network, and DWConv is a depth-wise convolution layer. We modify the original transformer decoder layer by replacing the first residual connection with a concatenation operation (Equation 5; the blue box in Figure 3 (b)), and move the feature projection layer after this to transform the feature to the original dimension. We find this modification benefits the downstream tasks in different sizes of models. We take inspiration from Ren al. (2022b) and add a depth-wiss wise convolution the end of the GP Block to improve the locality property of the features (Equation 6; the yellow box in Figure (b)). Finally, GP Block outputs Z as its final output.
Table 1: GPVIT architecture variants.
Model
Channels
Param (M)
FLOPs (G)
GPVIT-LI
216
9.3
5.8
GPVIT-L2
348
23.6
15.0
GPVIT-L3
432
36.2
22.9
GPVIT-LA
624
75.4
48.2
3.2 ARCHITECTURE VARIANTS OF GPVIT
In this paper we study four variants of the proposed GPVIT. We present their architectural details in Table 1. These four variants largely differ in the number of feature channels used (i.e. the model width). We the recently proposed LePE attention (Dong al., 2022) local attention by default. The FLOPs are counted using 224x224 inputs. Please refer to Section B. in our Appendix for detailed architectural hyper-parameters and training recipes for these variants.
5

---

Published as a conference paper at ICLR 2023
3.3 COMPUTATIONAL COSTS HIERARCHICAL AND HON-HIERARL VITs.
We visualize both the non-hierarchical and hierarchical VIT in Figure (a), where the non-hieen non-hierarchical ViT sim- ply stacks attention blocks and the hierarchical ViT divides the network into several stages and down-samples the feature map each stage. Natu- rally, with the same resolution input, the non- non-hierarchical ViT will have higher computation cost. The cost is divided into parts shown in Figure 4 (b): the self-attention mod- ule and the FFN module. Our GPViT largely reduces the computation of in- formation propagation by using our GP Block instead of self-attention. However, the cost of the FFN stays high for high resolution image fea- tures. Therefore we will expect higher FLOPs from GPVIT compared hicrarchical VIT given similar model parameters. However, believe non- hierarchical ViTs are still a direction worthy of exploration given their simplicity in extracting high-resolution features and the removal of the need to study the design of efficient downstream models that utilize multi-scale features as required for a hierarchical ViT. This helps to maintain the independence of the model's pre-training and fine-tuning designs (Li al., 2022a). In our experiments, we show that our GPVIT can achieve better detection and segmentation performance compared ttate-ofthe-as hicrarchical ViTs with similar FLOP counts.
Atti Block
x L
O(n)
FFN
Patch Merge
Norm
L
Attn Block
L
Patch Mergo
Altention
O(n3)
Attn Block
Altn Block
L
Norm
Input Image
Input Image
Non-Hierarchical
Hierarchical
Attention Block
Complexity
(a) Two types of vision transformer
(b) Complexity Analysis
Figure 4: (a). Structure comparison between non oiehhiearee and hierarchical ViTs. (b). The computation cost of an attention block.
EXPERIMENTS
4.1 IMAGENET-IE K. CLASSIFICATION
Setting: To ensure fair comparison with previous work, we largely follow the training recipe of Swin Transformer (Liu et al., 2021). We build models using the MMClassification (Contributors, 2020a) toolkit. The models are trained for 300 epochs with a batch size 2048 using the Adam optimizer with weight decay of 0.05 and a peak learning rate of 0.002. A cosine learning rate schedule used to gradually decrease the learning rate. We use the data augmentations from Liu et al. (2021): these include Mixup (Zhang et al., 2017), Cutmix (Yun et al., 2019), Random erasing (Zhong al., 2020) and Rand augment (Cubuk et al., 2020).
We compare GPVIT with hierarchical and non- hicrarchical vision transformers on the ImageNet-IK classification task and report the results in Table 2. As shown in the table, because of our high-resolution design and effective global information propagation via the grouping mechanism, our GPVIT outperforms outperforms the non-hierarchical baseline DeiT (Tou- vron et al., 2021a). In addition, GPVIT also outper- forms Swin Transformer (Liu al., 2021) and two recently proposed hierarchical counterparts Region- ViT (Chen et al., 2022a) and DWVIT' (Ren al., 2022a). This result showcases the potential of non- hierarchical vision transformers and suggests that the hierarchical design inherited from the ConvNet era is not necessary for obtaining a performin visual recognition model. This corroborates the work of Li et al. (2022a). That said, we do note that the FLOPs of our models are higher than most alternatives for
Table Comparison between GPVIT recent proposed models on ImageNet-1K
Params FLOPs Top-1 (M)
Model
(G) Acc
Ilieranchiaal
Swin-T (Liu et al., 2021)
29.0
4.5
81.3
Swin-B (Liu et al., 2021)
88
15.4
83.5
Region (Chen et al., 2U22a)
30.6
82.6
Region VTT-B (Chen el al., 2022a)
72.7
13.0
83.2
DW-T (Ren al., 2022a) DW-B (Ren et al., 2022a)
30.0
5,2
82.0
91.0
17.0
83.8
Non-hierarchical
DeïT-S (Touvron et al., 2021a)
22.1
4.6
79.9
DeiT-B (Touvron et al., 2021a)
86
16.8
81.8
ConV (d' Ascoli et al., 2021)
27.0
81.3
ConV (d' Ascoli
2021)
86.0
17.0
82.4
GPVIT-LI
9,3
5,8
80.5
GPVIT-L22
23.8
15.0
83.4
GPVIT-L3
36.2
84.1
GPVIT-LA
75.4
48.2
84.3

---

Published as a conference paper at ICLR 2023
Table 3: Mask R-CNN object detection and instance segmentation on MS COCO mini-val using xx and (or longer) + MS schedule.
Params FLOPs
1x
3x more
Backbone
(M)
(G)
AAP
A
A
APou
Hierarchical
RegionVET-B (Chen et al,, 2022a)
92
287
44.2
40.8
47.6
43.4
Swin-B (Liu et al,, 2021)
107
496
45.5
41.3
DaViT-Base (Ding al., 2022)
107
491
48.2
43.3
49.9
44.6
DW-B (Ren al., 2022a)
111
505
49.2
44,0
CSwin-B (Dong et al., 2022)
97
526
48.7
43.9
50.8
44,9
MPVIT-B (Lee al., 2022)
95
503
49.5
44.5
Non-hierarchical
ViT-Adapter-B (Chen et al., 2022b)
102
47.0
41.8
49.6
43.6
ViTDet-SUP (Li et al., 2021a)
111
800
47.6
42.4
ViTDet-MAE' (Li et al., 2021a)
111
800
51.6
45.2
GPViT-LI
33
457
48.1
42.7
50.2
44.3
GPVIT-L2
50
690
49.9
43.9
51.4
45.1
GPViT-L3
64
884
50.4
45.2
GPVIT-LA
109
1489
51.0
45.0
52.1
45.7
:: ViTDel (Li ata, 2021a) mudels were traindd for 100 epochs with advanced regulatisaiion techniquees.
similar parameter count. However, for a similar FLOP count we observe that GPViT can achieve comparable top-1 accuracy, but with many fewer parameters than the alternatives. For example, GPViT-L2 (15.0 G) has similar FLOPs to the Swin Transformer-B (15,4 G) and VhiftViT-B (15.6 G), but it achieves similar accuracy with significantly fewer parameters (23.8 M v.s. 88 M and 89 M).
4.2 COCO ORJECT DETECTION AND INSTANCE SEGMENTATION
Setting: We follow Chen al. (2022b) to Mask R-CNN and RetinaNet models for the COCO object detection and instance segmentation tasks. We use ViTAdapter (Chen et al., 2022b) to generate multi-scale features as FPN inputs and evaluate the model for both 1 and 3x training schedules.
Results: We compare GPVIT to state-of-the-art backbones, all pre-trained on ImageNet-IK. K. We report the results in Table 3 and Table 4. For competing methods we report the performance of their largest-sized models. For both detectors our GPVIT able to surpass the other backbones by large margin for a similar parameter count. With Mask R-CNN (Table 3), our smallest GPViT-L1 sur- passes its Swin Transformer-B (Liu et al., 2021) counterpart by 2.6 AP' and 1.4 Apmk for the 1x training schedule with fewer FLOPs and only 30% as many parameters. When comparing with models that are also equipped with ViTAdapter (Chen et al.,
Table 4: RetinaNet object detection MS COCO mini- with and 3x +MS schedule.
Params FLOPs
Backhone
(M)
(G)
Ap Apel
Hierarchical
PVT- (Wang al., 2021)
71
345
42.6
PVTv2-B5 (Wang et al., 2022h)
91
335
46.1
Swin B (Liu et al., 2021)
98
477
44.7
Region VTTB (Chen et al., 2022a)
83
308
43.3
46.1
MPViT B (l.ce et al., 2022)
95
503
48.3
DaViT-Base (Ding et al., 2022)
103
471
46.7
48.7
Non-hierane hor-hierarchiall
GPVITLI
21
317
45.8
48.1
GPVIT-L2
37
542
48.0
49.0
GPVIT-L3
52
731
48.3
49.4
GPVIT-LA
96
1319
48.7
49.8
2022b), we observe that GPViT achieves a better AP with fewer parameters, e.g. our smallest GPViT-LI outperforms ViT-Adapter-B in both training schedules. These results showcase GPViT's effectiveness at extracting good regional features for object detection and instance segmentation. A similar conclusion can be drawn from the single-stage RetinaNet detector, with RetinaNet (Table 4), GPViT-LI has FLOPs similar to the recently proposed Region ViT-B (Chen al., 2022a), but outperforms Region ViT-B by 2.5 and 2.0 Apo in both 1x and 3x schedules with only 25% as many parameters. In Table 3, we also compare our Mask R-CNN with the recently proposed ViTDet (Li et al., 2021a) that also uses a mon-hierarchical VIT as the backbone network. Here we continue to use the standard 3x (36 epochs) training recipe for GPVIT. The results show that under similar FLOPs, even if ViTDet is equipped with more parameters (111M), advanced masked-ame aote---uttt--dddee (MAE) pre-training (He al., 2022), longer training schedule (100 epochs), and heavy regularizations like large-scale jittering (Ghiasi et al., 2021), )), our model can still achieve comparable performance, which further validates the effectiveness of GPVIT.
7

---

Published as a conference paper at ICLR 2023
Table 5: Comparison between GPVIT and other vision transformers on ADE20K semantic segmentation task.
UperNet
SegFormer
Params (M)
FLOPS
Params FLOPs (M)
Backbone
(G)
mloU
Backbone
(G)
mlnti
Hierarchical
Hicrarchical
Swin-B (Liu et al., 2021)
121
1188
49.7
MiT B2 (Xie et al., 2021)
24
64
46.5
DAT-T (Xia et al., 2022)
121
1212
50.5
MIT-B4 OXie et al., 2021)
61
96
50.4
DaVIT Base (Ding al., 2022)
121
1175
49.4
Stunned-S (Ren et al., 2022h)
25
78
48.3
MPVTT-B (l.ce et al., 2022)
105
1186
50.3
CSwin-S(Dong al., 2022)
37
78
49,9
DW-B el al., 2022a)
1200
HRVIT-h3(Gu et al., 2022)
29
68
50.2
Non-hierarchical
(leung al., 2022)
31
Deit B (Touvron al., 2021a)
120
786
45.3
HILA+MiT-B2 (Leung et al., 2022)
31
76
46
ViT-Adapter-B B (Chen et al., 2022b)
134
48.1
Non- hin-hiirarchical
GPVITII
568
49.1
GPVIT-LI
9
34
46,9
GPVIT-L2
53
775
50.2
GPVIT-L2
24
83
49.2
GPVIT
66
944
51.7
GPVIT-L.3
36
123
50.8
GPVIT-L4
1469
52.5
GPVIT-L4
76
249
51.3
4.3 ADE20K SEMANTIC SEGMENTATION
Setting: We follow previous work (Liu et al., 2021) and use UperNet (Xiao et al., 2018) as the segmentation network. We also report performance when using the recently proposed SegFormer (Xie et al., 2021) model. For both models, we train for 160k iterations.
Results: We summarise the segmentation performance of GPVIT and other state-of-the-as backbone networks in Table 5. For UperNet, report results with the largest available model size for the competing methods to show how far we can go in the segmentation task. Thanks to its high-resolution design, GPVIT outperforms all competing methods in mloU with fewer FLOPPs and fewer parameters. For example, GPViT-LI only has 37M parameters but it can achieve comparable mIoU to methods with only half the number of FLOPs. This result tells us that for tasks requiring the perception of fine-gras une-grained details, scaling-up feature resolution is a better strategy than scaling up model size. GPVIT also excels when used with SegFormer. Specifically, GPVIT achieves better mIoU than recently proposed vision transformers with similar parameter counts, including HRVIT (Gu et al., 2022) that was specifically designed for semantic segmentation. We attribute these promising results to GPVIT's high-resolution design and its effective encapsulation of global information.
4.4 ABLATION STUDIES
Setting: We conduct ablation studies using two types of local attention: the simple window atten- tion (Liu et al., 2021) and the more advanced LePE attention (Dong al., 2022), which used in previous experiments. We use the L. level models (Param < 10 M) for all experiments. All models are pre-trained ImageNet classification task for 300 epochs using the same setting in Section 4. We report both ImageNet Top-1 1 accuracy and ADE20K SegFormer mIOU. Please refer to our appendix for more ablation experiments.
Building GPVIT step by step. Here we show how we build GPViT step by step and present the results in Table 6. We start building our GPVIT from a low- resolution vanilla DeiT with patch sizes of 16 and embedding channels of 216 (same as GPViT-Tiny). It achieves 77.4 t accuracy on ImageNet and 42.2 mIoU ADE20K. Then we increase the resolution by shrinking the patch size to 8. The FLOPs of the ImageNet and ADE20K models increase by 4.4x and 7.0x respectively. ImageNet accuracy increases to 79.2 but training this model for segmentation proves to be unstable. We see that enlarging the feature resolution using global self-attention leads to the number of FLOPs exploding and makes convergence difficult. We now replace self-attention with window attention (Liu et al., 2021) and the more advanced LePE attention (Dong et al., 2022). For both local attention mechanisms, the FLOPs of the ImageNet and ADE20K models drop to 5.8G and 34G respectively. We then incorporate GP Blocks, and observe that the accuracy and mloU improve for both types of local attention and FLOPs remain unchanged. These results showcase the effectiveness of using oighresoouution features as well as the importance of our combination of local attention blocks and GP Blocks to maintain a reasonable computation cost.
Table 6:
Ablation studies on feature resolution
and GP Block using two different local attentions.
FLOPs
Top-1
FLOPs
Setting
(G)
Acc
(G)
mloU
VTT-D216-P16 VIT-D216-P8
1.8
77.4
16
42.2
8.8
79.2
113
+ Win alln
5.8
78.1
34
41.7
+ GP Block
5.8
79.8
34
45.5
+ LcPE attn
5.8
79.5
34
46.2
+ GP Block
5,8
80.5
34
46.9
8

---

Published as a conference paper at ICLR 2023
Global information exchange. Here, com- pare our GP Block with other blocks that can exchange global information between image features. 'Tee competing blocks include the global attention block, the convolution propa- gation block (Li et al., 2022a), and the shifting window-attention block (Liu et al., 2021) de- signed for window attention. We follow ViT- Det (Li al., 2022a) to build the convolution propagation block that stacks two 3x3 convo- lution layers with a residual connection. We use the original version of the shifting window- attention block as in Liu et al. (2021). The result- ing models are acquired by putting competiting blocks in the same place as as our GP Block. We report the results Table 7. We observe that simply replacing the local attention layers with convolution layers causes severe performance drops for both types of local attention. We also observe that replacing local attention with global attention can improve performance for a very large increase in FLOPs. For window attention, found that using the shifting window strategy slightly hurts the performance. We postulate that this is caused by deficit of shifting window layers; half of Swin Transformer layers are shifting window layers, but we only use four here. For both types of local attention, GP Block achieves the best performance on ImageNet and ADE20K. These results show GP Block's effectiveness in propagating global information.
Table 7: Ablation study on different global information propagation methods.
Global
FLOPs
Top-1
FLOPss
Attention
Info
(G)
Acc
(G)
mloU
None
78.2
34
41.7
Conv
6.6
75.8
38
39.5
Window
Global attn
6.8
78.8
62
44.0
Win shift
5.8
78.1
34
40.7
Block
5.8
79.8
34
45.5
None
79.5
34
46.2
Conv
6.6
77.7
38
45.5
LePE
Global atln
6.8
80.4
62
46.7
GP Block
5.8
80.5
34
46.9
Number of group tokens. Here we study how the different combinations of the number of groups tokens in GP Blocks affect the overall model performance. We report the results in Table 8. We find using a large number group tokens across the whole network can give higher accuracy on ImageNet but at additional computational cost. However, using too few group e.g. 16 tokens will harm performance, In GPVIT we choose progressively decrease the number of group tokens from 64 to 16. This strategy gives us a good trade-off between accuracy and computational cost.
Table 8: Ablation study the group tokens number combinations.
Combination
FLOPs (G)
Top-1 1 Acc
{16, 16, 16, 16)
5.7
79,9
(32, 32, 32, 32)
5.8
80.3
(64, 64, 64, 64
6.0
80.7
{16, 32, 32, 64)
5.8
80.0
[64, 32, 32, 16)
5.8
80.5
Grouped features propagation. In Table 9 we compare different methods for global informa- tion propagation. The results show that even when we add nothing to explicitly propagate global information the model can still achieve a good performance (79.8% accuracy on Ima- geNet). The reason is that in this case the image features are still grouped and ungrouped so the global information can still be exchanged in these two operations. We also find that self-attention achieve a slightly better accuracy than MLPMixer (80.7 v.s. 80.5), but is more expensive. In GPViT we use MLPMixer for propagating global information.
Table Ablation study on the propagation approach
of grouped features.
Method
FLOPs (G)
Top-1 Acc
None
5.7
79.8
SelfAttn
6.2
80.7
MLPMixer
5.8
80.5
CONCLUSION
In this paper, we have presented the Group Propagation Vision Transformer (GPViT): a non- hierarchical vision transformer designed for high-resolution visual recognition. The core of GPViT is the GP Block, which was proposed to efficiently exchange global information among high-resolution features. The GP Block first forms grouped features and thxen updates them through Group Propaga- tion. Finally, these updated group features are queried back to the image features. We have shown that GPVIT can achieve better performance than previous work on ImageNet classification, COCO object detection and instance segmentation, and ADE20K semantic segmentation.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENT
Prof. Xiaolong Wang's group was supported, in part, by gifts from Qualcomm and Amazon Research Award. Chenhongyi Yang was supported by a PhD studentship provided by the School of Engineering. University of Edinburgh.
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. In Proceedings of the European conference on computer vision, 2020. 3
Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. In International Conference on Learning Representations, 2022a. 3, 6, 7, 18, 19
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang. Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 1906009156., 2019. 16
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions, arXiv preprint arXiv:2205.088 085544, 2022b. 7, 8, 16, 19
Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 589-598, October 2021. 18
Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas Huang, and Lei Zhang, High- erhrnet: Scale-aware representation learning for bottom-up human pose estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, June 2022.
MMClassification Contributors. Openmmlab's 's image classification toolbox and benchmark. https: //github. com aopen-mmlabe hww.llaammmelsssiiiiii....
2020a. 16
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. attps::/githii com/ cwtpm//peemmmmmmmm ab wtt.aaaaeammeeeeetttiiaa.oe 2020b. 16
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020. 6, 16
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In Proceedings of the European conference on computer vision, 2022. 7, 8, 18, 19
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped win- dows. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12124 12134, June 2022. 3, 5, 8, 18, 19
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL /ttpps://penreee //tp///eeee/....//////// net forum? i d=Yi cbFdNTTy. 2,4
10

---

Published as a conference paper at ICLR 2023
Stéphane d' Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference Machine Learning. PMLR, 2021. 6, 18
Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems, 2022. 3
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li. Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6824 6835, 2021. 3
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2918 2928, 2021.7 7
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: A vision transformer in convnet's s clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 3
Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z. Pan. Multi-scale high-resolution vision transformer for semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 3, 8
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12175-12185, June 2022. 3, 19
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34:15908-15919, 2021. 3
Ali Hatamizadeh, Hongxu Yin, Jan Kautz, and Pavlo Molchanov. Global context vision transformers.
arXiv preprint arXiv:2206.0990000 2022. 3, 18, 19
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition, June 2016. 19
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference Computer Vision, Oct 2017. 2
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022. 7
Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 603 612, 2019.
Drew Hudson and Larry Zitnick. Generative adversarial transformers. In International conference machine learning, pp. 4487-4499. PMLR, 2021. 3
Andrew Jaegle, Sebastian Borgeaud, lean-Baptiss Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver 10: A general architec- ture for structured inputs & outputs. In International Conference Learning Representations, 2022. 3
Thomas Kipf, Gamaleldin F. Elsayed. Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. In International Conference Learning Representations (ICLR), 2022.
11

---

Published as a conference paper at ICLR 2023
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Interaational Conference on Learning Representations, 2017.
Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7287-7296, June 2022. 3.7.8 18,
Gary Leung, Jun Gao, Xiaohui Zeng, and Sanja Fidler. Hila: Improving semantic segmentation in transformers using hierarchical inter-level attention. arXiv:220.002126, 2022. 8
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429 202 7
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2022a. 2,3, 6,
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4804-4814, June 2022b. 3, 4, 18, 19
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. LocalViT: Bringing Locality to Vision Transformers. arXiv.org, April 2021b. 18
Yin Li and Abhinav Gupta. Beyond grids: Learning graph representations for visual recognition. Advances in Neural Information Processing Systems, 31, 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 1, 3, 4, 6, 7, 8, 9, 16, 18, 19
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 1525 11538, 2020. 3
Haozhi Qi, Xiaolong Wang, Deepak Pathak, Yi Ma, and Jitendra Malik. Learning long-term visual dynamics with region proposal interaction networks. In International Conference on Learning Representations, 2021.
Pengzhen Ren, Changlin Li, Guangrun Wang, Yun Xiao, Qing Du, Xiaodan Liang, and Xiaojun Chang. Beyond fixation: Dynamic window visual transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11987-11997, June 2022a. 6, 7, 8, 18, 19
Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted self-attention via multi-scale token aggregation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10853-10862, June 2022b. 3, 5, 8, 16, 18, 19
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. (693-5703) 2019. 3
llya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34: 24261-24272, 2021. 2,5
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles. and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference Machine Learning, 2021a. 2, 6, 18
12

---

Published as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021b. 18
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention All you Need. In I Guyon, V Luxburg. Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017. 4, 5
Guangting Wang, Yucheng Zhao, Chuanxin Tang, Chong Luo, and Wenjun Zeng. When shift operation meets vision transformer: An extremely simple alternative to attention mechanism. In AAAI Conference Artificial Intelligence, 2022a. 18
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, +3(10):3349-3366 2020. 3
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 1, 3, 7, 18, 19
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. Computational Visual Media, 2022b. 7
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of the European conference on computer vision, pp. 399-417, 2018. 3
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, 30, 2017. 3
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021. 1,3,
Zhuofan Xia, Xuran Pan, Shiji Song, Erran Li, and Gao Huang. Vision transformer with de- formable attention. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4794 4803, June 2022. 8, 18, 19
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision, pp. 418-434, 2018. 2, 8
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 14:17777- 2021. 8
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated Residual Transformations for Deep Neural Networks. In The IEEE Conference on Computer Vision and Pattern Recognition, July 2017. 19
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Weijian Xu, Yifan Xu, Tyler Chang. and Zhuowen Tu. Co-scale conv-attentional image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9981-9990, 2021a. 3
13

---

Published as a conference paper at ICLR 2023
Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in Neural Information Processing Systems, 34: 28522 28533, 2021b. 2
Chenhongyi Yang, Zehao Huang, and Naiyan Wang. Querydet: Cascaded sparse query for accel- erating high-resolution small object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 33668-13677, June 2022. 3
Jianwei Yang, Chunyuan Li, Pengchuan Zhang. Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. In NeurIPS, 2021. 18
Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang- Hrformer: High-resolution vision transformer for dense predict. Advances in Neural Information Processing Systems, 34:7281-7295. 2021. 3
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rX1v:1710.04112, 2017. 6, 16
Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, Sercan Ö Arik, and Tomas Pfister. Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 3417-3425, 2022.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In Proceedings of the AAAI conference on artificial intelligence, 2020. 6,
14

---

Published as a conference paper at ICLR 2023
10
sePatt
Wheett
Wheett
432
Whettt
Wireth
dFSlod-44
GFRtod-04
PLOP% (C)
chttach-22
333
GF83o01-32
PLOP
330
230
20
139
1000
1000
Channel Dimension
(b) Input (224, 224)
(c) Input (800, 1280)
(d) Input (512, 2048)
Figure 5: wise FLOPs comparison between different attention blocks on high-resolution features. "GPBlock-NV denotes GP Block with N group tokens. (a) FLOPs v.s. Input size when using feature channel dimension of 348 (same as GPViT-sI); (b-d): FLOPs v.s. feature channel dimension on three typical input sizes using by ImageNet classification, COCO object detection and instance segmentation, and ADE20K semantic segmentation. Our GP Block can effectively gather and propagate global information a self-attention block while having similar or even fewer FL.PPs with local attention blocks when the model or input scalc-up.
Table 10: Inference speed comparison on three typical input sizes: 224x224 for ImageNet- IK classification, 800x 1280 for COCO object detection and instance segmentation, and 512 x 2048 for ADE20K semantic segmentation. The results are evaluated on NVIDIA 2080Ti GPUs. ('OOM' denotes out of GPU memory.)
Param (M)
Inference Time (ms) 224 800x 1 280 x12x2048
Model
Low-resolution Baseline
ViT-D216-P16
7.4
0.9
155
150
ViT-D348-P16
18.5
1.4
167
162
ViT-D432-P16
28.5
1.8
177
173
ViT-D624-P16
57.9
2.9
206
203
High-resolution Baseline
VTT-D216-P8
7.4
7.5
OOM
OOM
ViT-D348-P5
18.5
9.5
OOM
OOM
ViT-D432-P8
28.5
11.3
OOM
OOM
ViT-D624-P8
57.9
15.9
OOM
OOM
GPVIT-LI
9.3
2.7
83
87
GPVIT-L2
23.8
5.1
132
137
GPVIT-L3
36,2
7.0
174
182
GPViT-L4
75.4
11.1
281
290
A FURTHER ABLATION STUDIES
A.1 STUDY ON RUNNING EFFICIENCY
In Table 10, we compare the inference speed of GPVIT with VIT baselines. Specifically, for each variant of our GPVIT, we compare to ViT models with patch size 16 (low-resolution) and patch size 8 (high-resolution) while keeping the channel dimensions the same. We report inference time using three different input sizes, which correspond to the three typical input sizes used by ImageNet- Ik clas- sification, COCO object detection and instance segmentation, and ADE20K semantie segmentation. We draw three observations from the results:
When using small-sized inputs, GPVIT runs slower than the low-resolution VIT. Despite the efficient design of GP Block and the of local attention, the high-resolution design still incurs a significant cost for forward passes, slowing down inference speed.
When the models applied to downstream tasks where they take larger-sized inputs, all but the largest GPVIT models are faster than their low-resolution ViT counterparts. For example, when the model channel number is 216, GPVIT takes 83 ms to process an 800x 1280 sized image, while ViT-D216-P16 takes 155 ms. In this case, the self-attention operations with quadratic complexity severely slow down the speed of the VIT even with low resolution features. On the other hand, the computations in GP Block and local attentions grow much less than self-an attfntttnnton when the input scales up.
GPVIT is faster than the high-resolution ViT baselines when using small inputs. addition, high-resolution VIT' are not even able to process large-sized inputs: got Out of Memory
15

---

Published as a conference paper at ICLR 2023
emrors when using a NVIDIA 2080Ti GPU with 11 GB memory. This highlights our technical contribution of efficiently processing high-resolution features with GPVIT.
We further study how the computation cost for high-resolution features changes when the model size and input scale up by examining FLOP counts. The results are shown in Figure 5 where we compare GP Block with different group numbers to self-attention and local-attention operations: Self-attention and GP Block can both exchange global information between image features, but the computational cost of GP Block grows much slower than self-atten attentntnnion. Local attention operations have similar level of efficiency to GP Block, but are unable to exchange global information because of their limited receptive field.
B IMPLEMENTATION DETAILS B.I MODEL DETAILS OF GPVIT
The model details of different GPVIT variants are presented in Table 11. Different GPVIT variants are main difference by their model width (channels) and share similar hyper-parameters' other architecture designs.
Table 11: GPVIT model details for different variants.
Variants
GPViT-LI
GPViT-L2
GPVIT-L3
GPViT-L4
Patch Size
8
Channel Dimension
348
432
624
Number Transformer Layers
12
12
LcPE Strip Size
Attention Heads
12
12
FFN Expansion
4
Block Positions
(1,4, 10)
(1.4, 7, 10)
4, 10)
(1, 10)
Block Group Numbers
(64, 32, 32, 16)
(64, 32, 32, 16)
(64, 32, 32, 16)
(64, 32, 32, 16]
Feature Grouping Attention Heads
6
Feature Ungrouping Attention Heads
6
MI.Mixer Patch Expansion
0.5
0.5
0.5
MLPMixer Channel Expansion
ImageNet Drop Path Rate
0.2
0.3
0.3
Parameters (M)
23.6
36.2
75.4
B.2 TRAINING RECIPE FOR IMAGENET
The ImageNet experiments are based on the MMClassification toolkit (Contributors, 2020a). The models are trained for 300 epochs with a batch size 2048; the Adam optimizer was used with weight decay of 0.05 and a peak learning rate of 0.002. The cosine learning rate schedule is adopted. The gradient clip set to 5.0 (we also tested 1.0 and found it worked well too); data augmentation strategies are from Liu et al. (2021) and include Mixup (Zhang et al., 2017), Cutmix (Yun et al., 2019), Random erasing (Zhong et al., 2020) and Rand augment (Cubuk et al., 2020).
B.3 TRAINING RECIPE FOR COCO
The COCO experiments are based on the MMDetection toolkit (Chen et al., 2019). Following commonly used training settings, both Mask R-CNN and RetinaNet models are trained for 12 epochs (Ix) and 36 epochs (3x). For the 3x schedule, we follow previous work (Liu et al., 2021; Ren et al., 2022b) to use multi-scale inputs during training. The AdamW optimizer was used with an initial learning rate of 0.0002 and weight decay of 0.05. We used ViTAdapter (Chen et al., 2022b) to generate multi-scale features and followed the default hyper- parameter settings in Chen al. (2022b).
B.4 TRAINING RECIPE FOR ADE20K
The ADE20K experiments are based the MMSegmentation toolkit (Contributors, 2020b). Follow- ing commonly used training settings, both UperNet and SegFormer models are trained for 160000 iterations. The input images cropped to 5 x12x512 during training. The AdamW optimizer used with an initial learning rate of 0.00006 and weight decay of 0.01. We did not use ViTAdapter (Chen al., 2022b) for segmentation experiments.
16

---

Published as a conference paper at ICLR 2023
ADE20K Figure Feature grouping visualisation using model trained ImageNet- Ik, COCO and ADE20K.
ImageNet
COCO
C
VISUALIZATIONS
In Figure 6, we visualise the feature grouping results using models trained on ImageNet, COCO and ADE20K. We observe that the feature grouping can separate image's 's foreground and background in all three datasets. When the model receives fine- grained supervision like bounding boxes and semantic masks, the feature grouping correspond to more details in the image.
17

---

GPVIT-L4 Published as a conference paper at ICLR 2023
D COMPREHENSIVE COMPARISON
In Table 12 and Table 13, we provide a more comprehensive comparison between GPVIT and other visual recognition models on ImageNet- Ik classification and COCO Mask R-CNN object detection and instance segmentation.
Table 12: Comprchensive comparison between GPVIT and the recent proposed models on ImageNet- IK..
Model
Params (M) FLOPs (G) Top-1 Acc
Hierarchical
PVT-T (Wang et al., 2021)
13.2
1.9
75.1
PVT-S (Wang et al., 2021)
24.5
3.8
79.8
PVT-L (Wang et al., 2021)
64.1
9.8
81.7
CvT-13 (Wu et al., 2021)
20.0
4.5
81.6
Focal-Tiny (Yang et al., 2021)
29.1
4.9
82.2
Focal-Base (Yang et al., 2021)
89.8
16.0
83.8
Swin-T (Liu ct al., 2021)
29
4.5
81.3
Swin-S (Liu al., 2021)
50
8.7
83.0
Swin-B (Liu et al., 2021)
88
15.4
83.5
ShiftViT-T (Wang et al., 2022a)
29
4.5
81.3
ShiftViT-B (Wang et al., 2022a)
89
15.6
83.3
CSwin-T (Dong al., 2022)
23
4.3
82.7
CSwin-B (Dong et al., 2022)
78
15.0
84.2
Region (Chen et al., 2022a)
13.8
2.4
80.4
Region VIT-S
ViT- (Chen et al., 2022a)
30.6
5.3
82.6
Region VTT-B (Chen et al., 2022a)
72.7
13.0
83.2
MViTv2-T (Li al., 2022b)
24.0
4.7
82.3
GC ViT-XXT (Hatamizadeh et al., 2022)
12.0
2.1
79.8
GC ViT-XT (Hatamizadeh al., 2022)
20.0
2.6
82.0
GC ViT-T (Hatamizadeh et al., 2022)
28.0
4.7
83.4
GC ViT-S (Hatamizadeh et al., 2022)
51
8.5
83.9
GC ViT-B (Hatamizadeh et al., 2022)
90
14.8
84.4
DaViT-Tiny (Ding al., 2022)
28.3
4.5
82.8
MPVIT-S (Lee et al., 2022)
22.8
4.7
83.0
Shunted-T(Ren al., 2022b)
11.5
2.1
79.8
Shunted-S(Ren et al., 2022b)
22.4
4.9
82,9
DW-T (Ren et al., 2022a)
30.0
5.2
82.0
DW-B (Ren et al., 2022a)
91.0
17.0
83.8
DAT-T (Xia al., 2022)
29.0
4.6
82.0
DAT-B (Xia et al., 2022)
88.0
15.8
84.0
Non Non-hierarchical
DeiT-S (Touvron al., 2021a)
22.1
4.6
79.9
DeÎT-B (Touvron et al., 2021a)
86
16.8
81.8
CaiT-XS24 (Touvron et al., 2021b)
26.6
5.4
81.8
CaiT-S48 (Touvron et al., 2021b)
89.5
18.6
83.5
LocalViT-S (Li et al., 2021b)
22.4
4.6
80.8
Visformer-S (Chen et al., 2021)
40.2
4.9
82.3
Vin (d' Ascoli et al., 2021)
27.0
5.4
81.3
Con ViT-B Ascoli et al., 2021)
86.0
17.0
82,4
GPVIT-L1
9.3
5.8
80.5
GPVIT-L2
23.8
15.0
83,4
GPVIT-L3
36.2
22.9
84.1
75.4
48.2
84.3
18

---

Published as a conference paper at ICLR 2023
Table 13: Comprehensive comparison of Mask R-CNN object detection and instance segmentation MS COCO mini-val using x and 3 MS schedule.
Params FLOPs
1x
3x
Backbone
(M)
(G)
APM Apms
Aph
Ammk
Hierarchical
ResNet-50 (He al., 2016)
44
260
38.0
34.4
41.0
47.1
ResNet101 (He et al., 2016)
63
336
40.4
36.4
42.8
38.5
ResNeXt111-32x44 (Xic et al., 2017)
62
340
41.9
37.5
44.0
39.2
Region ViT-S (Chen et al., 2022a)
50.1
171
42.5
39.5
46.3
42.3
RegionViT-B (Chen et al., 2022a)
92
287
44.2
40.8
47.6
43.4
PVT-Tiny (Wang al., 2021)
32.9
36.7
35.1
39.8
37.4
PVT-Small (Wang et al., 2021)
44.1
40.4
37.8
43.0
39.9
PVT-Medium (Wang et al., 2021)
63.9
42.0
39.0
44.2
40.5
PVT-Large (Wang et al., 2021)
81.0
364
42.9
39.5
44.5
40.7
Swin-S (Liu al., 2021)
69
354
44.8
40.9
47.6
42.8
Swin-B (Liu et al., 2021)
107
496
45.5
41.3
MViTv2-T (Li al., 2022b)
44
279
48.2
43.8
MVITv2-S (Li et al., 2022b)
326
49.9
45.1
MViTv2-B (Li al., 20226)
71
392
51.0
45.7
DaViT-Small (Ding al., 2022)
351
47.7
42,9
49.5
44.3
DaViT-Base (Ding et al., 2022)
107
491
48.2
43.3
49.9
44.6
DAT-T (Xia et al., 2022)
272
40.4
47.1
42.4
DAT-S (Xia ct al., 2022)
69
387
47.1
42.5
49.0
44.0
DW-B (Ren et al., 2022a)
111
505
49.2
44.0
CMT-S (Guo et al., 2022)
44
231
44.6
40.7
CSwin-S (Dong et al., 2022)
54
342
47.9
43.2
50.0
44.5
CSwin-B (Dong et al., 2022)
97
526
48.7
43.9
50.8
44.9
MPVIT-B (Lee et al., 2022)
28
216
44.8
41.0
MPViT-B (Lee al., 2022)
231
46.6
MPViT-B (Lee et al., 2022)
43
268
48.4
47.6
MPViT-B (lee et al., 2022)
503
49.5
44.5
Shunted-S( -Sitted-SSRenneeee et al., 2022b)
42
47.1
52.1
49.1
43.9
Shunted -BRenn al., 2022b)
48.0
43.2
50.1
45.2
GC ViT-T (Hatamizadeh et al., 2022)
47
263
46.5
41.8
Non-hierarchical
ViT-Adapter-T (Chen et al., 2022b)
28
41.1
37.5
46.0
41.0
VTT-Adapte-S (Chen et al., 2022b)
47
44.7
39.9
48.2
42.8
ViT-Adapter-B (Chen et al., 2022b)
102
41.8 47.0
49.6
43.6
GPVIT-LI
33
457
48.1
42.7
50.2
44.3
GPVIT-L2
690
49.9
43.9
51.4
45.1
GPVIT-L3
64
884
50.4
44.4
51.6
45.2
GPVIT-L4
109
1489
51.0
45.0
52.1
45.7
19


Reviews:
[{"review_id": "z1H5SGMmNgI", "reviewer": "Reviewer_6Qyc", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a non-hierarchical transformer model for visual recognition tasks (detection, segmentation). Unlike recently proposed hierarchical methods like Swin Transformer that use an hierarchical transformer architecture, exchanging global information between features is computationally expensive for non-hierarchical transformers. To deal with this challenge, the paper proposes an efficient  Group Propagation Block (GP Block) to exchange global information between high-resolution features. In a GP block, grouped features formed by learnable group tokens and then global information is exchanged between grouped features. Finally, global information in updated grouped features is returned to the image features through the transformer decoder. GPVIT is evaluated on  image classification, semantic segmentation, object detection, and instance segmentation and obtains state-of-the-art performance. Keeping parameters or FLOPs constant, GP-ViT shows improved performance over previous methods in all cases.  \n", "summary_of_the_review": "The paper proposes a novel, yet uncomplicated, architecture for non-hierarchical transformers for visual recognition tasks and obtains SOTA performance on these tasks. ", "strength_and_weaknesses": "Strengths:\n1. The paper is clearly written. The method is explained well and carefully evaluated. \n2. The idea of group propagation is intelligent and is effective at reducing computational cost while keeping the architecture simple. \n3. The ablation studies and especially the explanation of how the architecture was built are informative and help explain the contribution of different components of the architecture.\n4. The method obtains SOTA performance across tasks, with impressive gains when compared to prior methods with similar number of parameters or FLOPs. \n\nQuestions:\nHow would you scale up GPViT beyond L3 to obtain better performance? Are there marginal gains to be had on current datasets by further scale up?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "o3ecdowd7NC", "reviewer": "Reviewer_oVeK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new transformer-based architecture that enables the extraction of high resolution feature maps from an images, while avoiding hierarchical downsampling in intermediate layers. To deal with the quadratic complexity of normal attention layers, a set of latent group representations is learned that collects information from the high resolution tokens. The latent group representations then mix their information based on an MLPMixer block and in turn the image patch tokens can cross attend to the update group features. Furthermore local attention is used for self-attention of the patch tokens. Using these principle, the complexity is linear in the number of groups and in the number of tokens, allowing a higher number of used input tokens. The resulting architecture is evaluated on several tasks (classification, detection, segmentation) and achieves good results compared to other methods with similar FLOP counts.", "summary_of_the_review": "To me the core approach is interesting, but I'm not very convinced by the FLOP constrained evaluations. Furthermore some important related work is missing. Together these things limit the novelty and contributions quite a bit in my opinion and as such I am leaning more towards rejecting the paper. But I'm interested to see the other reviews and the rebuttal and would be willing to change my mind here.", "strength_and_weaknesses": "Strengths:\n- While the idea to use such groups to reduce the complexity of attention is not really very novel, the application in this specific architecture is very interesting and show some promising results. Furthermore, a lot of the design choices seems rather general and similar systems could be applicable to other modalities such as point clouds, or potentially even multi-modal data.\n- The overall direction of not using intermediate downsampling is interesting and worth investigating.\n- The paper is easy to read and quite straightforward to follow.\n\nWeaknesses:\n- I think there is quite some rather related work that is not discussed. For example the Perceiver models by DeepMind have a different focus, but they are pretty similar in the underlying idea. Mask2Former also has a similar transformer part, albeit it works on top of a backbone to extract basic features and similar ideas where also explored in \"Generative Adversarial Transformers\". These are just a few I can recall off the top of my head. I wouldn't be surprised if there are a lot of other similar approaches. I think it is crucial that such related work is discussed with more care. Given that this is one main part of the contributions, the paper loses quite a bit of novelty in my mind. (And just because this uses an MLPMixer instead of vanilla attention does not make the model inherently different!)\n- I'm quite sad to see yet another paper that simply claims a method achieves state of the art results on some task, when it's obviously not true. Simply looking at other approaches, it becomes clear that the numbers are far from state of the art. I know the focus here is likely on models with similar FLOP counts, but in most of the sentences where claims about being state of the art are made, this fact is simply omitted. It would be very important to clarify this! And apart from that, it would also be interesting to see how well this type of architecture generalizes to bigger versions of a model with similar FLOP counts than the actual state of the art. This should clearly be clarified more!\n- Just looking at parameter counts and FLOP counts is not really giving a clear picture, but the actual throughput of a model is also important. (Have a look at the paper called \"The Efficiency Misnomer\".) Considering this architecture, I wouldn't be surprised if such a comparison would actually be favorable for the model, but I think it's sad that we don't see such numbers.\n- I think the idea of the approach is interesting and it would be great to see a bit more experiments with respect to the design choices that went into the different building blocks. E.g. how many groups should be used, or why is the MLPMixer used instead of normal self-attention? Indeed some of these things are discussed in the appendix, but I think they should be featured more prominently in the main paper in order to highlight the important aspects of this architecture design.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6mF-5ZONLBF", "reviewer": "Reviewer_vWHX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present GPViT, a non-hierarchival transformer model for visual recognition. The authors argue that by not using a hierarchical approach, and by relying only on high resolution features they achieve better recognition when fine grained details are needed for recognition. The authors introduce the Group Propagation Block to exchange global information tokens which achieves runtime efficiencies under these scenarios. They evaluate their approach on image classification. semantic segmentation object detection and instance segmentation.", "summary_of_the_review": "As I indicated above, the results are good and I think the ICLR audience would be interested in the paper. The effort made to decrease the number of parameters should be appreciated by the ICLR audience.", "strength_and_weaknesses": "Strengths:\n\nS1: tested on a variety of tasks\n\nS2: a good discussion on past transformer work, and the paper's place within this past work is discussed\n\nS3: thorough testing and a thorough ablation study is performed. The authors show that their model achieves good accuracy. For imagenet however it is not always clear to me that there is an improvement in terms of the number of parameters used. Discuss this in more detail in the results\n\n\n\nWeaknesses:\n\nW1: unless I missed it, will source code be provided. I did not notice a discussion on whether source code will be provided to make it easy for someone to replicate these results.\n\nW2: section 2: clarify what you mean by \"inductive biases\".\n\nW3: Eq 2: where is the concat operation shown in Figure 2?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%