Group Vision Transformer

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-10 05:06:24

Abstract: The Vision Transformer has attained remarkable success in various computer vision applications. However, the large computational costs and complex design limit its ability in handling large feature maps. Existing research predominantly focuses on constraining attention to small local regions, which reduces the number of tokens attending the attention computation while overlooking computational demands caused by the feed-forward layer in the Vision Transformer block. In this paper, we introduce Group Vision Transformer (GVT), a relatively simple and efficient variant of Vision Transformer, aiming to improve attention computation. The core idea of our model is to divide and group the entire Transformer layer, instead of only the attention part, into multiple independent branches. This approach offers two advantages: (1) It helps reduce parameters and computational complexity; (2) it enhances the diversity of the learned features. We conduct comprehensive analysis of the impact of different numbers of groups on model performance, as well as their influence on parameters and computational complexity. Our proposed GVT demonstrates competitive performances in several common vision tasks. For example, our GVT-Tiny model achieves 84.8% top-1 accuracy on ImageNet-1K, 51.4% box mAP and 45.2% mask mAP on MS COCO object detection and instance segmentation, and 50.1% mIoU on ADE20K semantic segmentation, outperforming the CAFormer-S36  model by 0.3% in ImageNet-1K top-1 accuracy, 1.2% in box mAP, 1.0% in mask mAP on MS COCO object detection and instance segmentation, and 1.2% in mIoU on ADE20K semantic segmentation, with similar model parameters and computational complexity. Code is accessible at https://github.com/AnonymousAccount6688/GVT.

Main paper: 
 introduction: 
 Convolutional neural networks (CNNs) have become prevailing architectures for computer vision applications, encompassing tasks such as image classification, object detection, and semantic segmentation. Following the remarkable success of AlexNet [17] in ImageNet-1K classification, there has been a continuous evolution toward developing deeper and more efficient CNNs [12, 27, 28] to enhance image recognition capabilities. In pursuing greater efficiency within CNN models, a succession of methodologies [14, 16, 29, 35] has been developed to mitigate model parameters and floating-point operations (FLOPs).Meanwhile, inspired by the remarkable success of Transformer [31] in the field of natural language processing (NLP), a multitude of studies have embarked on incorporating self-attention mechanisms into computer vision tasks. The Vision Transformer has emerged as a predominant model in computer vision applications. ViT [9] initially partitions an input image into multiple patches, transforming each patch into a token; subsequently, a Transformer encoder is employed to process these tokens utilizing the self-attention (SA) mechanism, enabling capturing both global and local relationships within the image.The Vision Transformer is composed of multiple stacks of basic blocks, each consisting of self-attention and a multilayer perceptron (MLP). The computational mechanism of self-attention shows linear complexity concerning the dimension of the token but exhibits quadratic complexity with respect to the number of tokens. Consequently, a series of methodologies sought to mitigate the computational complexity associated with self-attention. Another disadvantage of the Vision Transformer, when compared to Convolutional Neural Networks (CNNs), is the absence of inductive bias, as it lacks spatial hierarchies and translational invariance. Note that the Vision Transformer attempts to implicitly learn inductive bias from a large amount of training data, such as JFT-300. Recently, local attention Transformers have emerged to establish local inductive bias while concurrently reducing computational demands.In the Swin Transformer [21] , a non-overlapping window attention mechanism was introduced to concentrate on a pre-defined window area. Additionally, a shifted window mechanism was proposed for token mixing between two adjacent windows. Wang et al. [32] utilized a progressive shrinking pyramid to alleviate computation associated with large feature maps. Dong et al. [8] introduced a cross-shaped attention that performs self-attention in both horizontal and vertical stripes by dividing multi-heads into parallel groups. Ramachandran et al. [25] applied self-attention to replace traditional convolutional operations, where each pixel attends to the attention computation within a window, utilizing a sliding window. Furthermore, Hassani et al. [10] presented an efficient and memory-friendly local attention mechanism to reduce attention computation while preserving local inductive bias.Despite the ability of the aforementioned approaches to reduce computation by concentrating on small regions, they tend to overlook the channel dimension. In the Vision Transformer, each Transformer block comprises two main parts: the attention layer and the feed-forward network (FFN). The computational complexity of the attention part is quadratic concerning the token length and linear with respect to the channel dimension, while the computational complexity of the FFN part is quadratic concerning the channel length. The model complexity induced by the FFN layer is even larger than the self-attention layer. For instance, the NAT tiny model [10] comprises 28M parameters and 4.3G FLOPs, of which 12.461M parameters and 1.796G FLOPs are attributed to the selfattention layer, with the remaining generated by the feed-forward network. In a recent work [20] , a cascade group attention was proposed to reduce computation and diversify features by segmenting the feature map into multiple heads; attention is then computed for each head without sharing parameters. However, the emphasis of these studies remains predominantly on attention computation, with insufficient attention directed to the feed-forward network.In this paper, we propose a new Group Vision Transformer (GVT). We first partition the input feature tokens into multiple groups along the token dimension. Next, we apply self-attention and feedforward network (FFN) operations independently on each feature group. Simultaneously, one group is dedicated to capturing the global dependencies of the shallow layer. This design offers several advantages. First, the grouped feature tokens result in a reduction of computation in the FFN, which accounts for the majority of the Transformer Block computation, by a factor of 𝑔, where 𝑔 denotes the number of groups. Second, each feature group attends to selfattention and FFN computations independently, thereby enhancing the diversity of the feature groups. To incorporate inductive bias and global-range dependencies in the shallower layers, one group is utilized to capture global contours while the remaining groups are responsible for capturing local details. Experimental results across various computer vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation, demonstrate that the proposed Group Vision Transformer achieves competitive performance compared to baselines while maintaining comparable model parameters and computational complexity (e.g., Fig. 1 ). 
related work: 
 This section gives a brief review of Vision Transformers, highlights notable instances of Group Convolutions, and discusses recent advances of efficient Vision Transformers.The Vision Transformer was first proposed in [9] , where an input image is divided into small non-overlapping patches. These patches are then linearly embedded and treated as sequences of tokens. The tokenized patches undergo processing through multiple layers of Vision Transformer (ViT) blocks to capture spatial relationships among the feature tokens and learn relevant features.A ViT block, a fundamental building unit in the Vision Transformer architecture, comprises two components: A self-attention layer and a feed-forward network (FFN). In the self-attention layer, attention scores for each token of the image are determined based on its relevance to all the other tokens. Given an input token 𝑋 ∈ 𝑅 𝑛×𝑑 , where 𝑛 is the length of the token and 𝑑 is the dimension of each token, three matrices 𝑄, 𝐾, and 𝑉 are defined to map the input token to a query and a key-value pair. The output is computed as the weighted sum of the value matrices [31] . The output matrix is calculated as:EQUATIONwhose computation has a complexity of 𝑂 (𝑛 2 𝑑).After the self-attention layer, a two-layer Position-wise feedforward network is added to introduce more non-linearity to the model and enhance the representation ability of the model, as well as for dimension interaction. This allows the model to learn complex, non-linear relationships within the feature token. Given an input token 𝑋 ∈ 𝑅 𝑛×𝑑 , the FFN is computed as:EQUATIONwhere 𝑊 1 and 𝑊 2 denote the parameters of the two MLP layers in the FFN, 𝑏 1 and 𝑏 2 represent the biases of the two MLP layers respectively, and 𝛿 denotes a non-linear activation operation (e.g., GELU [13] ). The FFN has a computational complexity of 𝑂 (𝑛𝑑 2 ).Group convolution was initially introduced in AlexNet [17] , where it was employed to distribute the model across two GPUs. In [26] , depth-wise separable convolution was proposed to reduce parameters and computation by first conducting convolutions on each feature map independently and then mixing the features across different feature maps using a 1 × 1 convolution [18] . Xie et al. [35] introduced a homogeneous multi-branch structure utilizing group convolutions to enhance performance while preserving model parameters and computational complexity. Afterwards, group convolution was applied to develop more efficient models for edge devices, such as MobileNet [14] and Xception [4] , as replacements for traditional convolutions. In [39] , a shuffle unit was proposed to enhance information flow among different groups, thereby improving the overall performance.The original Vision Transformer (ViT) model [9] computes the attention scores of a token with all the other tokens, resulting in quadratic computational complexity concerning both the token length and embedding dimension. Afterwards, researchers have sought to reduce the computation of the Vision Transformer and develop more efficient ViT models.In the Swin Transformer [21] , attention computation is focused only on a small window rather than the entire pixel space. In PVT [32] , spatial reduction is employed on the key and value matrices, thereby reducing the computation required for attention. In SASA [25] , a sliding local attention module was proposed, replacing all convolutions in ResNet [12] with a self-attention layer, resulting in improved performance with less parameters and computation. In [8] , a cross-shaped window self-attention was proposed, where token self-attention calculations are conducted in horizontal and vertical stripes in parallel. Each stripe is obtained by splitting the input features into stripes of equal width, leading to improved performance with the similar parameters and computational demands. Hassani et al. [10] improved SASA [25] by providing a memoryand speed-efficient implementation. 
method: 
 This section presents our Group Vision Transformer (GVT).The overall architecture of our Group Vision Transformer is shown in Fig. 2 . Following the common practice in NAT [10] , we split an input image into overlapped patches by a patch splitting module. Each patch is regarded as a token and is derived as the concatenation of the pixel RGB values. This is implemented by two layers of strided convolution, with a stride of 2 and a kernel size of 3. The patch size is set to 4. As a result, the feature dimension of each token is 4×4×8 = 128. To attain a hierarchical representation, the whole network consists of four stages. In each stage, a token will be processed by a convolution to reduce the spatial resolution and double the feature dimension using a strided convolution (kernel size 3 × 3, stride 2 × 2). Therefore, the 𝑖-th stage will generate a feature map of size𝐻 2 𝑖 × 𝑊 2 𝑖 × 𝐶 • 2 𝑖, where 𝐻,𝑊 , and 𝐶 represent the height, width, and channels of the feature map generated by the first stage. Each stage consists of 𝑁 𝑖 sequential Group Vision Transformer Blocks, and the number of tokens and feature dimension are maintained within each stage. Consequently, the proposed architecture can be applied to various downstream vision tasks, such as object detection, instance segmentation, and semantic segmentation.Despite the fact that many previous ViT studies sought to reduce the computational complexity by focusing the attention computation on small local windows, they neglected to consider the token dimension. For instance, in the NAT Transformer block [10] , the computation of self attention is 3ℎ𝑤𝑑 2 +2ℎ 2 𝑤 2 𝑑 and that of the FFN layer is 2ℎ𝑤𝑟𝑑 2 , where 𝑟 is the hidden layer expansion between the two feed-forward layers, 𝑑 is the token dimension, and the window size is ℎ × 𝑤. Thus, the total computation of each NAT Transformer block is as follows:𝐹 𝐿𝑂𝑃𝑠 (𝑁 𝐴𝑇 ) = 3ℎ𝑤𝑑 2 + 2ℎ 2 𝑤 2 𝑑 + 2ℎ𝑤𝑟𝑑 2 .(EQUATIONAs shown above, in the total computation of each Transformer block, two terms are quadratic in the token dimension 𝑑. Since the local attention already reduces the window size (ℎ and 𝑤), the dimension 𝑑 becomes the key factor that impacts the total computation. Based on the above observation, we propose the GVT block design, in which each input token is decomposed into 𝑔 groups (see Fig. 3 ). For each such group, its amount of computation (i.e., FLOPs) is calculated as:EQUATION)Thus, the total computation of the 𝑔 groups of a GVT block is:EQUATIONTypically, the computational load attributed to the first two terms in Eq. ( 4) constitutes the primary portion of the overall computational burden. For example, in the context of the NAT model [10] , the combined computation of the first two terms amounts to 4.1 × 10 6 FLOPs, while the third term amounts to 4.6 × 10 5 FLOPs. Thus, the reduction in computation as described in Eq. ( 5) significantly alleviates the total computational load.The group attention structure is shown in Fig. 3 . Specifically, for each input feature token 𝑋 ∈ R 𝑛×𝑑 , we divide the features of 𝑋 into 𝑔 groups along its token dimension 𝑑, resulting in 𝑔 subsets 𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑔 . On each subset 𝑋 𝑖 , we perform self-attention and FFN separately.Another advantage of grouping the attention computation is to enhance the diversity of features. As Fig. 3 shows, we split the features of 𝑋 along the token dimension within each GVT block. During the attention and FFN computation, the features do not communicate across different groups, and thus this process reduces the likelihood of redundancy of features generated by each GVT block.The details of the GVT block are as follows. Given an input feature token 𝑋 ∈ R 𝑛×𝑑 , with 𝑛 = ℎ × 𝑤 as the spatial resolution of the feature map, we first split the features of the token into 𝑔 groups, as:EQUATIONwhere 𝑑 𝑖 and 𝑑 𝑖+1 denote the starting indices of the 𝑖-th and (𝑖 + 1)-th groups along the token dimension, respectively. The slice operator [𝑑 𝑖 : 𝑑 𝑖+1 ] returns the segment from index 𝑑 𝑖 to index 𝑑 𝑖+1 , excluding the element at index 𝑑 𝑖+1 . Next, we compute attention for each group 𝑖, as:EQUATIONAfterwards, each 𝑋 (0) 𝑖 is processed by a two-layer feed-forward network, obtaining features 𝑋 (1) 𝑖 , as:EQUATIONwhere 𝑤 1 and 𝑤 2 are the parameters of the two-layer MLP respectively, 𝜙 and 𝛿 are GELU activation functions, and 𝑋(1) 𝑖 ∈ R ℎ𝑤× (𝑑 𝑖+1 -𝑑 𝑖 ) . Then, all the features are concatenated and averaged to a 1D vector, as:EQUATIONwhere 𝐻 (•) denotes concatenation along the token dimension, and 𝐴𝑉 𝐺 (•) denotes average pooling along the (ℎ, 𝑤) plane. Thus, 𝑋 (2) ∈ R 1×𝑑 . After the above operation, 𝑋 (2) is processed by a fully-connected layer, further emphasizing informative groups:EQUATIONwhere 𝑊 3 ∈ R 𝑑 × (𝑑 𝑖+1 -𝑑 𝑖 ) refers to the parameters of the fullyconnected layer, 𝜎 refers to sigmoid activation, and𝑠 𝑖 ∈ R 1× (𝑑 𝑖+1 -𝑑 𝑖 ) .The output of each group 𝑖 is obtained by rescaling 𝑋𝑖 , as: where 𝐹 (•) is token-wise multiplication between the feature mapEQUATION× N 1 × N 2 × N 3 × N 4 H × W × 3 H 4 × W 4 × C H 8 × W 8 × 2C H 16 × W 16 × 4C H 32 × W 32 × 8C Stage 1 Stage 2 Stage 3 Stage 4𝑋 (1) 𝑖 ∈ R ℎ𝑤× (𝑑 𝑖+1 -𝑑 𝑖 )of group 𝑖 (𝑛 = ℎ × 𝑤) and the vector 𝑠 𝑖 . The final output of the GVT block is:EQUATIONEQUATIONThe processed feature map 𝑋 (4) will be sent to the next GVT block for further processing (if there is a next block).Table 1 presents detailed instantiation information of the model variants of our Group Vision Transformer. Following common practice [10, 12, 21] , we construct our base model, denoted as GVT-B, to closely mirror the model parameters and computational complexity of the base model NAT-B in [10] . Also, we build GVT-Mini (GVT-M), GVT-Tiny (GVT-T), and GVT-Small (GVT-S), each tailored to exhibit comparable parameter quantities and computational complexities to their NAT counterparts: NAT-Mini (NAT-M), NAT-Tiny (NAT-T), NAT-Small (NAT-S), and NAT-Base (NAT-B), respectively. We conduct experimental trials to determine optimal configurations for block parameters 𝑁 1 , 𝑁 2 , 𝑁 3 , and 𝑁 4 (shown in Table 1 ) to achieve the best performance while retaining the aimed computational complexities. The patch size is set to 4, the token dimension is 𝑑 = 128, the query size is 𝑑𝑖𝑚 = 128 (see Table 1 ), and the MLP expansion ratio is 𝛼 = 4, for all the experiments. The number of groups is set as 𝑔 = 4. 
experiments: 
 We conduct experiments on the ImageNet-1K [7] , MS-COCO object detection and instance segmentation [19] , and ADE20K semantic segmentation [41] datasets. Our ablation study investigates the effectiveness and computation costs of various network variants of GVT. We employ NAT as the Local Attention block (see Fig. 3 ).We train our model variants on the ImageNet-1K dataset to show the effectiveness of our GVT model. ImageNet-1K provides 128K, 50K, and 100K images for training, validation, and testing. The images are categorized into 1000 classes. We use the timm package [33] to train our model. Following the practice of Swin Transformer [21] , several data augmentation methods are utilized: CutMix [37] , Mixup [38] , RandAugment [6] , and Random Erasing [40] . We train our model for 300 epochs, with 5 warm-up epochs and an extra 10 cool-down epochs. The initial learning rate of the warm-up stage is 10 -6 , the maximum learning rate is 10 -3 , and the minimum learning rate is 10 -5 . Besides, a cosine learning rate decay strategy is applied. We use AdamW [23] to optimize our model. Table 2 shows the experimental results comparing our method with several typical methods on the ImageNet-1K dataset: DeiT [30] , ConvNeXt [22] , PVT [32] , Swin [21] , NAT [10] , BiFormer [42] , STViT [15] , and CAFormer [36] . The symbols M, T, S, B, and L denote Mini, Tiny, Small, Base, and Large, respectively. Specifically, we use the NAT [10] , BiFormer [42] , STViT [15] , and CAFormer [42] blocks as our base block, and compare with the models without our Group mechanism.As Table 2 shows, our model demonstrates superior performances compared to the vallina NAT, BiFormer, STViT, and CAFormer, while operating under similar parameter settings and computational constraints. Specifically, our GVT-M, GVT-T, GVT-S, and GVT-B outperform CAFormer-S18, CAFormer-S36, CAFormer-M36, and CAFormer-B36 by 0.4%, 0.3%, 0.2%, and 0.1% in Top-1 Accuracy, respectively. This trend underscores the capability of our Group Vision Transformer to enrich the diversity of the learned features. Further, our GVT block achieves reductions in computations and parameters, enabling our model to preserve more feature maps with similar model parameters and computational costs. Throughout the training process, we partition features into distinct paths with limited inter-path communication. This strategy effectively mitigates feature redundancy, thus enhancing the overall model performance. In addition, we observe a notable enhancement in the Frames Per Second (FPS) of our model compared to its counterpart. This improvement is primarily attributed to splitting the entire Transformer block into multiple groups, each executed in parallel. Consequently, this parallel group processing scheme leads to a significant enhancement in inference speed.•••••• •••••• •••••• •••••• •••••• Group 1 Group 2 Group 3 Group g Local Attention Local Attention Local Attention Global Attention FFN FFN FFN Group 1 Group 2 Group 3 Group g FFN C C X X (1) 2 X (1) 3 X (1) g X (2) X (4) X (1) 1 X (3) 2 X (3) 3 X (3) g X (3) 1 s 1 s 2 s 3 s g Figure 3:The structure of our proposed GVT block. denotes summation, denotes token-wise multiplication, and C○ represents concatenation of feature groups along the token dimension. Local Attention corresponds to the NAT block [10] , whereas Global Attention corresponds to the vanilla ViT block [9] .To further demonstrate the effectiveness of our proposed GVT model, we pre-train our model on the ImageNet-1K dataset and evaluate it on the MS COCO 2017 and ADE20K datasets for object detection, instance segmentation, and semantic segmentation.We test our pre-trained model on the MS COCO 2017 dataset [19] , which contains 118K, 5K, and 20K images for training, validation, and testing, using the Mask R-CNN and Cascade Mask R-CNN frameworks, respectively. Specifically, we pre-train our backbone model on ImageNet-1K following the practice of NAT [10] , and apply the pre-trained backbone to the object detection and instance segmentation frameworks. We evaluate our method using the backbones of ResNet, Con-vNeXt, and Transformers (i.e., Pyramid Vision Transformer (PVT), Swin Transformer (Swin), Neighborhood Attention Transformer (NAT), STViT [15] , CAFormer [36] , and Cross-Shape Transformer (CSwin)). We utilize the mmdetection framework [3] to train our object detection and instance segmentation networks. We resize all the images to have a shorter side of 480 to 800 pixels and a longer side of ≤ 1,333 pixels. Table 3 presents the experimental results on the MS COCO 2017 dataset using the Mask R-CNN framework with the "1×" (12 training epochs) and "3× 𝐿𝑅" (36 training epochs with multi-scale training) schedules. Table 3 shows that our approach outperforms STViT-B [15] by 1.2% in box AP and 1.1% in mask AP, respectively. On the Tiny model, our approach outperforms STViT-S by 1.2% in box AP and 1.0% in mask AP, respectively.Table 3 : Experimental results on the MS COCO 2017 dataset using Mask R-CNN [11] with the 1× and 3× training schedules. Images are resized to have a shorter side of 480 to 800 pixels and a longer side of ≤ 1333 pixels. GVT (NAT), GVT (CAFormer), and GVT (STViT) mean that we use NAT, CAFormer, and STViT blocks as the base local attention, respectively. Bold numbers mark the best performances with different parameters and computations. "-" denotes that corresponding results were not reported in the original papers. Table 4 reports the object detection and instance segmentation results on the MS COCO 2017 dataset using the Cascade Mask R-CNN [2] framework. Likewise, we find that our GVT model achieves promising performance gains compared to the known methods. Specifically, our GVT-B (CAFormer) and GVT-S (CAFormer) outperform CAFormer-M36 and CAFormer-S18 by 0.8% and 0.7% in box AP, respectively. The mask mAP performances also demonstrate the superiority of our model over the known ones. Since our GVT model and the NAT model use the same Local Attention block, these performance gains manifest the effectiveness and consistent improvement of our GVT approach.To demonstrate the generalizability of our GVT on semantic segmentation, we train our model using UperNet [34] on the ADE20K dataset. Specifically, we pre-train our GVT on the ImageNet-1K dataset, and apply the pre-trained backbone to UperNet for semantic segmentation. For fair comparison, we follow the practice of Swin Transformer and NAT by using the mmsegmentation framework [5] for training and testing. We train all the models for 160K iterations and set the batch size to 16. The training images are randomly resized and cropped to 512 × 512 in the training stage. During the test stage, a multi-scale testing using resolutions which are [0.5, 0.75, 1.0, 1.25, 1.5, 1.75] multiplied with that used in the training stage is employed.Table 5 presents the experimental results on ADE20K semantic segmentation in the mIoU measure. We compare our method with ConvNeXt [22] , Swin Transformer [21] , CSWin Transformer [8] ,Table 5 : Experimental results on the ADE20K dataset of semantic segmentation using UperNet [34] . Bold numbers mark the best performances with different parameters and computations. Training images are randomly resized and cropped to size 512×512. In the test stage, a multi-scale testing using resolutions which are [0.5, 0.75, 1.0, 1.25, 1.5, 1.75] multiplied with that used in the training stage is employed. and NAT [10] . As Table 5 shows, our GVT outperforms the state-ofthe-art models in different variants. Specifically, our GVT-T, GVT-S, and GVT-B achieve 1.2%, 1.2%, and 1.1% performance gains over the counterparts of CAFormer [42] with similar parameters and computational complexities.To further demonstrate the capability of our method in diversifying feature maps, we compare the dissimilarity [1] of the generated feature maps by our model and the vanilla base models (NAT-B [10] , CAFormer-M36 [36] , BiFormer-B [42] , and STViT-L [15] ). We use the cosine distance to measure the dissimilarity between two features 𝑓 𝑖 and 𝑓 𝑗 in a feature map 𝑓 ∈ R 𝐶 ×𝐻 ×𝑊 , with 𝑓 𝑖 , 𝑓 𝑗 ∈ 𝑅 𝐻 ×𝑊 . The diversity of 𝑓 is calculated as:EQUATIONEQUATIONEQUATION)Table 6 : Feature map dissimilarity between our method and previous methods. GVT (NAT), GVT (CAFormer), GVT (Bi-Former), and GVT (STViT) denote using NAT, CAFormer, BiFormer, and STViT blocks as the local attention, respectively.Method Dissimilarity NAT-B [10] 0.364 GVT (NAT) 0.487 CAFormer-M36 [36] 0.649 GVT (CAFormer) 0.798 BiFormer-B [42] 0.761 GVT (BiFormer) 0.884 STViT-L [15] 0.774 GVT (STViT) 0.896where 𝑚 is the number of feature pairs, and 𝑐𝑜𝑠 (𝑓 𝑖 , 𝑓 𝑗 ) is the cosine distance between two features 𝑓 𝑖 and 𝑓 𝑗 in 𝑓 .The feature map dissimilarities of NAT-B, CAFormer-M36, BiFormer-B, STViT-L, and our models are shown in Table 6 . Observably, our Group mechanism consistently enhances the diversity of the extracted feature maps due to the fact that the Group mechanism reduces the likelihood of redundancy of features generated by each GVT block. A qualitative example of feature map comparison between CAFormer-M36 and our GVT with the CAFromer block as the base local attention is shown in In this section, we conduct ablation study on the proposed GVT approach. Specifically, we explore how the Self Attention (SA) Grouping, FFN Grouping, and Re-scaling module affect the model performance respectively on the ImageNet-1K dataset. The results are shown in Table 7 . Note that we adjust the number of channels of each stage to closely mirror the model parameters and computational complexity of CAFormer-S18. Observably, combining the We further examine the efficacy of global and local grouping. Employing CAFormer-M36 as the local attention model, we designate the total number of groups to be 8. The results are shown in Table 8 . Notably, a performance enhancement is observed as the number of local attention groups initially increases. Furthermore, the performance saturates when the number of local attention groups reaches 6, thereafter experiencing a decline as the number of local attention groups continues to increase. This can be attributed to the complementary roles played by both local attention, which captures fine-grained details, and global attention, which encapsulates broader contextual information, thereby contributing to the overall image representation capability [24] .For fair comparison, we integrate our GVT mechanism into the backbone architectures of both Swin Transformer [21] and NAT [10] , respectively. Our evaluations are conducted on the ImageNet-1K dataset.Compared to Table 8 that studies the effect of the ratio of Local Attention Groups, in Table 9 , we present the effects of the total number of groups and the token dimension of each group, denoted as "GVT_𝑁 × 𝑊 ", where 𝑁 denotes the number of groups and 𝑊 denotes the token dimension of each group. Note that for an input feature token 𝑋 , the token dimension 𝑑 of 𝑋 is 𝑑 = 𝑁 × 𝑊 (for all the 𝑁 groups). As Table 9 shows, more groups give rise to performance gains under similar parameter numbers and computational budgets. On the backbone of Swin Transformer, our GVT (𝑁 = 4,𝑊 = 32) outperforms its counterparts by 1.3%, 0.5%, and 0.4% in Top-1 Accuracy, respectively. On the backbone of Neighborhood Attention Transformer (NAT), our GVT (𝑁 = 4,𝑊 = 32) outperforms its counterparts by 0.6%, 0.6%, 0.5%, and 0.5% in Top-1 Accuracy, respectively. These experimental results demonstrate the performance effectiveness and computational advantages of our approach.Notably, performance enhancements are observed with increased group numbers, attributing to the capability of our Group Vision Transformer to enrich the learned feature representations. Furthermore, our model exhibits more reductions in parameters and computations when more groups are used. Consequently, under comparable parameters and computational budgets, our model can accommodate a larger token dimension, thereby generating more feature maps and yielding superior performance. 
conclusions: 
 In this paper, we introduced the Group Vision Transformer (GVT), a novel and efficient fundamental framework. By conducting selfattention and utilizing feed-forward network on distinct groups of feature tokens, we reduced computational complexity and model parameters. Concurrently, the utilization of multi-group self-attention fosters enhanced feature diversity and reduces feature redundancy. Experiments on several computer vision tasks, including ImageNet-1K classification, MS COCO 2017 object detection and instance segmentation, and ADE20K semantic segmentation, validated the efficacy of our proposed GVT model. Ablation study conducted on these tasks and various backbones verified the effectiveness of the proposed GVT mechanism.  
 
 Appendix:  


Decision: This paper introduced a simple and efficient variant of Vision Transformer that divides and groups the entire Transformer layer to improve attention computation, showing competitive performances in common vision tasks and outperforming CAFormer-S36 with similar parameters and complexity. The reviewers provided positive feedback on the paper's novelty, methodology, and results, with some suggestions for further analysis and clarification. Overall, the paper is technically solid and will have high impact on multimodal perception research. Given the resolution of raised concerns and the unanimous positive reviews, the paper is accepted for publication.

Reviews: review: paper_summary: paper_summary: The authors conducted self-attention and utilized a feed-forward network on distinct groups of feature tokens for reducing computational complexity, and model parameters. The utilization of multi-group self-attention fosters enhanced feature diversity and reduces feature redundancy. Experiments on several computer vision tasks, including ImageNet-1K classification, MS COCO 2017 object detection and instance segmentation, and ADE20K semantic segmentation, validated the efficiency of the proposed GVT model. The core idea of this model aims to divide a global transformer operation into multiple independent branches that effectively disentangle information entanglement. And, it is more useful for vision tasks due to its part-whole relations.
strength_weakness: strength_weakness: The model demonstrates superior performances compared to the NAT, BiFormer, STViT, and CAFormer while operating under similar parameter settings and computational constraints.  This strategy effectively mitigates feature redundancy, thus enhancing the overall model performance.
limitations: limitations: This kind of idea has been a lot in the CNN era. I want to know what the author thinks about transferring the idea from the CNN era to the transformer. What is the difference or what are the advantages and disadvantages?
paper_summary: paper_summary: This paper introduces a new visual backbone network called the Group Vision Transformer (GVT). The primary idea is to independently divide the Vision Transformer into multiple groups and then perform self-attention and feed-forward network (FFN) operations independently within each group.
strength_weakness: strength_weakness: 1. Through grouped operations, the Group Vision Transformer (GVT) can significantly reduce the number of parameters and GFLOPS of the original Vision Transformer. In the design of model frameworks with equal numbers of parameters and GFLOPS, GVT surpasses other visual backbone networks in various visual tasks.

2. The method proposed in the paper is simple and effective, with sufficiently comprehensive experiments.
limitations: limitations: 1. In the Transformer, there has been some prior work on mimicking Group Convolution to perform grouped operations on transformer blocks, such as [1] and [2]. The authors should compare and analyze their work with these papers.

2. With the same number of parameters and GLOPS as other models, the introduction of the Group operation enables GVT to add larger feature dimensions and design deeper network. However, this approach results in a substantial increase in GPU memory usage. For instance, as shown in Table 2, the memory consumption of GVT-M is 48% higher compared to CAFormer-S18, and the memory usage for other model sizes has increased by more than 20%. This comparison seems to be somewhat unfair.

[1] Park S, Kim G, Lee J, et al. Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model[C]//Proceedings of the 28th International Conference on Computational Linguistics. 2020: 6883-6893.

[2] Liu X, Peng H, Zheng N, et al. Efficientvit: Memory efficient vision transformer with cascaded group attention[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14420-14430.
paper_summary: paper_summary: The paper points out the importance of reducing the computation demands of the Feedforward layer, which most researchers have overlooked. They introduce the Group Vision Transformer (GVT)  block to split the features into G groups, and further, each group is processed independently during the Attention and Feedforward layer. By doing this, the computation complexity is significantly reduced and also achieves comparable performance. This work emphasizes the importance of reducing the compute complexity of the Feedforward layer and producing more diverse features which may contribute to better performance. Extensive experiments demonstrate their effectiveness.
strength_weakness: strength_weakness: 1.	The paper introduces a method reducing parameters and computational complexity by splitting token embedding into different groups and computing them independently during the subsequent blocks.

2.	The proposed model, GVT, outperforms existing models on several benchmarks, providing solid empirical evidence for the effectiveness of their approach.

3.	They found that the independent branches design can produce more diverse feature maps and benefit downstream tasks.
limitations: limitations: 1.	The proposed method found that independent branches produce more diverse features. They lack experiments or theoretical guarantees to justify whether total independent or partial dependent groups are better. Will different group communication strategies further improve the model performance?

2.	For group splits, the authors should further clarify the influence of different group numbers and the correlation between group numbers and other factors such as model size and dataset size.

3.	There is a lack of analysis of different group sizes that influence attention mechanisms and a lack of ablation study of the global attention branch. Will a simple non-overlap split cause information loss for the following attention blocks?