Revisiting the Integration of Convolution and Attention for Vision Backbone
Authors: Lei Zhu
Xinjiang Wang
Wayne Zhang
Rynson W. H. Lau
Venue: NeurIPS 2024 poster (2024)
Keywords: convolution
attention
vision backbone
Primary area: deep_learning_architectures
TL;DR: Apply attention and convolution at different granularity levels for more efficient representation learning.

Abstract: Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such a fine-grained level. In fact, this is the root cause of the scalability issue w.r.t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel \textbf{at different granularity levels} instead. Specifically, in each layer, we use two different ways to represent an image: a fine-grained regular grid and a coarse-grained set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus 
enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named \textit{GLMix}: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e.g., 64) semantic slots to match the performance of recent state-of-the-art backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces a meaningful semantic grouping effect with only IN1k classification supervision, which may induce better interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available at \url{https://github.com/rayleizhu/GLMix}.

Revisiting the Integration of Convolution and Attention for Vision Backbone
Lei Zhu City University of Hong Kong ray. 1ii.huoooo@o. com
Xinjiang Wang Sensetime Research wangxinj wangaaiiiiaaneeessiiimm..e..me com
Wayne Zhang Sensetime Research wayne. wwwneeeesennnennneee..m
Rynson Lau City University of Hong Kong Rynson Lau@cityu edu hk
Abstract
Convolutions (Convs) and multi-head self-attentions (MHSAs) typically sidered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need include the heavy MHSAs at such fine-grained level. In fact, this is the root cause of the scalability issue w.r.t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel at different granularity each layer, we use two different ways to represent
levels instead. Specifically, an image: a fine-grained regular grid and a coarse granned set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named GLMix: by offloading the burden of fine- grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e.g., 64) semantic slots match the performance of recent state-of-the-ar backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces meaningful semantic grouping effect with only IN1k classification supervision, which may induce hetter interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available wttp:/loooooallllaaeeae.le.c http://wwwhbbb.ommmo////ee//a/aa
Introduction
Since the renaissance of deep learning over a decade ago, CNNs had dominated image analysis, until recently when transformers become popular in vision tasks. CNNs and transformers differ in how they model spatial feature interactions: CNNs use convolutions (Convs), while transformers use multi-head self-attention (MHSAs). Both have their own advantages and limitations. For example, Convs have an inductive bias of translation equivariance, which matches the image property and enables decent performances with less data. They also have linear complexity w.r.t. pixel number, making them scalable to high-resolution input. However, they have a limited receptive field, which cannot be remedied simply by stacking more layers together (39]. In contrast, MHSAs can model
Rynson Lau the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ACMix
INIk Top- Ace vs. Throughputs
Locol
85.0
Pustoon
84.0
83.5
83.0
(our))
STL (ours)
K2.5
SMT (00V13))
SG-Foemer (ICC)2))
K2.
MavVIT (EOCV
MHIRA
(CVPR'))
81.5
Swin (ICCV21)
100
200
300
468
600
700
500
Currra
Thoughnueg@2Me22 (imh)
Figure 1: Existing integration schemes, e-g.. ACMix 40 apply MHSAs and Convs the same granularity (top). contrast, we affirm that by offloading the burden of extracting fine-grained features to lightweight Convs, MHSAs can ag- gressively applied to coarse semantic slots to make spatial mixing more efficient (bottom).
Figure 2: INIk top-1 accuracy vs. FLOPs.
While several recent state-of-the-ard models (i.e.. Max ViT [47], CSWin [4] and SG-Former 45 SMT 34 lie in almost the same Pareto frontier,
our GLNet models move the frontier further to the upper-right with clear margin.
long-range dependency flexibly, but suffer from a quadratic complexity w.r.t. input resolution and require more data to compensate for the lack of inductive bias. Besides, some discussions [4]] also point out that MHSAs play the role of low-pass filters, while Convs play the role high-pass ones. Hence, they are complementary to each other.
There are indeed some works that use both Convs and MHSAs to build vision backbones. Some of them alternate Convs and MHSAs across different stages/blocks [29 47), forming a loose col- integrate Convs and MHSAs tightly in each block. Specifically, they apply Convs and MHS As in parallel at the same granularity level and fuse their outputs for further processing, as shown in Figure top).. With Convs responsible for fine mene-graied feature extraction, we ask if we still need to apply the heavy MHSAs at the pixel level] Meanwhile, recent vision vision-language language models [ 633 have shown that an image can be described as a fixed number of visual tokens regard- less of its resolution, possibly stemming from the low-rank property of natural signals. Inspired by these works, we propose global-local mixing (GI.Mxx) block, which uses Convs and MHSAs at different granularities for different roles: while Convs focus on extracting local features, MHSA As focus on learning global inter-object relations. Specifically, both fine-grained regular grid and coarse-grained set of semantic slots, and then apply Convs to the grid and MHSAs to the slots in parallel. To enable local-global feature fusion, we introduce a pair of conjugated soft clustering and dispatching modules bridge the grid and set representations. In this way, we achieve highly efficient local-g -llobal modeling by using lightweight Convs to extract high-resolution features and heavy MSHAs process fixed number of semantic slots.
laboration. Others 40 14 3
each block, we represent an image as
To verify the performance of the proposed integration scheme for Convs and MHSAs, referred as GL.Mx, we start by building a GLMix blocks. GLNet-STTL. achieves 82.5% top-1 accuracy on ImageNet- 1k. It surpasses Swin-T (81.3% top-1 accuracy) significantly by 1.2%. Besides, we note that the macro architectural designs are also important factors for the performance vision backbones. For example, PoolFormer [64] and ConvNext (36] reveal that with deeper architecture, vision backbones can still achieve strong performances with simple token mixers such as average pooling and depth-wise convolution. Hence, further adopt several macro designs from recent state-of-the- the vision backbones [34 43 and scale the model up to derive a family of 3 models: GLNet-4G/9G/166) As a result, the GLNet- 4G/9G/16G models achieve 83. 33..%%/44.5544 0.445%%/55.000 .0% top-1 accuracy, while being more efficient than recent wate-of-the-ane works (as shown in Figure D Evaluations on downstream dense prediction tasks such as object detection, instance segmentation, and semantic segmentation demonstrate the strengths Ge. consistently. We also observe that a meaningful semantic grouping effect has emerged in the soft clustering module, even with only image-level classification supervision.
Layout model, referred to as GL.Net-TTL, based on the
Here, refer to the pixels the feature maps instead of the input image.

---

To summarize, our contributions are fhree-fold:
We revisit existing integration approaches for Convs and MHSAs, and propose to integrate the two operations at different granularities. Such integration combines the strengths of Convs (e.g.. the inductive bias of translation equivariance) and MSHAs (e.g., global interactions, data adaptivity) while avoiding the scalability issue w.r.t. the imput resolution.
We introduce pair of conjugated. fully differentiable clustering and dispatching modules bridge the set and grid representations of image features, hence enabling the fusion of the global features extracted by MHSAs and local features extracted by Convs, An advantage of the soft clustering module is that it produces meaningful semantic grouping effects without direct dense supervision. Through extensive experiments on various computer vision tasks, such as image classification, object detection, and instance/semanti segmentation, we empirically verify our proposed approach. Specifically, a new family of vision backbones, Ge.Net, demonstrates favorable performance- computation trade-off to existing state-of-the-arts, under ImageNet- Ik supervised training.
2
Related Works
Efficient Attention Mechanisms, Vanilla MHSAs 48 have a quadratic complexity w.r.t. the number of input tokens, causing huge computation burden and heavy memory footprints, especially in vision applications where the feature maps are in high-resolution. A large volume of works have been conducted to develop efficient variants to overcome such a limitation. These works can be roughly categorized as sparse approximations 66 [5] 70), low-rank approximations [50] [T] and kernel-basee methods [27 19. The global branch in the proposed GL.Mx block, which is a combination of soft clustering and dispatching modules and an MHSA, can be used independently as a low-rank attention approximation with not only key-value pairs but also queries being down- dlwnn-smmped. However, according our experiments, such a usage produces poor performance (Table 6). possibly due to losing too many details and the lack of inductive bias. We find that using MSHAs and Convs in a complementary way is crucial to the success of our proposed GLNet family.
Hybrid Vision Backbones. Many works indicate that hybrid vision backbones, which use both Convs and MHSAs, can achieve better performances than pure transformers and CNNs. Among these works, some of them use Convs and MHSAs alternatively across different blocks or stages [34] 29 47 59 [2). forming a loose collaboration between the two operators. Another approach adopted by several recent sate-offthheaar works [4
55 is to integrate Convs and MHSAs in each block tightly. Different from these works that apply Convs and MHSAs at the same granularity level, we find that by offloading the burden of extracting fine-grained and location- preeeeriig features to lightweight depth-wise Convs, As can be applied aggressively on coarse semantic slots while achieving compelling performances with higher efficiency.
Clustering for Representation Learning, Clustering is a type of unsupervised learning method used to find meaningful structure, explanatory underlying processes. generative features, and groupings inherent in set of examples. Existing works, such as [58 66 (0 [7] have explored clustering for representation learning in deep neural networks. However, unlike ClusTR (58) and TCFormer [66),
which use DPC-KNN [66 for the clustering, the soft clustering module in our work is fully learnable and does not rely on predefined rules. In comparison with ClusterFormer 30 and PaCaViT [17].
which perform cross attsstattention between the feature grid and cluster representationssslots our work performs self-attention over the slots (i.e., queries and key-value pairs are both from the slots), making the attention even more lightweight. Besides, our soft clustering module is hardware-efficient because is designed to be non-iterative and mainly involves a dense matrix multiplication.
3 Methodology
Modern vision backbones are usually built by alternating spatial modeling modules (e.g., Convs, MHSAs, spatial MLPs) and per-location feed-forward networks (FFNs, i.e,, embedding MLPs). Much research has been dedicated to developing spatial modeling modules, which is also the primary focus of this work. Specifically, we seek an integration scheme for Convs and MHSAs, which can utilize the strengths of both and scale to high-resolution inputs. Without modifying the standard design of the two basic operators, our key idea is to represent input features twice as both a regular feature grid and a set of semantic slots, and then process the feature grid with Convs and the semantic slots with MHSAs (Sec. 3.I. A pair of fully differentiable soft clustering and dispatching modules is
3

---

Clustering
MM
mainix mulippication
columi-wise sottmax
row-wise sohtmax
Pooling Flatten
Cosine
Correspondence
Similarity
Logits
Dispatching
Flatbon
MHBA
Dony
Figure 4: Structure of our GLMix block. At the core is a pair of conjugated soft clustering and dispatching modules to bridge the and grid representations and enable local-global fusion.
introduced bridge the two representations, enabling local-globil fusion (Sec. 3.2 Based on such an integration scheme we propose new family of vision backbones named GLNet (Sec. 3.3
3.1 Convs and MHSAS at Different Granularities
Image features are usually organized as a regular grid in vision backbones. Such a representation preserves the spatial correspondence between features and the input image, which is necessary for downstream dense prediction tasks (e.g., semantic segmentation). Besides, extracting local features with the grid representation is convenient and efficient.
In addition to the grid representation, we also create an intermediate set representation composed of a fixed number of semantic slots to enable efficient global context modeling. The reason is that although global interactions are usually expensive to compute, it is feasible to use a small amount (e.g., 64 image [[] 63], as images with heavy spatial redundancy [22]. Notably, the set of semantic slots that we use here is different from the sequence of visual tokens in plain ViTs [15] 46). While each visual token corresponds to a hard-divided regular patch (e.g., 16 16 pixels), semantic slots are an abstraction of some "soft" irregular semantic regions, shown in Figure
in experiment) of semantic slots to summarize
natural signals
We apply Convs to the grid representation to extract local features they are lightweight and thus efficient in processing the fine-grained fea- ture grid. To model global context, we apply MHSAs on semantic slots. This is a natural choice as MHSAs are ceemmutttiooeequivann operators, thus naturally suitable for the set rep- resentation. The scalability issue w.r.t. input resolution avoided, as we have only small number of semantic slots. Hence, the drawback of MHSAS is overcome.
Computer vision is awsome
Computer vision is awsome
Figure 3: The semantic slots correspond to "soft" irregular semantic regions (left). Compared to ing hard-divided regular patches (as adopted by plain ViTs 155 46) the right, our formulation closer to tokenization in NLP.
Next, illustrate how the set and grid repre- sentations are bridged by a pair of soft cluster- ing and dispatching modules, that local and global features can be fused.
3.2 Bridging The Set and Grid Representations
To establish connection between coarse-grained semantic slots and fine-grained feature grids, we need to create a correspondence between them. Although the classical k-means clustering is applicable for this purpose [49] 66), it is suboptimal for two reasons. First, it an iterative algorithm, which inefficient GPUs. Second, it is heuristic approach, which cannot be end-to-end optimized. Hence,

---

Table 1: System-to- tystem-to-system comparison with existing works under the Twin-Tiny-Laye
Table 2: Model configurations of the GLNet family. C: base channels (i.e., feature channels of the first stage). e: FFN expansion ratio. #Blocks: the 4-stage block numbers. FLOPSs are measured at 224 x 224 resolution.
protocol 77 t: implemented by us with
modules provided by timm 54
Models
Params (M)
FLOPs
Thron.
INIK Top-1
(im/s)
Model
C |e]
#Blocks
AAdv. designs
FLOPs
Swin-T
29
755.2
81.3
DAT-T
10
29
4.6
82.0
GL.Ne--TTL
[2. 2, 6, 2]
No
4.4G
Swin-ACMixTT
Hatten-Swin-T
19
29
4.5
82.1
GI.Net-4G
64
3
[4, 4, 18, 4]
Yes
4.5G
NAT-STL
20
29
81.4
GLNet-9G
96
[4, 4, 18, 4]
Yes
9.7G
STX
47
28
4.5
763.4
81.8
GI.Net-
128
[4, 4, 18,
Yes
16.7G
GLNet-STL (ours)
835,9
82.5
introduce simplified and fully differentiable clustering module and the conjugated dispatching module to address these two problems, as shown in Figure We illustrate the process below.
Clustering (feature grid -> semantic slots). Given an input feature grid, X E Hx where C is the number of channels, H is the height, and W is the width, we first initialize M semantic slots, Sint E IR MFC via average pooling followed by shape flattening,
Sinit Flatten Flatten(Avg (AvgPool(X))
We then compute the correspondence logits, A R as scaled cosine similarity between the initial semantic slots Sint and the flattened input features X R HW x as:
A = CosineSimilarityy (S.itt X)/o.
(2)
Here, the learnable scale factor smooths the distribution of A. preventing dominance by salient entrances. With the correspondence logits, we perform a 1-step update to derive refined semantic slots, S E RR as the weighted sum of flattened features X, as:
S Softmax(AAX
(3)
The refined semantic slots S are then fed to MHSAs as input.
Dispatching (semantic slots - feature grid). After transforming S to propagate global context with an MHSA module, the transformed semantic slots S' dispatched spatial locations for fusion with local features. Specifically, the dispatched features, G RCxHXx are computed as:
G Unflatten(Softtaal (A A)')).
(4)
G can be readily fused with the feature grid processed by Convs due to shape compatibility. We follow the Feature Pyramid Network [32] to use additive fusion, as it is sinple/lightweihht and provides a regularization that aligns the global and local features the same semantic space.
Discussion. The soft clustering and dispatching operations are highly efficient as the main computa- tions lie in gordwware-riiendde dense matrix multiplications, and we perform only 1-step instead of iterative updates of the semantic slots. They are fully differentiable as we do not use hard assignments like k-means. The combination is similar to soft-routing SoftMoE 42 which aims build large mixture-of conture-of-aae of-expert models. However, as we target a better balance between cost/performance, our design has several differences: (1) slots are initialized with different strategy (i.e., per-image average pooling instead of learned parameters shared by all images): (2) the clustering module is placed at different position (i.e., pair with token mixers instead of FFNs); and (3) significantly fewer slots are used (i.e., 64, instead of 4096 which is even more than the number of tokens for an image).
3.3 GLNet
To verify the performance of the GLMix block proposed above, we start by creating a Swin- Tiny- Layout architecture, named GLNet-STL, which follows the macro architectural designs the Swin- T 33 model but with the spatial mixing modules (window attention and shift-window attention) replaced. Specifically, we use the ML block in the first three stages of GLNettSTTL where the feature maps in high-resolution ( 8. of the input resolution). At the 4th stage, which is
Swin-Tiny-
5

---

Table 3: Comparison with different otate-of-the-art backbones on ImageNet- 1K classification. All models are trained and evaluated on 224 x 224 input resolution. Top-1 refers to top-1 accuracy (%). We compare models trained with a standard supervised recipe and those trained with an advanced distillation recipe
26
Method
FLOPs
#Param,
Top-1
Method
FLOPs
#Param.
Top-1
Standard supervised training
Standard supervised training
Swin-T
35
pCcv21)
4.5G
29M
81.3
Swin
35
15.4G
88M
83.5
PVT-S
51
3.8G
25M
79.8
CrossFormer-I 53
16.1G
92M
84.0
CSWin-T
14
ICV98700
4.5G
23M
82.7
CSWin-B
14
15.0G
78M
84.2
CMT-
(EVER1S0)
4.0G
25M
83.5
CMT-L
18
19.5G
75M
84.8
RegionViT
(C
5.7G
31M
83.3
RigionViT-B
13.6G
74M
83,8
FrossForme--S
MaxViT-T
17
(CCU)22))
5.6G
5.3G
31M
31M
83.6
82.5
MaxVTT-B
MOAT-2
59
(BCC912]]
23.4G
17.2G
120M
73M
84.7
84.9
MOAT-0
59
5.7G
28M
83.3
NAT-B
20
13.7G
90M
84.3
NAT-T
20
[CVPR22))
4.3G
28M
83.2
IntermmaaeeB
166i
97M
84,9
Internlnage-T
52
[EVPR/20)
SG
30M
83.5
Flatten B
19
15.0G
75M
84.5
Flatten-1
pCCMAa
4.3G
21M
SG-Fomee-BB
44
15.6G
78M
84.7
S
(CCV2N
4.8G
23M
83.2
GLNet- (ours)
16.7G
106M
85.0
GeNNet-Gi (ours)
4.5G
27M
83.7
Swin-S
8.7G
50M
83.0
Trained with distillation supervision
CSwin-S
[
6.9G
35M
83.6
Uniforner-S* 29
(2
4.2G
24M
83.4
VeTM
7.9G
42M
83.4
Wave-ViT-S* 61
4.7G
23M
83.9
MaxViT
17
11.7G
69M
84.4
Dua
62
JRNNEE 21
5.4G
25M
84.1
MOAT-1
59
(CER229)
9.1G
42M
84.2
BiFormer S
70
4.5G
26M
84.3
NATS
[EVER
7.8G
51M
GLNet-4G* (ours)
4.5G
27M
84,4
InternImage88
52
(CVP879)
8G
50M
84.2
BiFormer-B
Flatten-S
70
19
(EVP9725)
6.9G
9.8G
57M
35M
83,8
Uniformer H
Wave-ViT-B*
61
29
[0CEV/13]
(1T1
7.2G
8.3G
34M
5UM
84.8
85.1
SG-Former-M
43
7.5G
39M
84.1
DualViT-R H
62
[TРAX]7]]]
9.36
43M
85.2
SMT-B
x4
[CCC)22)
7.7G
32M
84.3
BiFormer-B
70
[CV/R:23
9.8G
58M
85.4
GLNet-9G (ours)
9.7G
61M
84.5
GLNet-9G* (ours)
9.7G
61M
85.3
32 of the input resolution, we use full attention because this is affordable, and beneficial for the performance [29]] 41) As shown in Table [ our GLNet-STL achieves competitive 82.5% Top-1 accuracy at the highest throughput of 835.9 im/s among several comparable architectures.
The compelling performance of GLNet-STL encourages to build stronger vision backbones based on it. We therefore investigate several recent -tate-of-the-arts 14
47 70 43 [4], and incorporate
the following advanced architectural designs adopted by them: (1) Overlapped patch embedding: use overlapped convolutions (3 x 3 Cony with stride 2) for image/feature
wamplian instead of non-overlapped ones (2 2 Conv with stride 2) as in Swin- Transformer; (2) Hybrid stage 3: alternate full MHSAs and GLMix in consecutive blocks of stage 3; (3) Convolutional position encoding: add 3 x 3 residual depth-wise convolution prior each spatial mixing block; (4) Deeper layout: increase the depth ([2, 2, 6, 2] - [4, 4, 18, 4)) while reducing the width (base channel 96
64 and FFN expansion ratio 4 - 3); and (5) Convlutional FFN: add a 3 3 residual depth-wise convolution between the two linear projections of FFN.
Note that all designs above are widely used in the vision transformers. In addition, as this work is mainly to propose an effective as well as efficient integration scheme for MHSAs and Convs, we do not further incorporate some possibly useful designs, such as the squeeze- squecze-and-exeineee and-excitation (SE) block (23] used by Max VIT [47] and the gated linear unit (GLU) 44] used by SMT [34]. An ablation study for the incorporated architecture designs can be found in the Supplemental. After applying these modifications sequentially to GLNet-STL. we derive GLNet-4G, a model with 4.5G FLOPs. We scale up GLNet-9G and GLNet-16C FLOPs by increasing the number of base channels (96
for GLNet-9G and 128 for 16Nett166), The model specifications are summarized in Table
4 Experiments
We first empirically evaluate our proposed GLNet on a series of computer vision tasks, including ImageNet-IK Ik [3] image classification (Sec. 4.1. COCO (31) object detection and instance segmen- tation (Sec. 4.2 and ADE20k (69] semantic segmentation (Sec. 4.3). Following existing works, first train the models for image classification from scratch and then the trained weights
For 224 x 224 input image used in INIK classification, full attention at stage 4 is equivalent to 7 x window attention used by Swin- Transformer.

---

Table 4: Results on the COCO 2017 dataset using the RetinaNet [33 framework for object detection,
and Mask R- R-CNN [21 framework for instance segmentation. 1x refers to 12 epochs, and 3 refers to epochs. MS means multi-scale training. mAp and mAp denote box mAP and mask mAP, respectively. FLOPs are measured at 800 x
1 1280 resolution.
#Param.
FLOPs
RetinaNet 1x
RetinaNet 3 + MS
Backbons
(M)
(G)
mAp
AP
AP
AP
AP
APP
mAp
AP
AP3s
AP3
APP
APE
PVT-Small
34
226
40.4
61.3
43.0
25.0
42.9
55.7
42.2
62.7
45.0
26.2
45.2
57.2
Swin-T
355
39
245
41.5
62.1
44.2
25.1
44.9
55.5
43.9
64.8
47.1
28.4
47.2
57.8
Twins-SVT-S
34
43.0
64.2
46.3
28.0
46.4
57.5
45.6
67.1
48.6
29.8
49.3
60.0
CrossFormet-S
272
44.4
65.8
47.4
28.2
48.4
59.4
MaxViT-T 47
46
263
44.7
66.3
47.7
28.0
48.3
58.9
RiFormer-S 70
35
243
45.9
66.9
49.4
30.2
49.6
61.7
SMT-S 34
30
247
47.3
67.8
50.5
32.5
51.1
62.3
GLNet-IG (ours)
37
214
47.1
68.6
50.5
30.8
51.1
62.9
47.9
68.8
50.8
32.7
51.6
63.5
Swin-S 35
60
44.5
65.7
47.5
27.4
48.0
46.3
67.4
31.1
50.3
Twins-SVT-B
67
326
45.3
66.7
48.1
28.5
48.9
60.6
46.9
68.0
50.2
31.7
50,3
61.8
CrossFormer.
62
389
46.2
67.8
49.5
30.1
49.9
61.8
ScalableViT-B
60
85
330
45.8
67.3
49.2
29.9
49.5
61.0
48.0
69.3
51.4
32.8
51.6
62.4
MaxViT-S
47
79
46.1
68.0
49.5
28.9
50.2
61.4
BiFormer
67
356
47.1
68.5
50.4
31.3
50.8
62.6
GLNet-9G (ours)
70
292
47.7
69.0
51.6
31.8
51,6
63.5
48.8
69.6
52.5
33.5
52.9
63.9
#Paam.
FLOPs
Mask R-CNN
Mask R-CNN 3x MS
(M)
(G)
mAP'
AP!
AP
mAP
AP
AP
mAP
AP
AP,s
mAP"
AP
AP%
331
47.8
264
64.6
50.2
44.8
Twins-SVT-S
8
44.0
248
43.4
66.0
47.3
40.3
63.2
43.4
46.8
69:3
51.2
42.6
66.3
45.8
CSWin:T [4
42
279
46.7
68.6
51.3
42.2
65.6
45.4
49.0
70.7
53.7
43.6
67.9
46.6
BiFormer-S
70
45.2
262
47.8
69.8
$2.3
43.2
66.8
46.5
SGTommer-S S
43
41
47.4
69.0
$2.0
42.6
65.9
46.0
49.6
71.1
54.5
44.0
68.3
46.9
SMT-S
40.0
265
47.8
69.5
43.0
66.6
46.1
49.0
70.1
53.4
43.4
67.3
46.7
GL.Net-G (ours)
46.6
233
48.3
70.3
53.3
43.6
67.3
46.9
49.9
71.6
54.7
44.5
68.8
48.1
Swin-S 35
69.1
354
44.8
66.6
48.9
40.9
63.4
44.2
48.5
70.2
53.5
43.3
67.3
46.6
CrossFormer-B
53
72
408
47.2
69.9
$1.8
42.7
66.6
46.2
CSWin [14
54
342
47.9
70.1
52.6
43.2
67.1
46.2
50.0
71.3
54.7
44.5
68.4
47.7
BiFormer-B 70
76.3
375
70.5
3GFvvmer-M
43
51
48.2
70.3
53.1
43.6
66.9
47.0
50.5
71.5
54.9
45.4
68.8
48.2
SMT-B 34
51.7
328
49.0
70.2
53.7
44.0
67.6
47.4
49.8
71.0
54.4
44.0
68.0
47.3
Gl.Net-9 (0ors)
79.5
311
71.4
56.1
46.2
for model initialization when performing downstream dense prediction tasks. Note that for dense prediction tasks which take high-resolution inputs, keep the number of semantic slots to 64, which consistent with that of image classification. We have found that 64 slots are sufficient to achieve state-of-the-art performances while increasing the number does not help. We then visualize the semantic slots to demonstrate that a meaningful semantic grouping effect emerges in the proposed soft clustering module (Sec. 4.4). Finally, conduct ablation study on the design choices of the GLMix integration scheme, which is the core of GLNet (Sec. 4.5)
4.1 Image Classification on ImageNet-1k
Settings. For a fair comparison with existing works, we conduct image classification experiments on the ImageNet-1k dataset [3 using the standard training recipe provided by Swin-Transformer [35] and the advanced distillation recipe provided by LV-ViT [26). The training details can be found in Appendix B We then evaluate the models for classification accuracy and benchmark their throughputs with the script provided by the timm library [53], following the same hardware (a single Tesla V100 32G GPU) and batch size (128) configurations used in Swin-Transformer (35)
Results. In Table 3 we compare GLNet with several closely related methods and/or recent state- of-the-arts. Under the setting of standard supervised training, our GLNet-4G/9G/16 consistently show comparable superior performances to existing pest-performing models across different model scales. With dense distillation supervision, the potential of GLNets is further unleashed compared to standard supervised training. For example, the accuracy of the GLNet-4G model increases from 83.7% to 84.4%, a significant performance improvement of 0.7%. Both GLNet-4G and GLNet-9G provide a more competitive performance-Fil
trade-off than other distilled models. As FLOPs is an indirect metric for practical inference speed and does not consider the memory access cost, we also plot the performance-througee curve in Figure 2 The improvements become more pronounced when viewed w.r.t. throughputs. Interestingly, several of the latest vision backbones (i.e., SG-Former [43.
7

---

Mix 47, and CSwin (14] lie in almost the same Pareto frontier, while our GLNet models further move the frontier the upper-right corner with a clear margin. Such a result demonstrates the superiority of our integration philosophy: by applying the heavy MHSAs at coarse granularity and light-weight Convs at fine granularity, spatial modeling can be both effective and efficient.
4.2 Object Detection and Instance Segmentation
Settings. We evaluate the backbones for object detection and instance segmentation on COCO 2017 31 All experiments are conducted using the MMDetection [4] toolbox to ensure fair comparison with existing works. The RetinaNet [33 framework is used for object detection, and the Mask R-CNN (2]] framework is used for instance segmentation. During training, initialize the backbone with weights trained on ImageNet-1K while leaving all other layers randomly initialized. Input images resized by fixing the shorter side 800 pixels while restricting the longer side no more than 1,333 pixels. We train the RetinaNet and Mask R-CNN detectors with x schedule (12 epochs) and 3x schedule (36 epochs) provided by MMDetection. More training details are provided in Appendix B We report the widely used average precision (AP) metric family, such as mean average precision (mAP), average precision at different thresholds (AP75 and AP50), and average precision for objects of different sizes (APs, APM and A Details of these metries can be found MMDetection
Results. We show the results for object detection and instance segmentation in Table Our method achieves the best performances among the compared methods across all metrics and the two model sizes in both cases. These results indicate that local- sgoaalllobal modeling with the GLMix block benefits object/instance-levll tasks.
4.3 Semantic Segmentation ADE20K
Settings. Our semantic segmen- tation experiments are conducted on the ADE20K dataset using the MMSegmentation (10) tool- box. We evaluate our approach using two frameworks Seman- tic FPN [2]] and UperNet [57]. In both cases, the backbone initialized with ImageNet- weights, while the other layers are randomly initialized. For follow the same setting as PVT [51] to train the model 80k steps in our Se- mantic FPN experiments. On the
Table Performance comparison of different backbones on the ADE20K segmentation task. We report mloU with no test-time augmentation. FLOPs are computed at 512 2048 resolution.
Semantic FPN 80k
UperNet 160k
Backbone
#Param. FLOPs mloU
#Param. FLOPs mloU
(M)
(G)
(%)
(M)
(G)
(%)
PVT-S [51]
28.2
161
39.8
Swin-T 35
31.9
182
41.5
59.9
945
44.5
Twins- SVT-S
28.3
43.2
54.4
901
46.2
CSWin-T
43
26.1
202
48.2
59.9
959
49.3
fair comparison,
BiFormer-S 70
29.3
173
48.9
55.3
930
49.8
SG-Former-S 43
25.4
205
49.0
52.5
989
49.9
SMT-S 34
50.1
935
49.2
GLNet-4G (ours)
30.7
1,50
49.6
56.8
907
50.6
other hand, for UperNet periments, we follow the settings used in Swin Transformer (35) and train the model for 160k iter- ations. More training details are provided in Appendix [ We re- port the mean intersection over union (mloU) metric with no test- time augmentation.
PVT-M [51]
48.0
219
41.6
Swin-S 335
53.2
274
45.2
81.3
1038
47.6
Twins-SVT-B 8
60.4
261
45.3
88.5
1020
47.7
CSWin-S [4
38.5
271
49.2
64.6
1027
50.4
BiFormer-B [70
60.4
282
49.9
88.5
1041
51.0
SG-Former-N 43
38.2
273
50.1
68.3
1114
51.2
SMT-B 62]
61.8
1004
49.6
GLNet-9G (ours)
63.6
230
51.3
91.7
988
51.4
Results. Table T shows the results of two different frameworks. Our GLNet-4G/16G achieve 13.6//11. mloU with the Semantic FPN framework, improving the previous best SG-Former-S/M by 0.6/1.2 mloU. similar performance gain for the UperNet framework also observed. The enhancements demonstrate the benefits of utilizing GLNet for high-resolution wixel-wise predictions.

---

Figure Visualization semantic slots. For each sample, we show the input image (left), assignment maps of all semantic slots (middle), and four representative slots (right). We use the k-medoids algorithm to select the four representative slots automatically.
4.4 Visualization of Semantic Slots
As mentioned in Sec. 3.2 the conjugated clustering and dispatching modules construct correspon- dence between the semantic slots and the feature grid. Such a formulation allows us to visualize which regions the semantic slots correspond to. Specifically, Eq. Jadd split them into M scalar maps of shape H x W. These scalar maps are then pseudo-colored for visualization. In addition, we the k-medoids algorithm to select four representative slots for a closer look automatically. We find that a meaningful semantic grouping effect emerges in the first block of stage as shown in Figure 5 Note that use ImageNet- 1 trained GeNet-STL. for visualization. Hence, the model receives no dense supervision. Visualization for more samples, more blocks and different epochs can be found Appendix C
extract the clustering weights in
4.5 Ablation Study
global We ablate our GLMix integration scheme using the GLNet-STL model. By default, branch with 64 semantic slots and a local branch with x 5 depth-wise conv in parallel in the GLMix blocks, as shown in Figure 4 With this default setting, we investigate the effect of (a) local-global collaboration, (b) the clustering strategy, (c) Conv kernel size in the local branch, and (d) number of slots in the global branch. Table shows the experimental results. We summarize our findings below.
Local-global collaboration. First, using both local and global branches together is crucial. With the global/local branch removed, the model has significantly degraded accuracy of 8...%% 1.8%/78.0P5 indicating that both coarse-grained inter-object relationship and fine-grained per- per-pixel local context important. Second, using global and local branches in parallel instead of sequentially is important. possible explanation is that the parallel layout provides a regularization for the global branch from the local branch. Otherwise, the global branch is difficult to optimize due the lack of inductive bias. Finally, using Convs the local branch is better than window MHSAs, as the latter are heavier and significantly decrease the throughput from 835.9 im/s to 660.9 im/s. This may be because Convs can implicitly bring position information via padding [25] while window MHSAs cannot.
Clustering strategy. The soft clustering approach is an important component of the GLMix block. Using the k-means clustering results do not only produce a significantly lower throughput (835.9 im/s - 440.6 im/s) but also incurs unstable training. This be attributed to the fact that k-means is an iterative, non-differentiable algorithm, as mentioned in Sec. 3.2 We also observe that initializing the semantic slots as learnable parameters decreases the accuracy from 82.5% to 82. 1 This implies that per-image adaptive initialization is better than static initialization. Possibly, there are difficulties to learn diverse contexts for each image with shared parameters as the slot initialization, according to visualizations in Appendix C
Convolution kernel size in the local branch. The model is robust to the convolution kernel size in the local branch. Using a kernel size of 3 or produces similar accuracy (82.4%) to the kernel size of 5 (82.5%). This is because the global branch has provided sufficient large receptive field.

---

Table Ablation study on the GLMix design choices. We investigate the effect of (a) local-glon -lobbll collaboration, (b) the clustering strategy. (c) convolution kernel size of the local branch, and (d) number of slots the global branch. W-MHSA stands for window MHSA; we use a window size of because size divisibility is required. $: It is implemented with the official release of Clustered Attention 49). NaN loss occurred during training.
Slot init.
Slot number
Conv FLOPs
Params (M)
Throu. (im/s)
INIk T (%)
Model
k.s.
(G)
GLNet-STL
pooling
64
5
4.4
30.3
835.9
82.5
local branch only
pooling
5
3,8
26.4
999.7
81.8
global branch only
pooling
64
3.8
28.3
982.4
78.0
sequential (global - local)
pooling
64
5
4.4
30.3
860.1
80.6
sequential (local + global)
pooling
64
5
4.4
30.3
825.9
79.6
local branch w/ W-MHSA
pooling
64
w7
5.0
32.2
660.9
81.1
k-means clusteringt static slot initialization
hashing param.
64
5
5.2
30.3
440.6
N/A
64
5
4.4
30.5
852.0
82.1
local w/ 7 x 7 DWConv local w/ 3 x 3 DWConv
pooling pooling
64 64
7
4.4
30.3
855.2
82.4
3
4.3
30.4
823.9
82.4
global w/ 9 slots
pooling
5
3.9
30.3
893.6
81.9
global w/ 25 slots
pooling
25
5
4.0
30.3
880.8
82.1
global w/ 36 slots
pooling
36
5
4.1
30.3
880.0
82.3
global w/ 49 slots
pooling
49
5
4.2
30.3
866.6
82.3
global w/ 81 slots
pooling
81
5
4.5
30.3
790.0
82,4
Number of semantic slots in the global branch. Using 64 semantic slots is sufficient to achieve a good performance. Although the accuracy decreases to 82.3% with fewer semantic slots (e.g., 25, 36 49), increasing the number to 81 also incurs small performance drop 82.4%. We hypothesize that this is due to the optimization difficulty caused by too many similar/near-duplica. slots (68]
Conclusion
In this paper, we have revisited the existing integration approaches for Convs and MHSAs, and proposed to apply the two operators at disferent granularity levels. We discover that by offloading the task of extracting fine-grained features the lightweight Convs, the heavy MHSAS can be aggressively applied to a few semantic slots. Such an integration scheme, named GL.Mix, enables highly efficient local-global modeling to build nigh-peerformnce vision backbones. A key component of GLMix is pair of conjugated soft clustering and dispatching modules for bridging the feature grid and the set of semantic slots. Meaningful semantic grouping effects, which may induce better interpretability and inspire new in the clustering process.
semantic segmentation approaches, are observed
Currently, we only consider using a static number of semantic slots (i.e., 64 in our experiments) for all images. This may cause many redundant slots representing the same content, shown in Figure 5 may be interesting to design a dynamic slot pruning mechanism for more efficient computation and end-to-end weakly-supervised segmentation. Another drawback of GLNet is that still incorporates many hardware-inefficcee depth-wise convolutions with low arithmetic intensity. Seeking more hardware-fin harddare-friendly alternatives will further improve its throughputs on modern hardware.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 15:23716-2337 2022.
[2] Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High high-performance large-scale image recognition without normalization. In International Conference Machine Learning, pages 1059-1071
10

---

PMLR, 2021.
[3] Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit:
Regional to-tooo-ola attention for vision transformers. In International Conference on Learning Representations. OpenReview.net, 2022. URL
https://openrevier net/forum?id=T. V3uLix7V _..uuiiii
[4] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et Mmdetection: Open mmlab detection toolbox and benchmark. arXiv:1906 150.07996.0715,, 2019.
[5] Qiang Chen. Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang. Mixformer: Mixing features across windows and dimensions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5249-5259, 2022.
[6] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv: 10509, 2019.
[7] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arxiv:2009. J4794, 2020.
[8] Xiangxiang Chu. Zhi Tian, Yuqing Wang, Bo Zhang. Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention vision transformers. Advances in Neural Information Processing Systems, 1449355-93666 2021.
[9] XiangXiang Chu, Zhi Tian, Bo Zhang. Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. In International Conference on Learning Representations, 2023.
[10] MMSegmentation Contributors. Openmmlah semantic segmentation toolbox and benchmark. https 7/ //githu. wwwethhu...omopeeeeeeeee www.mmmmmmmmmmm. httpe//ammmmmmmmmmgtttttttiiiiia 2020.
[11] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition workshops, pages 702-703, 2020.
[12] Zihang Dai, Hanxiao Liu, Quoe V Le. and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances neural information processing systems, 34:3965-3977, 2021.
[13] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei. Imagenet: large-scalo hierarchical image datahase. 2009 IEFE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[14] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12124- 12134, 2022.
[15] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL http:: net/forum? id-YicbFaRTTy
[16] Mingjing Du. Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Baud Baxed Systems, 99:135 145, 2016.
[17] Ryan Grainger, Thomas Paniagua, Xi Song, Naresh Cuntoor, Mun Wai Lee, and Tianfu Wu. Paca-vit: learning patch-to-cluster attention in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18568-18578, 2023.
[18] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12175-12185, 2022.
[19] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5961-5971, 2023.
11

---

[20] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6185- 6194, 2023.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2969, 2017.
[22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-160000 2022.
[23] Jie Hu, Li Shen, and Gang Sun. Squuere-and-excittaiion networks. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 7132-7141, 2018.
[24] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In Proceedings of the European conference computer vision, pages 646 661, 2016.
[25] Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? arXiv preprint arXiv:2001, 008248, 2020.
[26] Zi- Hang Jiang. Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. tokens matter: Token labeling for training better vision transformers. Advances in Neural Information 2021.
Processing Systems, 34:
[27] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165. PMLR, 2020.
[28] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6399-6408, 2019.
[29] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[30] James Liang, Yiming Cui, Qifan Wang. Tong Geng, Wenguan Wang, and Dongfang Liu. Clusterfomer: clustering as a universal visual learner. Advances in neural information processing systems, 36, 2024.
[31] Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European conference computer vision, pages 740- 1000-55..65 Springer, 2014.
[32] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 21 2117-2125, 2017.
[33] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In Proceedings of the IEFF International Conference on Computer Vision, pages 2980-2988, 2017.
[34] Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scale-aware modulation meet transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6015-6026, 2023.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[36] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-11986, 2022.
[37] Loshchilov and F Hutter. Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, pages 1-16, 2017.
[38] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
12

---

[39] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
[40] Xuran Pan. Chunjiang Ge. Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration of self-attention and convolution. In Proceedings of the IEFE//VF conference on computer vision and pattern recognition, pages 815-825, 2022.
[41] Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations. OpenReviow.net, 2022. URL aattpe://openae wwwpperrrvieeennt/fouuulull D78Go4hVcx0
[42] Joan Puigeerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. arXiv preprint 23xiv:230.00095/ 2023.
[43] Sucheng Ren, Xingyi Yang, Songhua Liu, and Xinchao Wang. Sg-former: Self-guided transformer with evolving token reallocation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6003 6014, 2023.
[44] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:200.005202 2020.
[45] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2818-2826, 2826, 2016.
[46] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[47] Zhengzhong Tu, Hossein Talebi. Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Proceedings of the European conference on computer vision, 2022.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[49] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33:21665-21674, 2020.
[50] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, arXiv:2006.04700 2020.
[51] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.
[52] Wenhai Wang. Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring targe-scale vision foundation models with deformable convolutions. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14408-14419, 2023.
[53] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer: versatile vision transformer hinging cross-scale attention. In International Conference on Learning Representations, 2022. URL https::/ wwtp://wwww...t.toodddddddd---- Pftym_LLidddd____Lxxl
[54] Ross Wightman. Pytorch image models. https://githi: http:///ghhhhbboooommaaaaaaaaaap www.eeh--iiaam--mmmmmeeeee models 2019.
[55] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 22-31, 2021.
[56] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 4794 4803, 2022.
[57] Tete Xiao, Yingcheng Liu, Bolei Zhou. Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings the European conference computer vision, pages 418-43444444 4344, 2018
13

---

[58] Yutong Xie, Jianpeng Zhang, Yong Xia, Anton van den Hengel, and Qi Wu. Clustr: Exploring efficient self-attention via clustering for vision transformers. arXiv preprint arXiv:2208. 13138, 2022.
[59] Chenglin Yang. Siyuan Qiao, Qihang Yu, Xiaoding Yuan, Yukun Zhu, Alan Yuille, Hartwig Adam, and Liang-Chiec Chieh Chen. Moat: Alternating mobile convolution and attention brings strong vision models. In International Conference on Learning Representations. 2023.
[60] Rui Yang. Hailong Ma, Jie Wu, Yansong Tang. Xuefeng Xiao, Min Zheng, and Xiu Li. Scalablevit: Rethinking the context-oriented generalization of vision transformer. European Conference on Computer Vision, pages 0900-99...6. Springer, 2022.
[61] Ting Yao, Yingwei Pan, Yehao Li, Chong- Wah Ngo, and Tao Mei. Wave-vit: Unitying wavelet and transformers for visual representation learning. In Proceedings of the European conference on computer vision, pages 328-345. Springer, 2022.
[62] Ting Yao, Yehao Li, Yingwei Pan, Yu Wang, Kiao-Ping Zhang, and Tao Mei. Dual vision transformer. IEEE transactions on pattern analysis and machine intelligence, 2023.
[63] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[64] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang. Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819-10888 10829, 2022.
[65] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 6023-6032, 2019.
[66] Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human-eentric visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1 11101-1111, 2022.
[67] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz, mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https://
openreview.
[68] Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, and Kai Chen. Dense distinct query for end-to-end object detection. In Proceedings of the IERF/CVF Conference on Computer Vision and Pattern Recognition, pages 7329-7338, 2023.
[69] Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 32233330--321, 2019.
[70] Lci Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson WH Lau. Biformer: Vision transformer with bi-level routing attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10323-10333, 2023.
14

---

A Effect of Advanced Architecture Designs
Architecture design
Params
FLOPs Throu.
INIk Top-1
(M)
(G)
(im/s)
(%)
Swin-T layout (GLN
30.3
4.4
835.9
82.5
+ overlapped patch emb.
32.3
4.7
782.4
82.7
hybrid stage
31.4
4.8
784.0
83.1
+ convolution pos. enc.
31.4
4.8
762.6
83,2
deeper layout
26.8
4.5
630.7
83.5
conv. FFN (GLNet-4G)
27.0
4.5
541.2
83.7
Table 7: The evolution path from GLNeI-STL to GLNet-4G, Modifications are applied sequentially.
As mentioned in Sec. 3.3, we incorporate several advanced architecture designs adopted by recent vision backbones our GLNet family to achieve tate-of-the-art performance. Here, list the effects of these designs in Table 7 decrease the throughput.
All these designs improve the accuracy. However, they also
B Training Details
This section provides more training details for ImageNet-1k image classification, COCO object detection and instance segmentation, and ADE20K semantic segmentation.
Image classification. For the standard supervised training recipe, training details are in Table g When training with the advanced distillation recipe (26]. we add an distillation head to the
GLNet-4G/9G model and use the NFNet-F6 [2] to generate distillation targets; other training details are shown Table Experiments are run on 16 Tesla V100 SXM2 (32GB) GPUs. Each experiment
takes 2-4 days, depending on model size.
Object detection and instance segmentation. For COCO experiments, all models are trained using the AdamW (38 optimizer with a batch size of 16. We use a linear schedule with 500 warm-up iterations and set the peak learning rate le 4. The weight decay is 0.05 for Mask R-CNN [21] and 0.001 for RetinaNet [33. Experiments are run on 8 or 16 Tesla V100 SXM2 (32GB) GPUs. Each experiment takes 1-2 days, depending on model size.
Semantic segmentation. For the ADE20K semantic segmentation task, we apply the AdamW optimizer with batch size of 32. In Semantic FPN (28) experiments, we the cosine annealing learning rate schedule with 1000 warm-up iterations and a peak learning rate of 2e-4. The weight decay is le 4. In UperNet [57] experiments, polynomial learning rate schedule is employed with a linear warm-up phase of 1500 iterations. We set the learning rate as Ge 4 and weight decay as le Experiments run 8 Tesla V100 SXM2 (32GB) GPUs. Each experiment takes 1-2 days, depending on model size.
C More Visualization Results
In this section, we provide more visualization results, including (1) visualization of semantic slots of blocks at different depths (Figure 6). (2) visualization of slot evolution over training epochs (Figure 7
and (3) visualization of slots using learned parameters as clustering initialization (Figure g) We summarize the main observations below:
The semantic slots at the lower block (2nd block) tends group pixels according color cues. At the middle block (5th block), an object-level grouping effect has emerged. The upper block (10th block) pays attention discriminative local regions.
During the training, we found that at the end of the 1s epoch we can already distinguish the foreground objects and the backgrounds, although the grouping has not very concentrated patterns, this is possibly due to the fact that even random projection can preserve distances/similarities well. At the end the 5th epoch, the semantic grouping becomes more concentrated and similar to that of the final stage.
15

---

Table 8: Training details of standard supervised training for ImageNet- Ik classification.
Table 9: Training details of the advanced distilla-
tion recipe [26] for ImageNet- 1k classfication.
config
value
config
value
optimizer
AdamW (38]
optimizer
AdamW
learning rate
2e-3
learning rate
2e-3
weight decay
0.05
weight decay
0.05
optimizer momentum
81. f3 0.9, 0.999
optimizer momentum
81,
0.9, 0.999
batch size
2048
batch size
2048
learning rate schedule
cosine decay
learning rate schedule
cosine decay
warmup epochs
warmup epochs training epochs
5
training epochs
300
310
augmentation
RandAug(9, 0.5) I|I
augmentation
RandAug(9, 0.5)
label smoothing 45
label smoothing mixup
0.1
mixup [67
0.8
0.8
cutmix [65
1.0
cutmix
1.0
gradient clip
5.0
gradient clip drop path
5.0
drop path 24
0.15/0.3/0.4
0.1
When the semantic slots are initialized with learned parameters instead of adaptive average pooling proposed, the slots quite more chaotic and tend to focus foreground objects only, indicating that there are difficulties to learn diverse and global contexts. This possibly accounts for the degraded classification accuracy with such design (82.5% - 82. %, Table 6).
16

---

Input image
nd Block
5th Block
10 Block
Figure 6: Visualization of semantic slots of blocks at different depths. The setting is the same as in Figure 5 except that we add the visualizations for a shallower block (columns 2-3) and deeper block (columns 6-7).
17

---

Input image
Epoch
Epoch 5
Epoch 300
Figure 7: Slot evolution over training epochs. The setting is the same as in Figure 5 except that the checkpoint is replaced. #; slot assignments of epoch are normalized to range 0, 1] for better visibility, otherwise most of them will look like either empty or randomly scattered patterns.
Figure 8: Visualization of slots using learned parameters clustering initialization. The setting is the same as in Figure except that in the soft clustering initialization is modified.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The scope is stated at the beginning of the abstract. The contributions are summarized the end of the Introduction (Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We mentioned the limitations at the end of the conclusion (Sec. 5)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
19

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducíbility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Experimental settings provided in Sec. More details given in Appendix (Sec. B).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No]
Justification: All datasets used in this paper are existing public ones. We will release the code upon the acceptance of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (tttps://nippscc//pllll
quidess/CoeeSubmi ssionPolicy)) for more details.
While we encourage the release of code and data, understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for
open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: fttttp::/nipsscc/ppbbb issiissionn guides/Codeen wwt.bissssooooiiii oniooiccy for more details.
The authors should provide instructions data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
20

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits. hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings are provided in Sec. More details are given in Appendix (Sec. B).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is
necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report eπтоr bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Datasets such as ImageNet-1K COCO, and ADE20K are usually considered large- scale. The experimental results are not sensitive to random initialization. And running these experiments multiple times is too costly. Following existing works, this paper does not report
error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: The information on the computer resources included in the appendix (Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
21

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make into the paper).
Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics htttt::/ http:///ltitt/ttiiiiiicii.ii..a.
Answer: [Yes]
Justification: This paper does not involve human subjects or participants. We only use existing and publicly available datasets for evaluations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper focuses on designing general vision backbones. It is too broad to discuss the societal impacts of such general topic.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied par- ticular applications, Ict alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper focuses designing general vision backbones. It poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
22

---

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We only use existing and publicly available datasets or tools for evaluations. These works are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information
available online, the authors are encouraged reach out the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and documents will be released at attttpss//ggthhb..om////elezaa GLMiX
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codd/moodll as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does involve crowdsourcing research with human subjects.
Guidelines:
23

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equívalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
24


Reviews:
[{"review_id": "X8FMhS16Ti", "reviewer": "Reviewer_fhiH", "summary": "The paper discusses the use of Convolutions (Convs) and multi-head self-attentions (MHSAs) in vision backbones. Traditionally considered alternatives, the authors question the need for both to operate at the finest pixel granularity, particularly highlighting the scalability issues this causes in vision transformers. They propose a novel approach where Convs and MHSAs work in parallel but at different granularities: Convs handle local features on a fine-grained grid, while MHSAs manage global features on a coarse-grained set of semantic slots. The integration is facilitated by soft clustering and dispatching modules that bridge these representations, enabling local-global fusion. Their method, GLMix, leverages lightweight Convs for fine details and uses MHSAs on fewer semantic slots.", "questions": "Please refer to the weakness.", "limitations": "This paper will address the limitations in the conclusion section.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper re-evaluates current methods of integrating Convs and MHSAs and proposes a novel integration at different granularities. This approach harnesses the advantages of Convs, such as translation equivariance, and MHSAs, such as global interactions and data adaptivity, while mitigating scalability issues related to input resolution.\n2. The paper presents a pair of fully differentiable clustering and dispatching modules that connect the set and grid representations of image features. This enables the fusion of global features from MHSAs and local features from Convs. A notable advantage of the soft clustering module is its ability to produce meaningful semantic groupings without requiring direct dense supervision.", "weaknesses": "1. In Line 6, this paper mentions \"the scalability issue,\" but it only contains experiments on the IN-1K size dataset. Such a dataset cannot evaluate \"scalability.\" I suggest that the authors revise this argument.\n2. The novelty that introduces clustering into attention is limited. [1,2,3,4] have already studied the clustering in attention. Please compare with these methods and highlight your novelty and contributions.\n\n[1] Xie, Y., Zhang, J., Xia, Y., Hengel, A. V. D., & Wu, Q. (2022). Clustr: Exploring efficient self-attention via clustering for vision transformers. arXiv preprint arXiv:2208.13138.\n[2] Liang, J., Cui, Y., Wang, Q., Geng, T., Wang, W., & Liu, D. (2024). Clusterfomer: clustering as a universal visual learner. Advances in neural information processing systems, 36.\n[3] Grainger, R., Paniagua, T., Song, X., Cuntoor, N., Lee, M. W., & Wu, T. (2023). PaCa-ViT: learning patch-to-cluster attention in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18568-18578).\n[4] Zeng, W., Jin, S., Liu, W., Qian, C., Luo, P., Ouyang, W., & Wang, X. (2022). Not all tokens are equal: Human-centric visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11101-11111).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: IN-1k size dataset cannot evaluate \"scalability.\" It is advisable to revise this argument.\n\nWe thank this reviewer for the suggestion. However, in Line 6, we actually refer to the scalability w.r.t. the input size, instead of the dataset size. This can be reflected by the FLOPs-performance tradeoff on dense prediction tasks such as semantic segmentation (Table 5) and object detection (Table 4), where high-resolution inputs are used. We apologize for the confusion and will clarify it in our revision: \"... the scalability issue w.r.t. the input resolution for vision transformers\".\n\n---\n\nW2: Clarify the novelty and contribution in comparison with existing clustering attention works.\n\n* As an integration scheme of attention and convolution, GLMix distinguishes itself from existing works in using Convs and MHSAs at different granularity levels, i.e., Convs on fine-grained feature maps and MHSAs on coarse semantic slots. We have found that with Convs applied on the feature grid, MHSAs can be aggressively applied to a few semantic slots while achieving comparable and even better performances than existing state-of-the-arts.\n* Unlike ClusTR [1] and TCFormer [4], which use DPC-KNN for the clustering, the soft clustering module in our work is fully learnable and does not rely on predefined rules.  In comparison with ClusterFormer [3] and PaCaViT [4], which perform cross-attention between the feature grid and cluster representations/slots, our work performs self-attention over the slots (i.e., queries and key-value pairs are both from the slots), making the attention even more lightweight. We will incorporate this discussion in the Related Work section in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pAQvXEZmCo", "reviewer": "Reviewer_CazW", "summary": "The authors propose to leverage the strengths of convolution layers and MHSA block to improve the performance of vision transformers. They propose to apply them in parallel at different granularity, such that the convolution layers are applied to the grid of local features and MHSA to slots for global features. Slots can be considered as clustering of patch features, where patches corresponding to the same object would be associated to a slot. To create slots, they use soft clustering that enables local-global fusion across patch features such that it produces some meaningful semantic grouping effect.\nTo connect the slots back to the patch features, they propose a dispatching module, which is then fused with a convolution block os 1x1 convolution and depth-wise convolution operation (similar to Convnext). They terms this entire block as GLMix for Global-local mixing block. \n\nFinally, to validate the effectiveness of their approach, the authors propose steps to adapt it to Swin-tiny-layout module and empirically show the performance for image classification on Imagenet-1k, semantic segmentation on ADE20K, object detection and instance segmentation on MS-COCO. The authors provide a comprehensive list of baseline methods (Table 3, Table 4) for different tasks and also provide visualization of the slots attending to different regions of the image. They also discuss different experimental settings and provide additional visualizations of the slots in the Appendix.", "questions": "Apart from the ones mentioned in the Weakness section, I have the following questions\n\n- Can the authors please clarify their architecture? I see references to Swin, ConvNext, LV-ViT, etc. But Im unsure about what a single layer of GLNet looks like? Im also curious if the authors use a pretrained backbone, because Im unsure about the performance gains with just 1 V100 GPU and training the slots over just 1 iteration. Maybe Im missing some detail here.\n- Is there any specific reason apart from the smaller feature resolution that the GLMix block is applied to the 3rd layer? \n- There are missing references to hybrid ViTs, which the authors can add and discuss.\n\n[2*] Venkataramanan et al., Skip-Attention: Improving Vision Transformers by Paying Less Attention, ICLR 2024  \n[3*] Pan et al., Less is more: Pay less attention in vision transformers., AAAI 2022  \n[4*] Mehta & Rastegari, Separable self-attention for mobile vision transformers, arxiv  2022  \n[5*] Graham et al.,  Levit: a vision transformer in convnet’s clothing for faster inference, ICCV 2021  \n\n- Could the authors also provide the throughput details for table 3,4 and 5", "limitations": "Yes the authors have discussed the limitation, but one can reduce the redundancy in slots by trying to empirically understand the performance of the proposed method with fewer slots, as suggested above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The following are the strengths of the paper:\n- The authors propose an idea to integrate the strengths of convolution i.e. ability to encode inductive bias, alog with MHSA block with ability to learn global representation, to improve vision transformers. The idea is quite interesting and while the use of convolution and transformer layers have been applied in Hybrid architectures, their use in parallel is under-studied and is an exciting topic.\n- The visualizations provided in Figure 5 of the object level representations learned by the slots and additional such visualizations in the Appendix, gives a nice insight into representations encoded by slots. Its also interesting to see the slots learning different regions of the image, not only focusing on the object, but also regions in the background that might be of interest.\n- I also really liked the through and extensive experiments, along with comparisons to different SoTA methods, not only in terms of performance but also in efficiency.  \n- The paper is also easy to understand, with exhaustive mentions of related works and their comparisons.", "weaknesses": "I have the following concerns about the work:\n\n- In L59, the authors state that they have applied several macro designs from SMT [32]. However, comparing the performance of SMT, SMT-S achieves similar performance as GLNet-4G i.e. 83.7% accuracy, but has much fewer parameters (20.5M vs 27M), while having similar FLOPs. Additionally, the variant SMT-B has 2G fewer flops and half the number of parameters and yet achieves the same performance  i.e 84.3% s. 84.5%. Furthermore, SMT-S is also better than GLNet-4G on object detection on MS-COCO, which using RetinaNet. The authors of SMT also provide visualizations of the attention heads clearly attending to the object, same as in GLNet. This means that the scale-aware-module (SAM) achieves a much more efficient integration of convolution with MHSA. Could the authors please comment on effectiveness of GLNet against SMT.\n- Maybe I missed it, but Im unable to understand the intuition of applying self-attention operation on the slots (L161). Instead of just having a cosine similarity between the slots and patch features as in eq(2), a softmax over the slots would ensure that the slots do not collapse to the same object. Is this the reason why authors apply MHSA on S to obtain S'? \n- Continuing from the above point, I further cannot understand the significance of the dispatching module. Could the authors please explain the intuition behind going from slots to patch features? Is it because, it would be easier to integrate with the features obtained after conv layers? \n- In Figure 4, the clustering before applying the GLMix block and after are exactly the same. Its not clear to me what is being learnt by the GLMix block.\n- A popular work that uses slots to perform object centric learning is DINOSAUR [1*], which show in Figure 6 of their paper that increasing the number of slots results in a drop in performance. The authors however find that varying the slots has almost no effect in performance. Could the authors please comment of this difference in observation? Additionally, by carefully looking in figure 5, we can observe that there are many slots that represent the same object. This leads to some redundancy in learning. Can the authors vary the slots to somewhere between 5-10 like the one performed in [1*] to see if they have the same observation?\n\n[1*] Seitzer et al., Bridging the Gap to Real-World Object-Centric Learning, ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: Please comment on the effectiveness of GLNet against SMT.\n\n* For classification, the throughput/efficiency of SMT is not as good as its parameters and FLOPs indicate. This is because its core design, scale-aware modulation (SAM), relies heavily on depthwise convolutions (DWConvs), which cannot utilize the GPUs well due to low arithmetic intensity. Note that although we also use DWConvs, we only use it once in each GLMix block.\n\n| Model |#Params(M)|FLOPs(G)|Acc.@IN1k(%)|Throu.(im/s)|\n|---|---|---|---|---|\n|SMT-S|20.5|4.7|**83.7**|484|\n|GLNet-4G|27|4.5|**83.7**|**541**|\n| | | | | |\n|SMT-B|32.0|7.7|84.3|298|\n|GLNet-9G|61|9.7|**84.5**|**345**|\n\n* For object detection (Table 4) and semantic segmentation (Table 5), GLNet performs better and has lower FLOPs. This implies that GLNet has better scalability to high-resolution inputs than SMT.\n\n|Backbone|#Params(M)|FLOPs(G)|mAP (RetinaNet 3X+MS @ COCO) |\n|---|---|---|---|\n|SMT-S|30|247|47.3|\n|GLNet-4G|37|**214**|**47.9**|\n\n|Backbone|#Params(M)|FLOPs(G)|mIoU (UperNet 160k @ ADE20K) |\n|---|---|---|---|\n|SMT-S|50.1|935|49.2|\n|GLNet-4G|56.8|**907**|**50.6**|\n| | | | | |\n|SMT-B|61.8|1004|49.6|\n|GLNet-4G|91.7|**988**|**51.4**|\n\n---\nW2: What's the intuition of applying self-attention operation on the slots? Is it to ensure that slots do not collapse to the same object?\n\n* The self-attention operation on the slots is for inter-object/inter-region relation learning.\n* We have tried removing the self-attention over slots, and the slots do not collapse to the same object, possibly because the column-wise and row-wise SoftMax operations (Eq. 3 and Eq. 4) have introduced both inter-slot and inter-position competitions. However, the IN1k accuracy drops from 82.5% to 82.0% for the GLNet-STL. \n\n---\nW3: Could the authors please explain the intuition behind the dispatching module? Is it because it would be easier to integrate with the features obtained after conv layers?\n\nThe goal of the dispatching module is to propagate inter-region/object relation to spatial positions in the feature grid. Indeed, it is crucial for the fusion with the Conv layer processed features, as you point out here.\n\n---\nW4: In Figure 4, the clustering before and after applying the GLMix block are exactly the same. It's not clear what is being learnt by the GLMix block.\n\nThe GLMix block performs feature transformation. The input and output of the GLMix block are both feature maps. In Figure 4, we colorize both the input and output feature maps according to the expected clustering so that they look the same, which may be the reason for the confusion. We will revise the figure to be more clear (e.g., by removing the clustering colorization on the input feature map). Thanks for pointing out the issue.\n\n---\nW5: The observation on the effects of varying number of slots differs from DINOSAUR. Could the authors please comment on this difference? Can the authors vary the slots to somewhere between 5-10?\n\n* There are at least two aspects that make our observation on slot numbers differ from DINOSAUR:\n    * Supervision signal. DINOSAUR is an unsupervised framework for object discovery while our GLNet is trained under a supervised learning framework. DINOSAUR may require setting a proper number of slots to serve as a prior on how many objects can emerge in an image from their datasets.\n    * Evaluation metric. DINOSAUR uses the foreground adjusted rand index (FG-ARI), a no-reference metric for clustering quality, while we use the IN1k accuracy in the ablation study.\n* We have tried to use 9 slots (initialized with 3 $\\times$ 3 pooling) in GLNet-STL; the IN1k accuracy drops from 82.5% to 81.9%.\n\n---\nQ1: Can the authors please clarify their architecture? What does a single layer look like in GLNet? Do the authors use a pretrained backbone?\n\nThe architecture design is specified in Section 3.3. To summarize, \n* The macro architecture of GLNet follows most existing hierarchical transformers: it has four stages with gradually smaller spatial resolutions (1/4, 1/8, 1/16, 1/32) and larger hidden dimensions (C, 2C, 4C, 8C). Each layer contains a spatial mixing block (the GLMix block or MHSA block) followed by a channel mixing FFN/MLP. The detailed configurations can be found in Table 2.\n* We do not use a pretrained backbone. Since there are architectural differences, it is impossible to inherit the weights from existing architectures. We train the model from scratch on ImageNet and then use the weights to initialize the backbone for downstream dense prediction tasks, following existing works such as SwinTransformer.\n\n---\nQ2: Is there any specific reason apart from the smaller feature resolution that the GLMix block is applied to the 3rd layer?\n\nThe GLMix block is applied to the 1st, 2nd, and 3rd stages, and each stage has several layers according to the model size (see Table 2). The 4th stage uses a standard attention block as the resolution (1/32 input size) is sufficiently small so that the standard attention is affordable and beneficial for the performance. See also Section 3.3 for more details.\n\n---\nQ3: There are missing references to hybrid ViTs, which the authors can add and discuss.\n\nGLNet distinguishes itself from existing works, including the recommended references, by applying MHSAs and Convs at different granularity levels, i.e., the object/region level and pixel level. Thanks for bringing these works to our attention. We will reference and discuss them in our revision.\n\n---\nQ4: Could the authors also provide the throughput details for table 3,4 and 5.\n\n* It isn't easy to provide throughputs for these tables because most of the compared works did not report the data.\n* For classification (Table 3), Figure 2, together with the SMT throughputs in W1, has covered the latest SOTA models for comparison.\n* For object detection (Table 4) and semantic segmentation (Table 5), the throughputs may not be that useful because the task-specific heads, instead of the backbone, consume most computations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FzvtkFNqjr", "reviewer": "Reviewer_txiT", "summary": "This paper addresses the scalability issue in vision transformers by integrating convolutions (Convs) and multi-head self-attentions (MHSAs) at different granularity levels, rather than at the finest pixel level. The authors propose using Convs for fine-grained per-pixel feature extraction and MHSAs for coarse-grained semantic slots in parallel. They introduce a pair of fully differentiable soft clustering and dispatching modules to bridge the grid and set representations, enabling effective local-global fusion. The proposed integration scheme, named GLMix, offloads fine-grained feature extraction to light-weight Convs and utilizes MHSAs in a limited number of semantic slots. Through extensive experiments, they demonstrate the efficiency and effectiveness of their approach, showing improved performance and interpretability in various vision tasks. The paper also highlights the potential of their method to induce better semantic segmentation with weak supervision.", "questions": "1.How sensitive is the overall performance of the GLMix model to the choice of clustering strategy? Have the authors tested alternative clustering methods, and what were the outcomes?\t\n\n2.What are the potential limitations of using a static number of semantic slots, and have the authors considered dynamic allocation of semantic slots based on the complexity of the input image? \n\n3.How adaptable is the GLMix integration scheme to different types of vision tasks, such as object detection, instance segmentation, and semantic segmentation? Are there any task-specific adjustments needed for optimal performance?\n\n4.What are the scalability limits of the GLMix models when dealing with very high-resolution images or videos?", "limitations": "1. Scalability to Higher Resolutions: While the method shows good performance on standard benchmarks, its scalability to very high-resolution images or videos remains to be fully explored.\n\n2. Hardware Inefficiency: The use of depth-wise convolutions, despite their low arithmetic intensity, might still be inefficient on certain hardware, limiting the practical applicability of the model.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Integration Approach: The idea of integrating Convs and MHSAs at different granularity levels is innovative and addresses the scalability issues inherent in vision transformers.\n\n2. Efficient Local-Global Fusion: The use of soft clustering and dispatching modules enables efficient local-global feature fusion, which is a significant advancement.\n\n3. Extensive Empirical Validation: The method achieves state-of-the-art results on ImageNet-1k, COCO, and ADE20K benchmarks, showing a favorable performance-computation trade-off. Furthermore，the semantic grouping effect observed in the clustering module enhances the interpretability of the model.", "weaknesses": "1. Complexity of Implementation: The introduction of soft clustering and dispatching modules adds complexity to the implementation, which might pose challenges for practical deployment.\n\n2. Static Number of Semantic Slots: The use of a static number of semantic slots for all images may lead to redundancy and inefficiency in certain cases.\n\n3. Limited Scope of Clustering Strategy: The clustering strategy, though effective, could be further optimized. The current implementation might still be computationally intensive for real-time applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: Introducing the soft clustering and dispatching modules adds complexity to the implementation, which might pose challenges for practical deployment.\n\nThe clustering and dispatching modules involve only standard and widely used operators such as Matrix Multiplication and SoftMax, which are supported by all deep learning libraries across different hardwares, such as CPUs and GPUs. Therefore, the complexity of the implementation should not be a problem. \n\n---\n\nW2: The use of a static number of semantic slots for all images may lead to redundancy and inefficiency in certain cases.\n\nThe static number of semantic slots is indeed a limitation of our method, as we mentioned in Section 5. However,\n\n* On the one hand, we have observed that such a simple strategy is already sufficiently efficient to achieve a good efficiency-performance tradeoff, as demonstrated in Figure 2.\n* On the other hand, dynamic allocation of the slots still has the limitation on efficiency in the typical batched inference scenarios: an image with fewer slots needs slot padding to align an image with more slots.\n\nAlthough it may not be easily addressed, it would be interesting for future work to consider the dynamic allocation of slots.\n\n---\n\nW3: The clustering strategy, though effective, could be further optimized. The current implementation might still be computationally intensive for real-time applications.\n\n* Our clustering strategy is designed to be lightweight and only involves a pooling operation, a matrix multiplication followed by SoftMax. The three operations are widely used and highly efficient in all deep learning libraries.\n* The clustering takes less than 5% FLOPs of the whole models. It is far from being a computation bottleneck.\n\n---\n\nQ1: How sensitive is the overall performance of the GLMix model to the choice of the clustering strategy? Have the authors tested alternative clustering methods, and what were the outcomes?\n\nThe design choice of the clustering strategy is significant. According to our ablation study on the clustering strategy in L312-317 and Table 6,\n* using k-means clustering not only produces a significantly lower throughput (835.9 im/s -> 440.6 im/s) but also incurs unstable training;\n* clustering initialization with per-image adaptive pooling improves the performances over static initialization with learnable parameters (82.5% vs. 82.1%).\n\n---\n\nQ2: What are the potential limitations of using a static number of semantic slots, and have the authors considered the dynamic allocation of semantic slots based on the complexity of the input image?\n\nWe have addressed this issue in W2. Please refer to our reply to W2 above.\n\n---\n\nQ3: How adaptable is the GLMix integration scheme to different types of vision tasks, such as object detection, instance segmentation, and semantic segmentation? Are there any task-specific adjustments needed for optimal performance?\n\n* Keeping the number of slots consistent (i.e., 64) with image classification is good enough for dense prediction tasks. For object detection (Table 4 top group）, instance segmentation (Table 4 bottom group), and semantic segmentation (Table 5), we do not apply any task-specific adjustments and still obtain state-of-the-art performances.\n* We have also tried using more slots in the GLNet-4G backbone for semantic segmentation with UperNet. Using more slots, such as 100 or 256, did not improve the performances (50.6 mIoU -> 50.5 mIOU or 50.6 mIOU).\n\n---\nQ4: What are the scalability limits of the GLMix models when dealing with very high-resolution images or videos?\n\n* With GLMix, memory and computation grow linearly w.r.t. the input size, as the clustering and dispatching take linear complexity, and the attention among slots takes constant complexity. Therefore, the scalability should be similar to linear attention models such as SwinTransformer.\n* In practice, the scalability limits depend on memory and latency constraints, as well as engineering optimizations. For reference, below we show the peak memory occupancy and latency w.r.t. different input resolutions and batch sizes. Note that the single-input cases (batch size = 1) are heavily unoptimized in Pytorch and can be further improved with advanced inference engines like Nvidia Triton.\n\n| Batch Size | Input Size | GLNet-STL | GLNet-4G | GLNet-9G | GLNet-16G |\n|---|---|---|---|---|---|\n|128|$224^2$|1957MB/149ms|1553MB/233ms|2373MB/392ms|3230MB/586ms|\n|128|$448^2$|7469MB/562ms|5889MB/1012ms|8762MB/1690ms|11682MB/2517ms|\n|1  |$896^2$|346MB/22ms|433MB/59ms|722MB/95ms|1057MB/137ms|\n|1  |$1792^2$|1108MB/108ms|5027MB/628ms|7599MB/970ms|10218MB/1329ms|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive comments and suggestions. We are glad to see that reviewers consider our work novel/interesting/innovative (Reviewers fhiH/CazW/txiT), appreciate the semantic grouping effect brought by our design (Reviewers fhiH/CazW/txiT), and highlight the extensive experiments (Reviewers CazW/txiT). We address each reviewer's main concerns separately below.

Meta-review: This paper presents a new method for integrating the benefits of Convs for local features and MHSA for global features, which is integrated using soft clustering. The novelty is well-recognized by the reviewers. The reviewers raised concerns about clarity, additional visualization, sensitivity to the choice of clustering strategy, etc. The author's rebuttal addressed most of the reviewers' concerns. Reviewer CazW suggests that most of his/her concerns have been addressed in the rebuttal and do not provide further feedback or rating updates. The meta-reviewer appreciates the additional observation that the authors' additional experiments suggest, which well explains the remaining issues by reviewer CazW.
Decision: Accept (poster)