ClusterFomer: Clustering As A Universal Visual Learner
Authors: James Chenhao Liang
Yiming Cui
Qifan Wang
Tong Geng
Wenguan Wang
Dongfang Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Universal Model
Clustering

Abstract: This paper presents ClusterFormer, a universal vision model that is based on the Clustering paradigm with TransFormer. It comprises two novel designs: 1) recurrent cross-attention clustering, which reformulates the cross-attention mechanism in Transformer and enables recursive updates of cluster centers to facilitate strong representation learning; and 2) feature dispatching, which uses the updated cluster centers to redistribute image features through similarity-based metrics, resulting in a transparent pipeline. This elegant design streamlines an explainable and transferable workflow, capable of tackling heterogeneous vision tasks (i.e., image classification, object detection, and image segmentation) with varying levels of clustering granularity (i.e., image-, box-, and pixel-level). Empirical results demonstrate that ClusterFormer outperforms various well-known specialized architectures, achieving 83.41% top-1 acc. over ImageNet-1K for image classification, 54.2% and 47.0% mAP over MS COCO for object detection and instance segmentation, 52.4% mIoU over ADE20K for semantic segmentation, and 55.8% PQ over COCO Panoptic for panoptic segmentation. This work aims to initiate a paradigm shift in universal visual understanding and to benefit the broader field.

CLUSTERFORMES Clustering As A Universal Visual Learner
James C. Liang
Yiming Cui University of Florida
Qifan Wang Meta Al
Rochester Institute of Technology
Tong Geng University of Rochester
Wenguan Wang
Dongfang Liu" Rochester Institute of Technology
Zhejiang University
Abstract
This paper presents CLUSTERFORMER universal vision model that is based the CLUSTERing paradigm with TranSFORMER. comprises two novel designs: recurrent cross-attention clustering, which reformulates the cross-attention mecha- nism Transformer and enables recursive updates cluster centers to facilitate strong representation learning; and @ feature dispatching, which uses the updated cluster centers to redistribute image features through similarity- -baeed metrics, re- sulting in a transparent pipeline. This elegant design streamlines explainable and transferable workflow, capable of tackling heterogeneous vision tasks (i.e., image classification, object detection, and image segmentation) with varying levels of clustering granularity (i.e., image- box- and pixel-level). Empirical results demonstrate that CLUSTERFORMER outperforms various well-known specialized architectures, achieving 83 441 % top-1 acc. over ImageNet-II 1K for image classifi- cation, 54.2% and 47.6 .0% mAP over MS COCO for object detection and instance segmentation, 52.4% mloU over ADE20K for semantic segmentation, and 55.8% PQ over COCO Panoptic for panoptic segmentation. For its efficacy, we hope our work can catalyze paradigm shift in universal models in computer vision.
1 Introduction
Computer vision has seen the
Object
instrnce
Pancecis
Olsspration
Scomentation
emergence of specialized solu-
tions for different vision tasks
200z
AS
52.6
66.
541
(e.g.. ResNet (34] for image
Restict
Foster R--NNN
FON
Mosk R-CW
Panoptic Segformer
classification, Faster RCNN [70] for object detection, and Mask
Transformer
Deformable DETA
Figure 1: CLUSTER FORMER is a clustering-basee universal model,
RCNN 333 for instance segmenta-
offering superior performance various specialized architectures.
has shown that the human perceptual system exhibits exceptional interpretive capabilities for complex visual stimuli, without task- specifcc constraints. This trait of human perceptual cognition diverges from
tion), aiming for superior performance. Nonetheless, neuroscience research 73
65
current computer vision techniques 195 44 46), which often employ diverse architectural designs.
Human vision possesses unique attention mechanism that selectively focuses relevant parts of the visual field while disregarding irrelevant information [8] 40). This can be likened to clustering approach 2 3 89), in which individual pixel points are decomposed and reorganized into relevant concepts to address various tasks. This essentially is hierarchical process that involves combining
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

basic visual features, such as lines. shapes, and colors, to create higher-level abstractions of objects, scenes, and individuals 779 59 66 27] Inspired by the remarkable abilities of the human vision system, this work aims to develop universal vision model that can replicate the unparalleled prowess.
To this end, we employ clustering-based strategy that operates varying levels of granularity for visual comprehension. By solving different vision tasks (i.e., image classification, object detection, and image segmentation), we take into account the specificity which visual information is grouped (i.e., image- box-, and pixel-level). We name our approach, CLUSTER ($3.2), as it utilizes a CLUSTERing mechanism integrated within the TrameRorMeR architecture to create a universal network. The method begins by embedding images into discrete tokens, representing essential features that are grouped into distinct clusters. The cluster centers are then recursively updated through a recurrent clustering cross-attention mechanism that considers associated feature representations along the center dimension. Once center assignments and updates are complete, features are dispatched based on updated cluster centers, and then both are fed into the task head for the target tasks.
CLUSTERFORMER enjoys few attractive qualities. 1 Flexibility: CLUSTERFORMER a clustering- anchored approach that accommodates a broad array of visual tasks with superior performance (see Fig. D under one umbrella. The core epistemology is to handle various tasks with different levels of granularity (e.g.. image-level classification, box-level detection, pixel-level segmentation, etc.), moving towards a universal visual solution. Transferability; The cluster centers generated by the CLUSTERFORMER encoder are directly employed by the task head as initial queries for clustering, allowing the entire urchitecture transfer underlying representation for target-task predictions (see Table. This elegant design facilitates the transferability of knowledge acquired from the upstream task (i.e., encoder trained on ImageNet (72) to downstream tasks (e.g., decoder trained on instance segmentation on COCO 49 @ Explainability: Regardless of the target tasks, Cl.UsEERFoRMER s decision-making process is characterized by a transparent pipeline that continuously updates cluster centers through similarity- based metrics. Since the reasoning process is naturally derivable, the model inference behavior is ad-hoc explainable (see $4.2). This differs CLUSTER from most existing unified models [17 44 95]
that fail to elucidate precisely how a model works.
To effectively assess our method, we experimentally show: In $4.1.I] with the task of image classification, CLUSTERFORMER outperforms traditional counterparts, e.g., 0.13 ~ 0.39% top-1 accuracy compared with Swin Transformoer [53] on ImageNet (72). by training from scratch. In 1...2 when using our ImageNet-preetainee our method can be expanded to the task of object detection and greatly improve the performance compared to Dino [96] over Swin Transformer on COCO 49 (0.8 ~ 1.1% mAP). In addition, our method can also adapt to more generic per-pixel tasks, a.k.a, semantic segmentation (see 1..1, instance segmentation (see $4.1.4), and panoptic segmentation (see 1.. For instance, we achieve performance gains of 0.6 ~ 1.3% mloU for semantic segmentation on ADE20K [TOT], 1.0 1.4% MAP for instance segmentation on MS COCO 49 and 1.5 ~ 1.7% PQ for panoptic segmentation
COCO Panoptic [42] compared with Mask2Former [17] over Swin Transformer. Our algorithm are extensively tested, and the efficacy for the core components is also demonstrated through series of ablative studies outlined in $4.2
2 Related Work
Universal Vision Model. Transformers [1] have been instrumental in driving universal ambition, fostering models that are capable of tackling tasks of different specificity with the same architecture
86] in the 94 The encoder is centered around trained on extensive data that can be adapted and fine-tuned to diverse downstream tasks. For instance, Swin Transformer (53) capably serves as a general-purpose backbone for computer vision by employing a hierarchical structure consisting of shifted windows; VIT-22B (22), parameterizes the architecture to 22 billion and achieves superior performance on a variety of vision tasks through learning large-scull scale data. Conversely,
and embody the potential of these recent developments 23
16
95
96
80
30
57
the field. In the vision regime, mainstream research endeavors have been concentrating
development of either encoders [53] 88 or decoders
the effort of developing foundation models 4
553 24
22
93 76 37 99 25 48 is designed to tackle homogeneous target tasks, by using queries to depict visual patterns. For instance, Mask2Former 17 incorporates mask information into the Transformer architecture and unifies various segmentation tasks (e.g., semantic, instance, and panoptic segmentation); Mask-DINO 44 extends the decoding process from detection to segmentation by directly utilizing query embeddings
research on decoders (23 I7 16] 95, 94
dEl 96 87 50 20, 52
19 21
[1

---

for target task predictions. Conceptually different, we streamline an elegant systemic workflow based on clustering and handle heterogeneous visual tasks (e.g., image classification, object detection, and image segmentation) at different clustering granularities.
Clustering in Vision. Traditional clustering algorithms in vision
39
61 6 58
can be categorized into the hierarchical and partitional modes, The hierarchical methods [62 38
involve the modeling of pixel hierarchy and the iterative partitioning and merging of pixel pairs into clusters until reaching a state of saturation. This approach obviates the necessity of a priori determination of cluster quantity and circumvents the predicaments arising from local optima. 198 12]
However, it exclusively considers the adjacent pixels at each stage and lacks the capacity to assimilate prior information regarding the global configuration or dimensions of the clusters. 64). In contrast, partitional clustering algorithms (78 36) directly generate a flat structure with predetermined number of clusters and exclusively assign pixels to a single cluster. This design exhibits a dynamic nature, allowing pixels to transition between clusters [ 63]. By employing suitable measures, this approach can effectively integrate complex knowledge within cluster centers. a powerful system, human We possess the capability of grouping analogous entities at different scales. Meanwhile, we can also effectively categorize objects purely based on their shape, color, or texture, without having the hierarchical information. Drawing on the above insights, we reformulate the attention mechanism ($3.2) ) in Transformer architectures [8]) from the clustering's 's perspective to decipher the hierarchy of visual complexity.
vision incorporates the advantages of both clustering modes [89 83 67
3 Methodology
3.1 Preliminary
Clustering. The objective of clustering is to partition a set of data points, denoted by X E RRxxd into C distinct clusters based on their intrinsic similarities while ensuring that each data point belongs to only one cluster. Achieving this requires optimizing the stratification of the data points, taking into account both their feature and positional information, form coherent and meaningful groupings. Clustering methodologies typically employ advanced similarity metrics, such as cosine similarity, measure the proximity between data points and cluster centroids. Additionally, they consider the spatial locality of the points to make more precise group assignments.
Cross-Attention for Generic Clustering. Drawing inspiration from the Transformer decoder architecture [8], contemporary end-to-end architecture [17 [ utilize a query-based approach in which a set of K queries, C (c cx) E RRKD are learned and updated by series of cross- attention blocks. In this context, we rethink the term "C" to associate queries with cluster centers at each layer. Specifically, Pross-attention is employed at each layer adaptively aggregate image features and subsequently update the queries:
C + C. + softmaxx ( (K!)TIV'
where KRD V'ERHWED K'ERHW'D represent linearly projected features for query, key, and value, respectively. The superscripts "C" and "T" denote the features projected from the center and image features, respectively. Motivated by [95], we follow reinterpretation of the cross-attention mechanism as a clustering solver by considering queries as cluster centers and applying the softmax function along the query dimension (K) instead of the image resolution (H))::
C C softmaxx (Q (K')TVVV'
(2)
3.2 CLUSTERFORMER
In this subsection, we present CLUSTERFORMER (see Fig. 2 The model has serial of hierarchical stages that enables multi-scale representation learning for universal adaptation. At each stage. image patches are tokenized into feature embedding []] 533 24 which are grouped into distinct clusters via a unified pipeline first recurrent pross-ateention clustering and then feature dispatching.
Recurrent Cross-Attention Clustering. Considering the feature embeddings I R "W"" and initial centers ( encapsulate the iterative Expectation-Maximizatiii (EM) clustering process,
3

---

(a)
Recoment Attentioon
Restur Digsacchig
(c)
Feature Dispatching (Bq.
Stage
Stage I.
Pr
xd
Universal ddatatioon
1 Σ
K
X
MLP
Similarity
(b)
Recurrent Cross-Attennin Clustering (Eq 3)
E-step M-step
T iterations
(
(
M (0)
M
C
M
Figure 2: (a) Overall pipeline of CLUSTERFORMER (b) Each Recurrent Cross-Attention Clustering layer
carries out T 'terations of cross-attention clustering (E-step) and center updating (M-step) (see Eq. The
feature dispatching redistributes the feature emheddings the top of updated cluster centers (see Eq consisting of T iterations, within Recurrent EM Cross-Artention layer (see Fig. 2b)):
E-step: (1) softmax (Q (( (K')T),
(3)
M-step: ((++1) (1 v KXD
where i E [1,.... 7 and M E [0, 1] represents the "soft" cluster assignment matrix (i.e., probability maps of K clusters). As defined Section 3.1 Q e Kx denotes the query vector projected from the center C, and V',K' / E #2 correspond to the value and key vectors, respectively, projected from the image features The Recurrent Cross-Attentic approach iteratively updates cluster membership M (i.e., E-step) and centers C' (ie., M-step). This dynamic updating strategy embodies the essence of partitional clustering. It enjoys few appealing characteristics:
Efficiency: While the vanilla self-attention mechanism has a time complexity O(H=W2 the Recurrent Cross-Artention approach cxhibits a lower bound of O(TКHWD D). This is primarily due to the fact that TK < HW (i.e., 4165 in Swin [53] US. 1200 in ours). Specifically, considering the nature of the pyramid architecture 88 53 during the encoding process, TK can indeed be much smaller than HW, especially the earlier stages. It is important to note that during each iteration, merely the Q matrix requires update, while the K and V matrices necessitate single computation. Consequently, the whole model enjoys systemic efficiency (see Table Table(oc).
Transparency: The transparency hinges on the unique role that cluster centers play in our Recurrent Cross-Artention mechanism. The cluster centers, derived through our clustering process, act as 'prototypes' for the features they cluster. These 'prototypes' serve as form of representative sample for each cluster, reflecting the most salient or characteristic features of the data points within that cluster. Moreover, the Recurrent Cross-Attention method adheres to the eidely-establishedd EM clustering algorithm, offering a lucid and transparent framework. This cluster center assignment behaves a human- uuman-understandable manner (see Fig. 3 during representation learning and fosters ad-hoc explainability, allowing for a more intuitive understanding of the underlying relationships.
Non-parumetric fashion: The Recurrent Cross-AIe Altentionn mechanism achieves a recursive nature by sharing the projection weights for query, key, and value across iterations. This approach effectively ensures recursiveness without the introduction of additional Iearnable parameters (see Table 6b).
Since the overall architecture is hierarchical, Recurrent Cross-Altention is able thoroughly explore the representational granularity, which mirrors the process of hierarchical clustering:
C² (T' C.),
where RCA stands for the recurrent cross-attention layer. I''s the image feature map at different layers by standard pooling operation with H/2 XW/2 resolution. C* the cluster center matrix for tth layer and C/ the initial centers at 1th layer. The parameters for Recurrent Cross-Attention at different layers, i..., (RCA') 1 are not shared. addition, we initialize the centers from image grids:
(0)
FFN[Adptive (1)),
(5)

---

where FFN stands for Position-wise Feedforward Network which is an integral part of the Transformer architecture. It comprises two fully connected layers along with an activation function used in the hidden layer. Adptive_Poolle Poolingx (T) refers to select K feature centers from I using adaptive sampling, which calculates an appropriate window size to achieve a desired output size adaptively, offering more flexibility and precision compared to traditional pooling methods.
Feature Dispatching. After the cluster assignment, the proposed method employs an adaptive process that dispatches each patch within a cluster based on similarity (see Fig. [()), leading to more coherent and representative understanding of the overall structure and context within the cluster. For every patch embedding P: E I, the updated patch embedding P; computed as:
P4 = Pc + MLP
sim(Cke pc) Ck)
(6)
This equation represents the adaptive dispatching of feature embeddings by considering the similarity between the feature embedding and the cluster centers (C), weighted by their respective similarities. By incorporating the intrinsic information from the cluster centers, the method refines the feature embeddings, enhancing the overall understanding of the image's 's underlying structure and context. All feature representations are utilized for handling the target tasks in the decoding process. In $3. we discuss more details about the implementation of the ending tasks.
3.3 Implementation Details
The implementation details and framework of CLUSTERFORMER are shown in (Fig. 2p). We followed the architecture and configuration of Swin Transformer (33 The code will be available at here
Encoder. The encoding process is to generate presentation hierarchy, denoted as (I') with 1 [1,2,3, 3, 4), for a given image I. The pipeline begins with the feature embedding to convert the images into separate feature tokens. Subsequently, multi-head computing [8] 53 is employed
to partition the embedded features among them. Center initialization (Eq. 5) is then adopted as a starting for initializing the cluster centers, and the recurrent cross-attention clustering (Eq. 3 is utilized to recursively update these centers. Once the centers have been updated, the features are dispatched based on their association with the updated centers (Eq. (o. The further decoding process leverage both the centers and the features, which guarantees well-rounded learning. Adaptation to Image Classification. The classification head a single-layer Multilayer Perceptron (MLP) takes the cluster centers from the encoder for predictions.
Adaptation to Detection and Segmentation. Downstream task head has six Transformer decoder layers with the core design of recurrent cross-attention clustering (Eq 4p. Each layer has 3 iterations.
Experiment
We evaluate our methods over five vision tasks viz. image classification, object detection, semantic segmentation, instance segmentation, and panoptic segmentation on four benchmarks.
ImageNet-1K for Image Classification. ImageNet-1K[72] includes high-resolution images spanning distinct categories (e.g., animals, plants, and vehicles). Following conventional procedures, the dataset split into 1. 1.2M/50K/1
images for brain/validatiinn/eet
splits.
MS COCO for Object Detection and Instance Segmentation. COCO [49] dataset features dense annotations for 80 common objects in daily contexts. Following standard practices [49], the dataset
is split into 115K/5K/20K images for ADEZOK for Semantic Segmentation. ADE20K [01 dataset offers an extensive collection of images with pixel-level annotations, containing 150 diverse object categories in both indoor and outdoor scenes. The dataset comprises 20K/2K/3K images for train/val/test. splits.
splits.
COCO Panoptic for Panoptic Segmentation. The COCO Panoptic dataset [42] includes 80 "thing" categories and a carefully annotated set of 53 "stuff" categories. In line with standard practices [42), the COCO Panoptic dataset is split into 1 115K/5K/20K images for the train/val/test splits well.
The ensuing section commences by presenting the main results of each task ($4.1). succeeded by a series of ablative studies ($ 4.2) which aim to confirm the efficacy of each modulating design.
5

---

4.1 Main Results
4.1.1 Experiments on Image Classification
Training. We use mmclassification as codebase and follow the default training settings. The default configuration for our model involves setting the number of centers to 100. To optimize the model's 's performance, we employ cross-entropy as the default loss function, which widely used in classification tasks and helps in minimizing the difference between predicted probabili- ties and ground truth. For the training details, we run the model for 300 epochs, allowing suf- ficient time for the model to learn and converge. To manage the learning rate, we initialize it at 0.001 as default. The learning rate is then scheduled using a cosine annealing policy, which gradually decreases the learning rate over time. Due to limitations in our GPU capacity, constrained to set the total batch size at 1024. Models are trained from scratch on sixteen A100 GPUs. Results ImageNet. Table illustrates our compelling results over different famous meth- ods. CI.USTER FORMER exceeds the Swin Transformer ( by 0.13% and 0.39% on Tiny- based and Smsel-based models with fewer pa- rameters (i.e., 27.85M us. 28.29M and 18.77M us. M),, respectively. On top-5 accuracy, our approach also outperforms the Swin-Ting Tiny and Swin-Small with gains of 0.71% and 0.84%, respectively. In addition, our margins over the ResNet family (34] are 3.44% 4.76% on top-
Table 1: Classification top- and top-5 accuracy on ImageNet [72 val (see g4.1.1 for details).
Method
Paraams top-1
top-5
Context Cluster-Tiny
588
5.3M
71.68%
90.49%
DeiT-Tinymosaa
80
5.72M
74.50%
92.25%
WViG-Tiny/Naaire
31
9.46M
78.38%
94.38%
ResNet-50.vvant
34
25.56M
76.55%
93.06%
Swin- Swin-Tiny ncevore
133
28.29M
81.18%
95.61%
CLUSTERFORMER Tiny
27.85M
81.31%
96.32%
Context Slusttr-Smaallonn
188
14.0M
77.42%
93.699
DeiT-Small
80
22.05M
80.69%
95.06%
WWiG-Small:vente
31
29.02M
82.00%
95.97%
ResNet-101 cevenmnd
34
44.55M
77.97%
94.06%
I accuracy with on-par parameters (i.., 27.85M us. 25.56M and 48.71M vs. 44.55M).
Swin-Small Smallorevar
53
49.61M
83.02% 96.29%
CLUSTER COETTRRFORMER-SAEE
48.71M
83.41% 97.13%
4.1.2 Experiments on Object Detection
Training. We use mmdetection]] as codebase and follow the default training settings. For a fair comparison, we follow the training protocol in 17 1) the number of instances centers set to 100; 2) a linear combination of the L1 loss and the GloU Loss is used as the optimization objective for bounding box regression. Their coefficients are set to 5 and 2. respectively. In addition, the final object centers are fed into small FFN for object classification, trained with binary cross-entropy loss. Moreover, we set the initial learning rate to x 10-5, 5 the training epoch to 50, and the batch size to 16. We use random scale jittering with a factor in 1, 2.0] and a crop size of 1024 x 1024.
Test. We use one input image scale with shorter side as 800. Metric. We adopt AP, AP APrs. APs, AP AM. and AP,. Performance Comparison. In Table 2 we present the numerical results for CLUSTERFORMER for object detection. We observe that it surpasses all counterparts 170, 7 56 77 9 75 60 102 71 96 with remarkable gains with respect to mAP. In particular, CLUSTER ELUSTERFFRREES FORMER- Tiny exceeds the vanilla Deformable DETR [102], Sparse- (71]. and DINO 1961 over Swin-T [53] by 6.5%, 3.4%, and 0.8% in terms of mAP, respectively. In addition, our approach also outperforms these methods over Swin-S [53] i.e., 54.2% vy 48.3% vs 49.9% vs 53.3% in terms of mAP, respectively. Notably, CUSTERRFORMER achieves impressive performance without relying on additional augmentation.
4.1.3 Experiments on Semantic Segmentation
Training. We use rmsegmentaatorff as codebase and follow the default training settings. The training process for semantic segmentation involves setting the number of cluster centers to match the number of semantic categories, which is 150 for ADE20K [101]. Following the ap- proach employed in recent works [97] 17 (7), we adopt a combination of the standard cross- entropy loss and an auxiliary dice loss for the loss function. By default, the coefficients for the cross-entropy and dice losses are set to 5 and 1, respectively. In addition, we configure the ini- tial learning rate 1 x 10-5 the number of training epochs to 50, and the batch size to 16.

---

Table 2: Quantitative results COCO 49 test-dev for object detection (see $..1.2 2
for details).
Algorithm
Backhone
Epoch
mAP
1
1
APs
AP
APr
Faster R. R-CNN vearsos
70
ResNet- 101
36
41.7
62.3
45.7
24.7
46.0
53.2
Cascade R- CNNivvnnnn
7
ResNet-100 101
36
42.8
61.1
46.7
24.9
46.5
56.4
Grid CoNre
RexNet-50
43.9
DeficientDetevmann
07
Emicient-B3
300
45.4
63.9
49.3
27.1
49.5
61.3
DETRaccva
1
ResNet-50
150
39.9
60.4
41.7
17.6
43.4
59.4
Sparse r-CeN
ResNet-101 101
46.2
50.4
29.5
49.2
Conditional DETR 2009211
60
ResNet-50
50
41.1
61.9
43.5
20.4
44,5
59.9
Deformable DETR
102
Swin-T
50
45.5
65.2
49.8
27.0
49.1
60.7 60.7::029
Swin-S
48.3
68.7+0.27
$2.1
30.5
51.640,22
64,4
Swin
48.62022
69.610.20
53.5
30.1
51.8 51.810.21
64.9 24.92
Sparse Sparse-DETRiue DETR
Swin-S
70.3 49.9
14.0
32.5
66,2 66.2.10.2 53.61.095
Swin-T
51.2:0.20 20
68.42..26
50.3
31.3
55.1
65.8. 20.26
DINO pC1a2])
96
50
Swin
53.3
20.9
57.6
33.8
56.4.0.32
66.9 40.2
CLUSTERFORMER
Ours-Small
Ours Tiny
50
54.2 .0.39
71.8
20.4
57.5
59.1
35.6
20.28
57.2
20.29
10.30
67.4 10:18
L0.22
Furthermore, employ random scale jit- tering, applying a factor within the range of [0.5, 2.0], and utilize crop size with fixed resolution of 640 x 640 pixels.
Table 3: Quantitative results on ADE20K [00
val for
semantic segmentation (sce $4.1.3 1.113.fo for details).
Algorithm
Backbone
Epoch
mloU1
54
ResNet--0
50
36.0
Deeplab 3
15
ResNet-50
50
42.7
Test. During the testing phase, we re-scale the input image with a shorter side to 640 pixels without applying any additional data augmentation at test time.
APCNetiovazaod
32
ResNet-50
100
43.4
SET
100
ViT-L
100
49.3
Segmenteniccvant
ViT-B
100
52.1
Segformen Narteaat
MIT-BS
100
51.4
Metric. Mean intersection-e ceee--ve--uueee over-union (mloU) is used for assessing image seman- tie segmentation performance.
kMaX Tee.--eeplbbbccceo acevan:
95
ConvNeXi-T
100
48.3 #0.15
ConyNeXi-S
$1.6#9.29
Swin-T
48.5 10.24
Musk2Former.wwoee 1v
100
Swin-S
51.1 +0.21
Performance Comparison. Table 3 shows the results on semantic segmentation. Em- priailly, our method compares favorably to recent transformer-bai
Ours Tuny
49.
10.19
CLUSTERFORMER
100
Ours-Small
52.4 20.28
approaches (54 15 32
100
74
90 95 17 For instance, CLUSTER-
FORMER Tiny surpasses both recent advancements, i.e., kMaX- Deeplab 955 and Mask2Former [17 with Swin-T [53] (i.e., 49.1% vs. 48.3% vs. 48.5%), respectively. Moreover, CLUSTERFORMER- Small achieves 52.4% mloU and outperforms all other methods in terms of mloU, making it competi- tive with state-of of the-art methods as well.
4.1.4 Experiments on Instance Segmentation
Training. We adopt the same training strategy for instance segmentation by following $4.1.2 For instance segmentation, we change the training objective by utilizing a combination of the binary cross-entropy loss and the dice Loss for instance mask optimization.
Test. We use one input image scale with a shorter side 800. Metric. We adopt AP, AP50. AP7s, APs, AP Performance Comparison. Table presents the results of CLUSTERFORMER against famous instance segmentation methods (33 g [4 13
and AP
presens
[8 17 44 on COCO test-dev. CLUSTER FORSSTRRRRRRRRR shows clear performance advantages over prior arts. For example, CLUSTERFORMER Tiny outper- forms the universal counterparts Mask2Former (17 by 1.4% over Swin-T
26
23
(53 in terms of mAP and on par with the stale-of-the-art method, Mask-Dino 44 with Swin-T backbone. Moreover, CLUSTER SORMlR-Small surpasses all the competitors, e.g. yielding significant gains of 1.0% and 0.5% mAp compared to Mask2Former and Mask- Dino over Swin-S, respectively. Without bells and whistles, our method establishes new state-of the-art on COCO instance segmentation.
4.1.5 Experiments on Panoptic Segmentation
Training.Folie Following the convention 84 17
we use the following objective for network learning:
Panopric
hh + L aux
CaUX
hh and L represent the loss functions for things and stuff, respectively. To ensure fair comparison, we follow [95] 85) and incorporate an auxiliary loss calculated as a weighted sum of four different loss terms, specifically, PQ-style loss, mask-ID cross-entropy loss, instance discrimination
7

---

Table 4: Quantitative results COCO
49
test-dev for instance segmentation (see $4.1.4 1..4 for details).
Algorithm
Backborc
Epoch
AP::1
APar
AP2
Mask R-CNN
ResNet-101
Cascade MR-CNN
ResNet-100 101
12
37.3
58.2
40.1
19.7
40.6
$1.5
HTCrvssm
14
ResNet-101
39.6
61.0
42.8
21.3
42.9
55.0
PointRend cvesoo
33
ResNett
36.3
56.9
38.7
19.8
39.4
48.5
BlendMask conaor
B
ResNet-101 101
38.4
60.7
41.3
18.2
41.5
53.3
QueryInstacevoo
30
ResNet-101
41.0
63.3
44,5
21.7
44.4
60.7
SOI SQLQQ Queiitsonn
23
Swin-1.7
46.7
72.7
50.6
29.2
50.
60.9
Sparselmst evetaer
18
ResNet 50
37,9
59.2
40.2
15.7
39.4
56.9
Mask2Fommrr consoet
Swin-T
44.5 44.510
15
16.7
23.9.10.0
48.1 18.1
Swin-S
46.0 10:21
68.4
25.4:20.1
49.7
20:22
67.4
Swin-T
45.8 10:08
69.6
50.2....m Lo.90
50.2
16.0
48.7
19.7
66.4
Mask Dino
Swin-S
46.5 10.30
70.1 g0.1
52.2.0.2 10.22
52.2
27.6 g0.54
49,9
-0.25
69.5 20:20
Ours- Tiny
45.9 1..36
69.1 19.1
49.5 19.5
25.2 10.22
50.1 00.3
68.8 10:25
CLUSTERFORMER
Ours-Small
50
47.0
71.5 71.5
51.8
27.3.0.30
50.5
72.6
Table 5: Quantitative results COCO Panoptic
42
val for panoptic segmentation (see
4.1.5 for details).
Algorithm
Backbone
Epoch
PQ7
PQ"1
PQ"
mAP
mloU,.."
Panoptic-PPN PNjjevoooo
41
ResNet-101
20
44,0
52.0
31,9
34.0
51.3
UPSNeticvw2ou
92
ResNet-101
12
46.2
52.8
36.5
36,3
56.9
Panoptic Deeplahieweso
16
Xception-71
41.2
44.9
35.7
31.5
55.4
Panoptic-FCNe FCNieveeso
45
ResNet-50
12
44,3
50.0
35.6
35.5
55.0
Deppaaaapaaic
85
Max-L
55
51.1
57.0
42.2
CMT-Deplaaicvveonn
94
Axial-R104'
55
54.1
58.8
47.1
ResNei-50
49.615.8
54.4_o.06
4241008
39.5
60.8
Panoptic Segformenevveeen
40
ResNet-101
24
50.6+001
53.5-0224
43.2+-0.0
40.4
62.0
Swii-T
33.2+0.5
59,1
433300.2
42.33.0.2
62.9...19
Mask
50
Swin
14.1
60.2 60.2.0.24.
45.6:11.
49.1
63.6
Swin
33.420.9
59.52..20
2400.0.4
"44.5
63.2.0.27
Mask Dinorevron
Swin-S
54.9 40.3
61.1-0.25
46.2 20.2
45.0
64.3
Ours: Tiny
54.7 10.22
60.8-0.31
46.1
43.4
24.0
CLUSTERFORMER
Ours-Small
55.8
10.38
61.9 00.08
47.2 10.23
44.2 (0.02
65.5 (0.21
loss, and semantic segmentation loss. More information about _ can be found in 85 95 The coefficients Ath, X², and \ x are assigned the values of 5, 3, and respectively. Furthermore, the final centers are input into small feed-forward neural network (FFN) for semantic classification, which is trained using a binary cross-entropy loss. Moreover, we set the initial learning rate to x 10 the number of training epochs to 50, and the batch size to 16. We also employ random scale jittering with factor range of 1, 2.0] and a crop size of 1024 1024.
Test. We use input image scale with shorter side of 800.
Metric, We employ the PQ metric [42) and report PQ and PQ for the "thing" and "stuff" classes, respectively. To ensure comprchensiveness, we also include mAPTh precision on "thing" classes using instance segmentation annotations, and mloUpan which calculates mloU for semantic segmentation by merging instance masks belonging the same category, using the same model trained for the panoptic segmentation task.
pan- which evaluates mean average
Performance Comparison. We perform a comprehensive comparison against two divergent groups
of thate-of-the-art methods: universal approaches [46] 17 (44 and specialized panoptic methods [4] 94] As shown in T CLUSTERFORMER outperforms both types of rivals. For
92
16
45
85
97
instance, the performance of CLUSTER FORMER- Tiny clear ahead compared to Mask2Former [17]
(i.e.. 54.7% PQ us. 53.2% PQ) and Mask-Dino
(i.e., 54.7% PQ us. 53.6% PQ) on the top of Swin-T [53] and CLUSTERFORMES -mall achieves promising gains of 1.7% and 0.9% PQ against Mask2Former and Mask- -ino over Swin-S, respectively. Moreover, in terms of mAPTh and mIoU pan- the CLUUSTERFORMER also achieves outstanding performance beyond counterpart approaches.
44
pan
4.2 Ablative Study
This section ablates CLUSTERFORMER'S key components ImageNet 72 and MS COCO 49 validation split. All experiments use the tiny model.
Key Component Analysis. We first investigate the two major elements of FORMMER specifically, Recurrent Cross-Attention Clustering for center updating and Feature Dispatehing for feature updating. We construct a BASELINE model without any center updating and feature dispatching technique. As shown in Table 6a BASELINE achieves 74.59% top-1 and 91.73% top- 5 accuracy. Upon applying Recurrent Cruss-Altention Clustering to the BASELINE, we observe consistent and substantial improvements for both top-1 accuracy (74.59% -> 80.57%) and top-5 accuracy (91.73% 95.22%). This highlights the importance of the center updating strategy
8

---

Table 6: A set of ablative studies on ImageNet 72 4.2).
validation and MS COCO test-dev split (see
The adopted designs are marked in red.
Algorithm Component
#Params
top-1
top-5
Numbers (T)
#Params
top-1
top-5
BASELINE
21.73M
74.59
91.73
81.06
96.23
Recurrent Cross-Attention Clustering
26.27M
80.57
95.22
2
81.22
96.29
27.85M
Feature Dispatching
23.46M
78.58
94.68
81.31
96.32
CLUSTERFORMER (both)
27.85M
81.31
96.32
4
81.33
96.33
(a) Key Component Analysiss
(b) Number of Recursion
Variant Cluster Center Updating Strategy
#Params
top-1
top-5
Head Dimension
#Params
top-1
top-5
Cosine Similarity
23.88M
78.79
94.36
17.25M
71.69
90.16
Vanilla Cross- Attention
g
35.48M
79.67
94.95
24
22.88M
75.37
92.45
Criss Cross-Attention
35
34.16M
79.91
95.24
32
27.85M
81.31
96.32
K-Means 95
27.71M
80.96
95.57
40
32.81M
82.21
97.09
Recurrent Cross-Attentin
27.85M
81.31
96.32
48
38.14M
82.40
97.22
(c) Recurrent Cross-Attntion Clustering
(d) Head Dimension
Feature Dispatching
#Params
top-1
top-5
Decoder Query Initialization
mAP
APsott
7s7
None
26.27M
80.57
95.22
Free Parameters
44.2
66.3
46.4
Vanilla FC Layer
27.14M
80.83
95.47
Direct Feature Embedding
!7
44.5
67.3
47.2
Confidence- Based 68
26,81M
80.69
95.30
Mixed Query Selection
44
44.9
67.9
47.8
FC w/ Similarity (58)
27.46M
80.96
95.84
Scene- Adoe--Aee Embedding
47
45.1
67.8
48.0
Ours (Eq. 6
27.85M
81.31
96.32
Centers from Encoder (Ours)
45.9
69.1
49.5
(e) Feature Dispatching
(I) Decoder Query Initialization for instance segmentation
and validates the effectiveness of our approach, even without explicitly performing clustering, Furthermore, after incorporating Feature Disputching into the BASELINE, we achieve significant gains of 3.99% in top-1 accuracy and 2.95% in top-5 accuracy. Finally, by integrating both core techniques. CLUSTERFORMER delivers the best performance across both metrics. This indicates that the proposed Recurrent Cross-Attention Clustering and Feature Dispatching can work synergistically and validates the effectiveness of our comprehensive algorithmic design.
Recurrent Cross-attention Clustering. We next study the impact of our Recurrent Cross-attention Clustering (Eq by contrasting it with the cosine similarity updating, basic cross- pross-attention [8]], Criss-attention [35] and K-Means cross-attention (95]. As illustrated in Table 6c] our Recurrent Cross-Allention proves to be effective - it outperforms the cosine similarity, vanilla, Criss and K- Means by 2.52%. 1.64%, 1.40% and 0.15% top-1 accuracy respectively, and efficient its #Params are significantly less than the other vanilla and Criss-attention and on par with -MMens, in line with our analysis in $3.2 To gain further insights into recursive clustering, recursion number in Table We discover that performance progressively improves from 81.06% to 81.31% in top-1 accuracy when increasing T from to 3, but remains constant after running additional iterations. We also observe that #Params increase as T increases. Consequently, we set T = 3 as the default to strike an optimal balance between accuracy and computation cost.
examine the effect of the
Multi-head Dimension. We then ablate the head embedding dimension for the attention head in Table/6d| We find that performance significantly improves from 71.69% to 82.40% in top-1 accuracy when increasing the dimension from 16 to 48, but #Params steadily increase as the dimension grows. For fair comparison with Swin (33 we set the head dimension to 32 our default.
Feature Dispatching. We further analyze the influence of our Feature Dispatching. As outlined in Table 6c| a standard manner without any dispatching method, the model attains 80.57% top-1 accuracy and 95. 55.22% top-5 accuracy. By applying a vanilla fully connected layer to update the feature, we witness marginal increase of 0.26% top-1 accuracy. Moreover, using the confidence- updating method (68) and fully connected layer with similarity, the model demonstrates a noticeable enhancement in 0.12% and 0.39% top-1 accuracy, respectively. Last, our method yields significant performance advancements across both metrics, i.e., 81.31 % top-1 and 96.22% top-5 accuracy.
Decoder Query Initialization. Last, we examine the impact of query initialization in the decoder on downstream task (i.e., instance segmentation) in Table 6f For free parameter initialization, the base model can achieve 44.2% in terms of mAP. By applying direct feature embedding, the method has slight improvement of 0.3% mAP. In addition, the model exhibits improvements in

---

Figure 3: Visualization of center-feature assignment at the last stage of recurrent cross-attention clustering with the resolution of by 7. The map displays distinct clusters, each containing features with similar representations. mAP, achieving 44.9% and 45. 1,, respectively, by employing the mixed query selection 44 and scene-adoptive embedding [47] Outstandingly, LLUSTERFORMER achieves the highest performance in all three metrices, i.e., 45.9% mAP, 69.1% AP and 49.5% AP7s, respectively. The empirical evidence proves our design - using the cluster centers from the encoder to derive the initial query for the decoder- that facilitates the transferability for representation learning.
Ad-hoc Explainability. We visualize the cluster assignment map for image classification in Fig. This figure provides an insightful illustration of how GLustERRFRmER groups similar features together. Each color represents a cluster of features that share common characteristics.
5
Conclusion
This study adopts an epistemological perspective centered on the clustering- paradigm, which advocates universal vision framework named FORSSERRORRMER, This framework aims to address diverse visual tasks with varying degrees of clustering granularity. By leveraging insights from clustering, customize the cross-attention mechanism for recursive clustering and introduce novel method for feature dispatching. Empirical findings provide substantial evidence to support the effecctivnness of this systematic approach. Based on its efficacy, proposed universal solution will have a substantial impact on the wider range of visual tasks when viewed through the lens of clustering. This question remains open for our future endeavors.
argue deductively that the
Acknowledgement. This research was supported by the National Science Foundation under Grant No. 2242243.
References
[1] Sameer Agarwal, Jongwoo Lim, Lihi Zelnik-Manor, Pietro Perona, David Kriegman, and
Serge Belongie. Beyond pairwise clustering. In CVPR, 2005.
[2] Merav Ahissar and Shaul Hochstein. The reverse hierarchy theory of visual perceptual learning. Trends in cognitive sciences, 8 8(10):457-464, 2004.
[3] Valerie Ahl and Timothy FH Allen. Hierarchy theory: vision, vocabulary, and epistemology. Columbia University Press, 1996.
[4] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprini arXiv:2 108. 108.07258, 2021.
[5] Gijs Joost Brouwer and David J Heeger. Categorical clustering of the neural representation of color. Journal of Neuroscience, 135495515454-15445, 2013.
[6] Xiao Cai, Feiping Nie, Heng Huang, and Farhad Kamangar. Heterogeneous image feature integration via multi-modal spectral clustering. In CVPR, 2011.
[7] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection.
In CVPR, 2018.
[8] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn; high quality object detection and instance segmentation. IEEE TPAMI, (483)1444114999 2019.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[10] Mathilde Caron, Piotr Bojanowski. Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[11] M Emre Celebi. Partitional clustering algorithums Springer, 2014.
[12] Antoni B Chan and Nuno Vasconcelos. Modeling, clustering, and segmenting video with
mixtures of dynamic textures. IEEE TPAMI. (0(5))909-992, 2008
[13] Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan.
Blendmask: Top-down meets bottom-up for instance segmentation. In CVPR, 2020.
10

---

[14] Kai Chen, Jiangmiao Pang, Jiagi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation.
In CVPR, 2019.
[15] Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[16] Bowen Cheng, Maxwell Collins, Yukun Zhu, Ting Liu, Thomas S Huang. Hartwig Adam, and Liang-Chieh Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, 2020.
[17] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
[18] Tianheng Cheng, Xinggang Wang, Shaoyu Chen, Wenqiang Zhang, Qian Zhang, Chang Huang. Zhaoxiang Zhang. and Wenyu Liu. Sparse instance activation for real-time instance segmentation. CVPR, 2022.
[19] Yiming Cui. Feature aggregated queries for transformer-based video object detectors. In CVPR, 2023.
[20] Yiming Cui, Liqi Yan, Zhiwen Cao, and Dongfang Liu. Ti-blender: Temporal feature blender for video object detection. In ICCV, 2021.
[21] Yiming Cui, Linjie Yang, and Haichao Yu. Learning dynamic query combinations for transformer-based object detection and segmentation. JCML, 2023.
[22] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302. 004.05442, 2023. [23] Bin Dong, Fangao Zeng, Tiancai Wang. Xiangyu Zhang, and Yichen Wei. Solq: Segmenting objects by learning queries. In NeurlPS, 2021.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR,
2021.
[25] Jiemin Fang, Lingxi Xie, Xinggang Wang, Xiaopeng Zhang, Wenyu Liu, and Qi Tian. Msg- transformer: Exchanging local spatial information by manipulating messenger tokens. In
CVPR, 2022.
[26]
Yuxin Fang, Shusheng Yang, Xinggang Wang. Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. In ICCV, 2021.
[27]
David Field, Anthony Hayes, and Robert F Hess. Contour integration by the human visual system: evidence for local "association field". Vision research, 33(2):173-193, 1993.
[28)
Hichem Frigui and Raghu Krishnapuram. A robust competitive clustering algorithm with applications in computer vision. IEEE TPAMI, 2 21(5):450-465, 1999.
[29] Yoram Gdalyahu, Daphna Weinshall, and Michael Werman. Self-organization in vision: stochastic clustering for image segmentation, perceptual grouping, and image database organi-
zation. JEEE TPAMI, 23( 3(10):1053-103 1074, 2001.
[30] Anmol Gulati, James Qin, Chung- Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-angmenned transformer for speech recognition. arXiv preprint arXiv:2005. 08100, 2020.
[31] Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: An image is worth graph of nodes. In NeurIPS, 2022.
[32]
Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. In CVPR, 2019.
[33] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[34]
[35] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang. Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2019.
[36]
Anil K Jain and Richard C Dubes. Algorithms for clustering data. Prentice-Hall. Inc., 1988. Varun Jampani, Deqing Sun, Ming- Yu Liu, Ming Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In ECCV, 2018.
[37]
[38] Stephen Johnson. Hicrarchical clustering schemes. Psychometrika, 32(3):241-255 1967. Jean-Michel Jolion, Peter Meer, and Samira Bataouche. Robust clustering with applications in computer vision. IEEE TPAMI, 13(8):99-882, 1991.
[39]
[40] Bela Julesz. A brief outline of the texton theory of human vision. Trends in Neurosciences,
7(2):41 45, 1984.
11

---

[41] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid
networks. In CVPR, 2019.
[42] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic
segmentation. In CVPR, 2019.
[43] Alexander Kirillov. Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR, 2020.
[44] Feng Li, Hao Zhang, Shilong Liu, Lei Zhang, Lionel M Ni, Heung- Yeung Shum, et al. Mask dino: Towards a unified transformer- banddd framework for object detection and segmentation.
CVPR, 2023.
[45] Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang. Zeming Li. Jian Sun, and Jiaya Jia.
Fully convolutional networks for panoptic segmentation. In CVPR, 2021.
[46] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In CVPR, 2022.
[47] James Liang. Tianfei Zhou, and Dongfang Liu. Clustseg: Clustering for universal segmentation. In ICML, 2023.
[48] Weicong Liang. Yuhui Yuan, Henghui Ding, Xiao Luo, Weihong Lin, Ding Jia, Zheng Zhang, Chao Zhang, and Han Hu. Expediting large-scale vision transformer for dense prediction without fine-tuning, NeurIPS, 2022.
[49] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[50] Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR, 2021.
[51] Dongfang Liu, Yiming Cui, Liqi Yan, Christos Mousas, Baijian Yang, and Yingjie Chen. Densernet: Weakly supervised visual localization using multi-scale feature aggregation. In
AAAI, 2021.
[52] Dongfang Liu, James Liang, Tony Geng, Alexander Loui, and Tianfei Zhou. Tripartite feature enhanced pyramid network for dense prediction. IEEE TIP, 2023.
[53] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[54] Jonathan Long, Evan Shelhamer, and Trevor Darrell, Fully convolutional networks for semantic
segmentation. In CVPR, 2015.
[55] Le Lu and René Vidal. Combined central and subspace clustering for computer vision applications. In /CML, 2006.
[56] Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid r-cnn. In CVPR, 2019. Yawen Lu, Qifan Wang, Siqi Ma, Tong Geng, Yingjie Victor Chen, Huaijin Chen, and Dongfang Liu. Transflow: Transformer as flow learner. CVPR, 2023.
[57]
[58] Xu Ma, Yuqian Zhou, Huan Wang, Can Qin, Bin Sun, Chang Liu, and Yun Fu. Image as set of points. In ICLR, 2023.
[59] Celeste McCollough. Color adaptation of edge-detectors in the human visual system. Science, :49(3688):1115-1111
1965.
[60] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li. Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In ICCV, 2021.
[61] Marius Muja and David G Lowe. Scalable nearest neighbor algorithms for high dimensional
data. IEEE TPAMI, (011):2227-22444
2014.
[62] Fionn Murtagh and Pedro Contreras. Algorithms for hierarchical clustering: an overview, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1):86-97, 2012. [63] Satyasai Jagannath Nanda and Ganapati Panda. A survey on nature inspired metaheuristic algorithms for partitional clustering. Swarm and Evolutionary computation. 16:1-18, 2014.
[64] Frank Nielsen and Frank Nielsen. Hierarchical clustering. Introduction HPC with MPI for
Data Science, pages 195-211, 2016
2016..
[65] Haluk Ögmen, Thomas U Otto, and Michael H Herzog. Perceptual grouping induces non- retinotopic feature attribution in human vision. Vision Research, 46( 1923333-32222 2006.
[66] C. Alejandro Parraga, Tom Troscianko, and David Tolhurst. The human visual system is optimised for processing the spatial information in natural visual images. Current Biology, 10(1):35-38, 2000.
[67] Yury Petrov, Matteo Carandini, and Suzanne McKee. Two distinct mechanisms of suppression in human vision, Journal of Neuroscience, 25(38):8704 8707, 2005.
12

---

[68] Yulei Qin, Juan Wen, Hao Zheng, Xiaolin Huang, Jie Yang. Ning Song, Yue-Min Zhu, Lingqian Wu, and Guung-Zhong Yang. narifocal-net: A chromosome classification approach using deep convolutional networks. IEEE transactions on medical imaging, 38(11):2569- -55811 2019. [69] Chandan K Reddy and Bhamukiran Vinzamuri. A survey of partitional and hierarchical clustering algorithms. In Data clustering, pages 87-110. Chapman and Hall/CRC, 2018.
[70] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
[71] Byungseok Roh, Jae JaeWoons Woong Shin, Wuhyun Shin, and Sachoon Kim. Sparse detr: Efficient end-to-end object detection with learnable sparsity. ICLR, 2022.
[72] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei
Li. Imagenet large scale visual recognition challenge. IJCV, (11(3):211-252 2015.
[73] Dov Sagi. Perceptual learning in vision research. Vision research,
2011. [74] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
[75] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In CVPR, 2021.
[76] Teppei Suzuki. Clustering as attention: Unified image segmentation with hierarchical cluster- ing. arXiv preprint arXiv:2205. 09949, 2022.
[77] Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object
detection. In CVPR, 2020.
[78] Yuliya Tarabalka, Jón Atli Benediktsson, and Jocelyn Chanussot. Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques. IEÉE transactions on geoscience and remote sensing, 47(8):2973-2987, 2009.
[79] Simon Thorpe, Denis Fize, and Catherine Marlot. Speed of processing in the human visual system. nature, 38 (5866))))522-555. 1996.
[80] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training fata-efficient image transformers and distillation through attention. In
ICML, 2021.
[81]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[82]
Mai-Anh T Vu, Tülay Adal, Demba Ba, György Buzsáki, David Carlson, Katherine Heller, Conor Liston, Cynthia Rudin, Vikaas S Sohal, Alik S Widge, et al. A shared vision for machine learning in neuroscience. Journal of Neuroscience, 38(7))1601-166)))
2018.
[83] George Wald. Human vision and the spectrum. Science, 101(2635):653--66 -588, 1945.
[84]
Huiyu Wang. Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chie Chen. MaX-DeepLab: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021.
[85] Huiyu Wang. Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chiee Chen. Max-deeplab:
End-to-end panoptic segmentation with mask transformers. In CVPR, 2021.
[86] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu. Visual recognition with deep nearest centroids. ICLR, 2023.
[87] Wenguan Wang, James Liang, and Dongfang Liu. Learning equivariant segmentation with insancce-nique querying. NeurlPS, 2022.
[88] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[89] Hugh R Wilson. Computational evidence for a rivalry hierarchy in vision. Proceedings of the National Academy of Sciences. 100924):14499-14553
2003.
[90] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 2021.
[91] Xuanli Lisa Xie and Gerardo Beni. A validity measure for fuzzy clustering. IEEE TPAMI, (048 1991.
[92] Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In CVPR, 2019.
[93] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR,
2022.
[94] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Cmt-deeplab: Clustering mask transformers for
13

---

panoptic segmentation. In CVPR. 2022.
[95] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hatwig Adam, Alan Yuille, and Liang- Chigh Chen. k-means mask transformer. ECCV, 2022.
[96]
Hao Zhang. Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.0366
2022.
[97] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified
image segmentation. NeurIPS, 2021.
[98] Ying Zhao, George Karypis, and Usama Fayyad. Hierarchical clustering algorithms for document datasets. Data mining and knowledge discovery, 10:141-168, 2005, [99] Minghang Zheng, Peng Gao, Renrui Zhang, Kunchang Li, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. BMVC, 2021.
[100] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang. Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from sequence-to-sequeee
perspective with transformers. In CVPR, 2021.
[101] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[102] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
14


Reviews:
[{"review_id": "1nHJ0yP0zR", "reviewer": "Reviewer_KGpU", "summary": "This paper presents ClusterFormer, a new module as a replacement of the self-attention module for vision transformers. The method involves performing (iterative) clustering between the input tokens and finally summarizes them into a few clusters for feature computation (using a method named feature dispatching). ClusterFormer was validated effective in a wide range of vision tasks including image classification, object detection, semantic/instance/panoptic segmentation, etc.", "questions": "Please address the concerns raised in the weakness part.", "limitations": "Overall, this is an incremental improvement over the original vision transformers. There are two main limitations, including the existence of similar prior methods and the lack of ability to deal with smaller units (e.g. if a token occupies two semantic regions, it is not possible to split the token -- it is also a weakness of prior methods).", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "+ The proposed ClusterFormer has been validated on a wide range of vision tasks.\n\n+ The paper is well-written and organized.", "weaknesses": "- The proposed ClusterFormer is yet another form of self-attention in which a few clustering tokens were constructed to collect information from visual tokens and then propagate information to them. The novelty is limited given the following works published previously (and it is possible to find more).\n\n[A] Zheng et al., End-to-End Object Detection with Adaptive Clustering Transformer, BMVC 2021.\n[B] Fang et al., Msg-transformer: Exchanging local spatial information by manipulating messenger tokens, CVPR 2022.\n[C] Liang et al., Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning, NeurIPS 2022.\n\n- The ablative study part mostly studies the design choices (e.g. how to perform feature dispatching), but it misses a study on how the method improves the baseline approaches (e.g. compared against Swin). I am not sure why the proposed method is better than the original vision transformers (the current explanations, including visualization, are insufficient to claim the advantages).\n\n- I think the paper somewhat overclaims the advantages, such as explainability. I am a bit conservative in saying that the proposed method is explainable because the essence is still self-attention.\n\n- I hope to make sure if all methods are compared fairly. For example, in Table 1, the ClusterFormer entries were trained with a batch size of 1024 on 16 A100 cards, what about others (e.g. the closest competitor, Swin)? I know that the batch size can largely impact the final results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. ClusterFormer is yet another form of self-attention. The novelty is limited.**\n\n**A1:** We appreciate the reviewer's insightful feedback and we will add discussion with these methods in our version. In the meanwhile, it is important to distinguish our method from the referenced works. Our ClusterFormer adopts the recurrent **cross-attention** mechanism from the perspective of EM clustering to unify the encoding process. Therefore, though our objective of clustering might be similar to [ref1], [ref2], and [ref3], the way in which we accomplish this is conceptually and operationally different. The novelty of our approach lies in the use of this advanced architecture to serve as a universal visual learner.\n\nSpecifically, the method in [ref1] uses an adaptive clustering transformer only after the implementation of CNN backbones for further decoding, a different approach than ours that integrates the clustering mechanism throughout the attention mechanism. In [ref2], their methodology involves using multi-head self-attention, a shuffle module, and an MLP to generate messengers in every region. While this approach shares a similarity in the usage of self-attention, it does not align with our novel use of expectation-maximization clustering in our cross-attention mechanism. [ref3] constructs their clustering layer by following the improved SLIC scheme, a distinctly different approach from ours.\n\nOur methodology not only incorporates a different process of implementation but also provides a significant leap forward in the way we leverage the attention mechanism, fundamentally changing the way we gather, process, and disseminate feature representations within our model. \n\nWe hope that this explanation provides more insight into the innovative nature of our approach. Thanks.\n\n[ref1] Zheng et al., End-to-End Object Detection with Adaptive Clustering Transformer, BMVC 2021. \n\n[ref2] Fang et al., Msg-transformer: Exchanging local spatial information by manipulating messenger tokens, CVPR 2022. \n\n[ref3] Liang et al., Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning, NeurIPS 2022.\n\n#### **Q2. Difference between original vision transformers.**\n\n**A2:** We appreciate the reviewer's feedback. Our work significantly diverges from the traditional vision transformer architectures such as Swin, ViT, and PVT. These architectures rely heavily on a self-attention mechanism, whereas our ClusterFormer employs recurrent cross-attention clustering. Traditional self-attention mechanisms may struggle with the encoding process which tends to be highly distributed and entangled, making it difficult to disentangle what aspects of the input contribute to the output. In contrast, by cross-attention clustering, our model offers an implicit way to generate the cluster center with high semantics and distribute it to the output token individually. Specifically, this unique process allows us to acquire cluster centers and then use feature dispatching to update the feature representations from corresponding cluster centers. \n\nWe believe that our distinct methodological differences inherently lead to improvements over these methods. The recurrent cross-attention clustering offers a more dynamic and flexible means of processing visual knowledge, allowing for more robust and accurate representations. \n\nThank you for your great suggestion. We will provide more discussion together with an ablation result to better explain and visualize the advantage of our model over traditional vision transformers.\n\n#### **Q3. Explainability.**\n\n**A3:** The explainability root on the nature of the centers for each cluster of feature representations (as shown in Fig 3). The key point is the centrality of the 'centers' for each cluster of feature representations in our model. The main idea is that these cluster centers, which are determined through our recurrent cross-attention clustering process, represent a 'prototype' of the features they cluster. This feature allows us to identify what representation is most salient within each cluster, providing interpretability not typically associated with traditional self-attention mechanisms.\n\n#### **Q4. Fair Comparison.**\n\n**A4:** Thank you for the great question. We used the same batch size and endeavored to maintain a consistent and fair environment for all models to be tested. All experiments are followed by the same training schedulers in mmclassification to ensure a fair comparison. We understand that this point might not have been clearly communicated in our paper. We will make sure to provide more explicit details about the setup of all methods in our experiments during revision.\n"}, {"review_id": "DIk0ngNgWK", "reviewer": "Reviewer_bUzy", "summary": "The paper proposes a new architecture ClusterFormer.  Instead of a transformer block, ClusterFormer uses recursive clustering implemented by cross-attention layers and MLP layers dispatching cluster information to tokens. ClusterFormer outperforms other architectures in the same number of parameters. \n", "questions": "- In recent architecture research, computation costs of architecture are the most important part.  How much computation does ClusterFormer require? \n\n- In line 142, the paper claim that ClusterFormer is efficient architecture because $TK << HW$. However, in experiments, $K=100, 150$ are used. Original ViT uses $HW=196$. Although $T$ is not given in the paper, I think it is not enough to argue $TK << HW$. Please explain this mismatch between the method and experiments.\n\n- In the first plot of Figure 1, the x-axis scale looks wrong. Improvement of ClusterFormer is just +0.4. But, it looks +4.0 in the plot. Please correct this. ", "limitations": ".", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- ClusterFormer uses cross-attention similar to k-means clustering, significantly different from a self-attention layer of ViT. It enhances the originality of research and makes ClusterFormer interesting.", "weaknesses": "- Computation (FLOPs) and throughput of ClusterFormer are not reported. Comparison in computation costs is necessary for architecture paper.\n- The paper lacks essential details regarding network architectures, such as network depth, channels, stage configuration, and number of iterations for clustering. I don't think the paper includes enough information to reproduce the results.\n- Writing should be improved. Because the paper only focused on detailed modules, there is no overall architecture description. It is really hard to figure out ClusterFormer as an architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. Computation Cost**\n\n**A1:** Thank you for the suggestion. The computation cost and inference speed are reported as follows. We will incorporate them in the appendix.\n\n|  Method  | Parameters | FLOPs  | inference latency | GPU memory | top-1 accuracy|\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| DeiT-Tiny |  5.72 M | 1.26 G  | 0.35 ms | 1884 MB | 74.50 |\n| ResNet-50 |  25.56 M | 4.12 G  | 0.96 ms | 7658MB | 76.55 |\n| Swin-Tiny |  28.29 M | 4.36 G  | 1.35 ms | 7990 MB | 81.18 |\n| ClusterFormer-Tiny | 27.85 M | 4.19 G  | 1.31 ms | 7786 MB | 81.31 |\n\n|  Method  | Parameters | FLOPs  | inference latency | GPU memory | top-1 accuracy|\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| DeiT-Small |  22.05 M | 4.24 G  | 1.04 ms | 5251MB | 80.69 |\n| ResNet-101 |  44.55 M | 7.85 G  | 1.68 ms | 9682MB | 77.97|\n| Swin-Small |  49.61 M | 8.52 G  | 2.41 ms | 13976 MB | 83.02 |\n| ClusterFormer-Small | 48.71 M | 8.24 G  | 2.24 ms | 13215 MB | 83.41 |\n\n#### **Q2. Lack essential details**\n\n**A2:** Thank you for the feedback. In Section 3.3 of the paper, we try to comprehensively outline the implementation details. Furthermore, Fig 2 of the paper provides a visual depiction of the overall network architecture, illustrating its structure in a more intuitive way.\n\nThe channel configuration, which you pointed out as a missing detail, is described in our ablation study. The study, presented in Table 6, explores the head dimension, and its impact on the results of our experiments. We also studied the number of iterations required for the clustering process and provided these details in the same table.\n\nIn an effort to provide additional clarity and aid in reproducing our results, we have included both pseudo-code and the actual code with anonymous links in the supplemental material.\n\n\n#### **Q3. TK<HW**\n\n**A3:** Thank you for the feedback and we acknowledge your concern about the perceived mismatch between the theoretical claim TK < HW. Instead of using ViT, we follow the most recent pyramid architecture (e.g., Swin Transformer or Pyramid Vision Transformer) for building our model. Considering the nature of the pyramid architecture during the encoding process, the effective HW varies across different stages, with values of 12544, 3136, 784, and 196. The value of HW=196 applies to the final stage, while for earlier stages, HW can be significantly larger. Considering this, the efficiency of our model should be understood in the context of these pyramid architectures, where TK can indeed be much smaller than HW, especially in the earlier stages.\n\n#### **Q4. Figure 1**\n\n**A4:** Thank you for the feedback. We will update the figure in the revised version to make it more clear.\n"}, {"review_id": "bo0WIganw7", "reviewer": "Reviewer_RD2x", "summary": "In this paper, ClusterFormer, which is a network for various vision tasks, is proposed. The proposed algorithm consists of two major components: recurrent cross-attention clustering and feature dispatching. The recurrent cross-attention clustering module groups similar patches by combining EM algorithm with the attention technique. Also, feature dispatching module updates the feature based on clustering results. These two modules are alternately employed. To evaluate the performances of the proposed algorithm, results on various vision tasks, including image classification, object detection, and some segmentations, are provided. In most tests, the proposed algorithm outperforms the conventional techniques.\n", "questions": "I think that the proposed algorithm has more strengths than weaknesses in overall, even though the each piece of the proposed algorithm may seems not very new to the vision community. It is mainly because the proposed algorithm achieves the improved scores in various tests. Also, it is because the proposed algorithm is simple in some way, but sound to me. However, I'm still willing to see other reviewers' comments and the author responses.  ", "limitations": "The authors have addressed the limitation and broader impacts in the Appendix.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed algorithm is simple, but technically sound. The design of each core module may not be very new to the vision community. However, I think it is still meaningful when considering the experimental results. \n\n2. It seems like reproducible since the authors provide the demo code as well.\n\n3. Extensive experimental results are provided. The proposed algorithm achieves the best scores in most tests. Also, the detailed ablation studies are included. ", "weaknesses": "1. In section 4, more detailed discussion on the experimetal results is needed. In 4.1.1-4.1.5, only the performance gains, which may be easily recognized by readers, are described continuously. It would be useful if the discussion about the major difference between the proposed algorithm and the conventional algorithms, which causes the performance gaps, is provided.\n\n2. The proposed algorithm omits the explanation on some parts. For example. in Eq. (5) on page 4, there is no definition for FFN. Then, L288  on page 7, it is described. Even though FFN is widely-used these days, it would be better to explain briefly at least. Similarly, there is no explanation for adaptive sampling or adaptive pooling in L158. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. Detailed discussion**\n\n**A1:** We thank the reviewer for the feedback. The performance gains we observed are predominantly due to our core design of recurrent cross-attention and feature dispatching mechanisms. This strategy enables an implicit generation of semantically rich cluster centers and subsequently distributes this information individually to each output token. In specific, this cross-attention clustering mechanism allows our model to identify cluster centers and utilize feature dispatching to update the corresponding cluster center's feature representations. This innovative process inherently elevates our model's performance compared to traditional methodologies, as observed in our experimental results.\n\nTo better address your concern, we will ensure that these key differences and their impact on the performance gaps are discussed in detail in our revised manuscript. Again, we greatly appreciate your constructive feedback.\n\n\n#### **Q2. Explanation of FFN and adaptive pooling**\n\n**A2:** To clarify, FFN stands for Position-wise Feedforward Network which is an integral part of the Transformer architecture. It comprises two fully connected layers along with an activation function used in the hidden layer. \n\nIn addition, adaptive pooling is a type of pooling that adjusts to the size of the input feature map. This differs from max pooling and average pooling, which require a fixed size of the pooling window. Adaptive pooling calculates an appropriate window size to achieve a desired output size, offering more flexibility and precision compared to traditional pooling methods. \n\nWe will ensure that a proper definition of these terminologies is provided in the revised manuscript.\n"}, {"review_id": "jMiqu7FnEw", "reviewer": "Reviewer_R2Rz", "summary": "The paper proposes a ClusterFormer approach for visual recognition. The ClusterFormer has a Recurrent Cross-Attention Clustering stage which aggregates patch-level images features by cross-attention to form so-called \"cluster centers\" which contains global context information, and a Feature Dispatching which adds global context information from cluster centers to local patch-level features. Results show that the proposed approach obtains better results than the previous approaches on multiple visual recognition tasks, including image classification, object detection, instance segmentation, semantic segmentation, and panoptic segmentation.", "questions": "What's the FLOPS, inference latency, and training/testing GPU memory costs of the proposed approach compared to other approaches?", "limitations": "The authors have adequately addressed the limitations in their supplementary material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed approach is interesting.\n\n+ Most parts of this paper are easy to understand.\n\n+ Promising results are obtained by the proposed approach.", "weaknesses": "- Over-claimed.\nThe paper claims a \"universal vision model\". However, adaptations are needed to make the proposed approach work on different tasks. And different fine-tuned models are needed for different tasks. This is not new and exciting for a visual backbone - It's common in visual backbone work that after some adaptation an ImageNet pre-trained model can be adapted to different tasks using different fine-tuned models, from CNN-based models (e.g., ResNet) to transformer-based models (e.g., Swin). Therefore, saying the proposed model as a \"universal vision model\" is over-claimed. The author(s) should remove this claim in their next version of paper.\n\n- Qualitative results of cluster centers.\nIt would be better to show some qualitative results of cluster centers to show that the learnt cluster centers are really \"cluster centers\" rather than just some feature vectors with global context information.\n\n- Others.\na) Should give definitions of abbreviations, e.g., RCA in Eq. (4). \nb) The term \"recurrent\" is not accurate. It's more like a recursive process instead of recurrent.\nc) The Recursive process may take a lot of time and computation. The author(s) should provide FLOPS, inference latency, and training/testing GPU memory comparisons with other approaches as well instead of just # parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. claims a \"universal vision model\"**\n\n**A1:** We appreciate your invaluable feedback. As you correctly point out, each task requires specific adaptations (a small head) and fine-tuning to maximize performance, which is a common practice in the field. Our intent behind the terminology was to highlight the unique aspect of our framework. Our proposed approach utilizes a straightforward clustering paradigm and this paradigm allows the model to simultaneously tackle heterogeneous tasks, which demonstrates a generic learning capacity --- we, therefore, called it a \"universal visual learner\".\n\nWe agree with your comments that using this term might lead to misunderstandings and may appear as an overclaim. We will revise our terminology to more precisely reflect the model's capability and will stress the necessity of task-specific adaptations in further versions.\n\nAgain, we greatly value your feedback as it helps improve the clarity and accuracy of our work.\n\n\n#### **Q2. Qualitative results of cluster centers**\n\n**A2:** We appreciate your careful review and the suggestion to provide qualitative results of cluster centers. In response to your feedback, we would like to draw your attention to Figure 3 in our paper. This figure provides a visualization of the center-feature assignment at the final stage of our recurrent cross-attention clustering process. Each color in the map corresponds to a distinct cluster, representing a grouping of features with similar representations. These visualizations are meant to demonstrate that our cluster centers are representative and serve as meaningful points of aggregation for related feature vectors. The cluster centers act as representative vectors around which similar features coalesce, which is a defining characteristic of \"cluster centers\" in many clustering algorithms.\n\n#### **Q3. Definition of abbreviations**\n\n**A3:** Sorry for the confusion. RCA stands for the recurrent cross-attention layer. In light of this, we will revise our paper to include more definitions of abbreviations. \n\n#### **Q4. 'Recurrent' is not accurate**\n\n**A4:** Thank you for your insightful feedback. We will use “recursive process” in the version.\n\n#### **Q5. Computation cost**\n\n**A5:** The computation cost and inference speed are reported as follows.\n\n|  Method  | Parameters | FLOPs  | inference latency | GPU memory | top-1 accuracy|\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| DeiT-Tiny |  5.72 M | 1.26 G  | 0.35 ms | 1884 MB | 74.50 |\n| ResNet-50 |  25.56 M | 4.12 G  | 0.96 ms | 7658MB | 76.55 |\n| Swin-Tiny |  28.29 M | 4.36 G  | 1.35 ms | 7990 MB | 81.18 |\n| ClusterFormer-Tiny | 27.85 M | 4.19 G  | 1.31 ms | 7786 MB | 81.31 |\n\n|  Method  | Parameters | FLOPs  | inference latency | GPU memory | top-1 accuracy|\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| DeiT-Small |  22.05 M | 4.24 G  | 1.04 ms | 5251MB | 80.69 |\n| ResNet-101 |  44.55 M | 7.85 G  | 1.68 ms | 9682MB | 77.97|\n| Swin-Small |  49.61 M | 8.52 G  | 2.41 ms | 13976 MB | 83.02 |\n| ClusterFormer-Small | 48.71 M | 8.24 G  | 2.24 ms | 13215 MB | 83.41 |\n"}, {"review_id": "ymhfrISkZZ", "reviewer": "Reviewer_x7UH", "summary": "This paper proposes a vision model based on the clustering paradigm with Transformer, named ClusterFormer.\nIt contains two modules, `recurrent cross-attention clustering` and `feature dispatching`. \nThis paper explains the `cross attention` mechanism from the perspective of `E-M` process.\nIt cleverly combines clustering and attention mechanisms, making feature fusion more accurate.\nThe `feature dispatching` module updates the patch embeddings.\nThe extensive experiments on classification, object detection, and image segmentation demonstrate ClusterFormer has superior accuracy.", "questions": "1. The reviewer wants to see the speed of this model on different task,  e.g. using throughput (image / s).\n2. The reviewer wants to know if further optimization can be made in the  initialization of the cluster center ?\n3. What is the value of `k`? Is `k` related to specific task?\nIf `k` is different in different layers, would the results be better?\n4. Can a larger model achieve better results at the same inference time?", "limitations": "The speed of this model may be slow.\nReviewers may question the practicality of the proposed model.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) Originality.\nThis paper explains the `cross attention` mechanism from the perspective of `E-M` process, which is novelty.\nIt cleverly combines clustering and attention mechanisms, making feature fusion more accurate.\n\n2) Quality. \nThe vision model is carefully design and  experiments show promising performance and ablation studies present interesting results.\nIt presents promising results on varying levels of clustering granularity (i.e., image-, box-, and pixel-level).\n\n3) Clarity\nThe paper is well-written and well-organized.\nThe citation of the paper is comprehensive.\n\n4) Significance\nThe paper may have a high significance. It propose a new backbone paradigm, integrating clustering.\nIt alleviates the problem of unrelated patches being associated with each other in the global attention form of the ViT series models, and may become a universal visual learner.\n", "weaknesses": "Because of the recurrent step, the model may have a high FLOPs and slow inference speed under the same parameter size.\nThis paper does not reveal the inference speed, nor does it compare accuracy under the same inference speed conditions.\nThis may result in unfair experimental comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. Computation cost**\n\n**A1:** The computation cost and inference speed are reported as follows.\n\n|  Method  | Parameters | FLOPs  | inference latency | GPU memory | top-1 accuracy|\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| DeiT-Tiny |  5.72 M | 1.26 G  | 0.35 ms | 1884 MB | 74.50 |\n| ResNet-50 |  25.56 M | 4.12 G  | 0.96 ms | 7658MB | 76.55 |\n| Swin-Tiny |  28.29 M | 4.36 G  | 1.35 ms | 7990 MB | 81.18 |\n| ClusterFormer-Tiny | 27.85 M | 4.19 G  | 1.31 ms | 7786 MB | 81.31 |\n\n|  Method  | Parameters | FLOPs  | inference latency | GPU memory | top-1 accuracy|\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| DeiT-Small |  22.05 M | 4.24 G  | 1.04 ms | 5251MB | 80.69 |\n| ResNet-101 |  44.55 M | 7.85 G  | 1.68 ms | 9682MB | 77.97|\n| Swin-Small |  49.61 M | 8.52 G  | 2.41 ms | 13976 MB | 83.02 |\n| ClusterFormer-Small | 48.71 M | 8.24 G  | 2.24 ms | 13215 MB | 83.41 |\n\n#### **Q2. Initialization of the cluster center**\n\n**A2:** Thank you for your insightful question regarding the potential for further optimization in the initialization of the cluster centers. In our study, we employ the Forgy method for initialization, which involves randomly choosing K data samples as the initial centers. This approach has been selected because of its simplicity and the practicality it provides, allowing us to handle large datasets and complex structures efficiently. While it is random and thus prone to variability, it does, in many cases, provide a reasonable starting point for our EM clustering. Nonetheless, we recognize that this may not always guarantee the optimal solution due to the stochastic nature of the algorithm. As such, we are considering investigating the use of more advanced methods for the initialization of the cluster centers in our future work.\n\n#### **Q3. Value of K**\n\n**A3:** Sorry for the confusion. In our study, we set the value of k to 100 in the context of image classification. The selection of k is indeed related to the specific task. In a broader sense, the choice of k in our model is a balance between model complexity, performance, and computational efficiency. While it is generally true that increasing the value of k can potentially improve model performance by allowing it to capture more intricate patterns within the data, this comes with a trade-off. A larger k leads to a higher number of parameters, which in turn increases the computational demand and potentially the risk of overfitting. Therefore, the selection of k should be guided by the specific requirements and constraints of the task. For different k in different layers, we utilize a progressive pipeline that passes the centers directly between different layers. It may have a different performance if we change the number of k in different layers.\n\n\n#### **Q4. Better performance with the same inference speed**\n\n**A4:** This is a great question. As shown in the above table, our model achieves on-par inference speed compared with the Swin transformer while achieving better performance. A larger model might have higher inference latency.\n"}, {"review_id": "qdn93r7tAj", "reviewer": "Reviewer_QRN3", "summary": "This paper proposes the recurrent cross-attention clustering (RCA) that groups patch features by the traditional EM algorithm with soft-assignment, and also proposes feature dispatching that aggregates spatial information using the cluster center obtained from RCA.\nThe novel backbone model using RCA and feature dispatching, called ClusterFormer, is proposed.\nClusterFormer shows better accuracy than baseline architectures on several datasets and tasks, including image classification, object detection, and segmentation.", "questions": "- Since the proposed method recurrently computes the cross-attention, I think the FLOPs and latency will be higher than the baseline methods. I would like to know the FLOPs and latency of the proposed method. Also, how much memory budget does clusterformer require?", "limitations": "The proposed method would require computational cost, although I do not know for sure because there is no analysis of the FLOPs and the memory budget.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is evaluated on major tasks in the computer vision field.\n- The accuracy improvement is significant.\n- It is an interesting idea to aggregate spatial information based on the proposed feature dispatching.", "weaknesses": "1. The technical novelty of the clustering module is limited. The hierarchical clustering with neural networks is proposed in [1] and [2], and [2] also claims the explainability. The proposed recurrent cross-attention clustering is almost the same as the superpixel sampling network[3]. The paper needs to discuss the relationship between these methods.\n2. The experiments are conducted only with relatively small backbones.\n3. The authors claim the explainability of the recurrent cross-attention clustering. Still, I do not understand how useful it is because I think it does not contribute to the model's explainability, i.e., we cannot know how the model makes classification results.\n4. I think the paper does not provide sufficient information to implement ClusterFormer and reproduce results. At least, the detailed architectures of ClusterFormer-tiny, -small, and the overall pipeline of detection and segmentation should be described.\n\n[1] J. Xu et al. “GroupViT: Semantic Segmentation Emerges from Text Supervision.“, 2022  \n[2]T. Suzuki “Clustering as Attention: Unified Image Segmentation with Hierarchical Clustering.“, 2022  \n[3] V. Jampani et al. “Superpixel Sampling Networks.“, 2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. The technical novelty of the clustering module**\n\n**A1:** Thank you for your suggestion, and we will discuss the relationship between these methods. Our model improves the cross-attention mechanism from an Expectation-Maximization clustering perspective to unify the encoding process. This modification provides a unique approach to the task and contributes to our model's performance. To compare with the references mentioned, [ref1] jointly trains the model and a text encoder using a paired image-text training fashion and then transfers the trained model to the task of zero-shot semantic segmentation. Our method, however, does not rely on text encoders or paired image-text training and instead focuses directly on the vision task. In [ref2], a DCN-based kernel is used to generate the attention map for clustering. While our model also uses an attention mechanism, it does so in a fundamentally different manner by modifying the cross-attention mechanism from an EM clustering perspective. Finally, [ref3] employs the superpixel sampling network, which follows the SLIC scheme. This is conceptually different from our method, where the recurrent cross-attention mechanism is used for cluster formation.\n\nWe will ensure to discuss the relationship between these methods in a dedicated section of our revised manuscript, highlighting both the shared insights and distinct differences. Thank you again for the insightful suggestion.\n\n[ref1] J. Xu et al. GroupViT: Semantic Segmentation Emerges from Text Supervision, 2022\n[ref2] T. Suzuki Clustering as Attention: Unified Image Segmentation with Hierarchical Clustering, 2022\n[ref3] V. Jampani et al. Superpixel Sampling Networks, 2018\n\n\n#### **Q2. larger backbone**\n\n**A2:** Thank you for your suggestion. Due to the limited computation resources, we provide a base-sized model for image classification as follows. Based on our experience, it is generally true that increasing the size of the model -- by adding more parameters -- can lead to better performance. We conduct additional experiments on the Base-sized backbone and will incorporate the below results in the revision.\n\n|  Method  | Parameters | FLOPs | top-1 accuracy | top-5 accuracy |\n| :-: | :-: | :-: | :-: | :-: |\n| ResNet-152 |  60.19M  | 11.58G  | 78.61 | 94.15 |\n| Swin-Base |  87.77M | 15.19G  | 83.36 | 96.44 |\n| ClusterFormer-Base | 81.95M | 14.27G  | 83.62 | 97.36 |\n\n#### **Q3. Explainability**\n\n**A3:** Sorry for the confusion on the explainability. Our claim of the explainability hinges on the unique role that cluster centers play in our recurrent cross-attention mechanism. The cluster centers, derived through our clustering process, act as 'prototypes' for the features they cluster. These 'prototypes' serve as a form of a representative sample for each cluster, reflecting the most salient or characteristic features of the data points within that cluster.\n\nThe benefit of this is twofold. Firstly, it provides an avenue for interpreting the clustering mechanism itself, offering a glimpse into how the data is being partitioned and what features are being considered most pertinent within each cluster. Secondly, and perhaps more significantly, these prototypes can be associated with the final classification results, providing some degree of interpretability. Specifically, by examining which cluster a particular instance is associated with, and looking at the prototype of that cluster, we can gain some insight into what features the model deemed most relevant when classifying that instance.\n\nAgain, we appreciate your insight and will aim to make the level and limitations of our model's explainability more explicit in our revised version.\n\n#### **Q4. Implementation**\n\n**A4:** We appreciate your emphasis on implementation details. In response to your comment, we would like to point out that in Section 3.3 of our paper, we have endeavored to provide comprehensive information on implementation details and adaptation to different tasks. Moreover, to enhance reproducibility, we have provided both pseudo-code and the actual code via an anonymous link in our supplemental material. \n\nConcerning the specific architectures of ClusterFormer-tiny and -small, these are variations of our model followed by the same configuration (e.g., heads, embedded dimension, windows, and layers) of Swin Transformer.\n\nWe appreciate your constructive feedback and will enhance the description of detailed architectures."}]

Author rebuttal:
#### **To all reviewers：**

Thank you very much for your valuable time and suggestive comments. We will revise our paper according to your comments. The major changes are as follows:

1. We will offer a more detailed discussion regarding our novelty compared to existing methods and highlight the architecture distinction with original vision transformers as suggested by Reviewer KGpU and QRN3.

2. We will provide more insights on the explainability of our design from the perspective of clustering, as suggested by Reviewer KGpU.

3. We will further clarify the design of the recursive cross-attention mechanism compared with traditional self-attention in any pyramid-based transformer, as suggested by Reviewer bUzy.

4. We will add more descriptions and clarify some misleading terms, as suggested by Reviewer RD2x and R2Rz.

5. We will include more experimental results of the Base-sized model in the revision, according to the suggestion by Reviewer QRN3.

6. We will supplement additional implementation details together with the computational cost to ensure a more complete comparison, according to the comments from the reviewers.

We have strived to address each of your concerns comprehensively and welcome further discussions and insights.

Sincerely yours,

Authors


Meta-review: All but one reviewer agree to accept the paper. The reviewers agreed that while individual components of the proposed approach may not be novel, the overall system is a valuable contribution. 

Multiple reviewers raised the concern that the paper made exaggerated claims. The authors are encouraged to carefully delineate the claims, erring on the side of caution. The authors are also encouraged to discuss additional references that have been pointed out by the reviewers.
Decision: Accept (poster)