Image as Set of Points
Authors: Xu Ma
Yuqian Zhou
Huan Wang
Can Qin
Bin Sun
Chang Liu
Yun Fu
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Clustering
Image Processing
Context Cluster
Representation

Abstract: 
What is an image, and how to extract latent features? 
Convolutional Networks (ConvNets) consider an image as organized pixels in a rectangular shape and extract features via convolutional operation in a local region; Vision Transformers (ViTs) treat an image as a sequence of patches and extract features via attention mechanism in a global range. In this work, we introduce a straightforward and promising paradigm for visual representation, which is called Context Clusters. Context clusters (CoCs) view an image as a set of unorganized points and extract features via a simplified clustering algorithm. In detail, each point includes the raw feature (e.g., color) and positional information (e.g., coordinates), and a simplified clustering algorithm is employed to group and extract deep features hierarchically. Our CoCs are convolution- and attention-free, only relying on clustering algorithm for spatial interaction. Owing to the simple design, we show CoCs endow gratifying interpretability via the visualization of the clustering process.  
Our CoCs aim at providing a new perspective on image and visual representation, which may enjoy broad applications in different domains and exhibit profound insights. Even though we are not targeting SOTA performance, COCs still achieve comparable or even better performance than ConvNets or ViTs on several benchmarks.

Published as a conference paper at ICLR 2023
IMAGE AS SET OF POINTS
Xu Ma", Yuqian Zhou²; Huan Wang', Can Qin', Bin Sun', Chang Liu', Yun Fu' NNortheastern University 'Adobe Inc. (ma. xul, wang. huan, qin. ca, sun. bi, liu. yugzhou@adobe.com com yunfu@ece neu. neu. edu
ABSTRACT
What is an image and how to extract latent features?
Convolutional Networks (ConvNets) consider an image as organized pixels in a rectangular shape and extract features via convolutional operation in local region; Vision Transformers (ViTs) treat an image as a sequence of patches and extract features via attention mechanism in global range. In this work, we introduce a straightforward and promising paradigm for visual representation, which called Context Clusters. Context clusters (CoCs) view an image as set of unorganized points and extract features via simplified clustering algorithm. In detail, each point includes the raw feature (e.g., color) and positional information (e.g., coordinates), and a simplified clustering algorithm is employed to group and extract deep features hierarchically. Our CoCs are convolution- and attention-free, and only rely on clustering algorithm for spatial interaction. Owing to the simple design, we show CoCs endow gratifying interpretability via the visualization of clustering process. Our CoCs aim at providing new perspective on image and visual representation, which may enjoy broad applications different domains and exhibit profound insights. Even though we are not targeting SOTA performance, COCs still achieve comparable or even better results than ConvNets ViTs on several benchmarks. Codes are available at: https://ithbb.com/maau////Cnttet Cluuttrr.
INTRODUCTION
The way we extract features depends a lot how we interpret an image. As a fundamental paradigm, Convolutional Neural Networks (ConvNets) have dominated the field of computer vision and considerably improved the performance of various vision tasks in recent years (He et al., 2016; Xie et al., 2021; Ge et al., 2021). Methodologicalll. ConvNets conceptualize a picture as a collection of arranged pixels in rectangle form, and extract local features using convolution in sliding window fashion. Benefiting from some important inductive biases like locality and translation equivariance, ConvNets made to be efficient and effective. Recently, Vision Transformers (ViTs) have significantly challenged ConvNets' hegemony in the vision domain. Derived from language processing, Transformers (Vaswani al., 2017) treat image sequence of patches, and global-rang self-attention operation is employed to adaptively fuse information from patches. With the resulting models (i.e., ViTs), the inherent inductive biases in ConvNets are abandoned, and gratifying results are obtained (Touvron et al., 2021).
Recent work has shown tremendous improvements in vision community, which are mainly built on top of convolution or attention (e.g., ConvNeXt (Liu et al., 2022), MAE (He et al., 2022), and CLIP (Radford al., 2021)). Meanwhile, some attempts combine convolution and attention together, like CMT (Guo et al., 2022a) and CoAtNet (Dai et al., 2021). These methods sean images in grid (convolution) yet explore mutual relationships of a sequence (attention), enjoying locality prior (convolution) without sacrificing global reception (attention). While they inherit the advantages from both and achieve better empirical performance, the insights and knowledge are still restricted to ConvNets and ViTs. Instead of being lured into the trap of chasing incremental improvements, underline that some feature extractors are also worth investigating beyond convolution and attention. While convolution and attention are acknowledged to have significant benefits and an enormous influence on the field of vision, they are not the only choices. MLP-based architectures (Touvron
"Equal contribution

---

Published as a conference paper at ICLR 2023
et al., 2022; Tolstikhin et al., 2021) have demonstrated that pure MLP- based design can also achieve similar performance. Besides, considering graph network as the feature extractor is proven to be feasible (Han et al., 2022). Hence, expect new paradigm of feature extraction that can provide some novel insights instead of incremental performance improvements.
In this work, the classical algorithm for the funda- mental visual representation, clustering method (Bishop & Nasrabadi, 2006). Holisticallly, we view an image as a set of data points and group all points into clus- ters. In each cluster, we aggregate the points into center and then dispatch the center point to all the points adaptively. We call this design context cluster. Fig. illustrates the process. Specifically, we 5-dimensional
look back into
Figure 1: A context cluster in our network trained for im- age classification. We view an image as set of points and sample c centers for points clustering, Point features are aggregated and then dispatched within a cluster. For clus- ter center C₁ we first aggregated all points (sf. x x..... xx²) in ith cluster, then the aggregated result is distributed to all points in the clusters dynamically. See & 3 for details.
consider each pixel data point with the information of color and position. In a sense, convert an image as a set of point clouds and utilize methodologies from point cloud analysis (Qi et al., 2017b; Ma et al., 2022) for image visual representation learning. This bridges the representations of image and point cloud, showing strong generalization and opening the possibilities for an easy fusion of multi-modalities. With a set of points, we introduce simplified clustering method to group the points into clusters. The clustering processing shares similar idea as SuperPixel (Ren & Malik, 2003), where similar pixels are grouped, but they are fundamentally different. To our best knowledge, we are the first ones to introduce the clustering method for the general visual representation and make it work. On the contrary, SuperPixel and later versions are mainly for image pre- pre-processing (Jampani al., 2018) or particular tasks like semantic segmentation (Yang et al., 2020; Yu et al., 2022b)
We instantiate our deep network based on the context cluster and name the resulting models as Context Clusters (CoCs). Our new design is inherently different from ConvNets or ViTs, but also inherit some positive philosophy from them, including the hierarchical representation (Liu et al., 2022) from ConvNets and Metaformer (Yu et al., 2022c) framework from ViTs. CoCs reveal distinct advantages. First, by considering image as a set of points, CoCs show great generalization ability to different data domains, like point clouds, RGBD images, etc. Second, the context clustering processing provides CoCs gratifying interpretability. By visualizing the clustering in each layer, we can explicitly understand the learning in each layer. Even though our method does not target SOTA performance, it still achieves on par or even better performance than ConvNets or ViT's on several benchmarks. We hope our context cluster will bring new breakthroughs to the vision community.
2
RELATED WORK
Clustering in Image Processing While clustering approaches in image processing (Castleman, 1996) have gone out of favor in the deep learning era, they never disappear from computer vision. A time-honored work is SuperPixel (Ren & Malik, 2003), which segments an image into regions by grouping a set of pixels that share common characteristics. Given the desired sparsity and simple representation, SuperPixel has become common practice for image preprocessing. Naive application of SuperPixel exhaustively clusters (e.g., via K-means algorithm) pixels over the entire image, making the computational cost heavy. To this end, SLIC (Achanta et al., 2012) limits the clustering operation in a local region and evenly initializes the K-means centers for better and faster convergence. In recent years, clustering methods have been experiencing a surge of interest and are closely bound with deep networks (Li & Chen, 2015; Jampani et al., 2018; Qin et al., 2018; Yang et al., 2020). To create the superpixels for deep networks, SSN (Jampani et al., 2018) proposes a differentiable SLIC method, which is end-to-end and trainable and enjoys favorable runtime. Most recently, tentative efforts have been made towards applying clustering methods into networks for specific vision tasks, like segmentation (Yu et al., 2022b; Xu et al., 2022) and fine-grained recognition (Huang & Li, 2020). For example, CMT-Deepl.al Lab (Yu al., 2022a) interprets the object queries in segmentation task as
2

---

Published as a conference paper at ICLR 2023
cluster centers, and the grouped pixels are assigned to the segmentation for each cluster. Nevertheless, to our best knowledge, there is no work conducted for a general visual representation via clustering. We aim to make up for the vacancy, along with proving the feasibility numerically and visually.
ConyNets & ViTs ConvNets have dominated the vision community since the deep learning era (Simonyan & Zisserman, 2015; He et al., 2016). Recently, ViTs (Dosovitskiy et al., 2020) introduce purely attention-baut based transformers (Vaswani al., 2017) to the vision community and have set new SOTA performances various vision tasks. A common and plausible conjecture is that these gratifying achievements are credited the self-attention mechanism. However, this intuitive conjecture has soon been challenged. Extensive experiments also showcase that ResNet (He et al., 2016) can achieve on par or even better performance than ViT', with proper training recipe and minimal modifications (Wightman al., 2021; Liu et al., 2022). We emphasize that while convolution and attention may have unique virtues (i.e., ConvNets enjoy inductive biases (Liu et al., 2022) while ViTs excel generalization (Yuan et al., 2021b)), they did not show significant performance gap. Different from convolution and attention, in this work, we radically present a new paradigm for visual representation using clustering algorithm. With both quantitative and qualitative analysis, we show that our method can serve as a new general backbone and enjoys gratifying interpretability.
Recent Advances Extensive efforts have been made to push up the vision tasks' performances within the framework of ConvNets and ViTs (Liu et al., 2021b; Ding et al., 2022b; Wu et al.. 2021). To take advantage of both convolution and attention, some work learns to mix the two designs in hybrid mode, like CoAtNet (Dai et al., 2021) and Mobile-Former (Chen al., 2022b). We also note that some recent advances explored more methods for visual representation, beyond convolution and attention. MLP-like models (Tolstikhin al., 2021; Touvron et al., 2022; Hou et al., 2022; Chen et al., 2022a) directly consider MLP layer for spatial interaction. Besides, some work employs shifting (Lian et al., 2021; Huang et al., 2021) or pooling (Yu al., 2022c) for local communication. Similar to our work that treats the image unordered data set, Vision GNN (ViG) (Han al., 2022) extracts graph-level features for visual tasks. Differently, we directly apply the clustering method from conventional image processing and exhibit promising generalization ability and interpretability.
METHOD
Context Clusters forgo the fashionable convolution at- tention in favor of novelly considering the classical al- gorithm, clustering. for the representation of visual learn- ing. In this section, we first describe the Context Clus- ters pipeline. The proposed context cluster operation (as shown in Fig. 2) for feature extraction then thoroughly explained. After that, up the Context Cluster architectures. Finally, some open discussions might aid individuals in compre- hending our work and exploring more directions following our Context Cluster.
clatee
MLP
cluser
Input It sot data points Figure 2: A Context Cluster block. We context cluster operation to group set data points, and then communicate the points within clusters. An MLP block is applied later.
Context Cluster
MLP block
aggroggate cisputic within cluster
chamnel communicaiinn
3.1 CONTEXT CLUSTERS PIPELINE
From Image to Set of Points. given an input image I E R 33owcch we begin by enhancing the image with the 2D coordinates of each pixel I.,, where each pixel's coordinate is presented as
[L 0.5, 0.5]. is feasible to investigate further positional augmentation techniques to potentially improve performance. This design taken into consideration for its simplicity and practicality. The augmented image is then converted to collection of points (i.e., pixels) P 5x* where x is the number of points, and each point contains both feature (color) and position (coordinates) information; hence, the points could be unordered and disorganized.
3

---

Published as a conference paper at ICLR 2023
(Ca n
Stage [Ci
Stage 2
Stage 3
Stage 4 Co 1024
[5.m]
Image Points Set
Points Reducer
Context Cluster Blocks
Points Reducer
Context Cluster
Points Reducer
Context Cluster
Points Reducer
Context Cluster Blocks
Blocks
Blocks
XN2
XN
Figure 3: Context Cluster architecture with four stages. Given a set of image points, Context Cluster gradually reduces the point number and extracts deep features. Each stage begins with points reducer, after which a succession of context cluster blocks is used to extract features.
We are rewarded with excellent generalization ability by offering fresh perspective of image, a set of points. A set of data points can be considered universal data representation because data in most domains can be given as a combination of feature and position information (or either of the two). This inspires us conceptualize image set of points.
Feature Extraction with Image Set Points. Following the ConvNets methodology (He et al., 2016; Liu et al., 2022), extract deep features using context cluster blocks (see Fig. for reference and $ 3.2 for explanation) hierarchically. Fig. 3 shows our Context Cluster architecture. Given a set of points P R we first reduce the points number for computational efficiency, then succession of context cluster blocks are applied to extract features. To reduce the points number, we evenly select some anchors space, and the nearest k points are concatenated and fused by a linear projection. Note that this reduction can be achieved by a convolutional operation if all points are arranged in order and is properly set (i.e., and 9), like in ViT (Dosovitskiy et al., 2020). For clarity on the centers and anchors stated previously, we strongly suggest the readers check appendix & B.
Task-Specific Applications. For classification, we average all points of the last block's output and use a FC layer for classification. For downstream dense prediction tasks like detection and segmentation, we need to rearrange the output points by position after each stage to satisfy the needs of most detection and segmentation heads (e.g., Mask-RCNN (He et al., 2017)). In other words, Context Cluster offers remarkable flexibility classification task, but limited to a compromise between dense prediction tasks' requirements and our model configurations. We expect innovative detection & segmentation heads (like DETR (Carion et al., 2020)) can seamlessly integrate with our method.
3.2 CONTEXT CLUSTER OPERATION
In this subsection, we introduce the key contribution in our work, the context cluster operation. Holistically, we first group the feature points into elusters; then, feature points in each cluster will be aggregated and then dispatched back, as illustrated in Fig. 1.
Context Clustering. Given a set of feature points P E Ix we group all the points into several groups based on the similarities, with each point being solely assigned to one cluster. We first linearly project P to P, for similarity computation. Following the conventional SuperPixel method SLIC (Achanta al., 2012), we evenly propose e centers space, and the center feature is computed by averaging its À nearest points. We then calculate the pair-wise cosine similarity matrix S E Roi between P, and the resulting set center points. Since each point contains both feature and position information, while computing similarity, we implicitly highlight the points' distances (locality) as well as the feature similarity. After that, we allocate each point to the most similar center, resulting in clusters. Of note is that each cluster may have different number of points. In extreme cases, some clusters may have zero points, in which case they are redundant.
Feature Aggregating. We dynamically aggregate all points in a cluster based on the similarities to the center point. Assuming cluster contains M points (a subset in P) and the similarity between the m points and the center is s E R" (a subset in S ), we map the points to a value space to get P, mindd" where d' is the value dimension. We also propose center v. in the value space like the
4

---

Published as a conference paper at ICLR 2023
clustering center proposal. The aggregated feature g R is given by:
g = Vc Σ sig (as; + +/ s.l. I= Σ sig (ass + +
(1)
Here and B are learnable scalars to scale and shift the similarity and sig (-) is a sigmoid function to re-scale the similarity to (0. 1). v, indicates i-th point in P... Empirically, this strategy would achieve much better results than directly applying the original similarity because no negative value involved. Softmax is not considered since the points do not contradict with one another. We incorporate the value center V. in Eq. for numerical stability' as well as further emphasize the locality. To control the magnitude, the aggregated feature is normalized by factor of C.
Feature Dispatching. The aggregated feature g is then adaptively dispatched to each point in a cluster based on the similarity. By doing so, the points communicate with one another and shares features from all points in the cluster, as shown in Fig. 1. For each point pi, we update it by
p P: + FC (sig (as; + B) x))
Here follow the same procedures to handle the similarity and apply fully-comnected (FC) layer to match the feature dimension (from value space dimension d' to original dimension d).
Multi-Head Computing. We acknowledge the multi-head design in the self-attention mecha- nism (Vaswani et al., 2017) and use it to enhance our context cluster. We consider / heads and set the dimension number of both value space P, and similarity space P, to d' for simplicity. The outputs of multi-head operations are concatenated and fused by a FC layer. The multi-head architecture also contributes to satisfying improvement our context cluster, as we empirically demonstrate.
3.3 ARCHITECTURE INTIIALIZATION
While Context Cluster is fundamentally distinct from convolution and attention, the design philoso- phies from ConvNets and ViTs, such as hierarchical representations and meta Transformer architec- ture (Yu et al., 2022c), are still applicable to Context Cluster. To align with other networks and make our method compatible with most detection and segmentation algorithms, we progressively reduce the number of points by a factor of 16, 4, 4, and 4 in each stage. We consider 16 nearest neighbors for selected anchors in the first stage, and we choose their nearest neighbors in the rest stages.
An underlying issue is computational efficiency. Assume we have d-dimensional points and clus- ters, the time complexity to calculate the feature similarity would be O (ned), which is unacceptable when the input image resolution is high (e.g., 224 224). To circumvent this problem, introduce region partition by splitting the points into several local regions like Swin Transformer (Liu et al., 2021b), and compute similarity locally. As a result, when the number of local regions is set r, noticeably lower the time complexity by factor of r, from O (ncd) to O (r4 fd). See appendix A for detailed configurations. Note that if we split the set of points to several local regions, we limit the receptive field for context cluster, and no communications among local regions are available.
3.4 DISCUSSION
Fixed or Dynamic centers for clusters? Both conventional clustering algorithms and SuperPixel techniques iteratively update the centers until converge. However, this will result in exorbitant computing costs when clustering is used as a key component in each building block. The inference time will increase exponentially. In Context Cluster, view fixed centers inference efficiency, which can considered as a compromise between accuracy and speed.
alternative for
Overlap or non-overlap clustering? We allocate the points solely to specific center, which differs from previous point cloud analysis design philosophies. We intentionally adhere to the conventional clustering approach (non-overlap clustering) since we want to demonstrate that the simple and traditional algorithm can serve as a generic backbone. Although might produce higher performance, overlapped clustering is not essential to our approach and could result in extra computing burdens.
If there were involved and no points are grouped into the cluster coincidentally. C would be zero, and the network cannot he optimized. In our research, this conundrum occurs frequently. Adding small value like le does not help and would lead to the problem of vanishing gradients.
5

---

Published as a conference paper at ICLR 2023
Table I: Comparison with representative backbones ImageNet- Ik benchmark. Throughput (images s) is measured on single V100 GPU with a batch size of 128, and is averaged by the last 500 iterations. All models are trained and tested 224x224 resolution, except VYT-B and ViT-L.
Throughputs (images/s)
Method
Param.
GFLOPs
Top-1
ResMLPP112 (Touvron et al., 2022)
15.0
3.0
76.6
511.4
ResML.P-24 (Touvron al., 2022)
30.0
6.0
79.4
509.7
ResMLP-36 (Touvron al., 2022)
45.0
8.9
79.7
452.9
MLP
MLP-MMxer-BBi B/16 (Tolstikhin et al., 2021)
59.0
12.7
76.4
400.8
MLL--Mixer-L116 (Tolstikhin al., 2021)
207.0
44.8
71.8
125.2
gMLP-Ti (Liu et al., 2021a)
6.0
1.4
72.3
511.6
gMLP-S (Liu al., 2021a)
20.0
4.5
79.6
509.4
ViT-B/16 (Dosovitskiy et al., 2020)
86.0
55.5
77.9
292.0
Attention
ViT-L/16 (Dosovitskiy et al., 2020)
307
190.7
76.5
92.8
PVT-Tiny (Wang et al., 2021)
13.2
1.9
75.1
PVT-Small (Wang al., 2021)
24.5
3.8
79.8
T2T-ViT-7 (Yuan et al., 2021a)
4.3
1.1
71.7
DeiT-Tiny/16 (Touvron et al., 2021)
5.7
1.3
72.2
523.8
DeiT-Small/16 (Touvron al., 2021)
22.1
4.6
79.8
521.3
Swin-T (Liu et al., 2021b)
29
4.5
81.3
Res.Net18 (He al., 2016)
12
1.8
69.8
584.9
Cluster Convolution
ResNer50 (He et al., 2016)
26
4.1
79,8
524.8
Conv-Mixer-512/16 (Trockman al., 2022)
5,4
73.8
Con-Mixer-1022/12 (Trockman et al., 2022)
14.6
77.8
Conv-Mixer-766/22 (Trockman al., 2022)
21.1
80.16
142.9
Context-Cluster-1 (ours)
5.3
1.0
71.8
518.4
Context-Cluster-Ti (nurs)
5.3
1.0
71.7
510.8
Context-Cluster-Smalll (ours)
14.0
2.6
77.5
513.0
Context-Cluster-Mlcciun (ours)
27.9
5.5
81.0
325.2
4 EXPERIMENTS
We validate Context Cluster ImageNet-1K (Deng al., 2009), ScanObjectNN (Uy al., 2019), MS COCO (Lin et al., 2014), and ADE20k (Zhou et al., 2017) datasets for image classification, point cloud classification, object detection, instance segmentation, and semantic segmentation tasks.
Even we are not in pursuit of state-of-the-art performance like ConvNeXt (Liu et al., 2022) and DaViT (Ding et al., 2022a), Context Cluster still presents promising results on all tasks. Detailed studies demonstrate the interpretability and the generalization ability of our Context Cluster.
4.1 IMAGE CSASTFIIAAIION ON IMAGENET-
We train Context Clusters on the ImageNet-I K training set (about 1.3M images) and evaluate upon the validation set. In this work, we adhere to the conventional training recipe in (Dai al., 2021; Wight- man, 2019; Touvron et al., 2021; Yu et al., 2022c). For data augmentation, we mainly adopt random horizontal flipping, random pixel erase, mixup, cutmix, and label smoothing. AdamW (Loshchilov & Hutter, 2019) is used to train all of our models across 310 epochs with momentum of 0.9 and weight decay of 0.05. The learning rate schedular (Loshchilov & Hutter, 2017). By default, the models are trained on 8 A A10 GPUs with 128 mini-batch size (that 1024 total). We use Exponential Moving Average (EMA) to enhance the training, similar to carlier studies (Guo et al.. 2022b; Touvron et al., 2021). Table reports the parameters used, FLOPs, classification accuracy, and throughputs. denotes a different region partition approach that used to divide the points into 499.49,
0.001 by default and adjusted using cosine
in the four stages.
Empirically, results in Table indicate the effectiveness of our proposed Context Cluster. Our Context Cluster is capable of attaining comparable even better performance than the widely-used baselines using a similar number of parameters and FLOPs, With about 25M parameters, our Context Cluster surpasses the enhanced ResNet50 (Wightman et al., 2021) and PVT--small by 1.1% and achieves 80.9%

---

Published as a conference paper at ICLR 2023
(attentioon mup
VIT-B/16
(chrstering map) (clustering maap) (class act. maap)
ResNet50
CoC-M
wi relie purtitiin
CoC-T
Figure 4: Visualization of activation map, class activation map, and clustering map for VYT-B/16, ResNet50, our CoC-M, and CoC-T without region partition, respectively. We plot the results of the last block the four stages from left to right. For ViT-B/16, we select the [3rd, 6th, 9th, 12th] blocks, and show the cosine attention map for the cls-token. The clustering maps show that our Context Cluster able to cluster similar contexts together, and tell what model learned visually.
top-1 accuracy. Additionally, our Context Cluster obviously outperforms MLP-based methods. This phenomenon indicates that the performance of our method is not credited to MIP blocks, and context cluster blocks substantially contribute to the visual representation. The performance differences between Context-Clustery and Context-Cluster-TT
negligible, demonstrating the robustness of our Context Cluster to the local region partitioning strategy. We recognize that our results cannot match the SOTA performance (e.g., CoAtNet-O arrives 81.6% accuracy with a comparable number of parameters as ConyTiny), but we emphasize that we are pursuing and proving the viability of a new feature extraction paradigm. We successfully forsake convolution and attention in networks by conceptualizing image as a set of points and naturally applying clustering algorithm for feature extraction. In contrast convolution and attention, our context cluster has excellent generalizability to other domain data and enjoys promising interpretability.
Component Ablation. Table 2 reports the results on ImageNet-IK 1K of eliminating each individual compo- nent in Context-Cluse Cluster-Smann variant. To remove the multi-head design, we utilize one head for each block and set the head dimension number to [16, 32, 96, 128] in the four stages, respectively. When the positional information is removed, the model becomes untrainable since points are disorganized. A similar phenomenon can also be seen from cifar (Krizhevsky et al., 2009) datasets. Performance dropped 3.3% without the text cluster operation. Besides, multi-head design is
Table 2: Component ablation studies of Context-Cluster-SSaall on ImageNet-1k:
position info.
context
multi head
top-1
cluster
acc.
X
X
X
74.2(13.3)
/
X
76.6(10.9)
/
77.5
able boost the result by 0.9%. Combining all the components, we arrive 77.5% top-1 accuracy.
4.2 VISUALIZATION CLUSTERING
To better understand Context Cluster, draw the clustering map in Fig. 4, and we also show the attention map of ViTs and the class activation map (i.e., CAM) (Zhou et al., 2016) of ConvNets. Notice that the three kinds of maps conceptually different and cannot be compared directly. We list the other two (attention and class activation) maps for reference and demonstrate the inner operations in ViTs, ConvNets, and Context Cluster. Detail settings can be found in the caption of Fig. 4.
As the number of points is reduced, the details are merged to form a context cluster. Three observations justify the correctness and effectiveness of our Context Cluster. First, our method clearly clusters the
7

---

Published as a conference paper at ICLR 2023
goose one object context in the last stage and groups the background grass together. A similar phenomenon can also be observed from the previous stages but in more detailed and local regions. Second, our context cluster can even cluster similar contexts in the very early stages (e.g., the first and second stages). Zoom in the details in the red boxes, we can see that the points belonging to the goose's neck are clearly clustered together, suggesting the strong clustering ability of method. Last, we notice that most clusters emphasize the locality, while some (colored in bright green) show the globality lot, as shown in the clustering map of the last stage. This further demonstrates the design philosophy; we encourage similar points to be grouped but make no restriction to the receptive field. Visual clustering map and detailed analysis indicate that our Context Cluster is effective and exhibit promising interpretability. Notably, our method demonstrates promising clustering results in SuperPixel-style when removing the region partition operation. See appendix for more examples.
4.3
3D POINT CLOUD CLASSIFICATION ON SCANORIECTNN
Context Clusters are nat- ural fit for point clouds Qi et al. (2017b); et al. (2022). Therefore we also examine
Table 3: Classification results ScanObjectNN. All results
reported on the most challenging variant (PB T50 RS).
Method
mAco(%)
OA(%)
our method for the task of point cloud classification. We
SpiderCNN (Xu et al., 2018)
69.8
73.7
choose PointMLP (Ma et al.,
DGCNN (Wang et al., 2019)
73.6
78.1
2022) as the foundation for our model because of its per- formance and ease of use. In detail, we only consider one
PointCNN (Li et al., 2018)
75.1
78.5
GBNet (Qiu al., 2021)
77.8
80.5
PointBert (Yu et al., 2022d)
83.1
Point-MAE (Pang et al., 2022)
85.2
head and set the head dimen- sion number to min (5- 32) where indicates the chan- nel number in each layer. We place our Context Clus- ter block before each Resid- ual Point Block in PointMLP.
Point-TnT (Berg et al., 2022)
81.0
83.5
PointNet (Qi et al., 201
63.4
68.2
PointNet++ (Qi et al., 201 7b))
75.4
77.9
BGA-PN++ (Uy et al., 2019)
77.5
80.2
PointMLP (Ma et al., 2022)
83.9
85.4
PointMLP-elite (Ma et al., 2022)
81.8
83.8
The resulting model termed PointMLP-CoC. Note that
PointMLP-CoC (ours)
84.4
86.210.
better settings would result improved performance, but that is not the focus of our study. We report the mean accuracy over all classes (mAcc) and overall accuracy over all examples (OA) in Table 3.
In Table 3, we present the mean accuracy across all classes (mAce) and the overall accuracy across all samples (OA). Experimental results show that our method can substantially increase PointMLP's performance, with improvements in mean accuracy of 0.5% (84.4% vs. 83.9%) and overall accuracy of 0.8% (86.2% vs. 85.4%). Note that the promising gain has only been made by the introduction of one head in the context cluster; with more heads and elaborate settings, performance would be improved. Most importantly, the outcomes show that our approach is highly generalizable different domains, such point clouds. We anticipate that our Context Cluster will operate satisfactorily when applied to more domains with little to no modifications.
4.4 OBJECT DETECTION AND INSTANCE SEGMENTATION ON AS-COCO
Next, we investigate Context Cluster's generalisability to downstream tasks, including object detection and instance segmentation. We conduct our experiments on the MS COCO 2017 benchmark (Lin et al., 2014), which has 118k images for training and 5k images for validation. Following previous work, we train and test model integrating with Mask RCNN (He et al., 2017) for both object detection and instance segmentation tasks. All models are trained with Ix scheduler (12 epochs) and initialized with ImageNet pre-trained weights. For comparison, consider ResNet representative for ConvNets and PVT for ViTs. We report evaluation metric mean Average Precision (mAP) in Table 4.
We notice that owing the differences in image resolution, directly adopting the Context Cluster configuration for ImageNet may not be appropriate for the downstream tasks. For classification task, would have 49 points and 4 centers in local region. The detection and segmentation tasks
8

---

Published as a conference paper at ICLR 2023
Table 4: COCO object detection and instance segmentation results using Mask-RCNN (1x).
Family
Backbone
Params
Apbox
AP
A
Apmask
50
AP
Conv.
ResNet-18 18
31.2M
34.0
54.0
36.7
31.2
51.0
32.7
Attention
PVT-Tinn
32.9M
36.7
59.2
39.3
35.1
56.7
37.3
CoC-Small/4
33.6M
35,9
58.3
38.3
33,8
55.3
35,8
Cluster
CoC-Small/25
33.6M
37.5
60.1
40.0
35.4
57.1
37.9
CoC-Smmll/49
33.6M
37.2
59.8
39.7
34.9
56.7
37.0
would have 1000 points with the same configuration for image size (1280, 800). is obvious that grouping 1000 points into clusters would produce inferior result. To this end, we investigate 25, and 49 centers for a local region, and we refer to the resulting models as Small/4, Small/25, and Small/49, respectively. Results in Table 4 indicate that Context Cluster demonstrates promising generalisability to downstream tasks. Our CoC-Smal1/25 outperforms the ConvNet and ViT baselines on both detection and instance segmentation tasks when properly configured (25 centers region). In line with our expectations, only 4 centers cannot accurately model the large local region, and unnecessary centers cannot further enhance the performance. See appendix C for more results.
local
4.5 SEMANTIC SEGMENTATION ON ADE20K
We examine Context Cluster equipped with semantic FPN (Kirillov et al., 2019) for semantic segmentation task on the ADE20K (Zhou et al., 2017) dataset. For training, validation, and testing, ADE20K includes 20k, 2k, and 3k images, each of which corresponds categories. For a fair comparison, we train all of our models for 80k iterations with a batch size of 16 on four V100 GPUs and adopt the standard data augmentation methods used in PVT (Wang et al., 2021). With an initial learning rate of 2x 10-4, the Adam W optimizer is used to train all of our models. We use a polynomial decay schedule with a power of 0.9 decrease the learning rate.
one of 150 semantic
Experimental results on ADE20K are reported in Ta- ble We show that our Context Clusters clearly out- perform PVT and ResNet using similar number of parameters. The promising improvements can be cred- ited to our novel context cluster operation. Our context cluster is similar to the SuperPixel, which is an over- segmentation technology. When applied for feature extraction, we expect context cluster can over-segment the contexts in intermediate features, and show im- provements for semantic segmentation tasks. Unlike in object detection and semantic segmentation tasks, the centers number shows little influence on the results, More results can be found in appendix $ C.
Table Semantic segmentation perfor- mance of different backbones with Seman- tie FPN on the ADE20K validation set.
Backbone
Params
mloU((%)
ResNet18
15.5M
32.9
PVT--Tiny
17.0M
35.7
CoC-Small/4
17.7M
36.6
CoC-Small/25
17.7M
36.4
CoC-Small/49
17.7M
36.3
5 CONCLUSION
We introduce Context Cluster, a novel feature extraction paradigm for visual representation. Inspired by point cloud analysis and SuperPixel algorithms, view image as set of unorganized points and employ the simplified clustering approach to extract features. In terms of image interpretation and feature extraction operation, Context Cluster is fundamentally distinct from ConvNets and ViTs, and no convolution or attention is involved in our architecture. Instead of chasing SOTA performance, we show that our Context Cluster can achieve comparable or even better results than ConvNet and VIT baselines multiple tasks and domains. Most notably, our method shows promising interpretability and generalization properties. We hope our Context Cluster can be considered as a novel visual representation method in addition to convolution and attention.
As discussed at the end of $ 3, our new perspective and design for visual representation also come with challenges, primarily in the compromise between accuracy and speed. Better strategies worth exploring. Departing from the current framework of detection and segmentation to apply our context cluster philosophy other tasks is also worthwhile direction to pursue.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
In our paper, we strictly follow the ICLR ethical research standards and laws. All datasets we employed publicly available, and all related publications and source codes cited appropriately.
REPRODUCIBILITY STATEMENT
We adhere to ICLR reproducibility standards and ensure the reproducibility of our work in multiple ways, including:
We upload our codes, pre-trained models, and the training log files a GitHub repository, as stated in the abstract. A clear description is presented to ease the reviewing work.
Besides the main results each task, we also upload the codes and checkpoints for all our ablation studies to ensure each experiment is strictly proved.
We clearly present the design of Context Cluster in Section 3.
Detailed framework and more experiments are presented in appendix A and & C.
By doing so, each experiment our submission easy reproduce. We always open-source our research work for each submission to help the community better understand our work.
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Stisstrunk. Slic superpixels compared to state-of-the-art superpixel methods. TPAMI, 34(11):2274-2282. 2012.
Axel Berg, Magnus Oskarsson, and Mark O'Connor. Points to patches: Enabling the use of self- attention for 3d shape recognition. arXiv preprint arXiv:2204.0 03957, 2022.
Christopher Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume Springer, 2006.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020.
Kenneth R Castleman. Digital image processing. Prentice Hall Press, 1996.
Shoufa Chen, Enze Xie, Chongjian GE, Runjian Chen, Ding Liang, and Ping Luo. CycleMLP: MI.Plike architecture for dense prediction. In ICLR, 2022a.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengehen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. CVPR, 2022b.
Zihang Dai, Hanxiao Liu, Quoe Le, and Mingxing Tan. attention for all data sizes. NeurIPS, 2021.
Coatnet: Marrying convolution and
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. ECCV, 2022a.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In CVPR, 2022b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2020.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv: 1020100004444 2021.
10

---

Published as a conference paper at ICLR 2023
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. CVPR, 2022a.
Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. arXiv preprint urXiv:2202.09741, 2022b.
Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: image is worth graph of nodes. NeurIPS, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable mip-like architecture for visual recognition. IEEE TPAMI, 2022.
Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu. Shuffle transformer: Rethinking spatial shuffle for vision transformer. arXiv preprint arXiv:2 2021.
Zixuan Huang and Yin Li. Interpretable and accurate fine- fine-grained recognition via region grouping. In CVPR, 2020.
Varun Jampani, Deqing Sun, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In ECCV, 2018.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. NeurIPS, 2018.
Zhenggin Li and Jiansheng Chen. Superpixel segmentation using linear spectral clustering. In CVPR, 2015.
Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architecture for vision. In ICLR, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV. Springer, 2014.
Hanxiao Liu, Zihang Dai, David So, and Quoe Le. Pay attention to mlps. NeurIPS, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In CVPR, 2022.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Tao Lu, Chunxu Liu, Youxin Chen, Gangshan Wu, and Limin Wang. App-net: Auxiliary-point--on push and pull operations for efficient point cloud classification. arXiv preprint wrxiv:2205.0084. 2022.
Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: simple residual mlp framework. ICLR, 2022.

---

Published as a conference paper at ICLR 2023
Yatian Pang. Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self--uuerrvied learning. ECCV, 2022.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. +ointnet++: Deep hicrarchical feature learning point sets in metric space. NeurIPS, 2017b.
Can Qin, Maoguo Gong, Yue Wu, Dayong Tian, and Puzhao Zhang. Efficient scene labeling via sparse annotations. In AAAIW, 2018.
Shi Qiu, Saeed Anwar, and Nick Barnes. Geometric back-projection network for point cloud classification. IEEE Transactions on Multimedia, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Xiaofeng Ren and Jitendra Malik. Learning a classification model for segmentation. In ICCV, 2003.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. NeurIPS, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. ICML, 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedfor- ward networks for image classification with data-efficient training. IEEE TPAMI, 2022.
Asher Trockman et al. Patches are all you need? arXiv preprint arXiv:2201.0979: 2022.
Mikaela Angelina Uy, Shang-HHie Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. ICCV, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Beng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. /CCV, 2021.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. Acm Transactions On Graphics (tog), 38,5):1-2, 2019.
Ross Wightman.
Pytorch image models. pytorch- imaaimmmamoss models, 2019.
httts:://githhb..cmmmmmmg
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. In NewrlPS 202/ Workshop on ImageNet, 2021. URL UtL.ittps://ooperrvv.e peet/forummiddNGGt)nWLLL
Lemeng Wu, Xingchao Liu, and Qiang Liu. Centroid transformers: Learning to abstract with attention. arXiv preprint 0rXiv:2102.08666 2021.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 2021.
12

---

Published as a conference paper at ICLR 2023
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon. Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In ECCV, 2018.
Fengting Yang, Qian Sun, Hailin Jin, and Zihan Zhou. Superpixel segmentation with fully convolu- tional networks. In CVPR, 2020.
Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chies Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022a.
Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hatwig Adam, Alan Yuille, and Liang-
Liang-Chich Chich Chen. k-means mask transformer. ECCV, 2022b.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. CVPR, 2022c.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu, Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In CVPR, 2022d.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021a.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432. 2021b.
Bolei Zhou, Aditya Khosla, Agata Lapedriza. Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
13

---

Published as a conference paper at ICLR 2023
A MODEL CONFIGURATIONS
We first introduce the detailed configurations of our Context clusters in Table 6. Point reducer block consistent with image downsize blocks, like in PVT and ConvNeXt. In the context of points, select k_ neighbors of the nearest points for a proposed anchor, and fuse all the points using FC layer. We reduce the number of points by factor of downsample__t The key contribution in our work is context cluster blocks. We first evenly split the whole set of points into regions local regions in the space. In each local region, we propose Local_center dimension of each head to heads and head dim, respectively, in our context cluster operation. The channel number is expanded by factor of mlp_7 (i.e.. mlp_r's x dim) and then reduced to dim in the MLP block. The Context Cluster block would be repeated by several times in each stage. We mark all variables in blue for easy understanding. For the Context-Cluster Tiş variation, shares the same network structure as partition and local center numbers differently. particular, the number region partitions set to [49, 49, 1, I] and the number of centers in each local region is set to [16, 4, 49, 16] for the four stages.
for clustering. We set the heads number and
with the exception that we configure the region
Table 6: Detailed configurations for our Context Cluster. We initialize three variants, CoC-Time Tiny, CoC-Small, and CoC-Medium, with different model capacities.
Stage
Points
Block
CoC-Tiny
CoC-Small k neighbors downsample_ dim
CoC-Medium k k_neighoors downsample dim 64
& neighbors downsample
Point
50176
S1
Reducer
dim 32
regions 64
regions 64 pecul_centess heads
regions 64
_ocu__cettrr-
lncul_centers heads head dim 32
Context
heads 4
3136
Cluster Blocks
head dim 24
x
head dim 32
x
x
mip_r.
mlp_r.
mlp_r. dim 64
dim 32
dim 64
Reeighbors
k_neighbors
k_neighbrrs downsample__ dim 128
Point Reducer
3136
downsample_r dim 64
downsample_ 4
$2
dim = 128
regions 16
regions = 16
regions 16
local centers
local ceal_ceetersss
cecal_centers
Context
heads 4
heads = 4
heads =
784
Cluster Blocks
head. dim 24
head dim 32
x2
head dim 32
x
mlp_r.
mlp_r. dim = 128
mip_r. dim = 128
dim = 64
k neighbors =
k neighhbors = downsumple_r
k neighhorrs = downsumple_r
Point Reducer
784
downsample_rs
dim = 196
dim = 320
dim = 320
regions =
regions =
regions =
locd__enterss centers 4
local
local_centers centers 4
local
Jucal_center
Context
heads = 8
heads = 8
heads = 12
196
Cluster
head, dim 24
head dim 32
head dim
Blocks
mip_r.
mlp_r.
mlp_r.
dim 196
dim 320
dim = 320
k neighbors =
k neighbors =
k neighbors =
Point Reducer.
196
downsample y
downsample. r =
downsample r
$4
dim = 320
dim = 512
dim = 512
regions =
regions =
regions =
local local_centers centers
local centers
local_centers heads head dim
Context
heads =
heads =
49
Cluster
head dim
x2
head dim
x2
X4
Blocks
mip_r =
mip =
mlp = dim 512
dim 320
dim 512
B DETAIL EXPLANATIONS
One may be confused about how to specify the anchors in our point reducer block and the centers in context cluster block. We provide illustrative and thorough explanations of them in this section.
For both anchor and center, they are generated evenly in the space. In order to better illustrate this, plot organized image points in Fig. 5.
14

---

Published as a conference paper at ICLR 2023
FC
(a) Illustration anchors points reduction.
(b) Demo of centers CoC.
Figure 5: Detail explanations the anchors in points reducer block and centers for context cluster block. The image points (represented in patch) are organized for easy understanding and illustration. In sense, the anchors are for reducing point numbers, and centers are used for clustering. Both of them are evenly distributed in design. On the left, we evenly propose 4 anchors (marked in blue dot) with 4 neighbors for each anchor. On the right, we evenly sample 9 centers (marked in red block), hence leading to 9 irregular clusters. The center feature value is achieved by averaging its k neighbors. In this figure, we show the neighbors the big blue circle for the second center.
On the left, we display 16 points with 4 proposed anchors for point reduction, each of which takes its closest 4 neighbors into account. All neighbors concatenated along the channel dimension, and FC layer is used to lower the dimensional number and fuse the information. After reducing the number of points, we arrive at new set of points with the
number of proposed anchors.
On the right, we show centers (red blocks) generated from the set of image points and corresponding 9 clusters. The feature of generated centers will be given by averaging the k neighbors (for the second center, we average the points in the big blue circle).
The number of neighbors can be of any value. We set it to 4 9 due to three reasons. First, follow the design of ConvNets and pyramid ViTs to ensure the set of points can be reorganized to rectangular feature map. Second, this strategy eases our coding by employing convolution or pooling operations (which are equivalent to our description) and avoids heavy indices searching work. Last, rectangular feature map is required by most detection and segmentation methods.
C MORE EXPERIMENTS
We conduct more experiments to validate the effectiveness of our Context Cluster.
More results for Segmentation. We port the CoC-Small with Semantic FPN's performance on the ADE20K validation set in Table 7 under the same conditions as previous. We notice that performance increases modestly as more centers are added local region, but the compu- tational cost rises a lot. With 4 V100 GPUs, training CoC-Small/4 would take 9 hours. As comparison, it would take 1l hours for CoC- MoC-Small/25 and 14 hours for CoC-Small/49, The cost of computing increases linearly with the complexity of the computations in our context cluster blocks.
Table 7: Semantic segmentation results of different back- bones with Semantic-FPP on the ADE20K validation set.
Family
Backbone
Params
mloU(%)
Conv.
ResNet50
28.5M
36.7
Atten. PVT-Small
28.2M
39.8
Cluster
CoC-Medium/4
25.2M
40.2
Cluster CoC-Medium/25
25.2M
40.6
Cluster
CoC-Medium/49
25.2M
40.8
More results for Detection. We also conduct more results for object detection and instance segmen- tation task. Besides the experiments reported in Table 4, also conduct experiments based on the pre-tree trained CoC-Medium. Results are reported in Table In line with CoC- -mall, CoC- MedMMu shows comparable performance in ResNet50 and PVT-Small.
15

---

Published as a conference paper at ICLR 2023
Table 8: COCO object detection and instance segmentation results using Mask-RCNN (1x).
Family
Backbone
Params
Aphox
AP
AP
75
Apmask
APmask
AP
Conv.
ResNet-50
44.2M
38.0
58.6
41.4
34.4
55.1
36.7
Atten
PVT-Small
44.1M
40.4
62.9
43.8
37.8
60.1
40.3
Cluster
CoC-Medium/4
42.1M
38.6
61.1
41.5
36.1
58.2
38.0
Cluster
CoC-Medium/2. 25
42. IM
40.1
62.8
43.6
37.4
59.9
40.0
Cluster
CoC-Medium/49
42.1M
40.6
63.3
43.9
37.6
60.1
39.9
Stage 3
Stage 4
Figure 6: We visualize the clustering maps of all heads in the last block of stage3 and stage4 in our Context Cluster-Medium. While context cluster operation shows preference for the locality as we expected, notice that it also favors vertical horizontal directions.
Clustering map heads. To have a better understanding of our context clustering operation, we show the clustering maps of all heads in the last block of stage3 and stage4 in our Context Cluster- Meditu. As we expected, results in Fig. indicate that our method is able to cluster semantically similar contexts together and exhibits decent locality. An interesting observation is that the context cluster operation also tends to cluster contexts along vertical or horizontal directions. Similar phenomena also be observed in other images and model variants.
Table 9: Ablation study on region partition operation. Based on CoC-Tiny, we eliminated all region partition operations. To make the model trainable, the cluster numbers were changed 16 in the first two stages and to 4 in the last two stages. We train models on 8xA100 GPUs with a batch size of 32 on each GPU, and report the training memory demand of GPU. We test our model on one GPU.
Model
Partition?
Parameters
FLOPs
Infer. Memory
Train Memory Top-1 71.8%
5.3M
1.0G
1.58G
23.39G
CoC-Tiny
5.3M
1.0G
2.19G
34.76G
72.7%
Ablation region partition operation. While sacrificing the ability to model global interactions, region partition would introduce useful inductive biases like locality. We remove CoC-Tiny region partition operations to where the performance is coming from and report the results in Table 9, Experimental results indicate that without the region partition operation, the performance increased by 0.9% ImageNet, indicating the effectiveness of our Context Cluster methodology. However, the training time and the memory demands are significantly increased (as discussed in $ 3.3). Despite the fact that we agree that the region partition operation does introduce useful inductive bias, the results show that global interaction is also constructive to the success of our Context Cluster (as well as in other designs like ConvNets and ViTs).
Interestingly, we find that our Context Cluster even offers a meaningful clustering result in the early stages without the restriction locality from region partition, as shown in Fig.
16

---

Published as a conference paper at ICLR 2023
Figure 7: The clustering results of the last context cluster block in the first CoC-Tiny stage (without region partition). Without region partition, Our Context Cluster astonishingly displays "SuperPixel"" like clustering results, even the early stage. we pick the most intriguing one out of the four heads.
Input
Stage 1
Stage 2
Stage
Stage
Figure 9: Clustering results of CoC-Tiny without region partition operations.
Also, by eliminating the limitation of lo- cality, our Context Cluster presents more promising clustering results, across all stages, as shown in Fig. 9. This phe- nomenon supports our motivation and in- dicates gratifying interpretability (which not easy for convolution nor attention). Right figure shows example for the clustering results all four heads in the first stage, different clustering patterns learned (with noises).
Input
Clustering results in 4 heads (16 clusters)
Figure 8: sample of all groups' clustering results.
17

---

Published as a conference paper at ICLR 2023
(a) Discrete pixels
Masked image
(c) Irregular image
(d) RGB-D image
Figure 10: Four examples of image formats. Remember that there are no pixels in the white area.
In summary, these ablation studies region partition operation indicate great potential of our Context Cluster. It would fascinating challenge to figure out how to eliminate the region partition operation without introducing prohibitive computation and memory requirements.
Ablation on iteratively updating centers. As mentioned in the discussion section, we simplified the clustering algorithm for computational efficiency by fixing the centers without updating. We also conduct simple experiment to confirm the effectiveness of dynamic centers. By updating centers for two times, we attain 71.85 .85%% accuracy on ImageNet-1 IK based on CoC-Tiny (71. 77,88%), showing negligible improvements (0.02%).
D GENERALIZATION OUTLOOK
The clustering algorithm is general method not limited to particular input format. Previously, we validated the generalization ability Context Cluster on both image and point clouds. Here, we further outlook the generalization ability to different image formats as shown in Fig. 10.
We starts from the discussion on the discrete pixels. Because the clustering algorithm, our Context Cluster actually has significant advantage over ConvNets and ViTs in processing discrete images, In other words, our Context Cluster does not require an image pixel to be in a continuous space. In detail, given image consisting of discrete pixels, extract features like regular images but change the center proposal method. In our submission, we describe the center proposal method by evenly proposing centers in space, and the center feature is computed by averaging its nearest points (which can be easily implemented by pooling). For the discrete pixels, we can consider the Farthest Point Sampling (FPS) method (Qi al., 2017b) from point cloud processing. Notice that our method is inspired by point cloud methods, and an image with discrete pixels is naturally point cloud set with RGB information. In addition FPS, other discrete sampling techniques can also be investigated to propose centers for discrete pixels, including random sampling, grid sampling, etc.
In addition discrete pixels, our context cluster can also be used with variety of additional image formats. For masked images and irregular images, traditional ConvNets or ViT's require the image to be filled with white pixels. Differently, by conceptualizing an image as set of points, we escape this step. We interpret masked or irregular images as discrete points and handle them previously described. Thanks to the clustering algorithm, our Context Cluster exhibits great generalization ability to various image formats.
18


Reviews:
[{"review_id": "VjyZVxIM58", "reviewer": "Reviewer_QuSd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an alternative approach to image content representation called context clusters. These are compared as alternatives to convolutional networks and vision transformer approaches. \n\nThe work is well presented and well written. They argue for their approach in a scientific manner and set it in place for future use in other vision techniques and applications. ", "summary_of_the_review": "I enjoyed this paper. The authors write with enthusiasm and back up their claim that context clusters are worth further investigation through scientific means. \n", "strength_and_weaknesses": "The paper is well written (although requires an editorial read)  and the experiments provided are detailed. \n\nThe methodology could be more clearly presented though. It should be immediately understandable but the reader has to re-read to follow properly. As a case, I would like to be explicitly told how the context clusters are believed to capture the content better. \n\nThe citations are incorrectly used throughout. \nThe use of italics and bold in paragraphs seems unprofessional and I would prefer it not used. \n\nWhy are some figures bold in Table 1? It is not clear what this means.\n\nThe authors state \"See appendix for more experiments.\" It would read well if a short explanation of what is in the appendix is provided here.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RPD2fsBpcB", "reviewer": "Reviewer_cH7t", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is a pioneering work considering SuperPixel-like idea and point cloud modeling for 2D general visual backbone.\nThe main contributions are:\n- Proposing a model called CoC to explore the possibility of point-cloud-clustering modeling for 2D image processing, and initially verifiing its effectiveness.\n- Providing fresh insights into the design of general vision model, which are not only well suited to cope with dense prediction tasks (detection and segmentation), but also facilitate the combination with 3D point cloud data in order to promise a very general cross-modal vision model.", "summary_of_the_review": "This work is innovative, insightful, written with ample detail, and highly reproducible.\nI believe it deserves to be published and discussed in ICLR.", "strength_and_weaknesses": "**[strength]**\n\n- This is a very insightful work towards general visual backbone design. The fusion of SuperPixel idea as well as PointNet structure is natural and elegant. I believe this work reveals the potential of point cloud modeling for 2D image processing, which is certainly a great contribution to the community and is worth exploring.\n- I would say that CoC is modeled in a way that fits well with the nature of visual signal processing, as it has the inductive biases that are as useful as those of convolutional networks (such as locality and multi-scale).\n- The interpretability is also as good as convolution (CoC embodies spatial selectivity, while convolution exemplifies the idea of pattern matching). Besides, CoC acts like the deformable convolution and is clearly more flexible and general than ordinary convolution that operates on regular grids.\n\n\n**[weakness]**\n- **Potential representation weakness**. As an architecture similar to ConvNet and PointNet, CoC could be significantly weaker than Transformers in terms of global modeling capabilities (e.g., interacting two spatially distant objects). What are the authors' insights on this problem?\n- **Related work**. I find the first paragraph in Related Work is way more relevant to CoC than the latter two. It would be much appreciated if the authors could devote more space to the first paragraph, e.g., introduce [R1,R2,R3,R4] in more detail and discuss them with this paper in terms of method philosophy.\n- **Figure clarity**. I recommend using more contrasting colors when visualizing. Currently these two figures (Fig. 4 and 6) do not illustrate the clustering effect of CoC models very well.\n- **Font formats**. The spacing between characters in the headings is too small, making them look cramped. Also, there seems to be too much use of bolding or italics in the body text. I would suggest using a more concise format.\n\n------------\n\n**[open questions]**\n\nI have more questions as follows and would like to discuss them further with the authors:\n\n- From Fig. 5(b), the CoC block has similarities with DAT [R5]. Can the authors further analyze or compare them?\n- I don't think it's necessarily a disadvantage that the methods [R3, R4] can only be used for specific tasks like segmentation; in other words, a point-based (or SuperPixel-based) modeling might be inherently more useful for dense prediction tasks (detection, segmentation, etc.) than classification. The authors mentioned that they expected to see more work that seamlessly integrates CoC and DETR in the future. This is reasonable, but I still expect the authors to come up with some prototypes of CoC for dense prediction :).\n\n----------\n[R1] Varun Jampani, Deqing Sun, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In ECCV, 2018.\n\n[R2] Zixuan Huang and Yin Li. Interpretable and accurate fine-grained recognition via region grouping. In CVPR, 2020.\n\n[R3] Fengting Yang, Qian Sun, Hailin Jin, and Zihan Zhou. Superpixel segmentation with fully convolutional networks. In CVPR, 2020.\n\n[R4] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022a.\n\n[R5] Xia, Zhuofan, et al. Vision transformer with deformable attention. In CVPR, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ew6PM-Q20RC", "reviewer": "Reviewer_1Gqf", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a different formulation for vision tasks, i.e. considering image as a set of unorganized points instead of organized rectangular shape or patches sequence. Each point has raw feature and positional information and they are grouped with clustering algorithm. It shows comparable or sometimes better result as CNN or transformer. The code is also released. ", "summary_of_the_review": "To summary, I think this paper is very interesting and should be discussed at the main conference. It might be a good alternative architecture for vision tasks. The results are impressive and well-presented. I recommend acceptance.\n", "strength_and_weaknesses": "Strength:\n+ The idea is very interesting and this new architecture could be an alternative architecture for CNN, ViT. It provides a new perspective on how the information of an image is represented.\n+ The idea should be of great interest for the representation learning and computer vision community.\n+ The computation is a hard problem if using clustering on 224x224 image, while the authors use a smart way to address it, similar to Swin Transformer. \n+ The performance is impressive as compared with recent popular architectures.\n+ The propose method also generalizes well on downstream tasks like object detection and segmentation.\n+ The paper is well organized and the writing is good.\n\nWeakness:\n- The paper claims to bridge the gap between representation of image and point cloud. It seems to work well on PointNet. But it is kind of hard to understand. The authors use fixed centers (Fig.5) to save computation, how does this work if the discrete pixels are sparsely sampled from one image? It might be helpful if the authors can provide more explanation about this. It would also be interesting to see the performance of dynamic centers? For example, maybe use only 2 iteration for center updates?\n- The proposed method may have good interpretation ability, but not better than CNN or ViT. The figure 4 might need more explanation since the framework is very different from other architectures. The method seems to cluster similar contexts in early stage, but it is not easy for reader to get this insight from figure 4. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "42tbo1k57D", "reviewer": "Reviewer_Mp7d", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new view of images that considers each image as a set of points (the pixels) and uses a clustering algorithm to extract the features from it. The goal is to investigate the way to utilize this new form of visual representations and evaluation the performance that could be achieved. To this end, the paper introduces a novel backbone network that includes the proposed Context Clusters and evaluation this model on several vision tasks as well as a point cloud data application.", "summary_of_the_review": "This paper introduces a new form of image representation that considers each image as a set of points and proposes a clustering-based architecture for feature extraction. Both the idea of “image as set of points” and the proposed architecture are interesting and novel. The experiment result also shows that the method achieves comparable performance to ConvNets and ViTs. A small concern is that the role of the region partition mechanism is unclear since good performance could actually be attributed to this design.\n\n---\n\n## Post-rebuttal updates:\nI appreciate the detailed feedback and the extra experiments. It is very interesting to see that: \n\n1. Removing the partition operation actually improves the performance. This demonstrates the efficiency of the original design.\n2. The clustering behavior reveals superpixel-based grouping. This is anticipated, but demonstrating that it is actually working in this way in the paper is much appreciated.\n3. Slightly worse than Swin in performance. The answers to my questions and the performance difference are also acceptable for a novel model design.\n\nSince most of my concerns have been addressed, I decided to raise the score to 8. This is a good paper and worth acceptance", "strength_and_weaknesses": "Strengths:\n\n- To the best of the reviewer’s knowledge, the topic of considering an image as a set of points and extracting features from it for vision tasks is original and very interesting.\n- The proposed method that uses the clustering algorithm as the basic build block is novel and of significance to the community.\n- The evaluation plan of the paper is comprehensive. It provides experiments on standard vision tasks like image classification and object detection/segmentation and applications for point cloud inputs like object classification.\n- The evaluation results show that the method provides improvements on various tasks over the CNN and ViT baselines  (though not outperforming the state-of-the-art approach).\n\nWeaknesses:\n\n- By using the region partition mechanism, the set of points is no longer unorganized but becomes structured based on their locality. Additional experiments are required to clarify the role of the region partition.\n    - Before applying the context clusters operation, the region partition operation, which is similar to the shifting windows in Swin [1], is introduced to reduce the computational cost. The authors seem to imply that the region partition trades off performance for speed. However, the locality introduced by the region partition could also bring useful inductive bias for the encoder. Therefore, additional experiments are required to answer the following questions:\n        1. If the region partition operation is removed in the clustering process, could the model achieve similar or better performance? What would the clustering map be like in this case?\n        2. It would be nice to introduce Swin as one baseline to investigate this problem.\n\n[1] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. *arXiv*. https://doi.org/10.48550/arXiv.2103.14030", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%