SPFormer: Enhancing Vision Transformer with Superpixel Representation
Authors: Jieru Mei
Liang-Chieh Chen
Alan Yuille
Cihang Xie
Venue: Accepted by TMLR ()

Abstract: This work introduces SPFormer, a novel Vision Transformer architecture enhanced by superpixel representation. Addressing the limitations of traditional Vision Transformers' fixed-size, non-adaptive patch partitioning, SPFormer divides the input image into irregular, semantically coherent regions (i.e., superpixels), effectively capturing intricate details. Notably, this is also applicable to intermediate features, and our whole model supports end-to-end training, empirically yielding superior performance across multiple benchmarks. For example, on the challenging ImageNet benchmark, SPFormer outperforms DeiT by 1.4% at the tiny-model size and by 1.1% at the small-model size. Moreover, a standout feature of SPFormer is its inherent explainability — the superpixel structure offers a window into the model's internal processes, providing valuable insights that enhance the model's interpretability and stronger robustness against challenging scenarios like image rotations and occlusions.

Published in Transactions on Machine Learning Research (01/2025)
SPFormer: Enhancing Vision Transformer with Superpixel Representation
Jieru Mei
meijieru@gmaii.com com
Deportment of Computer Science Johns Hopkins University
Liang- Liang-Chich Chen
aquariusjay@@maailccom com
Byledonce
Alan Yuille
alon.
Department of Computer Science Johns Hopkins University
Cihang Xie
cilangpic306@@mmaill.co com
Deportment of Computer Science and Engineering University of California, Santa Cruz
Reviewed OpenReview: https: // openreview. net/ str/formm? id=na18/VgSuy
https://openrevvieu
Abstract
This work introduces SPFormer, a novel Vision Transformer architecture enhanced by super- pixel representation. Addressing the limitations of traditional Vision Transformers' fixed-size, non-adaptive patch partitioning, SPFormer divides the input image into irregular, semanti- cally coherent regions (i.e., superpixels), effectively capturing intricate details. Notably, this also applicable to intermediate features and our whole model supports end-to-end training, empirically yielding superior performance across multiple benchmarks. For example, the challenging ImageNet benchmark, SPFormer outperforms DeiT by 1.4% at the tiny-model size and by 1.1% al the small-model size. Moreover, standout feature of SPFormer is ils inherent explainability - the superpixel structure offers a window into the model's internal processes, providing valuable insights that enhance the model's interpretability and stronger robustness against challenging scenarios like image rotations and occlusions.
1 Introduction
Over the past decade, the vision community has witnessed remarkable evolution in visual recognition systems, from the resurgence of Convolutional Neural Networks (CNNs) in 2012 (Krizhevsky et al. 2012) to the cutting-edge innovation of Vision Transformers (ViTs) in 2020 (Dosovitskiy et al. 2021). This progression has instigated a significant shift in the underlying methodology for feature representation learning, transitioning from pixel-based (for CNNs) patch-based (for ViTs).
Traditionally, pixel-based representations organize an image as regular grid, allowing CNNs (He al., 2016 Tan & Le 2019; Mehta & Rastegari 2022 Sandler et al. 2018) to extract local, detailed features through sliding window operations. Despite the inductive bias inherent in CNNs, like translation equivariance, aiding their success in effectively learning visual representations. these networks face a challenge in capturing global-range information, typically necessitating the stacking of multiple convolutional operations and/or additional operations (Li et al. 2020 Chen et al. 2018 to enlarge their receptive fields.
regard an image a sequence of patches. These patch- based representations, usually of much lower resolution compared to their pixel-based counterparts, enable
In contrast, ViTs (Dosovitskiy al. 2021

---

Published in Transactions on Machine Learning Research (01/2025)
ImageNet
COCO
Visualization of learned superpixels with our SPFormer trained on ImageNet with category labels only. For each row, show input image; visualization of 196, 49, and 16 superpixels. The learned superpixel aligns well with the object boundaries even with 16 superpixels. The last row shows results from COCO image (not trained), demonstrating SPFormer's zero-shot ability.
Figure 1:
ranbel-aagg self-attention operations in a computationally efficient manner. While the attention mechanism successfully captures global interactions, it does so at the expense of losing local details, like object boundaries. Morcover, the low resolution patch-based based representations poses challenges adaptation for high-resolution dense prediction tasks such segmentation and detection, which require both local detail preservation and global context information.
This dichotomy raises an intriguing question: can we derive benefits from both preserved local details and effective long-range relationship capture? response, we explore superpixel-based solutions, which have been employed extensively computer vision prior to the deep learning era (Zhu & Yuille
1996 Shi & Malik
2000 Martin et al.] 2001]] Malik et al.] 2001]: Borenstein & Ullman| 2002 Tu & Zhu 2002: Ren & Malik
2003]. These solutions provide locally coherent structures and reduce computational overhead compared
to pixel-wise processing. Specifically, adaptíve to the input, superpixels partition image into irregular regions, with each region grouping pixels with similar semantics. This approach allows for small number of superpixels, making it feasible to model global interactions through self-attention.
To this end, we introduce a novel transition of ViT architecture from patch representation to superpixel representation, through our newly developed Superpixel Cross Attention (SCA). The resulting architecture, Superpixel Transformer (SPFormer), adeptly marries local detail preservation with global-range self-attention, enabling end-to-end trainability. In comparison to standard VIT architectures, SPFormer demonstrates strong enhancements across various tasks. For instance, it achieves impressive gains on the challenging ImageNet benchmark, such as 1.4% for DeïT-T and 1.1% for DeiT-S. Notably, the superpixel representation in SPFormer aligns seamlessly with semantic boundaries, even in unseen data. Moreover, the interpretability provided by our superpixel representation facilitates understanding of the model's 's decision-making process and enhances the robustness against rotations and occlusions. These findings highlight the potential of superpixel- based approaches advancing the field, and, hopefully, could inspire future research beyond traditional pixel and patch-based paradigms in visual representation learning.
2
Related Work
Pixel Representation. CNNs LeCun al. 1998
Krizhevsky et al. 2012
Simonyan & Zisserman 2015
Szegedy et. al. 2015 Toffe & Sxegedy 2015 He et al.
2016 Tan & Le 2019 Liu et al. 2022) process
2

---

Published in Transactions on Machine Learning Research (01/2025)
image grid of pixels in sliding-window window manner. CNN has been the dominant choice since the advent of AlexNet (Krizhevsky et al.) 2012], benefiting from several design choices, such as translation equivariance and the hierarchical structure to extract multiscale features. However, it requires stacking of several convolution operations to capture long-range information (Simonyan & Zisserman, 2015 He et al. (016), and it could not easily capture global-range information, as the self-allention operation (Vaswani et al. 2017
Patch Representation. The self-attention mechanism [Bahdanau al.) 2015 of Transformer architec- tures (Vaswani et al., 2017) effectively captures long- range information. However, its computation cost is quadratic to the number of input tokens. ViTs (Dosovitskiy et al.) 2021) alleviate the problem by tokenizing (or patchifying) the input image with a sequence of patches (e.g., patch size 16 x 16). Patch representa-
tion (Dosovitskiy et al. 2021) unleashes the power of Transformer urchitectures (Vaswani el al. 2017 7 in
computer vision, significantly impacting multiple visual recognition tasks (Russakovsky et al. 2015
C Carion
ct al. 2020 Zhu ct al.]
2021 Wang ct al.
2021a Touvron et al.) 2021a Radford ct al.) 2021
Bao ct al.
2022
He el al.
2022 Yu al.
2022c 2023). Due the lack of built-in inductive biases in CNNs, learning
with VITs requires special training enhancements, e.g., large-scale datasets (Sun et al.] 2017 better training
recipes (Touvron et al.
2021a Steiner et al. 2021), or architectural designs (Liu et al. 2021] Wang et al.
2021b). To mitigate the issue, some work exploits convolutions (LeCun et al.
1998; [Sandler et al. 2018 2021
to tokenize images, resulting in hybrid CNN- Transformer architectures (Wu et al. 2021:
Yuan et al.
2022 Yang et al., 2023). Unlike those works that simply gather knowlodge from existing CNNs and ViTs, we explore a diffcrent
Dai et al.] 2021 Xiao et al.) [022]: Mehta & Rastegari 2022 Guo et al., 2022 Tu et al.
superpixel representation in ViTs.
Superpixel Representation. Before the deep learning era, superpixel is one of the most popular rep- resentations in computer vision (Zhu & Yuille 1996 Shi & Malik 2000 Martin et al. 2001 Malik et al.,
2001
Borenstein & Ulman 2002 Tu & Zhu 2002 Ren & Malik 2003). i Ren and Malik (Ren & Malik,
2003
preprocess images with superpixels that are locally coherent, preserving the structure necessary for
the following recognition lasks. It also significantly reduces the computation overhead, compared to the
wixel-wise processing. The superpixel clustering methods include graph-based approaches (Shi & Malik
2000
Felzenszwalb & Huttenlocher 2004), mcan-shift (Comaniciu & Meer 2002 Vedaldi & Soatto 2008)
or k-means clustering (Lloyd] 1982 Achanta et al. 2012). Thanks to its effective representation, recently
some works attempt to incorporate clustering methods into deep learning frameworks (Jampani et al. 2018
Yang et al. 2020
Yu et al. 2022a Zhang et al. 2022 Yu et al. 2022b Ma et al. 2023 Huang et al., 2023 Zhu et al.) 2023). For example, SSN (Jampani et al. 2018)
Locatello et al.] 2020 Xu et al.] 2022
integrates the dillerentiable SLIC (Achanta el al. 2012) lo CNNs, allowing end-to-end training. Yu et al.
(2022a] b) regard object queries (Carion et al. 2020 Wang et al., (2211a) as cluster centers in Transformer decoders |Vaswani et al., 2017). SVIT (Huang al. 2023) clusters the tokens to form the super tokens, where the clustering process has gradient passed through] Consequently, their network is not aware of the clustering process and could not recover from the clustering error. CoCs (Ma et al. 2023) groups pixels into clusters, while aggregating features within each cluster by regarding the image as a set of points with coordinates concatenated. In contrast, our proposed method groups pixels into superpixels, and models their global relationship via self-attention. Furthermore, during clustering, CoCs uses a Swin-style window partition (Liu et al., 2021) that introduces visual artifacts, especially around the window boundarics.
3 Method
We introduce SPFormer, a novel image processing framework that integrates superpixel-based feature representation with a SCA mechanism. This approach effectively addresses the limitations of traditional pixel and patch-based methods by enhancing both computational efficiency and detail preservation. We begin with detailed discussion of our advanced superpixel representation (Sec. 3. followed by an explanation of the SCA mechanism in Sec. 3.2 which refines this representation by blending local details with global contextual information efficiently. Finally, we elaborate on the integration of these advancements into the SPFormer
architecture in Sec. 3.3
I From official code:
3

---

Published in Transactions on Machine Learning Research (01/2025)
Illustration the relationships between a pixel i and its neighboring superpixels N₁, well as the local window W, of superpixel p. In this framework, cach pixel is associated with adjacent superpixels within neighborhood radius of r. For the purposes of this study, we set r= 3. Conversely, each superpixel incorporates a set of pixels that fall within its local window Wy. These pixels are selected based on their proximity and linkage to the superpixel p, as determined by their membership in No
Figure 2:
3.1 Superpixel Representation: Bridging Pixel and Patch Approaches
In the evolving landscape of feature representation, the transition from pixel to patch-based methods in ViTs has opened new avenues for image processing. However, each method has inherent limitations, inspiring our exploration of more adaptive and efficient representation: superpixels.
Pixel Representation. Traditional pixel representation conceptualizes an image I as a grid of high- resolution pixels, expressed as I Rexhxw, Predominantly employed in CNN-based methods, this approach suffers from restricted contextual integration due to the limited receptive fields of the convolution operations. Although self-attention mechanisms could theoretically mitigate this limitation, their implementation is computationally prohibitive at this scale due the quadratic dependencies on the number of pixels.
Patch Representation. Contrastingly, ViIs adopt a patch- representation with reduced resolution, denoted by P E xxxxxXX>>........... This simplificatio reduces computational complexity and facilitates the application of self-attention mechanisms albeit at loss of finer detail and contextual richness due to the coarse granularity.
Superpixel Representation. Bridging the benefits of both pixel and patch-based approaches, our model employs superpixel representation, encompassing fcatures S E Rex* Xxw and an association matrix A [Raxhxw The employment of superpixels significantly reduces the number of feature components; for instance, in this study, we utilize a stride of h/s, = w/sw = 4, effectively leading to a reduction by a factor of 16 relative to the original pixel count. This association matrix establishes a competitive relationship among pixels and their corresponding superpixels, aspect that will be claborated in ensuing detailed discussions on the structured components of our approach.
1. Neighboring Superpirels (W)): Each pixel i is connected to its neighboring superpixels, comprising the nearest superpixel and those within its Moore Neighborhood, as shown in Fig. 2 Specifically, the neighborhood radius is r, resulting a total of 22 adjacent superpixels. In the context. of this study, we set r 3, thereby involving superpixels around cach pixel. This configuration not only facilitates localizod information sharing but also enhances the granularity of the feature representation. Moreover, these superpixels engage in competitive interaction to establish dominance over the association with the pixel, which dynamically affects the feature integration process.
2. Superpizel's Local Window (W,): For each superpixel p, a local window W, is defined, which encompasses pixels that are potentially owned by the superpixel, based on the Ar associations. This window configuration is critical for implementing the superpixel cross attention mochanism via a sliding window technique, as detailod in Sec. 3.2 It ensures that each superpixel contextually interacts with its relevant local pixel environment, enabling precise and adaptive refinement of attention dynamics.
4

---

Published in Transactions on Machine Learning Research (01/2025)
association
g
matrix
linear
linear
linear
P25 attention
S2P attention
superpixels
pixel
features
features
Illustration of our SCA module, demonstrating iterative refinement of superpixel and pixel features using a sliding based cross-attention mechanism. Each superpixel cross-attends to localized region of pixels, highlighted in the colored rectangle. On the left, we detail the P2S cross-altention process, while the S2P cross-attention is depicted similarly, albeit with reversed roles for superpixel and pixel.
Figure 3:
The superpixel representation can be effectively converted into a pixel representation using the equation:
I
Σ Aip Sp PEN;
(1)
In Eq. [ I, represents the feature vector corresponding to the pixel i, while S, represents the feature vector of the superpixel p. The term Aip denotes the association weight between the pixel i and the superpixel p. This transformation method not only preserves boundary information with high fidelity but also offers a finer granularity than direct patch upeampling. It facilitates a seamless transition between superpixel and pixel reprosentations, enabling the use of more manageable superpixel grid while ensuring meticulous retention of detailed pixel-level information.
Furthermore, our superpixel approach effectively retains crucial boundary information and demonstrates robustness against common image distortions such as rotation and occlusion, benefiting from the inherent adaptability superpixels. The reduced computational demands, along with semantic pixel clustering for improved explainability, and resilience against complex transformations, position this method as superior alternative traditional approaches.
3.2
Superpixel Cross Attention
The SCA module utilizes a dual attention mechanism: Susperiol (P2S) and Superpixel-to- Pixel -oxxel (S2P). This design aims at localized enhancement of feature representations, critical for applications demanding high-resolution detail and precise contextual understanding (Fig 3
3.2.1 Mechanisms of SCA
Both P2S and S2P mechanisms are engineered to optimize the interplay between pixel and superpixel features:
P2S
This process enhances the represcntation of superpixels by integrating contextual details from pixels within specific local windows.
S2P Cross-Attentin Attention: Conversely, this attention refines pixel features by drawing from the contextual data of adjacent superpixels, thereby enriching the detail and granularity of pixel features.
These bidirectional processes effectively bridge superpixel efficiency and the detailed granularity of pixel representations, offering versatile and refined feature set.
5

---

Published in Transactions on Machine Learning Research (01/2025)
3.2.2 Iterative Feature Refinement
Starting with initial pixel 1 E Rexxxx" and superpixel features S E RR=xxsx* each iteration t progressively refines these features. Updates are applied to both superpixel features S and association matrix A to ensure continual improvement of the superpixel representation.
In the S2P cross-attention phase, starting with pixel features I' and superpixel features S first recreate the association A between pixel i and superpixel as demonstrated:
A softmaxpeN, (qp-1- ks;-1)
(2)
where M2 comprises the neighboring superpixels of pixel i, as defined in Sec. 3.1 and q, k, and symbolize the query, key, and value vectors derived from lincar transformations of the previous itcration's features.
Building upon Eq. (1,, pixel updates are generated by mapping superpixel features Vsş-1 back to pixels using the freshly computed associations A in a residual manner:
I -
Σ A V
(3)
PEN,
Concurrently, superpixel features are updated through P2S cross-attention, as formulated below:
S = Spp + Σ
softmax; (9s:-1 . ky:11) Vi--
(4)
EEW,
This equation iterates pixels i located within the local window Wps defined in Sec. 3.1
These differentiable update equations are essential for refining both pixel and superpixel feature representations, enabling the gradients from either domain to be leveraged for end-to-end training.
3.2.3 Positional Embedding and Multi-head Attention
To enhance the spatial fidelity of superpixels, positional information is integrated into the SCA module using Convolution Position Embedding (CPE) (Huang et al.) 2023), which captures spatial relationships within the image. Both superpixel and pixel features are augmented with CPE implemented as 3 x 3 depthwise convolution supplemented with skip connection, applied prior to the execution of P2S and S2P cross-attention mechanisms. This augmentation further encourages the locality of the superpixels.
Given the propensity for see-seggmnnttionn in superpixel methods, multiple segmentations can emerge; thus, relying exclusively on a single superpixel segmentation may not suffice for an exhaustive image description. Contrary prior methodologies such as those proposed by Jampani et al. (2018) and Zhu et al. (2023), which generate singular type of superpixel segmentation incorporates multi-head attention. This technique facilitates intricate interactions between superpixels and pixels by leveraging global contextual information. As such, it produces spectrum of superpixel representations, depicted in Fig. These varied representations address distinct levels of granularity and ambiguity associated with superpixel seer-segmentttion, thus enhancing the method's utility and adaptability.
time, our approach within the SCA module
3.3 SPFormer Architecture
Our architecture capitalizes on the sophisticated capabilitios of our superpixel representation while adhering closely to the foundational structure of the standard ViT as delineated by Dosovitskiy et al. 2021 1, thus facilitating a direct and fair comparison.
Unlike the standard VIT that regards the images as 16 16 patches, we utilize a more granular approach for extracting initial pixel features 1o gexhxw opting for lightweight stem with stride of 4. This modification includes conventional stride 4 non-overlapping patchify layer. Additionally, in variants of our

---

Published in Transactions on Machine Learning Research (01/2025)
SCA
N VIT Block
Figure 4: Illustration of a single stage of the SPFormer architecture. It starts with initial superpixel features and pixel features as inputs. The SCA module iteratively refines superpixel features, enhancing their semantic richness. These features are then processed by the MHSA for global contextual understanding. The stage concludes by updating the pixel features based on the enriched superpixel information. These updated pixel and superpixel features are either fod into the subscquent stage, which starts with another SCA module, utilized for making final predictions.
model, denoted by 1, the patchify layer is replaced by two convolutional layers each with a kernel size of 3 and a stride of 2. This convolutional stem provides a more effective initialization of pixel features, aiding in better feature representation from the onset.
For the extraction of superpixel features, we implement stride = 4, producing superpixel features 8 E XgexXaxx*. directly from the pixel features I. This process involves a x 1 convolutionn followed by a 4 x 4 average pooling operation. With this configuration, the resultant number of superpixels is equivalent to the number of patches in the standard ViT. Such equivalence crucially ensures that the computational costs associated with the remainder of our network align closely with those of the classical ViT framework, apart from the minimal, yet strategically important, overhead introduced by the integrated Suporpixel Cross Attention (SCA) layers.
As shown in Fig. | So and 1sd undergo continuous refinement across multiple iterations t = 2 using the SCA module. This module enhances the semantic alignment of the superpixel representation by exploiting the local spatial context of each superpixel, as detailed in Sec. 3.2 Subsequently, the enriched superpixel features St are processed by multi-head self-attention to extract long-range dependencies and contextual nuances within the image.
We observed that even after multiple iterations (e.g., >> 2), the superpixel representations generated by a single SCA module may not fully align with the overarching context. This misalignment primarily stems from insufficient semantic information derived from the output of the lightweight stem. To tackle this challenge, we implement a gradual refinement strategy for enriching superpixel representations by interleaving multiple SCA modules with the VIT Blocks. We segment the original ViT architecture into multiple stages, each starting with a SCA module with t t= 2 itcrations). This design allows each SCA module to build on the semantically enriched superpixel features from the previous stage, progressively enhancing the semantic depth.
Specifically, at the end of the stage, superpixel features enriched with global attention are projected back into the pixel domain, outlined in Eq. ( and then added with the existing pixel features I'. This integration process ensures that the pixel features are augmented with comprehensive global contextual information. In the subsequent stage, these contextually enriched pixel and superpixel features are utilized as inputs, avoiding the need to relnitialize features from scratch. This continuous, integrative approach not only aligns the superpixels more closely with low-level boundaries but also enhances consistency with semantic contours, thereby improving the overall coherence and relevance of the image representation.
Conceptually, our network architecture is designed a dual-branch structure. One branch preserves a dense pixel representation with high resolution, while the other branch focuses on our compact superpixel representation. Minimal direct operations are applied to the dense pixel representatin, enabling us to allocate most computational resources to the more efficient superpixel representation. This dual-branch approach fosters computational efficiency without sacrificing the detailed preservation of local image features.
By integrating superpixel representation with SCA, SPFormer offers an advanced solution for boosting both the semantic interpretability and computational efficiency of image processing. This combination enhances detail retention and positions SPFormer as strong tool in diverse image processing applications.
7

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 5: The multi-head SCA design generates multiple superpixel representations, each capturing different semantic relationships and addressing the ambiguity in superpixel over-segmentation.
4 Experiments
4.1 Implementation Details
Different number of heads of the atlentin are used to enable multiple superpixel segmentation in a SCA module. For our architecture variants, we allocate two attention heads for the smaller models (SPFormer-T and SPFormer-S) and three heads for the base model (SPFormer-B).
As outlined in Sec. 3.3 SCA blocks are interleaved inlo the standard VIT architecture, specifically before the first and third self-attention blocks. We utilize the LayerScale technique, specified in Touvron et al. (2021b), to modulate gradient flow and enhance training stability and effectiveness. With the inclusion of residual connections as detailed in Eq. 4 and Eq. B our approach starts by mimicking standard patches but evolves to exploit superpixels more effectively as training advances.
Adhering to the protocols established in DeiT (Touvron et al. 2021a we implement robust data augmentations, use the AdamW optimizer, and follow cosine decay learning rate schedule. All models train on the ImageNet dataset (Russakovsky et al. 2015) for 300 epochs. During SPFormer-Bi BBrr training, significant overfitting challenges arose. Increasing the Stochastic Depth (Huang et al. 2016) rate from 0.1 to 0.6 effectively addressed these issues, highlighting a potential need for advanced regularization strategies tailored to superpixel representation for future studies.
4.2
Efficiency in Image Classification and Segmentation
4.2.1
Main Results ImageNet
Our evaluation of SPFormer on the ImageNet dataset demonstrates its superior efficiency and performance over the DeiT bascline under varying configurations as shown in Tab. [ Specifically, SPFormer-S, which employs the standard ViT configuration with 196 tokens, exceeds the performance DeiT-S by 1.1%, achieving a top-1 accuracy of 81.0% compared 79.9% for DeiT-S. Furthermore, SPFormer-T outperforms DeiT-T by 1.4%, recording 73.6% versus 72.2%.
The superior detail preservation afforded by superpixel representation enables SPFormer to exhibit scaling behaviors distinct from those of VIT, which utilizes patch representation. This advantage becomes increasingly evident with larger patch sizes, such as 32. While DeiT-S/32 suffers from performance degradation due to its coarser gramularty, SPFormer-S/20 S/32 demonstrates stable performance, achieving 76.4%, which a notable 4.2% improvement over DeiT-T and requires fewer FLOPPs.
The token reduction capability of the superpixel representation permits a strategic focus on increasing image resolution to capture finer details effectively. By increasing the stride from 4 to 8 and adopting higher resolution of 448 which produces the same number of of superpixels, SPFormer achieves 0.3% enhancement in performance relative to its standard SPFormer-S configuration, without incurring extra computational costs. Conversely, similar adjustments in DeiT-S only yield a negligible improvement of 0.1%, hindered by the granularity of its patch-based representation.
Employing a convolutional stem provides an enhanced initialization for pixel features, which subsequently leads improved representations of superpixels in the initial stage. This modification has been shown to
8

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Comparative analysis of SPFormer's performance on ImageNet classification against DefT baselines.
Variants augmented with two convolution layers of kernel size with stride 2 are denoted by
Model
# Params
#FLOPs
Top-1
SPFormer-S/56
22M
0.5G
72.3
DeiT-T Touvron et al.
2021a
5M
1.3G
72.2
SPFormer-T
5M
1.3G
73.6
SPFormer-Ti
5M
1.3G
75.0
DeiT-S/32 (Tourron 2021a
22M
1.1G
73.3
SPFormer-S -/33
22M
1.2G
76.4
SPFormer-S/38
22M
1.3G
77.9
DeñT-S Touvron et al.
2021a
22M
4.6G
79.9
SPFormer-S
22M
5.2G
81.0
SPFormer-St
22M
5.3G
81.7
DeiT-B TTovron al. 2021a
87M
17.5G
81.8
SPFormer-B
87M
19.2G
82.4
SPFormer-Bi
87M
19.2G
82.7
Table 2: Ablation study on the design choices in SPFormer.
Model
#Params
#FLOPs
Top-1
SPFormer-S/33
22M
1.2G
76.4
Single Iteration in SCA
22M
1.2G
75.4
SCA at Initial Layer Only
22M
1.2G
74.8
Single-Head SCA Learmable Position Embedding
22M
1.2G
75.6
22M
1.2G
76.
consistently enhance performance, as exemplified by SPFormer-S/321, This particular configuration achieved notable increase of 1.5% in ImageNet classification accuracy, attaining a total of 77.9%. These findings further underscore the significance of precise superpixel assignment and demonstrate its advantages over the traditional vanilla patch representation. Such enhancements validate the efficacy of modified approach in handling complex image processing tasks more effectively.
4.2.2 Ablation Study: Design Choices in SPFormer
We evaluates key design elements of SPFormer-S/33 on the ImageNet validation set. We investigate the impacts of iteration count in the SCA module, the placement of SCA within the architecture, the use of multi-head attention, and the choice of position embeddings.
The findings highlight the importance of multiple iterations in SCA for performance enhancement, with single iteration leading to a 1.0% drop in accuracy. The strategic placement of SCA across different layers is crucial, as restricting it to the initial layer causes a 1.6% accuracy reduction, indicating that higher-level features play a vital role in augmenting semantic depth and rectifying early-stage superpixel inaccuracies. Furthermore, employing multi-head attention SCA significant for capturing diverse superpixel relationships, with its absence leading a 0.8% docrease in accuracy. Lastly, using learnable position embeddings over CPE results in slight drop in performance.
This ablation study validates the effectiveness of the considered design choices in SPFormer, affirming their contributions to the overall performance of the model.
4.2.3 Semantic Segmentation: Utilizing SPFormer's High- Resoluten Feature Preservation
SPFormer's superpixel representation intrinsically maintains higher resolution features, making it particularly suitable for semantic segmentation tasks. This characteristic allows for detailed and context-rich segmentation outputs, key advantage over traditional patch-based methods.

---

Published in Transactions on Machine Learning Research (01/2025)
Table 3: Semantic segmentation on ADE20K val split.
Method
# Params
#FLOPs
Pretrained
mloU
DeiT-S
22M
32G
X
20.1
SPPormer-S
23M
35G
X
23.1
DeiT-S
22M
32G
/
42.3
SPFormer-S
23M
35G
/
46.5
Table Semantic segmentation on Pascal Conext val split.
Method
#Params
#FLOPs
Pretrained
mloU
DeiI'S
22M
27G
x
18.0
SPFormer-S
23M
30G
21.1
DeiT-S
22M
27G
48.3
SPFormer-S
23M
30G
51.2
Incorporating SPFormer into the SETR (Zeng et al. 2021 framework, we enhance segmentation performance
by directly classifying individual superpixels. This direct approach leverages SPFormer's high-resolution feature preservation allowing for more muanced segmentation. The final segmentation maps are generated by upscaling the superpixel-based logits using Eq. (1).
We evaluate SPFormer on the ADE20K (Zhou al. 2017 and Pascal Context (Mottaghi et al. 2014 datasets. Utilizing ImageNet-pretrainee models, SPFrrmer demonstrates significant improvements in mloU, highlighting its effectiveness in detailed segmentation tasks.
As shown in Tab. 3 and Tab. 1 the performance gains in mloU are noteworthy when using ImageNet- pretrained models: 4.2% improvement on ADE20K and 2.8% on Pascal Context. These results not only highlight the detailed nature of SPFormer's superpixel representation but also adaptability and complex datasets. To further validate the intrinsic segmentation capabilities of SPFormer, we conduct additional training from scratch. This approach reiterates the model's strength in maintaining high-resolution features independently of pretraining influences, leading mloU improvements of 3.0% on ADE20K and 3.1% on Pascal Context compared to baseline methods.
diverse
4.3 Unveiling SPFormer's Explainability
Integrating superpixel reprosentation into the Vision Transformer architecture adds significant layer of explainability compared to conventional fixed patch partition methods. This section first discusses the inherent explainability of our superpixel representation, followed by an evaluation of its semantic alignment and generalizability to unseen data.
4.3.1 Superpixel Representation as an Explainability Tool
Our method's superpixel representation can be visualized through the association matrix A, providing insights into the model's internal processing. In Fig. 1 we visualize the learned soft associations by selecting the argmax over the superpixels:
À argmax(A)
(5)
These visualizations reveal that, even with soft association, the superpixels generally align with image boundaries. This alignment is noteworthy as it emerges even though the network is only trained on image category labels. Thus, the model segments images into irregular, semantics-aware regions while reducing the number of Lokens needed for representation.
Furthermore, we assess the gencralizability our superpixel representation using the COCO dataset (Lin et al., 2014), which consists of high-resolution images with complex scenes. For this evaluation, we resize and center-crop COCO images to align with the ImageNet evaluation pipeline. Fig. showcases the
10

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 6: Zero-shot transferability on the COCO dataset. Trained solely on ImageNet, SPFormer demonstrates effective segmentation of unseen COCO images into detailed superpixels. 196 superpixels are used in this visualization.
Table 5: Evaluation of superpixel quality in zero-shot setting Pascal VOC 2012 and datasets, using 196 patches ssuprpiiiss.. Our SPFormer variants demonstrate notable improvements over traditional patch representations and are competitive with the SLIC method.
Pascal Voc2012
Method
mloU
mAce
mloU
mAcc
Patch
87.8
92.8
68.7
78.2
SPFormer-TI
91.5
95.7
71.5
79.9
SPFormer-S!
92.0
96.6
73.3
82.4
SPFormer-Bt
91.2
96.3
72.5
81.4
SLIC (Achanta et al. 2012
92.5
95.4
74.0
81.7
visual representation of superpixels on these images. Note that the superpixels generated by SPFormer, trained exclusively on ImageNet, adapt well to this unseen data, capturing intricate structures such as thin objects. This adaptability highlights the model's capability to preserve detail and generalize its superpixel representation to new contexts.
4.3.2 Semantic Alignment of Superpixels
Our evaluation of superpixel representation focuses on its ability align with ground truth boundaries in images, despite the model not being trained the datasets used for this assessment. This test involved quantitative analysis both object and part levels using the Pascal VOC 2012 dataset (Everingham et al. 2015) and Pascal-Part-58 (Zhao et al. 2019). Note that these assessments were performed without any training on these specific datasets, underscoring the model's generalization capabilities.
In our approach, each superpixel or patch's prediction is derived by aggregating the ground truth labels of the pixels it encompasses. We assign the most frequently occurring label within superpixel as its prediction, assuming optimal classification. This method leverages the soft associations produced by our SCA module, where predictions are formed by combining pixel labels with their corresponding weights and upscaled as per Eq. ①
Diverging from the single-superpixel outcome of traditional patch representation, our model employs a multi-head design in the SCA module. This allows for the creation of multiple, distinct superpixels for each head, enhancing the richness and diversity of the extracted features (see Fig. 5). For our evaluations, we computed an average of predictions across all heads. is notcworthy that effective feature extraction in our model deemed successful if even a single head accurately identifies superpixel.
The proficiency of our superpixel approach is demonstrated in its performance compared to vanilla ViIs that utilize patch representations with stride of 16. ViTs often suffer from a gramularity tradc-off, losing finer details in favor of broader patch representations. In contrast, the superpixels from our SCA module, as shown in Tab. 5 manifest substantial improvements - achieving a 4.2% increase in object-level and 4.6% in part-level mIoU with SPFormer-St. Furthermore, these superpixels display a quality comparable to those from traditional superpixel methods like SLIC (Achanta et al. (012)., highlighting our method's effectiveness in capturing detailed semantic information without direct training on the evaluation datasets.
11

---

Published in Transactions on Machine Learning Research (01/2025)
Table 6: Quantitative evaluation of SPFormer's robustness rotation, comparing performance at different
angles. Variants augmented with two convolution layers of kernel size 3 with stride 2 are denoted by 1.
Model
Clean
15
30
45
DeiT-S/32 Touvrro et al.
2021a
73.3
71.1
67.7
59.5
SPFormer-S/ar
77.9
75.2
73.4
66.9
Rotation
Occlusion
Figure 7: Visualization of SPFormer's superpixel representation under rotation and occlusion, highlighting the model's adaptability and robustness.
4.3.3 Explainabillty-Driive Robustness
The robustness of SPFormer is deeply intertwined with its explainability, particularly through the superpixel representation. This section explores how the model's transparent and interpretable features contribute to its resilience against image modifications like rotation and occlusion.
Robustness to Rotation. Our model's capacity to generate coherent superpixels even under rotational
transformations showcases the robustness afforded by its explainable structure. For example, Fig. illustrates the model's performance under rotation, and Tab. G quantifies this robustness, under patch size 32. By visualizing how superpixels adapt to rotated images, we gain insights into the model's stability in varied orientations. While SPFormer demonstrates heightened robustness to rotation, it still exhibits limilations, likely due to the learnable absolute position embeddings not being inherently rolation-invariant. These observations suggest potential avenues for enhancing rotational robustness, possibly through integrating rotation-invariant mechanisms within the superpixel representation or the network architecture.
Robustness to Occlusion. The occlusion robustness of SPFormer is another facet where explainability plays a key role. By examining superpixel behavior in occluded images, we observe the model's ability to distinguish between occluders and the object of interest. Unlike Iraditional patch-based representations, which tend to blend occluders with the object, our superpixel representation more effectively isolates and identifies obscured parts of the image. This nuanced differentiation a direct result of the model's explainable superpixel structure, which provides more detailed and contexl-aware interpretation of the image content, demonstrated in Fig. 7
5 Conclusion
In this work, we introduced SPFormer, novel approach for feature representation in Vision Transformers, emphasizing superpixel represcntation. This method showcases promising shift from traditional pixel and palch-based approaches, offering two distinct advantages: elliciency due to a reduced number of superpixels facilitating global self-attention and explainability through semantic grouping pixels. The empirical results demonstrate the potential of SPFormer in diverse computer vision tasks, under both the standard benchmarks and challenging recognition scenarios. We hope our findings can pave the way for future exploration in this direction, encouraging further research into superpixel-based visual representation learning.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Acknowledgment
This work was supported by ONR N00014-21-1-2690. We thank the Center for Al Safety for supporting our computing needs.
References
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-arr superpixel methods. TPAMI, 34(11):2274 2282, 2012.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2022.
Eran Borenstein and Shimon Ullman. Class-specific, top-down segmentation. In ECCV, 2002.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Liang-Chiee Ching Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4))834-848, 2018.
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. TPAMI, 24 (5):603-619 619,, 2002.
Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965 3977, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Mark Everingham, S. M. Ali Eslami, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 111 9-6., 2015.
Pedro Felzenszwalb and Daniel Huttenlocher. Efficient graph-based image segmentation. IJV, 59: 167-181, 2004.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In CVPR, 2022.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), ECCV, volume 9908 of Lecture Notes in Computer Science, pp. 646 661. Springer, 2016.
Huaibo Huang, Xiaoqiang Zhou, Jie Cao, Ran He, and Tieniu Tan. Vision transformer with super token sampling. In CVPR, 2023.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In TCML, 2015.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Varun Jampani, Deqing Sun, Ming- Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), ECCV, 2018.
Alex Krizhevsky, Πya Sutskever, and GeofTrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document rocognition. Proceedings of the IEEE, 86(11):227) 2222222444, 1998.
Yingwei Li, Xiaojie Jin, Jieru Mci, Xiaochen Lian, Linjie Yang, Cihang Xie, Qihang Yu, Yuyin Zhou, Song Bai, and Alan L. Yuille. Neural architecture search for lightweight non-local networks. In CVPR, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), ECCV, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In ICCV, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2):129 137, 1982.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances én Neural Information Processing Systems, 33:1 23:11525-11523 2020.
Xu Ma, Yuqian Zhou, Huan Wang, Can Qin, Bin Sun, Chang Liu, and Yun Fu. Image set of points. In ICLR, 2023.
Jitendra Malik, Serge Belongie, Thomas Leung, and Jianbo Shi. Contour and texture analysis for image segmentation. IJCV, 43:7 27, 2001.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, In ICCV, 2001.
Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. In ICLR, 2022.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasum, and Alan L. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, pp. 891-898. IEFE Computer Society, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Xiaofeng Ren and Jitendra Malik. Learning classification model for segmentation. In 1CCV, 2003.
Olga Russakovaky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenct large scale visual recognition challenge. IJCV, 115:211-22 255, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. TPAMT, 22(8):888 000888885558888 2000.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLRR 2015.
Andreas Steiner, Alexander Kolesníkov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. (XXi::1066112700 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. In ICCV, 2017.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent. Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, pp. 32 42. IEEE, 2021b.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
Zhuowen Tu and Song-Chun Zhu. Image segmentation by data-driven markov chain monte carlo. TPAMI, 24 (5):657-673, 2002.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Andrea Vedaldi and Stefano Soatto. Quick shift and kernel methods for mode seeking. In ECCV, 2008.
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang- Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021b.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In 1CCV, 2021.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in Newral Information Processing Systems, 34:30392 30400, 2021.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022.
Chenglin Yang, Siyuan Qiao, Qihang Yu, Xiaoding Yuan, Yukun Zhu, Alan Yuille, Hartwig Adam, and Liang-Chieh Chen. Moat: Alternating mobile convolution and attention brings strong vision models. ICLR, 2023.
Fengting Yang, Qian Sun, Hailin Jin, and Zihan Zhou. Superpixel segmentation with fully convolutional networks. In CVPR, 2020.
Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chior Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. CVPR, 2022a.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Qihang Yu, Huiyu Wang. Siyuan Qiao, Maxwell D. Collins, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang-Chies Chich Chen. k-means Mask Transformer. In ECCV, 2022b.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022c.
Yaodong Yu, Tianzhe Chu, Shengbang Tong, Ziyang Wu, Druv Pai, Sam Buchanan, and Yi Ma. Emergence segmentation with minimalistic white-box transformers. CoRR, abs/2308.16271, 2023.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In ICCV, 2021.
Yifan Zhang, Bo Pang, and Cewu Lu. Semantic segmentation by early region proxy. In CVPR, 2022.
Yifan Zhao, Jia Li, Yu Zhang, and Yonghong Tian. Multi-class part parsing with joint boundary-semantic awareness. ICCV, pp. 9176 9185. IEEE, 2019.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from sequence-to- sequence perspective with transformers. In CVPR, pp. 6881 6890. Computer Vision Foundation / TEEE, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, pp. 5122-5130. IEEE Computer Society, 2017.
Alex Zihao Zhu, Jieru Mei, Siyuan Qiao, Hang Yan, Yukun Zhu, Liang-Chieh Chen, and Henrik Kretzschmar. Superpixel transformers for efficient semantic segmentation. CoRR, abs/ (bs/2909.66889, 2023. doi: 10.48550/ ARXIV 220...1698)) URL /ttp::://00d.orr/110485555
2309 16889
Song Chun Zhu and Alan Yuille. Region competition: Unilying snakes, region growing, and bayes/mdl for multiband image segmentation. TPAMI, 18(9):884 900, 1996.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
16


Reviews:
[{"review_id": "7D3Z9WnUUJ", "reviewer": "Reviewer_dhve", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Fairly large improvements to ImageNet top-1 accuracy with minimal extra compute (especially nice for DeiT-S comparisons).\n- Quite large improvements for robustness to rotation (thought this was most interesting).\n- Enhanced explainability via superpixel representation.\n- Notable improvements to object-level and part-level classification on Pascal VOC 2012 (zero shot, trained only on ImageNet) compared to solely-patch-based ViT.\n\nWeaknesses\n- The method isn't entirely clear from Sec 3.1. It would be nice to define r and n more generally, surely these aren't fixed at r = 3 and n = 9 (n is the number of superpixels, I assume).\n- Not clear how to construct N and W (neighboring superpixels, superpixel local window) in Sec. 3.1 -- maybe this is more obvious if you're familiar with previous work on superpixels, but since this is the key contribution of the paper it would be good to be more precise.\n- Not sure what subscript ip is in Eq. 1 -- I'm guessing p is a (h,w) pair, and N_i is a set of such pairs?\n- Somewhat confusing that P2S is presented as a single equation (2) and S2P as two equations (3-4).\n- On that note, since the SCA module is the key contribution, would be nice to have its exact functional form specified (I guess I'm inferring from Fig. 2 that there are two activations S and I which are maybe concatenated later?)\n- When you say iteratively, you mean there are multiple S2P/P2S layers? Otherwise, unclear what is meant by \"iterations\"\n- In Sec S3.3, the fact that superpixel features come from a 1x1 conv followed by 4x4 average pooling seems quite important. I guess broadly speaking, the actual form of the architecture is a bit mysterious based on the paper, and could be more clearly laid out.\n- For example: it seems by the end of 3.3 that SCA isn't just part of the stem, but actually interleaved into the ViT. This could be more clear/precise.\n- Sec 4.1 contains more (crucial?) implementation details despite being in the Experiments section.\n- In 4.1, kind of sounds like we you still use 4x4 patches before processing superpixels (which is okay, just should be lifted into 3.3 / an architecture section)\n- (Fairly generic) experiment details are mixed into novel architecture details in Sec 4.1, these should be organized better\n- More architecture changes within results in 4.2.1, adding several 3x3 stride-2 convs unexpectedly -- and this seems quite crucial to performance...\n- There is generally quite a bit of discussion about the efficiency of this representation, but in the tables SPFormer seems to use more FLOPs for relatively small accuracy gains -- I know +1% is quite a bit for ImageNet, but it's also something like 10% more FLOPs. I think this is fine, but maybe deserves explicit discussion (e.g., how much accuracy would we expect to gain from increasing FLOPs by this much through another technique). This is minor, I appreciate the technique regardless.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a Superpixel Cross Attention operation to leverage a superpixel representation within Vision Transformers. They show that this leads to relatively large gains in accuracy (1%) on ImageNet-1k classification for a relatively small increase in FLOPs. The method also results in substantially better (4%) zero-shot performance on datasets such as COCO and Pascal VOC 2012. Interestingly, they show enhanced robustness to rotations up to +7% using the superpixel representation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Section 3.2, and to a lesser extent 3.1, could include a clearer explanation of the technique. This might be as simple as defining symbols such as N, W, n, r. The paper assumes familiarity with previous work on superpixel representations, and could reach a wider audience by being more self-contained. Further, the architecture is somewhat unclear, with details interspersed in Sections 3.3, 4.1, 4.2. The architecture details/SCA details should be gathered into clear sections rather than mixed with experimental details, IMO (this can work sometimes, but the overall view of the architecture is somewhat confusing in this case). Please also see Weaknesses, though I don't really expect every point to be addressed.\n\nI anticipated that this paper would be solely about changing the input representation, but I think the SCA layers are actually dispersed throughout the ViT architecture. If the authors want to present a more input-representation-centric perspective, they might be interested in mentioning works such as [0, 1, 2].\n\nOverall, I think this is a neat contribution, but the reach will be limited by lack of clarity / reproducibility. \n\n[0] Adaptive Input Representations for Neural Language Modeling, Baevski and Auli (more for language than vision, admittedly)\n[1] Patches are all you need? Trockman and Kolter (there is much discussion of the patch representation)\n[2] MetaFormer Is Actually What You Need for Vision, Yu et al (might show some support for the importance of input representation vs. architecture)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mixJCtRIay", "reviewer": "Reviewer_FofP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. It is interesting to integrate the superpixel representation in vison transformer.\n\n2. The proposed method can achieve better performance than that of DeiT on classification and semantic segmentation.\n\nWeakness:\n\n\n1. The experiments are significantly insufficient, with only one baseline method included for the classification and segmentation tasks. The proposed approach is not compared against some of the latest methods. Additionally, a comparison with the similar work [1] is also essential to provide a more comprehensive evaluation.\n\n2. The computational cost should be further justified by providing additional evidence such as actual running time, throughput or an analysis of time complexity.\n\n3. In Table 5, the performance of SPFormer-B is lower than that of SPFormer-S. Can authors explain why SPFormer would achieve lower performance despite having more parameters in this task?\n\n\n[1] Huang, Huaibo, et al. \"Vision transformer with super token sampling.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel vision transformer architecture that leverages a superpixel-based representation instead of the traditional patch-based approach. This superpixel representation is generated using the proposed SCA module, which enhances the interpretability of the vision transformer and improves its robustness to rotations and occlusions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "See Weakness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ReX4Z5KLL4", "reviewer": "Reviewer_RDXq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The algorithm proposed in this paper is based on super pixel segmentations, hence comes with better explainability than patch based ViT. The experimental results have shown robustness to rotation and Occlusion as well.\n\n- The overall structural of the paper is complete, with literature reviews on ViT and super pixel representations, and ablation studies showing the improvement of proposed methods.\n\n**Weacknesses:**\n\n- The super pixel learning part is not described in sufficient detail, making it hard for reader to comprehend how the pixel are grouped, and how much learning and engineering are involved to build the model. Specifically, $A_{ip}$, $S_{p}$ in EQ(1) are not defined nor discussed.\n\n- Given the unclear core method definition, it is hard the core method, hence hard to validate the adaptation ability. How to ensure the total number of super pixel, or are they fixed at all? How to ensure there are no empty super pixel? How much offline and online computations are required - are they learned via clustering or gradient decent?\n\n- The difference between super pixel learning in this paper with previous super pixel papers are not discussed, hence the novelty is hard to judge.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new ViT class algorithm that leverages super pixel to build attention. It claims to be more flexible and adaptable using the segmentation level attention.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The method needs to be clearly defined, and the novelty (difference) of this work with previous works needs to be further highlighted.\n\nMinor: Better explain in plain text rather than place jargons, e.g. mathematically describe Moore Neighborhood rather than refer to the name without explanation.", "broader_impact_concerns": "The method is not elaborated in sufficient detail, and detailed comparisons with previous methods are missing.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wCwCJAZvCM", "reviewer": "Reviewer_MvDf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* The idea of leveraging superpixel seems intriguing, albeit some previous works (Jampani et al. (2018) and Zhu et al. (2023)) have studied it.\n* The motivation is clearly conveyed.\n* The accompanying visualizations of seemingly semantically clustered patches nicely demonstrate the strengths of the proposed superpixel patches.\n\nWeaknesses\n\n**The utility of superpixels, in comparison to the vanilla patch-wise representations of ViT**\n\nMy main concern is stated above. Specifically:\n\n- If the main benefit of the superpixel embeddings is on the performance of the vision backbone, then comparing it with DeiT (2021) alone may be insufficient as of 2024, as there have been some more advanced architectures available, for instance, SwinTransformer and its variants.\n\n- Additionally, given that the concept of superpixel has been studied before (in fact, the author has mentioned some of them in the text, e.g., Jampani et al. (2018)), it poses the question as to why the author has not compared the performance of the proposed method with such work.\n\n- Subsequently, I would imagine another potential benefit of such arrangements is the drastic reduction in the number of tokens needed -- however, this would depend on the granularity of the embeddings. Yet, the author has not provided ablation studies on this aspect. It would be nice to do so (specifically, superpixel granularity *vs.* performance *vs.* inference cost).\n\n\n**Some notations are slightly confusing**\n\nIn Section 3.1, the author introduces the superpixel representations, however:\n\n- what will be the dimensions of $s_h$ and $s_w$ for $\\mathbf{S}$? Because given the visualizations, the superpixel patches in an image will no longer be of fixed quantity (as in the vanilla ViT patches where it is simply $N\\times N$). Also, is $\\mathbf{A}_{i\\times p} \\cdot \\mathbf{S}_p$ functioning as a weighted sum here -- given that in Equation 1 the pixel representation for pixel $i$ is obtained as such. Perhaps an example of the matrices would be nice (or a visualization of such transformations between pixel and superpixel representations).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes incorporating the superpixel representation into the vision transformer (ViT), thereby improving the patch-wise representation used by the vanilla ViT.\n\nImplementation-wise, this work introduces the Superpixel Cross Attention (SCA) module that is prepended ahead of the traditional attention layers, which performs iterative refinements in obtaining the embeddings of the superpixels. Though, interestingly, as the SCA modules output two sets of embeddings (superpixel, and pixel embeddings), the architecture is effectively a dual-branch design.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*This section largely is a brief summary of the weaknesses part above.*\n\n- The comparison of related work (both more state-of-the-art vision transformers such as SwinTransformer and more domain-specific works that are heavily related to the idea of superpixels) is a key missing point from my point of view. Perhaps the author could consider adding such comparisons to demonstrate that the proposed method is indeed superior.\n\n- Likewise, the ablation studies that highlight the efficiency of the superpixels would be a strong argument and a nice thing to have -- this is especially true since the authors repeatedly state that the computational cost is a benefit of the proposed model (Section 3).\n\nBased on the above two points, I am tentatively rating the **Claims And Evidence** as **No**.\n\n---\n\n- The modifications on the notations would be a relatively small improvement. Though, it will help readers understand the method better.", "broader_impact_concerns": "No broader impacts were identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers' scores are somewhat mixed. Although the authors have added experiments in their rebuttal by incorporating [1]'s superpixel representation into the SPFormer-S/32 model, they did not directly compare it with [1] or other related methods. This direct comparison is crucial for assessing the effectiveness of the proposed method, especially in the context of existing approaches that address similar challenges. Additionally, Table 1 appears insufficient in demonstrating the algorithm's efficiency advantages, particularly when compared to other similar methods. 

Despite the concerns mentioned above, reviewers agree that this work introduces a novel architecture that integrates super-pixel representations to address the limitations of fixed partitions in traditional Vision Transformers (ViT). Therefore, this paper has value for publication, but it would be beneficial to address the remaining concerns to make the claims more convincing.
Decision: Accept (poster)