CAST: Concurrent Recognition and Segmentation with Adaptive Segment Tokens
Authors: Tsung-Wei Ke
Jyh-Jing Hwang
Stella Yu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Recognizing an image and segmenting it into coherent regions are often treated as separate tasks.  Human vision, however, has a general sense of segmentation hierarchy before recognition occurs.  We are thus inspired to learn image recognition with hierarchical image segmentation based entirely on unlabeled images.  Our insight is to learn  fine-to-coarse features concurrently at superpixels, segments, and full image levels,  enforcing consistency and goodness of feature induced segmentations while maximizing discrimination among image instances.

Our model innovates vision transformers on three aspects.  1) We use adaptive segment tokens instead of fixed-shape patch tokens. 2) We create a token hierarchy by inserting graph pooling between transformer blocks, naturally producing consistent multi-scale segmentations while increasing the segment size and reducing the number of tokens.  3) We produce hierarchical image segmentation for free {\it while} training for recognition by maximizing image-wise discrimination.

Our work delivers the first concurrent recognition and hierarchical segmentation model without any supervision.   Validated on ImageNet and PASCAL VOC, it achieves better recognition and segmentation with higher computational efficiency.

Under review as a conference paper at ICLR 2023
CAST: CONCURRENT RECOGNITION AND SEGMENTATION WITH ADAPTIVE SEGMENT TOKENS
Anonymous authors Paper under double-blind review
ABSTRACT
Recognizing an image and segmenting it into coherent regions are often treated as separate tasks. Human vision, however, has general sense of segmentation hierarchy before recognition occurs. We are thus inspired to learn image recognition with hierarchical image segmentation based entirely on unlabeled images. Our insight is to learn fine-to-coarse features concurrently at superpixels, segments, and full image levels, enforcing consistency induced segmentations while maximizing discrimination among image instances.
Our model innovates vision transformers in three aspects. 1) segment tokens instead of fixed-shape patch tokens. 2) We create a token hierarchy by inserting graph pooling between transformer blocks, naturally producing consis- tent multi-scale segmentations while increasing the segment size and reducing the number of tokens. 3) We produce hierarchical image segmentation for free while training for recognition by maximizing image-wise discrimination.
We use adaptive
Our work delivers the first concurrent recognition and hierarchical segmentation model without any supervision. Validated on ImageNet and PASCAL VOC, it achieves better recognition and segmentation with higher computational efficiency.
INTRODUCTION
Convolutional neural networks (CNN) (eeCun al. 1989 Krizhevsky et al. 2012 He al. 2016)
and Vision Transformers (ViT) (Dosovitskiy et al.) (2020) have been very successful in computer vision. However, recognizing an image and segmenting it into coherent regions are treated as separate tasks or learned sequentially (Martin et al.) (001). Fig. iig[[siltstrttes
common practice: CNN (ViT) predicts the semantic class of an image based on the image-level feature from the output of the final convolutional layer (transformer block), and additional clustering based on earlier pixel-wise features is required to generate image segmentation (Hwang et al. 2019 Ke et al. 2022
However, human vision has a general sense of segmentation hierarchy, in terms of groups of pixels or segments, before recognition even occurs. This perceptual organization perspective (Witkin & Tenenbaum 1983; Biederman 1987) has been overlooked in CNN and ViT architectures: models optimized for image classification tend to latch onto discriminative parts (Selvaraju et al. (017)) such as faces, often missing inconspicuous body parts that go with the face. Previous methods seldom model how different parts such as face and body are organized for the whole animal explicitly.
To understand the connections between parts and wholes, visual information must be extracted locally and globally. There are three major approaches (Fig. [
1. Spatial downsampling: With pixels laid on a regular grid, features are extracted from patches. The granularity of visual information is determined by the patch size. Max or mean pool- ing (Krizhevsky et al., 20122 Simonyan & Zisserman 2014), uniform sub-sampling (striding) (He) lt al. (016) and patch merging (Liu al. | (021) are performed multiple times increase the effective receptive field size (Luo et al.) 2016). To generate segmentation, the output features need to be upsampled and clustered (c.g. K-Means (Hwang et al.) 2019)), often resulting in over-smoothed boundaries.
2. Attention: Inspired by Natural Language Processing (NLP), image patches treated visual word tokens of the entire image document. To extract more global information, ViT contextually updates feature representations based on pair-wise correlation among all the tokens of an image,

---

Under review as a conference paper at ICLR 2023
paliches
OIGE VTT
New CAST
Image discrimination
superpixels
hierarchical segmentation
Figure 1: We innovate vision transformer models to concurrently learn image recognition and hierarchical image segmentation from unlabeled images alone. Top: ViT (Dosovitskiy al. (020) takes patch tokens as inputs and maintains the same large number of tokens through all encoder blocks. Image segmentation would require additional pixel-wise clustering (e.g. K-Means) on the fixed patch-wise features. Bottom: Our model takes segment tokens inputs and hierarchically groups them into fewer coarsened region tokens. Unlike patch tokens, these segment tokes adapt to the image and vary in shape. We unify fine-to-coarse feature learning at multiple levels in single model to support not only recognition with maximum image-wise discrimination, but also segmentation with consistency across the hierarchy. Consequently, achieve better recognition and segmentation with higher computational efficiency.
using attention modules (Vaswani et al. 2017). However, VIT is computationally inefficient as all image tokens are kept every transformer block.
3. Significance-bases subsampling: To increase ViT's computational efficiency, tokens are sub- sampled at higher levels based on their significance scores. POWER-BIE BERT (Goyal et al.) 2020) and Token Pooling (Marin et al.) 2021) define the significance score as the total attention given each token from all other tokens. Downsampling then retains only the most dominant visual features in the image. Such methods only keep the most informative tokens in final output representations.
These existing methods have two major issues. 1) Both CNN and ViT models take regularly shaped patch features as inputs, regardless of what is in the image. Image segmentation derived from such representations often fails to align with contours. Image segmentation does not involve local-to- global feature extraction, which is treated as a separate visual task from image-wise recognition.
Our first insight is that pixel groupings not computational inconvenience (as opposed regular patches), but a natural structure to be exploited for better visual computing. Unlike existing CNN and VIT which extract features pixel groupings at an carly stage and develop feature representations subsequently. Our model takes segment features as input tokens and carries this adaptive segment representation through deeper layers. Post-processing with pixel-wise clustering methods is no longer needed.
regular grid throughout the entire model, directly get low-level
Our second insight to derive fine-to-coarse pixel groupings jointly with local-to-giobal feature extraction. Given a set of token features, we cluster them into fewer components. The next-level feature is the result of pooling current features within each cluster. Since our input tokens come from segments of an image, feature clustering turns fine-grained segments into coarse-grained regions. By repeating the procedure, we obtain a consistent fine-to-coarse (hierarchical) image segmentation and corresponding feature representations at each level of granularity.
We propose to integrate such data-driven perceptual organization into Vision Transformers (Doso- vitskiy et al.] 2020). We develop Concurrent recognition and segmentation with Adaptive Segment Tokens CAST). has three novel aspects (Fig. 2). 1) We use adaptive segment tokens instead of fixed-shape patch tokens. They no longer live on regular grid, and their shapes and numbers vary with the image. 2) We create a token hicrarchy by inserting graph pooling between transformer blocks, naturally producing consistent multi-scale segmentations while increasing the segment size
2

---

Under review as a conference paper at ICLR 2023
1) CNN
2) VIT
3) Token Pooling
4) Our CAST
models
CNN
ViT
Token Pooling
Our CAST
modes
patches
patches
patches
segments
edges
local fixed
global / dynamic
global dynamic
global dynamic
coarsening
X
segmentation
x
x
X
Figure 2: Our model bootstraps from low-level segment tokens, coarsens visual information by clustering, and merges fine-graine segment tokens into coarser-grainct region tokens. Top) We compare different models what they operate on and how they extract local-to-global information. CNN (Krizhevsky al. 2012 He et al. (016) computes features global information by spatial downsampling. VIT (Dosovitskiy et al.
regular grid and handles more 2020) takes regularly shaped 2017). Token Pooling (Marin et al. 2021) subsamples tokens by their significance scores. Our CAST takes segment tokens as inputs and coarsens them into larger region tokens, which adapt to the image. Bottom) We compare models from a graph perspective, in terms of nodes, edges (connections between nodes), and whether the graph coarsening is used and segmentation is produced. Our method is the only one that uses adaptive segment tokens with coarsening and outputs segmentations.
patches inputs, and updates features using attention (Vaswani et al.
and reducing the number of tokens. 3) We learn segmentation for free while training the model for unsupervised recognition by maximizing image-wise discrimination (Chen al. 202T). Neither recognition nor segmentation requires any labeled supervision.
Our experimental results demonstrate our superior computational efficiency and segmentation ac- curacy on ImageNet and PASCAL VOC. More importantly, our model delivers far more precise foreground masks which can be very useful wide range of dense pixel applications.
In short, our work makes three major contributions. 1) that can concurrently achieve image-wiss recognition and hierarchical image segmentation without any additional processing. 2) We outperform existing token coarsening methods for both image classification and segmentation tasks. We achieve a better trade-off berween model efficiency and task performance. 3) We deliver better attention maps that capture foreground semantics without supervision, with many potential applications beyond segmentation.
We develop the first vision transformer model
2 RELATED WORKS
Vision Transformers. Vision transformers (ViT) (Dosovisskiy et al. 2020) and its followups (Tou-
vron et al. 202T Yuan al. 202T Wang al. 202 adopt the transtormer architecture originally for NLP proposed in Vaswani et al. (2017). ViT achieves remarking performance on image classifica- tion (Deng et al. 2009). however. high computation costs limit applications. Its computational complexity is derived from two factors: the latent feature dimensions and the number of tokens.
To reduce the latent feature computation, one direction to restrict the attention connections by leveraging spatial relationships in the data (Parmar et al.) 2018 Ramachandran et al. 2019 [Blltayy
ct al. 2020 Child et al.) 2019 Zaheer et al. (2020) or utilizing hashing, sorting, or compression (Kitaey
et al. 2020; Vyas al. 2020 Tay et al. 2020 Liu et al. 2018 Wang et al. 2020). To reduce the number of tokens, two camps of approaches are proposed. The first is to apply the concepts of hierarchical convolutional neural nets to downsample tokens using various pooling methods (Liu
ct al.] 202TT: Heo et al., [202]] Dong et al.] (002).. The other attempts to measure the significance
among the tokens and drop prune tokens accordingly (Goyal al. 2020 Rao al. 2021
3

---

Under review as a conference paper at ICLR 2023
Marin et al. 2021). This camp is the most related to our work. Our work differs in that tokens not discarded but merged into coarser ones.
TCFormer (Zeng et al. 2022) and Group ViT (Xu et al. 2022) are two recently proposed hierarchical vision transformers. Both models still use patch tokens. TCFormer is only applied to supervised tasks, and Group ViT requires text supervision. Our work operates on adaptive segment tokens, which naturally induce hierarchical image segmentations. Our model does not require any human label.
Superpixels. Superpixels are referred to as sets of locally connected pixels that contain coherent structures (e.g. colors) (Ren & Malik] 2003). Superpixels have been applied to densely labeling
tasks, such body-part parsing (Mori et al. 2004), saliency detection (Ren et al.! 2013). image segmentation (Gould et al., 2008 Fulkerson et al., 2009; [harma et al. 2004 Gadde et al., 22166).
and hierarchical segmentation (Wei al.] 2018). Recently, Zhhnng et al. (2022) tackle semantic segmentation by replacing patch with superpixel tokens in ViT architectures. In contrast, our model further creates segment hierarchy and performs both classification and segmentation concurrently.
Image Segmentation and Clustering. Image segmentation is referred to as partitioning an image into coherent regions. Classic methods have two steps: extracting local features and clustering them based on different criteria, e.g., mode-finding (Comaniciu & Meer) 2002; Banerjee et al., 2005). or graph partitioning (Felzenszwalb & Huttenlocher] 2004| [Shi & Malik
Malik et al. 2001
Yu & Shi 2003a; 2004). A hierarchical segmentation is predicted
output for comparing against human perception (Arbelacz et al. 2010). The common approaches, to avoid ambiguities along object
boundaries, typically resort to contour detection (Hwang & |015 Xie & Tu 2015) and eliminate 2015
contours iteratively to form multi-scale segmentations (Arbelaez et al. 2010). Such approaches train on the finest level of ground-truth segmentation and hope to produce coarser levels of segmentation automatically for inference. Our work operates directly on segments as opposed to contour proxies.
Concurrent Recognition and Segmentation. This idea was explored before the deep learning era: Recognition by grouping compatible patches and segmentation by grouping visually similar pixels are solved together through detected pixel-patch relations, resulting object-specific segmentation (Yu et al. 2002. Yu & Shi 2003b) and figure-ground segmentation (Mare, 2010 Maire et al. 201T). These methods rely on not only handcrafted features and grouping cues, but also pre-trained object part detectors, whereas our work does not use any such priors or supervised training.
Self-supervised Segmentation and Representation Learning. Recent works can be categorized into three camps. 1) A straightforward approach is to leverage self- self-supervised image recognition and transfer the model to segmentation by increasing the location sensitivity (Wu et al., 2018 HH
iet al.
2020 Chen et al.) 2020 Wang et al.) 2021c), adding contrastive loss across views (Wang ctt al.
2021b), or by stronger augmentation and constrained cropping (Van Gansbeke et al. 202
Selvaraju et al. 2021). 2) A pixel-wise cluster predictor can be learned by maximizing the mutual information between cluster predictions on augmented views of the same instance at corresponding pixels (Ji et al. 2019 Ouali et al. 2020). 3) A pixel-level feature encoder can be learned directly by maximizing discrimination between pixels based on either contour-induced segments (Hwang al. 2019), pre-computed region hierarchies (Zhang & Maire| 2020), hierarchical clustering transformers (Ke et al., 2022). Segmentation is thus derived from pixel feature similarities. Our work unifies the first and the third camp, we train ViT with celf-super image recognition framework while naturally producing unsupervised hierarchical segmentation.
CONCURRENT RECOGNITION AND HIERARCHICAL SEGMENTATION
At the core of our method is to consider image recognition and segmentation as concurrent, not separate, tasks. The basic idea is to extract local-to-global visual information by producing fine-to- coarse image segmentations and corresponding feature representations. Meanwhile, such multi-scale feature representations should support final image-level recognition.
We ground our approach on a general perspective: images as graphs where pixels are nodes. We first generate low-level superpixels given input image and extract corresponding segment tokens. Then we integrate image segmentation into the ViT architecture, where transformer blocks and our proposed graph pooling modules take arbitrarily shaped segment, not fixed-shape patch, tokens as inputs and outputs coarsened segment tokens for the next transformer block. In other words, we augment ViT to cluster fine-grat fine-grained segments into coarse-grained regions w.r.t group-wise correlation
4

---

Under review as a conference paper at ICLR 2023
Hierarchical Segmentation
Superpoe Superpixels
Ented
Feature
Image
Position Encoding Epor
Figure 3: Our CAST jointly produces 1) a hierarchical image segmentation and corresponding multi-level features for an input image. Building on VIT, our model operates on segment, not patch, tokens. We oversegment the image concatenate with [CLASS] token, and sum together with position encoding Epos as inputs to subsequent transformer blocks. Followed by a graph pooling module, we group segments into coarser regions and aggregate features within each group. Let Pt indicate how fine segments map to coarse regions at level The coarsened tokens become the inputs next-level encoder blocks. Repeating the procedure, generate hierarchical image segmentation and multi-level features.
superpixels, and extract initial segment tokens x,. We
in the token feature space, resulting in hierarchical segmentation. By doing so, our model performs classification and hierarchical segmentation concurrently and more efficiently.
We describe the three components in our model in each subsection. 1) Oversegmentations of input images based on low-level visual cues. 2) Transformer encoder blocks to update token features. 3) Graph pooling modules to cluster segment tokens and generate next-level representations. See Fig.
3.1 THE FINEST-LEVEL PIXEL GROUPINGS AND TOKEN FEATURES
Existing methods tackle image segmentation separately from feature extraction. Such models extract patch features from the input image and then generate after- the-fact image segmentation by clustering the fixed patch features (similar to SegSort (Hwang al., 2019) and HSG (Ke al. 2022)). In stark contrast, our core idea is to involve image segmentation in feature extraction in the model architecture. We derive pixel groupings at early stage, where extract corresponding features for every segment. Our model carries such segment features to the subsequent layers and obtains image segmentations directly from the segment index of each pixel. Post-processing is thus needed.
Given an input image, we start with the finest-level pixel groupings. We perform pixel groupings based on low-level visual cues align segmentations with image contours. Specifically, apply oversegmentation methods, e.g. Seeds (Bergh et al.) 2012). to partition an image into locally connected and color- wise coherent eggions-superpiin image segmentation with precisely localized contours by progressively grouping these superpixels.
Detailed in section 3.3 we can produce hierarchical
To extract features of the superpixels, we first convolve the image with multiple convolutional layers, resulting in overlapping patch features. We then average pool those patch features within each superpixel to derive initial segment tokens x, (with dimensions of the number of superpixels and number of feature channels).
We then input the initial segment tokens x,, along with additional priors to our model. Following ViT (Dosovitskiy et al.] 2020), we append with learnable embedding ([CLASS] token Xclass) ) to encode the most prominent features an image. Xelass randomly initialized and shared among different input images. We also enforce spatial prior by adding x, with relative position encodings Epas- initiate Epos at the same resolution as the convolutional patch features and then average pool within each superpixel. To sum up, the input segment token is 2p X x3) + E Epos-
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: GraphPool
Algorithm 2: Overall framework
Input: Feature and number of clusters
Input: Initial segment token xa, class token Xelases
Output: Coarsened feature z and assignments P
position encoding Epos and grouping steps
/* Sample centroids.
*/
Output: Feature felass and feeg
Centroid indices 5 <- [2)
/a Input tokens with priors. +/
/* Refine features to encode correlation.
Zo Xalassi X.) + Epos
/+ ViT with Graph Pooling. */
Refined feature u t (2)) +
for = 1.... do
Normalized feature 11 4- 11 mean(1) (u) + bias
2: t ViI Eucodeerrr()))))
Centroid feature v <- {u,li E S)
if E A then
/x Assign new clustering.
f- GraphPool());
P + soflmax(( ||u [v]
uv
else end
i1:
/ * Output new features.
+/
Average pooled feature - PT 2/P
end
New centroid feature z + [2.|0 E S}
/* Outputs for classification. */
Updated centroid feature
FC(LN(Z))
[CLASS] token faius +- LN(ZZ))
FPS: Farthest Point Sampling.
/a Outputs for segmentation. ./
MSA: Multi-headed -ele-Atttn
Multi-level segment tokens
FC: Fully Connected Layer.
feeg + P) TeA
LN: Layer Norm. BN: Batch Norm.
3.2 GENERAL AND GLOBALIZED BACKBONE ARCHITECTURES
Most existing vision models digest pixels on regular grid and update features within a limited range of neighboring grids. Such methods have two limitations: 1) all features correspond to the same-shape patches in images, and 2) pixels/patches are locally connected and have little knowledge of global correlation. Yet, segmentations are adaptive to images: every segment has different shapes. Optimal segmentation also requires group-wise correlation among pixels (Shi & Malik, 2000). We thus prefer globalized architectures, such as ViT and GNN (Kipf & Welling 2016), over the grid-based models.
Specifically, select ViT as the backbone, which updates features in the context of all inputs without the need for regular grid. That is, ViT computes features from a cooneely-comeetted graph. It consists of multi-headed self-attention modules (MSA) /Vaswani et al.) 2017). where features are updated according to pair-wise correlation among all the input features. As result, ViT encodes global correlation and allows inputs to have different shapes, which ideal to enable optimal segmentation.
Building upon VIT, our model contains multiple encoder blocks that take segment tokens as inputs: level 1, the encoder block updates its inputs 1 to 2, Notably, to extract information different scales, we vary the shapes and sizes of segment tokens throughout the model. Coarser segmenta- tions have fewer segments (tokens), which improves our model efficiency. For the first block, Zo corresponds to superpixels. For all the other blocks, 2 corresponds to segments at different scales.
3.3 HIERARCHICAL GROUPINGS OF SEGMENT TOKENS
Starting with superpixels, we group fine-grained segment tokens into coarse-grained region tokens to obtain more global visual information. With fine-to-coarse segment groupings, we can directly induce hierarchical image segmentations of an input image. As we map superpixels into token features, hierarchical segment groupings become multi-scale feature clustering and pooling problem. We have two considerations: 1) The number of coarser segments (tokens) should be freely adjustable during training and inference. 2) The model should achieve optimal partitioning of inputs. We summarize our feature clustering algorithm, dubbed Graph Pooling Module, in Alg. [
Existing methods (Xu et al. 2022 Ke et al., 2022) perform clustering with a set of learnable embeddings, resulting in fixed segmentation granularity. Yet, the optimal number of segments varies with image scales: larger (smaller) images require more (fewer) segments. Instead, we conduct feature clustering with arbitrary number of centroids that are initialized by sampled inputs. The number of centroids corresponds to the segmentation granularity. We apply the Farthest Point Sampling (FPS) algorithm (Qi et al. 2017) to select subset of token features initial centroids. The FPS algorithm

---

Under review as a conference paper at ICLR 2023
enables the sampled centroids to cover the imput feature distributions, unbiased w.r.t dominant features. We can set the number of clusters flexibly during training and inference.
In particular, we predict the soft clustering assignments P. which indicates how input features assigned to sampled centroids. We calculate Pt as softmax- netmax-norrmaized pair-wise similarity among inputs and sampled centroids. We then generate coarsened tokens by weighted-average aggregating within a cluster. For detailed computation, please refer to Alg. T
3.4 TRAINING AND INFERENCE
We summarize our overall framework in Alg. 2 We conduct segment grouping at certain levels in the model. We train our CAST using an image-wise contrastive learning framework-Moco-v2 (Chen al. 2.2T The objective is to contrast each image against others in the latent feature space.
To predict classification, we follow MoCo-v3 to apply a 3-layer Multil Perceptron (MLP) head on output [CLASS] token. To predict segmentation, we fuse multi-level features as outputs (Hariharan) et al. 2015 and unpool higher-level features based on the grouping index w.r.t superpixels.
EXPERIMENTS
Datasets. 1) ImageNet (Deng et al. 2009) is an object-centric image classification dataset, annotated with 1, 000 object categories -N-riees--N-KK.... IN- Each image is labeled with object category, and objects mostly locate at the image center. The training and validation set includes 1.28M and 50K images, respectively. We follow Tian al. (2020) to subsample 100 object categories to create IN-100. The subset consists of 127K and 5K images for training and testing. 2) Pascal VOC 2012 (Everingham al., 2010) an object-centric semantic segmentation dataset that contains 20 object categories and a background class. We use the augmented training set (Hariharan et al., (2011) with 10, 582 images and the validation set with 1, 449 images. 3) MSCOCO (Lin et al. (0144) dataset. The scene contexts are more complex and include more objects in each image (7.3 vs. 2.3) than VOC. Following Van Gansbeke al. (202T), we train on 118, 287 images of train2017 split and test on the VOC validation set.
generie scene parsing
Model architecture. For most experiments, we base our architecture on ViT-S (Dosovitskiy et al. 2020), which has 384 channel dimensions of all encoder blocks. We follow Xiao et al. (2021) to 1) replace the patch-wise linear projection layer with a stack of four 3 x 3 and one 1 x convolutional layers; 2) use 11, not 12, encoder blocks to maintain similar model capacity. We adopt the same design choice for our vanilla ViT baselines. For vanilla ViT (our CAST), we set stride to among the first four (three) convolutional layers. Our models aggregate pixel features within superpixels, resulting in similar number of input tokens as ViT baselines. We set graph pooling step A = {3,6, 6, and 1 of initial inputs.
reduce the number of tokens to 3 E T2
K-Medoids clustering baselines. There is no released code for Token Pooling Marin et al. 2021) We thus re-implement the method by combining POWER-BERT (Goyal et al.) 2020) with K-Medoids clustering algorithm. We follow the same settings our CAST: we adopt segment tokens and use the same number of tokens at intermediate layers as our CAST.
Image resolution and token numbers. For training on all datasets and testing on ImageNet, we set crop_size to 224 and partition 196 superpixels from an image, resulting in around 196 input tokens. For testing on VOC, we generate 1024 (384) superpixels from 512 x 512 input image, resulting in 1024 (384) input tokens for semantic (figure-grouml segmentation. For vanilla ViT, we adopt the same image resolution and use 196 and 1024 input tokens on ImageNet and VOC.
Model training. Based on MoCo, we train all models from scratch without any human-labeled supervision. We mostly follow MoCo to set up the hyper-parameters. See Appendix for more details.
Model testing. 1) On ImageNet, we follow MoCo to apply linear probing to evaluate model performance. We freeze the trained model weights and replace the 3-layer MLP head with a randomly initialized linear projection layer as the classifier. We train the classifier with ground-truth labels. 2) On VOC, we follow Van Gansbeke al. (2021) to predict semantic segmentation via nearest neighbor search from the labeled VOC training set. We also evaluate performance by fine-tuning models the training set and testing on the validation set. 3) On figure-ground segmentation,
7

---

Under review as a conference paper at ICLR 2023
following DINO Caron et al. 2021). binarize multi- head attention maps and select the best binary segmentation among all attention maps for each image. See Appendix for more details.
backbone
method
# tokens
GFLOPS
IN-100
IN-Ik
83
Top-1 Acouracy
Vanilla
[196] 4
4.67
78.1
67.9
VIT-S
K-Medoids
196, 64, 32, 16
2.84
75.8
62.8
78
Our CAST
196, 64, 32, 16
3.12
78.9
66.1
Vanilla
[16] x 4 196, 62, 16
17.8
81.7
-V-VIT
CAST
VIT-B
K-Medoids
10.8
81.6
73
Our CAST
196, 64, 32, 16
11.7
82.0
0
10 GFLOPS
20
Table 1: Our model achieves better trade-off between model efficiency and task performance for unsupervised image classification on ImageNet val set. We report the top-1 accuracy of the linear classifier. Left: Image classification on IN-100 and IN-1k. Compared to vanilla ViT, our CAST reduces computation overhead by decreasing the number of tokens in the intermediate blocks. Right: Image classification on IN-100 val set with different model sizes. Our CAST outperforms vanilla ViT using the same computational budget.
Better accuracy on ImageNet classification. Our model achieves better trade-off between model efficiency and task performance for unsupervised image classification. We report top-1 accuracy on both IN-1k and IN-100 (see the left of Table D. On IN-100, using ViT-S and ViT-B backbone, our CAST achieves comparable performance as vanilla ViT, yet our model is 66.8% and 65.7% computationally more efficient. On IN-1k, our CAST maintains 97.3% performance of ViT baseline (66.1% 1% vs. 67.9% accuracy). Our graph pooling module consistently outperforms K-Medoids clustering. Due the computation limitations, we smaller batch size and training epochs on IN-IK. We also do not apply the pre-training (Marin et al., 2021) or model distillation (Rao
Result 1:
al.
2021 strategy. We can further improve the performance based on such settings.
On IN- 100, our model outperforms baselines under different model sizes (see the right plot of Table T
Our CAST outperforms vanilla ViT using the same computational budget.
image
VIT
CAST
Trained on MSCOCO, to be tuned on VOC
before
after
method
# tokens
GFLOPS
ImloU F-score mloU F-score
Vanilla VIT
[1024] x 4
34.1
30.9
16.1
65.8
40.7
K-Medoids 320, 160, 80
21.2
27.6
17.2
66.7
47.5
Our CAST
1024,320, 160, 80
23.6
35.9
24.7
66.8
48.1
Table 2: Our predicted segmentations are much more precise and better aligned with ground-truie (row 2 column image) semantic boundaries on VOC set, benchmarked with the regional mloU and boundary F-score metric. Segmentations are predicted based on segment-wise nearest neighbor retrievals (before fine-tuning, white-colored table columns, row 1 images) and fine-tuned models (after tuning with supervision, gray-colored table columns, row 2 images). Our model achieves better image segmentation with less computation.
Result 2: Better semantic segmentation on VOC. We evaluate unsupervised semantic segmentation on VOC in Table 2 We report the regional mean Intersection of Union (mloU) and boundary F- measure metric. For both segment retrieval and transfer learning, our segmentations are more precise (+5.0% and +1.0% mloU) and respect object boundaries better (+88.6 and +7.4% F-measure) than vanilla VIT, yet model efficiency is greatly improved. Notably, without applying CRF for post-processing, our model is still able to preserve thin structures (e.g. horse legs) well, whereas ViT results over-smoothed.
Result 3: Better figure-ground segmentation on VOC. We show that our latent attentions capture semantics more precisely. We report the Jaccard similarity between ground truths and predicted foreground masks. Summarized in Table our CAST outperforms DINO and ViT by +2.1%. Our foreground masks cover dominant foreground objects and better align with object boundaries.
8

---

Under review as a conference paper at ICLR 2023
method
loU
Our CAST
48.0
Vanilla ViT
45.8
DINO 45.9
Table 3: Our CAST attends to foreground semantics more precisely. All models are trained on IN-1K from scratch. Due to computation limitations, we train vanilla ViT and our CAST with a smaller batch size and fewer epochs. Left) Jaccard similarity between ground-truth and predicted foreground masks. Right) Visual comparison between our CAST (top row) and ViT (bottom row). Our predicted masks are more coherent within foreground regions and better align with object boundaries.
image
Vanilla ViT K-Medoids
CAST
image
Vanilla ViT K-Medoids
CAST
Figure 4: Our CAST generates higher-quality hierarchical segmentation. From left right, we show the input image and hierarchical segmentations generated by vanilla VIT, K-Medoids clustering, and our CAST. We also show the superpixels (white contours) generated from input images. From top to bottom, we hierarchically segment images into 8, 16, and 32 regions. VIT requires additional pixel-wise clustering (e.g. K-Means) on fixed features, yet the results are over-smoothed. Our CAST naturally generates hierarchical segmentations which capture semanties more precisely.
Result 4: Better hierarchical segmentation on IN-1K. Fig. ssowws that our hierarchical segmen- tations respect semantics different levels of granularity better than VIT and K-Medoids baselines. Our model directly produces hierarchical segmentations without needing additional clustering. Boot- strapping from superpixels, our segmentations (e.g. dog regions) align with image contours more precisely than vanilla ViT. Compared to K-Medoids clustering, our model maps similar-semantic fine segments into the same coarse regions (e.g. street-sign pole regions) more consistently..
Summary. We develop novel vision transformer that performs image-wise recognition atop of consistent hierarchical image segmentation, by learning fine-to-coarse features over adaptive segment tokens instead of regular patch tokens. We deliver the first concurrent recognition and hierarchical segmentation model without any supervision, achieving better accuracy and efficiency. The idea can be extended to supervised image classification, with hierarchical semantic segmentation for free.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7297-7306, 2018.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. JEEE transactions pattern analysis and machine intelligence, 33(5):898-99) 916, 2010.
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(Sep): 1345-13822 2005.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprini 05X1v:2004.05/50, 2020.
Michael Van den Bergh, Xavier Boix, Gemma Roig, Benjamin de Capitani, and Luc Van Gool. Seeds: Superpixels extracted via energy-driven sampling. In European conference on computer vision, pp. 13-26. Springer, 2012.
Irving Biederman. Recognition-by-comm bo-components: theory of human image understanding. Psycholog- ical review, 94(2):1 15, 1987.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IFFF/CVF International Conference on Computer Vixion, pp. 9650-9660, 2021.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. arXiv preprint arXiv:1606.00915. 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904. 10509, 2019.
Dorin Comaniciu and Peter Meer. PAMI, 2002.
Mean shift: A robust approach toward feature space analysis.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped win- dows. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12124-12134, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Mark Everingham, Luc Van Gool, Christopher KI Williams. John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
10

---

Under review as a conference paper at ICLR 2023
Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-basee based image segmentation. IJCV, 2004.
Brian Fulkerson, Andrea Vedaldi, and Stefano Soatto. Class segmentation and object localization with superpixel neighborhoods. In 2009 IEEE 12th international conference on computer vision, pp. 670-677. IEEE, 2009.
Raghudeep Gadde, Varun Jampani, Martin Kiefel, Daniel Kappler, and Peter V Gehler. Superpixel convolutional networks using bilateral inceptions. In ECCV, 2016.
Stephen Gould, Jim Rodgers, David Cohen, Gal Elidan, and Daphne Koller. Multi-class segmentation with relative location prior. International journal of computer vision, 80(3):300-118 2008.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sab- harwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference Machine Learning, pp. 3690-3699, PMLR, 2020.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 2011 International Conference on Computer Vision, pp. 991-998. IEEE, 2011.
Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns for object segmentation and fine-grained localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 447-456, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11936-11945, 2021.
Jyh-Jing Hwang and Tyng-Luh Liu. Pixel-wise deep learning for contour detection. arXiv preprint 150400504 2015.
Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chich Chen. Segsort: Segmentation by discriminative sorting of segments. ICCV, 2019.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Tsung-Was Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella X Yu. Unsupervised hierarchical semantic segmentation with multiview cosegmentation and clustering transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xi::1412.6990 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. 2016.
arXiv preprint
Nikita Kitaev. Lukasz Kaiser. and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445/ 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.

---

Under review as a conference paper at ICLR 2023
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint 1801. 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
Michael Maire. Simultaneous segmentation and figure/ground organization using angular embedding. In European Conference on Computer Vision, pp. 450-464. Springer, 2010.
Michael Maire, Stella X. Yu, and Pietro Perona. Object detection and segmentation from joint embedding of parts and pixels. In 2011 International Conference Computer Vision, pp. 2142- 2149. IEEE, 2011.
Jitendra Malik, Serge Belongie, Thomas Leung, and Jianbo Shi. Contour and texture analysis for image segmentation. IJCV, 2001.
Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers. arXiv preprint arXi::2 2021.
D. Martin, C. Fowlkes, D. Tal, and Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2, pp. 416-423, July 2001.
Greg Mori, Xiaofeng Ren, Alexei A Efros, and Jitendra Malik. Recovering human body configu- rations: Combining segmentation and recognition. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. II-II. IEEE, 2004.
Yassine Ouali, Celine Hudelot, and Myriam Tami. Autoregressive unsupervised image segmentation. In Proceedings of the European Conference Computer Vision (ECCV), August 2020.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pp. 4055-4064, PMLR, 2018
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. Advances in Neural Information Processing Systems, 32, 2019.
Yongming Rao. Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34, 2021.
Xiaofeng Ren and Jitendra Malik. Learning a classification model for segmentation. In Computer Vision, IEEE International Conference on, volume pp. 10-10. IEFE Computer Society, 2003.
12

---

Under review as a conference paper at ICLR 2023
Zhixiang Ren, Shenghua Gao, Liang-Tien Chia, and Ivor Wai-Hung Tsang. Region- Region-based saliency detection and its upplication object recognition. IEEE Transactions on Circuits and Systems for Video Technology, 24(5):769-779, 2013.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-611-66 626, 2017.
Ramprasaath R Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. Casting your model: Learning to localize improves self-supervised representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pp. 1058--1067, 2021.
Abhishek Sharma, Oncel Tuzel, and Ming- Yu Liu. Recursive context propagation network for semantic scene labeling. Advances in Neural Information Processing Systems, 27, 2014.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888-905 2000.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning, pp. 9438-9447, PMLR, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In European conference on computer vision, pp. 776-794. Springer, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Revis- iting contrastive methods for unsupervised learning of visual representations. arXiv preprint arXiv:2106.05967, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Apoory Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33:21665-21674 2020.
Sinong Wang. Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- celf-attention with linear complexity. arXiv preprint arXiv:2006.04700 2020.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 568-578, 2021a.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033, 2021b.
Xudong Wang, Ziwei Liu, and Stella X Yu. Unsupervised feature learning by cross-level instance- group diserimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12586 12595, 2021c.
Xing Wei, Qingxiong Yang, Yihong Gong, Narendra Ahuja, and Ming-Hsuan Yang. Superpixel hierarchy. IEEE Transactions on Image Processing, 27(10):4838-44 2018.
Andrew Witkin and Jay M Tenenbaum. On the role of structure vision. In Human and machine vision, pp. 481-543. Elsevier, 1983.
13

---

Under review as a conference paper at ICLR 2023
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. arXiv preprint arXiv:1805.1193, 2018.
Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. Early convolutions help transformers better. Advances in Neural Information Processing Systems, 34, 2021.
Saining Xie and Zhuowen Tu. Holistically- edge detection. In ICCV, 2015.
Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 18134-1811 18144, 2022.
Stella X. Yu and Jianbo Shi. Multiclass spectral clustering. In ICCV, 2003a.
Stella X. Yu and Jianbo Shi. Object-specific figure-ground grore--rounddd segregation. 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume 2, pp. II-39, IEEE, 2003b.
Stella X Yu and Jianbo Shi. Segmentation given partial grouping constraints. PAMI, 2004.
Stella X. Yu, Ralph Gross, and Jianbo Shi. Concurrent object recognition and segmentation by graph partitioning. Advances in neural information processing systems, 15, 2002.
Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokene-to-token vit: Training vision transformers from scratch on imagenet. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 558-567, 2021.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslic, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33: 33:17283-17297, 2020.
Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human-centrr visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11101-11111, 2022.
Xiao Zhang and Michael Maire. Self-supervised visual representation learning from hierarchical grouping. Advances Neural Information Processing Systems, 33, 2020.
Yifan Zhang, Bo Pang, and Cewu Lu. Semantic segmentation by early region proxy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1258-12221 2022.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
We develop the first vision transformer that concurvently achieves image-wise recognition and hierarchical image segmentation without additional processing. Our CAST outperforms existing token coarsening methods for image classification, semantic segmentation, and attention-induced figure-ground segmentation tasks. In this supplementary, we include more details as followings:
We present more visual results of hierarchical segmentation ImageNet A.1
We present more visual results of attention-induced figure-ground segmentations VOC in A.2
We present visual results attention maps from our CAST on ImageNet in A.3
We present the ablation study regarding the proposed graph pooling module in A.4
We present the ablation study inference latency of our CAST in A.5
We present the ablation study on effectiveness of superpixels in A.6
We present quantitative results of hierarchical segmentation on DensePose in A.7
We illustrate more details of hhper-parametee parameters for training in A.8
We illustrate more details of inference in A.9
image
Vanilla ViT K-Medoids
CAST
image
Vanilla VIT K-Medoids
CAST
Figure 5: Our CAST generates higher-quality hierarchical image segmentation. From left to right, we show the input image and hierarchical segmentations generated by vanilla ViT, K-Medoids clustering, and our CAST. We also show corresponding superpixels (white contours) generated from input images. From top to bottom, hierarchically segment the input image into 8, 16, and 32 regions. Vanilla ViT requires additional pixel-wise clustering (c.g. K-Means) on fixed feature representations, yet the results over-smoothed. Our CAST naturally generates hierarchical segmentations without any post-processing. Our segmentations align with image boundaries and capture semantics more precisely.
15

---

Under review as a conference paper at ICLR 2023
CAST (top) Vanilla VIT (bottom)
Figure 6: Our CAST (top row) attends to foreground semantics more precisely than vanilla VIT (bottom row) and DINO CCaron et al. 2021) on VOC. We adopt the same procedure as DINO to generate foreground segmentation masks from latent multi- attention maps. All models trained on I dataset from scratch. Our CAST and ViT are trained based on MoCo-v3 (Chen
al. 2021
A. VISUAL RESULTS ON HIERARCHICAL SEGMENTATION.
We present more visualization results of hierarchical segmentations induced by vanilla VIT, K- Medoids, and our CAST (see Fig. 5). We hierarchically segment the input image into 32, 16, and regions. Notably, Vit baseline requires additional pixel-wise K-Means clustering on fixed feature representations, yet the results are over-smoothed. Our CAST naturally generates hierarchical seg- mentations without any post-processing Our multi-scale segmentations align with image boundaries and capture semantics more precisely.
A.2 VISUAL RESULTS ON ATTENTION-INDUUS FIGURE-GROUND SEGMENTATION.
We present more visual results of figure-ground segmentations generated from multi-head attention maps on VOC. Our CAST attends foreground semantics more precisely than vanilla ViT, and the segmentations preserve object boundaries more accurately. See Fig.
A.3 VISUAL RESULTS ON MULTI-HEAD ATTENTION MAPS.
We visualize the multi-hear attention maps of the [CLASS] token to all the other segment tokens in vision transformer. As the [CLASS] token is optimized for image-wist wise discrimination. such attention maps indicate the most informative groupings of segments that will induce the most discriminative image-wisg wise representations. We visualize the same attention maps used to generate
16

---

Under review as a conference paper at ICLR 2023
Figure 7: Our multi-head attention maps reveal parts-of-the-whole information of the image on IN-100. From left to right: input images and corresponding 12 heads of attention maps of the [CLASS] token to all the other segments. We follow DINO (Caron et al. 202T) to binarize attention maps. We show that the same object parts are together attended the same head, e.g. face vs. ears vs. nose of the dog. Our model takes segment tokens, resulting in attention maps better aligned with object boundaries.
the figure-ground segmentation, which are the ones the 9th transformer encoder block. The layer takes 32 coarsened segment tokens inputs, resulting in 12 heads of 32 x 32 attention maps. We follow the same procedure as DINO (Caron et al., (202)) to display the binarized attention maps. The threshold is adjusted to keep 60% of the mass. See Caron et al. (2021) for more details.
As shown in Fig. 7 our attention maps reveal parts-of-the-whorl information of the image. We observe that the same object parts are together attended in the same attention head, e.g. face vs. ears vs. nose of the dog. It indicates that image-wise recognition requires oitiss-fffhh-whooo information. Additionally, model carries segment, not patch, tokens through the layers, resulting in attention maps better aligned with object boundaries.
Pooling
Acc.
Token
Pooling
Acc.
Our Graph Pooling
78.9
Patch
78. 1
K-Medoids: POWER-BERT
75.8
Segment
78. 1
K-Means: POWER-BERT
73.9
Segment
/
78.9
K-Medoids: Random Sampling
72.3
Table Our proposed graph pooling module and loss regularizations improve image classification on IN-100 val set. Left: Improved performance by adding each loss regularization. Right: Our graph pooling module outperforms K-Medoids and K-Means clustering algorithm by large margin. Cluster centroids are initiated by either BAWER-BERR random sampling.
17

---

Under review as a conference paper at ICLR 2023
A.4 ABLATION STUDY ON PROPOSED GRAPH POOLING MODULE.
Summarized in Table we demonstrate the efficacy of the proposed graph pooling module. We report top-1 accuracy results on IN-100 dataset. We show that our graph pooling module outperforms K-Medoids and K-Means clustering by large margin.
#. of Tokens
Encoder Blocks
GraphPool (FPS)
196
86.43
63.02 (37.64)
64
25,4
18.2 (9.7)
32
12.9
9.6 (3.0)
16
5
6.1 (1.5)
Table FPS in our graph pool module requires additional computation. report the inference time (ms) of each module with 384 channel dimensions and 256 batch sizes on IN-100. Optimizing the token sampling technique to increase model efficiency is our future work.
A.5 ABLATION STUDY ON INFERENCE LATENCY
We present the comparison of inference latency among our CAST and vanilla ViT architectures. We report the inference time (ms) of each module with 384 channel dimensions and 256 batch sizes on ImageNet- 100. As summarized in Table our graph pool module with FPS indeed requires additional computation. However, our method can also reduce inference time by decreasing the number of tokens in deeper layers.
Disregarding the Conv Stem, vanilla VIT and our CAST take 316.91 and 220.57 ms for inference, respectively. Our model 30.4% faster than ViT. Optimizing the token sampling technique to increase model efficiency is our future work.
A.6 ABLATION STUDY ON SUPERPIXEL
We next verify the superior efficacy of superpixel tokens dense pixel applications. As shown in Table [6] we compare patch tokens to superpixel tokens on image classification and semantic segmentation tasks. For both VIT baseline and our CAST, we observe significant performance gain for semantic segmentation, yet the performance gap for classification is negligible. We conclude that using superpixels can be very useful in wide range of dense pixel labeling tasks.
A.7 QUANTITATIVE RESULTS ON HIFRARCHICAL SEGMENTATION
To demonstrate the efficacy of our hierarchical segmentation results, compare CAST with K- Medoids on the DensePose dataset (Alp Güilrr et al. 2018). We process body-part-level labels into the head, torso, upper limb, and lower limb regions. We segment an image into 64, 32, and regions, and evaluate the region-win wise coverings (Arbelaez et al. 2010) with ground truths by F1-score. As shown Fig. X our CAST outperforms K-Medoids and is better at capturing semantics at every level of granularity.
A.8 HYPER-PARAMETERS FOR TRAINING
We list the hyper parameparseeters for training in Table Mostly follow MoCo, we set batch_ size to 256. learning rate to 1.5e-4 4, weight_ to 0.1, and momentum to 0.9. use AdamW (Kingma & Ba 2014) optimizer. For hyper-parameters of MoCo framework, we set temperature to 0.2, output dimension to 256, momentum coefficients to 0.99. The 3-layer MLP head has a hidden dimension of 4096. For IN-100, IN- 1k and COCO, set training epochs to 200. 100 and 400, along with cramupp_ppctee to 20, 10 and 40, respectively. Cosine decay schedule applied to adjust the learning rate.
18

---

Under review as a conference paper at ICLR 2023
image
ground-truth
Vanilla ViT
CAST (patch)
CAST
Lak
Classification
Segmentation
Method
GFLOPS
dim
Token
Acc.
mIoU
f-score
Patch
78.1
65.8
40.7
Vanilla ViT
65.4
384
Segment
78.1
66.5
46.7
Patch
78.1
66.3
41.4
Our CAST
42.7
384
Segment
78.9
66.8
48.1
Table 6: Using superpixel tokens improves the performance of dense pixel applications significantly. Visual examples of semantic segmentation by fine-tuned models on VOC val set. Bottom: Quantitative results on classification and semantic segmentation tasks. We compare patch tokens to superpixel tokens on image classification (IN-100) and semantic segmentation (VOC). We report the mloU and boundary f-score performance of semantic segmentation under the setting of transfer learning. We report the GFLOPS of segmentation models, where classification models use the same number of channel dimensions. For vanilla VIT and our CAST, observe significant performance gain for semantic segmentation, though the performance gap for classification negligible. In particular, superpixel-basee methods preserve thin structures (boundaries) of objects much better than their patch- based counterparts.
Top:
A.9 INFERENCE AND TESTING
For image classification, apply the linear probing procedure for evaluation. For semantic segmen- tation, we use the segment retrieval and transfer learning procedure to evaluate model performance.
Image classification: linear probing. We follow MoCo-v3 (Chen et al. 2021) to evaluate image- wise discrimination model performance using a linear probing protocol. We freeze the trained model weights and replace the 3-layer MLP head with randomly initialized linear projection layer as classifier. We train the linear classifier with ground -ound-truth labels and report the top-1 accuracy. Following Chen et al. (202T), we train the linear classifier with 90 epochs ImageNet dataset. We
19

---

Under review as a conference paper at ICLR 2023
30
K-Medoids
CAST
F1-Score
20
10
64
Number of Regions
Figure 8: Our CAST delivers better hierarchical segmentation than K-Medoids clustering algorithm. On DensePose (Alp Güler al. 2018). we parse oody-part-level labels into head, torso, upper and lower limb regions. Using CAST or K K-Medoids, we partition an image into 64, 32 and 16 regions, and evaluate the region-wise converings with ground-truths by F1-score. Our CAST captures semantics better at every level of granularity.
Parameter
IN-100
IN-1K
COCO
batch_size
256
256
256
learning_ran rate
1.5e
1.5e
1.5e
weight_decay
0.1
0.1
0.1
mmmentun
0.9
0.9
0.9
total epochs
200
100
400
warmup_epoon epochs
20
10
40
optimizer
Adam
Adam Cosine decay
Adam
learning_rate_poliey
Cosine decay
Cosine decay
MOCO temperature
0.2
0.2
0.2
MOCO output_diint dimension
256
256
256
MOCO momentum
carrenttumcofffciintsss coofficients
0.99
0.99
0.99
MOCO MLP hidden dimension
4096
4096
4096
Table 7: Hyper-paramede for training CAST, K-Medoids clustering, and vanillar VIT IN-100, IN-IK, and COCO dataset. We follow mostly the same set up as MoCo.
momentum to 0.9 and weight decay to 0 for all experiments. On IN-1k, set batch size to 1024, learning_rate rate to 30; on IN-100, we set batch_size to 256, learning_rate rate to 0.8. SGD is used as the optimizer.
Semantic segmentation: segment retrieval. We follow Hwang et al. (2019); Van Gansbeke et al. (202T); Ke et al. (2022) to evaluate semantic segmentation using segment retrieval. We partition an image into several segments and conduct nearest neighbor search to predict the label for each segment. We assign the majority labels from the 20 retrieved segments.
For ViT baselines, apply the MLP head on each token to generate unit-length output features and upsample the feature maps to the original resolution the input image. Followed by spherical K-Means clustering algorithm, partition the image into 36 segments using the output features.
Our CAST does not require additional upsampling and K-Means clustering, For segmentation, our model follows Hypercolumn design (Hariharan et al. 2015) unpool and fuse multi-level segment tokens. Our model generates the same number of output tokens as the superpixels. We gather pixel features from output tokens based the superpixel index. Without the need for spherical K-Means
20

---

Under review as a conference paper at ICLR 2023
clustering, CAST predicts image segmentation using the graph pooling modules. We compute normalized segment features according to such image segmentation.
Semantic segmentation: transfer learning. We follow Van Gansbeke et al. (2021 to evaluate model performance using transfer learning protocol. All models are unsupervisedly trained on MSCOCO, and supervisedly fine-tuned Pascal VOC. We replace the 3-layer MLP head with 2-layer 1 x 1 convolutional layers. We set the training steps to 30K, learning_ rate to 0.003, weight_ 0.0001, batch_sizze size 16, crop_size size 512. Following Chen al. (2016), we adopt poly learning iter
to
rate policy by multiplying base learning rate by maz_iter use only a single-scale image for inference.
0.9
We adopt the SGD optimizer. We
21


Reviews:
[{"review_id": "fNJ1qFoRhnI", "reviewer": "Reviewer_Jmwj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to replace the standard rectangular patch based tokens in transformers with superpixel based regions. Features for each superpixel are pooled to a token and these are used in a hierarchical transformer architecture. When reducing the number of tokens (GraphPool), furthest point sampling is used to select a subset of tokens and all the tokens from the previous layers are grouped to the sampled tokens with the highest similarity. The complete network is trained in an unsupervised fashion, meaning that a hierarchical segmentation can be extracted from the network without any labels. Using some additional supervision on top of this results in an approach that can perform semantic segmentation in a hierarchical way. The overall idea to modify a transformer architecture to obtain a hierarchical segmentation without the need for direct segmentation supervision is very interesting.", "summary_of_the_review": "Overall I like the core concept automatically learning a hierarchical segmentation without the need for labels, however, I have some doubts about how the model really works and how well it works compared to external baselines. I think adding some more experiments regarding these issues and giving the paper another good polishing will likely make it a lot better. As such I'm a little unsure whether to accept this or not. Depending on the other reviews and the rebuttal, I'm willing to change my opinion more towards the positive side though.", "strength_and_weaknesses": "Strengths:\n- The paper proposes an interesting architecture idea and  shows that it has some merit to it with quite some experiments. Especially, the Graph Pooling module seems to work fairly well in their instantiation compared to other baseline approaches.\n- Especially that this can be trained without labels makes it rather interesting.\n\nWeaknesses:\n- I'm mainly concerned about the training of the proposed model. Specifically the GraphPool block seems to perform a non-differentiable hard one_hot assignment. While I understand that this should be a hard assignment, you cannot propagate gradients to this into the features in such a way that pooling operation learns to pool together meaningful features. As such I also wouldn't really know why the additional losses should be effective and the ablation study also shows that they do not have a major effect on the performance. I would assume this could be fixed with something like a Gumbel softmax. Did you consider doing that? As it is right now, I think the model really only groups together similar superpixels based on similar features that become similar by accident, but not because the features of meaningful supersegments were trained to be more similar. I see this as one of the major weaknesses of this paper. Empirically it performs reasonably well, but it's not really clear if that is because it does what one would expect it to do. I get the same feeling about the superpixel segmentation. As far as I can tell, you apply SEEDS to the RGB image and thus the gradients are passed to the initial feature extraction layers through the regions, but the network can never learn better features to create better superpixel oversegmentations.\n- The paper states that the model can deal with varying numbers of input superpixels (and different numbers of coarser supersegments in the later layer) since the number of sampled tokens can just be changed. Sadly there are no real experiments to confirm that. I think this would actually be a super interesting experiment to truly learn something about the architecture. How does it behave when during inference these numbers are actually changed and how important is it that these numbers are selected in the right way? I think focusing a little more on these aspects would make the paper a lot stronger.\n- As far as I understand the paper, all of the reported numbers are based on own experiments, including all the baseline numbers. As such I really wonder how well these baselines were tuned. Simply applying the same schedule to some other network doesn't make it an inherently fair comparison. What makes it even worse is that in several cases the margin between the proposed model and the baselines is not even very large, thus raising the question if the model truly performs better than the VIT or k-mediod baselines, or it's just a simple matter of slightly different tuning of the model. To be fair though, the paper does not claim that the method achieves state-of-the-art results. A comparison with external baselines would be very good though!\n- The ablations not only show that the additional losses don't make a major difference, but even the fact whether or not superpixels are used instead of patches does not have a major impact on the performance. Did you ever check if the quality of the superpixels matters at all? Could you just use bigger or smaller superpixels?\n- Partially the paper is quite informally written and some sections just do not read very well. For example, I have never seen the abbreviation 'a.k.a.' used in a paper and it is consistently missing the last dot. Especially the sections dealing with training setups are not very nice to read and I think they would be better placed in the appendix, likely in tabular form. In general another pass over the paper, checking some spelling and grammar, would likely be good.\n\nQuestion:\n- A quick google search yielded the following CVPR'22 paper \"Semantic Segmentation by Early Region Proxy\". It shows that there is at least one other transformer-based method using superpixels. Even though I don't think it's extremely related, it could make a good addition to the related work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O61GhKsikEm", "reviewer": "Reviewer_5DAX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use superpixel-based tokens instead of fixed-shape patch tokens for image segmentation, and proposes a graph pooling between transformer blocks to create an increasing size of segmentation from the oversegmented superpixel map.", "summary_of_the_review": "Overall, this paper poses certain merits in terms of unsupervised image segmentation. However, there are some missing details/problems to answer mentioned in the [Strength And Weaknesses] tab to increase the soundness of the paper.", "strength_and_weaknesses": "Strength:\n1. The motivations to use adaptive segment tokens are natural and straightforward.\n2. The paper turns this segmentation problem into a hierarchical segment grouping problem which is very interesting.\n\nWeakness:\n\n1. The paper claims hierarchical segmentation as the main contribution. However, there is no further support for why this is important or useful due to they are outcomes naturally from hierarchical segment grouping.\n\n2. The technical contribution is rather limited. The main technical contribution is the GraphPool algorithm, but further experimental evaluation specifically designed to verify this module's effectiveness is missing.\n\n3. It's mentioned in Sec. 3.3, we have to set the cluster number. Any experiments or investigations on that? Comparison to mode-seeking algorithms such as meanshift may remove the need to set cluster number?\n\n4. Just curious, how can the model learn to correct the over-segmentation error in the superpixel stage? For example, a superpixel contains two or more three semantic different classes. Will the proposed model be able to separate it? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F8okWQr_HN5", "reviewer": "Reviewer_BmNe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to concurrently perform image classification and segmentation using adaptive segment tokens, which is demonstrated effective and superior to current methods. ", "summary_of_the_review": "The paper provides in-depth analysis of the human vision for image recognition and segmentation, and a novel method to deal with the task with superior performance.  ", "strength_and_weaknesses": "Strength\n\n1. The paper is well organized and written so that it is easy to follow and understand.\n2. The basic idea is reasonable and insight analysis is convincing.\n3. The proposed is technically sound and the novel.\n\nWeaknesses\n\n1. Not sure how the \"goodness of grouping and consistency across the hierarchy\" are measured and used in proposed method.\n2. More experiment with state-of-the-art segmentation and classification methods would be better.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5WT9jinYBS", "reviewer": "Reviewer_kF9A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for jointly computing region segmentation and image recognition, with any supervision to learn segmentations. The concept is based on hierarchical region segmentation to feed tokens into an image transformer, instead of fixed-size image patches. Experiments show that the method achieves recognition accuracy comparable to the baseline transformer but at half the computational cost, and considerable increases in segmentation accuracy.", "summary_of_the_review": "The paper is well written, with a novel idea that is validated through thorough experiments. It should be of interest to those working in image segmentation and recognition.\n", "strength_and_weaknesses": "Strengths\n\nGeneralizing vision transformers to incorporate tokens of arbitrary shape and size is a significant contribution. As the authors state, partitioning an image into fixed size and shape tokens is an artificial construct that ignores image content. Removing that constraint is a worthy goal and should lead to improved performance.\n\nConstructing the model to handle arbitrary image regions requires generalizing it to handle a variable number of tokens. Fig. 2 is a very clear, effective illustration of the important aspects of transformer-based segmentation algorithms and the advantages of the proposed approach. Explicitly computing and leveraging image regions presents complexities but is much more faithful to the image.\n\nThe method is based on hierarchical segmentation using superpixels, an interesting revisitation of traditional methods combined with transformers. Before deep learning such methods were popular and achieved SOTA performance on segmentation problems, and combining these techniques with deep learning is a worthwhile attempt. The primary benefit, as shown in the results, is an improvement in computational efficiency rather than overall accuracy. The results show modestly improved accuracy over the VIT baseline in most cases, with significant improvements in computational cost.\n\nCombining region segmentation and image recognition through jointly optimizing them is a compelling idea, and has been treated only lightly in recent years.\n\nWeaknesses\n\nOn page 1 the authors state that “Models optimized for image classification have no concepts of parts and wholes; recognition is achieved without understanding how different parts such as eyes and face are organized for the whole animal.” This claim seems hard to justify, as CNNs are designed to learn object parts and their relationships, and many works have shown this to be true. This statement should be qualified appropriately or softened considerably.\n\nThe results do not show a significant gain in segmentation accuracy from the proposed method compared to baseline VIT using standard (square) image tokens, on ImageNet, which is disappointing. However the primary strength of the method is in fine-grained accuracy, and this performs better as shown in tables 2 and 3 for VOC.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject