Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation
Authors: Abhishek Aich
Yumin Suh
Samuel Schulter
Manmohan Chandraker
Venue: ICLR 2025 Poster (2025)
Keywords: universal segmentation; efficient transformers
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose an efficient transformer encoder for universal segmentation architectures.

Abstract: A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions. With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses \~50% of its compute only on the transformer encoder. This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer. With this observation, we propose a strategy termed PROgressive Token Length SCALing for Efficient transformer encoders (PRO-SCALE) that can be plugged-in to the Mask2Former segmentation architecture to significantly reduce the computational cost. The underlying principle of PRO-SCALE is: progressively scale the length of the tokens with the layers of the encoder. This allows PRO-SCALE to reduce computations by a large margin with minimal sacrifice in performance (\~52% encoder and \~27% overall GFLOPs reduction with no drop in performance on COCO dataset). Experiments conducted on public benchmarks demonstrates PRO-SCALE's flexibility in architectural configurations, and exhibits potential for extension beyond the settings of segmentation tasks to encompass object detection. Code is available here: https://github.com/abhishekaich27/proscale-pytorch

Published as a conference paper at ICLR 2025
PROGRESSIVE TOKEN LENGTH SCALING IN TRANS- FORMER ENCODERS FOR EFFICIENT UNIVERSAL SEG- MENTATION
Abhishek Aich'; Yumin Suh Samuel Schulter Manmohan Chandraker
NEC Laboratories, America, USA, University of California, San Diego, USA, 3 AAmazon AGI, Atmanity Inc.
ABSTRACT
A powerful architecture for universal segmentation relies transformers that en- code multi-scale image features and decode object queries into mask predictions. With efficiency being high priority for scaling such models, state-of-the method Mask2 uses >50% of its compute only on the trans- former encoder. This is due to the retention of full-length token-level represen- tation of all backbone feature scales at each encoder layer. With this observation, we propose strategy termed PROgressive Token Length SCALing for Efficient transformer encoders (PRO-SCALE) that can be plugged-in in to the Mask2Former segmentation architecture to significantly reduce the computational cost. The un- derlying principle of PRO-SCALE is: progressively scale the length of the tokens with the layers the encoder. This allows PRO-SCALR to reduce computations by a large margin with minimal sacrifice in performance (~52% encoder and 27% overall GFLOPs reduction with no drop in performance on COCO dataset). Experiments conducted on public benchmarks demonstrates PRO-SCALE's flex- ibility in architectural configurations, and exhibits potential for extension beyond the settings of segmentation tasks to encompass object detection. Code available here: https: / git.bb..tubu.ggggggg com/ abhi sheksieekhoooo ch27 proscale prrossale-pytorcc pytorch
observed that the
INTRODUCTION
The tasks of image segmentation (instance (He et al., 2017), semantic (Tu, 2008), and panoptic (Kir- illov et al.. 2019)) are recently being addressed together under the paradigm of "universal" image segmentation (Cheng et al., 2021: 2022; Jain et al., 2023; Gu al., 2024; Cavagnero et al., 2024; Rosi et al., 2024). This is due to the evolution of transformer-based (Vaswani et al., 2017) approaches that can represent both stuff and things categories (Kirillov et al., 2019)) using general tokens, leading to a diminished distinction among the tasks of semantic, instance, and panoptic segmenta- tion. Success the state-of-the-ard universal segmentation framework Mask2Former (M2F) (Cheng et al., 2022) can be attributed to its DEtection TRansformer (Carion et al., 2020) DETR-style ar- chitecture. This DtTR-style segmentation (henceforth termed as M2F-style) framework exhibits exceptional performance across various segmentation tasks without the need for task task-specific design choices, setting them apart from preceding modern panoptic segmentation frameworks (Rashwan al., 2024; Hu al., 2023; Ammar et al., 2023; Sun et al., 2023).
The strong performance of M2F-style architecture, however, incurs significant computational over- head hindering their widespread deployment. In this paper, we address this important problem of designing an efficient M2F-style architecture for universal segmentation model. In particular, we present PROgressive Token Length SCALing for Efficient transformer encoders or PRO-SCALE that reduces the computational load occurring in the transformer encoder of such models with sur- prisingly low performance deterioration. M2F-style architectures contain a backbone (for multi- scale feature extraction from input images), pixel decoder or transformer encoder (to capture long- range dependencies and contextual relationships across the multi-scale backbone features), and transformer decoder (for predicting the masks and labels) along with segmentation module (see
"Corresponding author, email: aaich001 at uer dot edu; Work done by YS and SS while NECLA.

---

Published as a conference paper at ICLR 2025
encoder
decoder seg. module
backbone
(bakd
transformer Sayers
pixel embordding layer
class
Encoder
Mt
Doater
wit
S40rs
14.8%
mplit-scale
mask
features
(a)
(h)
Figure 1: Compute distribution. (a) Mask2Former-style segmentation model (b) In the Mask2Former model using Res50 (He al., 2016) and SWIN-T (Liu al., 2021) backbones, the transformer encoder contributes the most to the overall computation cost, accounting for 54.04% and 50.38%, respectively.
Fig. (a)). In efficient segmentation models, where the backbones traditionally lightweight (SWIN-T (Liu et al., 2021) and Res50 (He et al., 2016)), cross-scale feature attention has been shown as potent way to achieve high segmentation performance (Cheng et al., 2022: Li et al., 2023a). However, it has been shown in (Li et al., 2023a; Lv et al., 2023; Yao et al., 2021) that due to large number of query tokens introduced from multi-scale features of light backbones, the encoder incurs the highest computational cost. We make a similar observation in Fig. 1(b) for M2F where more than 50% of the compute comes from the encoder, given backbones SWIN-T and Res50.
Existing solutions (Li et al., 2023a; Lv et al., 2023) for making DETR-style architectures efficient are only designed to handle detection tasks. For example, Lite-DETR (Li al., 2023a) proposed to update larger and smaller scale features in different frequencies for efficient computation. RT- DETR (Lv et al., 2023) on the other hand, proposed a hybrid encoder that transforms multi-scale features into sequence of image features through intra-scale interaction and cross-scale fusion. Such detection- based strategies do not suit the segmentation task: either they propose to discard the features (as in (Lv et al., 2023)) or lack the ability to reduce computations induced due to constructing pixel embedding map (as in (Li et al., 2023a)). This significantly reduces their impact performance-efficienng trade-off for segmentation models (e.g. 333 GFLOPs reduction but 1 % segmentation accuracy drop).
of
To this end, introduce PRO- for M2F's transformer encoder that has two key properties: (7) progressively increase the token length or input size at each encoder layer by introducing larger scale features in the deeper layers (2) simplifying the pixel embedding layer by replacing with Light Pixel Embedding (LPE) module. The fundamental idea of PRO-SCALE is to address the pos- sible redundancy that arises from consistently maintaining a full-length token sequence across all layers in the encoder. For example as shown in Fig. 2(a), M2F uses tokens from multi-scale features across all encoder layers resulting expensive computations. With PRO-SCALE'S progressively expanding tokens, the reduction in sequence length leads to significant FLOPs savings with mini- mal to negligible degradation in segmentation performance (see Fig. 2(b)). Extensive experiments show that TRRO-SCALE based M2F architecture achieves ~52% reduction in transformer encoder GFLOPs while maintaining the same segmentation performance. with SWIN-T backbone achieves 52.82% PQ with 171.7 GFLOPs (vs. original performance of 52.03% PQ with 234.5 GFLOPs) on the COCO (Lin et al., 2014) dataset.
particular, PRO-SCALE-ME
To summarize, we present efficient transformer encoder PRO-SCALE for M2F universal seg- mentation architecture. PRO-SCALE operates on the fundamental idea of progressively expanding tokens along the encoder depth address the computation redundancy. It is further assisted by light pixel embedding module that effectively target computational reduction in the encoder module. Extensive experiments show that PRO-SCALE achieves the best performance--fficieee trade-off on two datasets across diverse settings.
2 RELATED WORKS
DETR (Carion et al., 2020) embraces an end-to-end object detection approach with a set-prediction objective, discarding the need for manually crafted modules like anchor design and non-maximum
2

---

Published as a conference paper at ICLR 2025
Enc. GFLOPs
Prior
Encoders
PRO-SCALE
51.98% GFLOP decrease
80.43% GFLOP decroase
PQ (%)
Input tokens
54
$3
53
5g
$3
52
52
100
depth of encoder
depth encoder
M2F
RT-M2F
(1111)
(3333
(a)
(b)
Figure 2: Key idea and performance comparison of PRO-SCALL w.r.f. prior works. (a) il- lustrates the key idea of PPPPPCCCLL SCALE progressively extend the token length in the transformer encoder. S3, Sa) represent different resolutions. In (b), we show two instantiates of our pro- posed transformer encoder PRO-SCALE compared with Mask2Former (M2F) (Cheng et al., 2022) and RT-M2F (an adaptation of (Lv et al., 2023)). PRO-SCALE eliminates 80.43% (with config- uration (p), P2, p3) = (1,1,1) and 51.98% (with configuration (P), P2, ps) (3,3,3) of encoder GFLOPs from M2F while maintaining the competitive performance. Results computed on the COCO (Lin et al., 2014) dataset.
suppression. Using the set prediction mechanism introduced in DETR, MaskFormer (Cheng al., 2021) proposed approach that converts any existing per-pixel classification model into mask classification model. This resulted a universal segmentation architecture that demonstrated state-of-the-art performance across all segmentation tasks public benchmarks in diverse settings (Cheng et al., 2022; Li et al., 2023b; Ding et al., 2022: Cavagnero et al., 2024; Rosi et al., 2024) without task-specift design choices. In particular, it employs transformer decoder to predict set of pairs (a class prediction, a binary mask). Mask2Former (Cheng et al., 2022) improved Mask- Former by using masked attention in the transformer decoder restrict the attention localized features centered around predicted segments. Recently, (Li et al., 2023b) presented a DETR-style multi-task architecture by extending DINO (Zhang al., 2022) for both detection and segmenta- tion tasks. PEM (Cavagnero et al., 2024) introduced a MaskFormer based model that includes multi-scale feature pyramid network extract high segg--sematicccomee self-modulation to ensure efficiency. PRO-SCALE focuses on making the Mask2Former universal segmentation model efficient.
features with context-based
In recent times, there have been some interesting efforts (Wang et al., 2019; de Geus et al., 2020; Hong et al., 2021; Hou et al., 2020; Sun et al., 2023; Šarié et al., 2023; Cavagnero et al., 2024; Xu al.. 2024; Rosi al., 2024) make task-specific or non-M2F-sty/ panoptic segmentation models efficient. For example, YOSO (Hu et al., 2023) introduced lightweight panoptic segmen- tation model by utilizing a feature pyramid aggregator and separable dynamic decoders. ReMaX (Sun et al., 2023) proposed a training pipeline to address the dominant impact of false positive mask assignments in panoptic segmentation by utilizing separate semantic prediction head. Above methods contain specific model design choices training strategies suited for the panoptic segmen- tation task. Unlike these, PRO-SCALE does not conform to one segmentation task and still shows competitive performances.
METHODOLOGY
Framework Overview. Following M2F, our proposed framework is composed of a lightweight backbone, our novel transformer encoder design PRO-SCALE and a transformer decoder with mask and class prediction heads (each described next). The overall model framework shown in Fig. 3. In particular, the input image is fed to the backbone to create multi-scale features. These features (introduced in Sec. 3.1) with gradual expansion input
are flattened and updated by tokens along with depth of intermediate encoder layers, allowing better performaacc--ffficeee trade-off. The transformer decoder uses the representations from PRO-SCALE and learnable object queries to compute mask embeddings. Finally, segmentation module the decoder output and per-pixel embeddings from PRO-SCALLE for mask predictions. PRO-SCALE provides these per-pixel embeddings required for mask prediction using parameter-free Light-Pixel Embedding
3

---

Published as a conference paper at ICLR 2025
54
O concat @ product addition
PRO-SCALE
$3
(Sec 3.1)
object queries
52
$1
Decoder
Seg. module
Backbone
Pi
classes
Pa
ReLU
image
Deformable Layers
Layer Norm
mask
upsample
MaxPool2D
LPE module
(Sec 3.1.1)
Figure Proposed framework. Our model includes our transformer encoder PRO- SCALE (Sec. 3.1), designed to reduce the computational load. {s represent the multi-scale backbone features. PRO-SCALE progressively scale the length of the tokens with the layers of the encoder. This allows PRO-SCALE to reduce computations by a large margin with minimal sacrifice in per- formance. Further, 81 goes through our efficient Light-Pixer Embedding (LPE) module (Sec. 3.1.1) to create pixel embeddings for mask prediction. Pt. P2 and p3 represent encoder layer frequency. {s', s", s" represent the outputs of respective layers.
module (described Sec. 3.1.1). We start by introducing notations and model components, before diving into the details.
A backbone network that extracts features from an input image "xwx3 The backbone network can provide multi-scale feature maps (5), 82, 83, J. The spatial resolutions are typically 1/82, 1/16?, and 1/322 of the input image, respectively. We will denote the token form of these features 81, $2, $3, and $4, respectively.
A pixel decoder transformer encoder, that enhances the image features {82, $3, 84 and also creates per-pixel embedding map from S₁ and S2. The feature enhancement is performed on token representation P kx where K E(HW /32² + HW AW/16* + HW /s*) and C' 256 (Cheng et al., 2022). This P is composed of concatenated {s2, 93, $1} obtained via flattening the spatial dimensions and fed several stacked transformer blocks. Following M2F, our framework also consists of six deformable attention transformer (Zhu et al., 2020) layers that includes self-attention block and a feed- forward block. The per-pixel embedding map, the other hand, is generated using convolutional layers $1 and $2, which enhance local spatial details to be used to create the segmentation mask.
transformer encoder. The transformer encoder usually consists of
A Transformer decoder along with segmentation module that decodes binary masks from the mod- ulated image features from the pixel decoder using set of randomly initialized object queries. The transformer decoder consists of three stages, with each stage consisting of three transformer layers (Vaswani al., 2017). Each layer includes self attentttnnion block, cross- crossaattnntion block, and feed-forward block. Each layer only handles tokens of one scale of features for efficiency (Cheng et al., 2022).
As described above, the original transformer encoder results in significantly expensive computations from global attention mechanism, as it maintains the complete token length of P for all layers. Sim- ply dropping larger scale features results in poor localization, especially in small objects (Li et al., 2023a). For example, removing features $1 and $2 results to original model. Moreover, the of large-scale scale features to produce pixel embedding map in the convolutional layers require processing an enormous number of tokens, leading to an extremely high computational demand. We will now explain how approach tackles these obstacles.
degradation ~ 5% AP compared
3.1 PRO-SCALE SCALE: PROPOSED TRANSFORMER ENCODER
Intuition. The bottleneck towards an efficient encoder are excessive large-scale features, where most of which are not informative but contain local details for different objects (Li et al., 2023a). In order to better handle their usage in the encoder, leverage the composition of multi-scale
4

---

Published as a conference paper at ICLR 2025
features from the backbone: (1) The limited quantity of small-scale features (e.g. Sa, 8a) captures abundant semantics. (2) The larger quantity of scage-scale features (e.g. $1, S2) captures crucial local features important for segmenting varying scale of objects. For example, assuming a SWIN-T back- bone based M2F, ($2, 83, 84} result in (76.19%, 19.04%, 4.76%) token contributions, respectively (also see analysis in Fig. in Appendix). Therefore, propose to prioritize enrichment of tokens from small scale features earlier than the large scale features.
Encoder structure. We build upon deformable attention due to its linear complexity with the number of feature queries (represented by P). We create 3 splits of P (following the three scales):
P1 = C(si) E RKix
(1)
P2 C(s', ss) E Rxxx
(2)
P3 C(s", E RRKx
(3)
Here, C(-) denotes the concatenation operation along the token size dimension, K3- = (HW/322 + HW HW/102 HW H K2 - ((HW/322 HW H and Ki HW /322.. These splits are then fed to three stages of the deformable attention transformer layers sequentially. The output of these stages are s', s" and s"" respectively. s" is subsequently fed to the decoder. Each stage is repeated P1. P2 and Pa times before propagating the tokens to the next stage. This results in updation of $4 for (p) + P2 + P3), 83 for (p2 + ps), and 82 for pl times, gradually expanding the token length of inputs to intermediate layers in PRO-SCALE, As we show in Sec. 4, this strategy reduces computing load by significant margins while maintaining performance. Note that, since we skip using tokens from token recalibration within enriches small-scale features with high-scale features to gain slightly better segmentation without significant computational overhead. Please see Sec. C in Appendix for more details and Fig. 7 for complete visualization.
-
large scale features in the initial layers, we also
that
3.1.1 LIGHT-PIXEL EMBEDDING (LPE) MODULE
Intuition. Strong performance of M2F depends on multi-scale features computed from the back- bone (Cheng et al., 2022). Tokens {s2, S1) are fed to the encoder layers compute s" in order to produce per-segment embeddings in the transformer decoder. S1, on the other hand, serves the purpose of creating the per-pixel embedding map &emb to enhance local details in the feature maps. However, due to the large size of $1, it results in high computational load from the use of convo- lutional layers in original M2F. Hence rather than dropping S), we weaken this inductive bias and assess the existing learnable module with a simpler module in our design.
LPE structure. We propose to use simple maxpooling layer followed by normalization and non-linearity to compute Eemb. The goal of our LPE module is to mitigate this overhead while keeping the advantages of s in producing Eemb. We observe it reduces almost 445% GFLOPs in the encoder architecture, but doesn't 't significantly harm the overall model's 's segmentation performance. In our implementation, we pooling kernel size of 3 (and stride is set to 1).
4 EXPERIMENTS
In this section, evaluate PRO-SCALE based M2F architecture on two benchmarks all segmen- tation tasks. First, we observe that PRO-SCALE is extremely effective in reducing computational load while providing best trade-off's compared to baselines (Tab. 1, 2, 9). Second, provide an tensive ablation analysis of PRO-SCALE in Tab. 7, and Fig. 4, 5. Third, we test the performane of PRO-SCALE on prevalent state-of-the-art frameworks for tasks like object detection, joint-task predictions, and open-vocabulary universal segmentation in Tab. 8, 10 and 11. Finally, we visualize some segmentation examples in Fig. 6. In the Appendix, we provide our training, and dataset (for cOCO (Lin et al., 2014) and Cityscapes (Cordts et al., 2016)) details. For brevity, we will denote our overall framework as PRO-SCALE.
Evaluation metrics. We evaluate PRO-SCALE and baselines in the universal segmentation set- ting. This means, following (Cheng et al., 2022), PRO-SCALE's evaluation is performed using model trained exclusively with panoptic segmentation annotations. For panoptic segmentation, the conventional PQ (Panoptic Quality (Kirillov et al., 2019)) metric is used. Following (Cheng al.,
5

---

Published as a conference paper at ICLR 2025
Table COCO evaluation. PRO-SCALL (with configuration (pt. P2, Ps)) is extremely competitive against the baselines on COCO with at least 51.99% GFLOPs reduction compared to M2F with no performance drop. Non-M2F-style architectures are colored in gray.
Performance (1)
GFLOPs (1). (% decrement)
Model
PQ
m
AP,
Total
Encoder
Backbone: SWIN-T
M2F (Cheng et al., 2022)
52.03
62.49
42,17
234.50
117.00
Lite-M2F (Li et al., 2023a)
52.70
63.08
41.10
188.00 (-19.83)
79.78 (-31.81)
RT-M2F (Lv al., 2023)
41.36
61.54
24.68
158.30 (-32.49)
59.66 (-49.01)
PRO-SCALE 1,
50.79
62.39
39.57
137.10 (-41.54)
22.89 (-80.44)
PRO-SCALE (2, 2)
52.12
63.21
41.58
154.40 (-34.16)
39.53 (-66.21)
PRO-SCALE (3, 3.33)
52.82
63.49
42.60
171.70 (-26.78)
56.18 (-51.99)
Backbone: Res50
M2F (Cheng et al., 2022)
51.73
61.94
41.72
229.10
135.00
MF (Cheng et al., 2021)
46.50
$7.80
33.00
181000 (-20.99)
-()
PEM (Cavagnero ct al., 2024)
46.38
55.95
34,25
110.90 (-51.59)
-(-
YOSO (Hu et al., 2023)
48.40
58.74
36.87
114.50 (-50.02)
-(-
RAP-SAM (Xu al., 2024)
46.90
123.00 (-46.31)
ReMaX (Sum el al., 2023)
53.50
169.00 (-26.23)
(-)
PRO-SCALE (1. 1, 1)
50.31
60.66
39.56
131 131.40 (-42.65)
30.25 (-77.59)
PRO-SCALE (2, 2)
51.21
61.53
40.66
148.80 (-35.05)
48.85 (-63.48)
PRO-SCALE (3, 3.
51.45
61.58
41.45
166.10 (-27.50)
67.45 (-50.03)
2022), we report AP, (Average-Precies Precision (Lin et al., 2014)) metric for instance segmentation. This computed the 'thing' categories from the instance segmentation annotations. For semantic seg- mentation, we report mloU, (mean Intersection-over-unee Union (Everingham et al., 2015)) by merging instance masks from the same category. Here, subscript p denotes evaluation after training with panoptic segmentation annotations. For computing the GFLOPs, we use image scale of (800, 1333) for COCO dataset, and (1024, 2048) for Cityscapes dataset. All models are trained and evaluated on the train and validation split, respectively.
with the original M2F, along with some recently proposed efficient transformer encoders for detection (Li et al., 2023a; Lv al., 2023; Cavagnero al., 2024), Specifically, we replace the transformer encoder of M2F with encoders proposed in Lite-DETR (Li al., 2023a) and RT-DETR (Lv et al., 2023). We call these Lite-M2F and RT-M2F, respectively. Note that, we opted for the "Lite-DETR H3L1-(6+1)x 1" setup without its key-aware deformable attention (I.i et al., 2023a). However, we modified this setup to (5+1) when implementing Lite- M2F. For completeness, we also compare with recent non-M2F-style panoptic segmentation models YOSO (Hu et al., 2023), RAP-SAM (Xu et al., 2024) and ReMaX (Sun et al., 2023).
Baselines. We compare
Architecture Details. We focus on standard lightweight backbones Res50 (He et al., 2016), SWIN-Tiny (Liu et al., 2021), and MViT2-Tiny (Li et al., 2022b), pre-trained on ImageNet-1K (Deng et al., 2009). We use PRO-SP -SALLECAEE as the transformer encoder. As per Sec. 3, we use dif- ferent integer values of (Pr.P2 Ps) to instantiate different depths of the stages in PRO-SCALE. We directly adopt the M2F's decoder that consists masked attention (Cheng et al., 2022) with 9 lay- ers in total and 100 learnable queries. Similar to M2F's round robin design, PRO-SCALE feeds (82, 83, } into successive transformer decoder layers.
4.1 MAIN RESULTS
The analysis on the COCO dataset is presented in Tab. 1. We use M2F (Cheng et al., 2022) as reference for performance and present the following insights. First, PRO-SCALE significantly computationally cheaper than baseline efficient M2F-style models such as Lite-M2F and RT-M2F. For example with SWIN-T backbone, PRO- SCALE achieves a PQ of 52.82 and with a computa- tional cost of 17 GFLOPs. This PQ is 1 11.46 points better than that of RT-M2F, while enabling a ~~52% lighter transformer encoder. Compared to Lite-M2F, PRO-SCALE shows a better overall segmentation while decreasing approximately ~52% of GFLOPs compared to ~32% of Lite-M2F. Similar performance improvements can be observed over M2F and MaskFormer (Cheng et al., 2021) with Res50 backbone. Second, PRO- outperforms non-M2F-style models YOSO (Hu al.,

---

Published as a conference paper at ICLR 2025
Table 2: Cityscapes evaluation. PRO-SCALE (with configuration (p1, P2. p3)) shows strong ef- ficiency trade-off compared to the baselines on Cityscapes, egg. 51.96% (SWIN-T) and 50.17% (Res50) GFLOPs reduction and little-to-no accuracy drop. Non-M2F-style models are colored in gray. f trained for 200K iterations.
Performance (1)
GFLOPs (J). (% decrement)
Model
PQ
IoU,
AP,
Total
Encoder
Backbone: SWIN-T
M2F (Cheng et al., 2022)
64.00
80.77
39.26
537.8
281.00
Lite-M2F (I.i al., 2023a)
62.29
79.43
36.57
428.70 (-20.29)
172.00 (-38.79)
RT-M2F (Lv al., 2023)
59.73
77.89
31.35
361.10 (-32.86)
130.00 (-53.74)
PRO-SCALE (1. I,
60.58
78.29
32.97
311.90 (-42.00)
055.14 (-80.38)
PRO-SCATA (2, 2, 2)
62.37
78.62
35,97
352.00 (-34.55)
095.24 (-66. I1)
PRO-SCALB (3, ,3)
63.06
77.94
37.81
392.10 (227.09)
135.00 (-51.96)
Backhone: Res50
M2F (Cheng el al,, 2022)
61.86
76.94
37.35
524.10
291.00
PEM (Cavagnero et al., 2024)
61.07
77.62
34.11
236.60 (-51.59)
-(-)
YOSO (Hu et al., 2023)
59.70
76.05
33.76
265.10 (-49.42)
-(-
ReMaXt (Sun et al., 2023)
65.40
294.70 (-43.77)
(-)
PRO-SCALE (1. I,
59.70
77.19
32.77
298.10 (-43.12)
65.21 (-77.59)
PRO-SCALE (2, 2,
60.89
76.61
34.95
338.20 (-35.47)
105.00 (-63.92)
PRO-SCALE (3, 3,
61.87
78.44
37.33
378.30 (-27.82)
145.00 (-50.17)
Table 3: Component ablation. Comparison of different cases with settings of PRO-SCALE for PQ and GFLOPs for the encoder with SWIN-T backbone on Cityscapes dataset. PRO-SCALE provides flexible way adjust performance and computational cost by varying its scaling factors.
Case #
Setting
PQ (1)
GFLOPs (4)
FPS (1)
M2F
64.00
281.00
5.10
C
M2F (completely remove 81)
60.65
132.00
7.28
C2
PRO-SCALE (1,1,1) +1
61.73
73.49
6.36
C3
PRO-SCALE (I,1,I) + (51 via LPE)
60.58
55.14
6.47
C4
PRO-SCALE (2,2.) + (81 via LPE)
62.37
95.24
6.07
C5
TRO-SCALE (3,3,3) + (81 via LPE)
63.06
135.00
5.71
2023) in panoptic segmentation by at least ~ points. YOSO has a slightly better computational load than PRO-SCALE while ReMaX (Sun et al., 2023) has competitive accuracy. Note that, ReMaX focuses the training pipeline of panoptic segmentation (Sun et al., 2023), which is orthogonal to our approach to design efficient segmentation architectures. Further, ReMaX limited by the in- herent efficiency of model and becomes ineffective larger models (see Appendix C in (Sun al., 2023)). Similarly, in comparison with a recent universal segmentation architecture PEM (Cavagnero al., 2024) with Res50 backbone, PRO- SCALE shows stronger results in severe GFLOPs reduc- tion settings (P1, P2. Ps = 1,1.1). The analysis on Cityscapes dataset segmentation is shown in Tab. 2. Similar to COCO, PROLECALE is computationally most efficient compared to both M2F- style models Lite-MMF and RT-M2F) and at-par with non-M2F-style models (YOSO and ReMaX (Sun al., 2023)) with extremely competitive performance. For example, PRO-SCALE with SWIN- T backbone achieves 63.06 PQ vs. Lite-M2F's 62.29 PQ while having comparatively ~27% less GLOPS than M2F Compared to non-M2F-style models in Res50 backbone, PRO- SCALE demon- strates competitive performance with no specific panoptic segmentation design like YOSO (Hu et al., 2023), or much fewer training iterations with specific panoptic segmentation training strategies like ReMaX (Sun al., 2023) (200K vs 90K iterations).
4.2 ABLATION STUDY
Ablation of components. We analyze the impact of PRO-SCALE with SWIN-T backbone in low GFLOPs budget scenario in Tab. 3. We make the following observations. Simply remov- ing $1 from M2F (Case C₁) reduced GFLOPs by 51.24% but also reduced PQ by 5.7%. Using PRO-SCALE (p1, P2, Ps 1,1,1) (Case C2) achieved 73.85% reduction in GFLOPss with a 4.7%
7

---

Published as a conference paper at ICLR 2025
53
46
53
100
44
PQ (%0
62
PQ (96)
522
CPLOPs
60
42
51
10
20
50
36
61
322
02
configg
config.
Eac. contig.
Enc. comlin
Figure 5: Impact of pre-trained weights. PRO-SCALE provides significant computational boosts, irrespective of backbone pre-trained weights. Here, backbone/dataset= SWIN-
Figure 4: Impact of LPE module. Per-pixel em- beddings produced by LPE do not significantly harm the performance but demonstrate a strong impact on the computational reduction. Here, T/COCO, weights supervised/ MoBY (Xie backbone= SWIN-T, PRO-SCALE configuration: et., 2021) on IN1K (Russakovsky et al., 2015), PRO-SCALE config.: Ci 11,1, C2 (3, 1), C3 = (1,3, 1), (1,,1)))) 13,
C1 (1, 1, 1), C2 (2, 2, 2), C3 = (3,3.3) mod- els w/o LPE and w/ LPE), dataset = COCO.
Table 5: Impact of different backbones.
Table 4: LPE module with Lite-M2F. LPE can PRO-SCALE can provide significant effi be flexibly integrated to improve the computa- ciency boosts across various backbones. Here, tional load of Lite-M2F (Li et al., 2023a) with minimal performance degradation. Backbone (P1.P2, P2, Ps) = (I, 1, 3), = SWIN-T.
Backhone
Performance (1)
GFLOPs (}). 1% decrement)
type
PQ
mlOU,
AP,
Total
Enewder
Dateet
Model
PQ
Perfermance (?)
mtOu,
AP;
GELOP& (% decmmmm)
Total
Eacoder
SWIN-T
52.03
52.40
62.66
62.49
41.62
42.17
164 (0229.7))
234.50
54.77 (-53.19)
117.00
Res50
51.35
51.73
61.94
41.72
(58.60()30.77)
229.10
$9.44 (-55.97)
135.00
Otyicapes
Lite M2F
-PP
61:35
62.29
79.43
7968
36.57
35.91
351 (-11000))
429.70
94.99 (:448))
172.00
MVTT-T
53.70
61.17
13.53
174.10(-28.8))
(-57.77)
PQ reduction. When 81 is added back via LPE to (P1, P2, Ps = 1,1,1) (Case C3), results in stronger performance-cfficcinnee trade-off than Ct. This performance further improves when differ- ent configurations of PRO- SCALE are used in case C4 and Cz.
Impact of LPE module. Our LPE module is aimed to drastically decrease the computational overhead without impairing the segmentation performance in PRO-SCALE. To analyze this, we compare LPE with the M2F's convolutional layer based embedding layer module in Fig. 4. Here, we can observe that compared to this convolutional unit with learnable parameters, LPE is extremely effective. For example, for PRO-SCALE SCALE configuration C3 = (3, 3, 3), LPE reduces the computa- tions by 400 with performance degradation of 0.12 PQ points. Similarly, 02 = (1, 1, 1) and C2 2, 2) provide similar computational benefits with 0.43 PQ points degradation. This demon- strates the favorable performance compuutttncoomttttttionn trade-off LPE provides in PRO- SCALE. To further demonstrate the potency of our proposed LPE module, we incorporate LPE in Lite-M2F and lyze the results in Tab, 4. We can observe that LPE shows similar impact on Lite-M2F: 455% less transformer encoder GLOPS with only ~0.5-1 PQ point trade-off.
Variations in encoder configuration. Here, we analyze the impact of PRO- SCALE configura- tion on model performance in Tab. 6. We can make some key observations. First, we observe that PRO-SCALE shows stronger performance any p; > 1. This is trivial as more computations are allowed for the corresponding layers. Second, increasing Pa and p2 have different performance- efficiency trade-off: for COCO, P2 vs. (p2 + 2) results only increment of 6.5 GFLOPs with 1.0% PQ improvement whereas P3 (pa 2) results in higher computational load of 32 GFLOPs increase with ~ 1.6% PQ performance improvement. We make similar observations for
the Cityscapes dataset for (p1 + ys. (pe + 2).
We analyze the performance PRO-SCALE with dif- ferent backbones in Tab. 5. Specifically, observe the impacts of SWIN-T, Res50, and MViT-T in the segmentation performance. PRO-SCALL is versatile in providing strong iiffooomanne-fffcciineee across diverse backbone types. We also present results with larger backbones namely ResNet101,
Variations in backbone architectures.
8

---

Published as a conference paper at ICLR 2025
Table 6: Analysis of configurations (pi, P2, P2.Pa). Increments in different Pi shows different
Table 7: Impact of large backbones. PRO-SCALE can provide significant effi- ciency boosts across various backbones. Here, (p1. P2. Ps) (3, 3, 3), dataset Cityscapes.
performanc--fficienny trade-off.
Performance (t)
GFLOPs (1), (% decrement)
PQ miou,
AP,
Total
Encader
Dataset Cityscapes. Backbone: SWIN-T
Backbome type
Performance (T)
GFLOPs LJ), 1% decremenn)
original
39,26
537.80
281.00
PQ
m10U, AP.
Total
Encoder
(.1.1 1)
60.50
78.29
12.97
311.90 (-2..00) (2.00)
35)
40.70
724.30
281.00
0.3.0 D
6163
61.38
78.58
78.67
35.26
33.75
314.70 480
(-59.0 33)
057.93 (-79.30 38)
069 62 (-75.22)
64,41
39.90
578.30 (-20:33)
(:5.96)
L,
02.32
79,49
36.97
314.80 (-30331)
118.00 (:38.01)
SWIN-B
65.27
66.10
83.70
81.97
42.80
41.12
(-13.7)))))8)
1051.20
(31.59))
283.00
Dataset:
COCO, Backbone: SWIN-T
original
$2.01
62.49
42.17
254.50
117.00
SWIN-L
65.93
66.60
82.90
82.77
43.60
41.30
107.9900:7.49
1949 70
141 (:5087)
287,00
29.57
135.300 (-41.44)
(-80.44
01.3. 1)
51.76
41.07
40.46
(-38 68)
(-40.85)
(-77.03)
(-72.40)
Res
61.28
62.40
76.50
78
37.70
36.19
(-2.45))
679.70
(50.17))))
291.00
0. 3)
52.40
62,66
41.62
(62.700--99777
$4.77 (-53.19)
SWIN-Small, SWIN-Base, and SWIN-Large Cityscapes dataset in Fig. 7. PRO-SCALE reduces GFLOPs in all cases while providing a strong accuracy-eficiency trade-off. The PRO-SCALE. con- figuration is (pI, P2. Pa) = (3,3,3). Note that, we can also employ a different PRO- SCALE configu- ration (P1, P2. P3) to further reduce the computations.
Variations in backbone pre-trained weights. We analyze the performance of PRO-SCALE with different pre- pre-training strategies of the backbones in Fig. 5. Specifically, we initialize SWIN-T with supervised learning (SL) and self-supervised learning (SSL) based ImageNet- IK weights. For SSL pre-training, we employ MoBY (Xie al., 2021). We can see that PRO-SCALE can gain perfor- mance robustness of SSL weights while providing the same trade-offs as SL weights. For example, when e₁ = (pi,2 P2, P2,P3/ P3) = 1,) MoBY shows an improvement of 11.6% AP, with the same efficiency gains. On average, integrating PRO-SCALA with the MoBY pre-trained backbone re- sults better (overall) performance compared to using SL backbone weights, especially in instance segmentation (Fig. 5, right).
FPS comparison. Tab. 9 compares M2F and PRO-SCALE against state-of-the-arr efficient en- coder for universal segmentation method PEM (Cavagnero et al., 2024). While M2F achieves the highest PQ (51.73) and lowest FPS (4.91) and PEM provides the highest FPS (7.31) but lower PQ (46.38), RROEECALE offers a competitive PQ (51.35) with a balanced FPS (6.25).
Impact beyond universal segmentation. We analyze PRO-SCALE on tasks beyond universal segmentation to test its effectiveness. In particular, we add PRO-SCALE to DINO (Zhang al., 2022) for the task of object detection, MaskDINO (Li et al., 2022a) for the joint task of instance segmentation-ddttettoon and FCCLIP (Yu al., 2023) and MaskCLIP (Zheng Ding, 2023) for the task of open-vocabulary segmentation. Note that we can use a different PRO-SCALE configuration to further reduce the computations in all cases.
Impact on DINO for object detection. As shown in Tab. 8, PRO-SCALE (integrated with DINO) stands out with the highest AP (49.4), showcasing exceptional performance compared to other models. It also has the lowest encoder GFLOPs (56. 166.18), making it more computationally efficient than other methods. We also show segmentation PQ for comprehensive overview.
Impact two ooen-vocab universal segmentation frameworks. We used recent state-of-the- art open- vocabularyy image segmentation methods MackCLIP (Zheng Ding, 2023) and FCCLIP (Yu et al., 2023). Our method FRO-SCALE can casily work with these frameworks and reduce the computations while having better performance as shown in Tab. 10. We followed the ex- act training and testing protocols of respective methods and trained model with COCO. We perform evaluation on COCO val set and cross-evaluation on ADE20K (Zhou et al., 2017) val set.
Impact on MaskDINO for multi-task prediction. We integrated and trained MaskDINO with P PROSSCALE, and the results are shown in Tab. 11. In particular, we trained the model on COCO panoptic annotations with the exact same training settings as MaskDINO for 50 epochs and eval- uated the model for segmentation and object detection. Clearly, PRO-SCALA effectively reduces the computational requirements of MaskDINO while maintaining overall performance.

---

Published as a conference paper at ICLR 2025
Table 8: PRO-SCALE for object detection. PRO-SCALE can easily work with DETR (Car- Table 10: PRO-SCALE for open-vocabulary ion et al., 2020) based object detection models segmentation.
PRO-SCALE can easily work and reduce the computations while having bet- with M2F based open-vocab universal segmenta- ter performance. The PRO-SCALE configura- tion models (Zheng Ding, 2023; Yu et al., 2023) tion is (P1, P2. Ps. P1) (3,3,3,3), backbone is and reduce the computations while having better Res50, training dataset is COCO, and trained for PQ performance. The PRO-SCALE configura- 12 epochs. The segmentation results are with tion is (Pi-P2, Pa) (3,3,3), backbone is Res50, SWIN-T backbone. LPE module is not required and training dataset is COCO.
for object detection.
Performance (1)
GELOPs
ADE20K
Total
Performance (1)
GFLOPs (1)
MaskCTIIP (7hang Ding, 2023)
29.08
549.30
Total
Encoder
280-BCALE
36.53
16.53
486.10
67.45
Lite DETR (Ly et ol., 2025)
DINO (2hang et al,, 2022)
49.10
49.00
52.70
NIA
188.00
N/A
79.78
NIA
FCCLIP (1u et 2023)
55.81
56.40
25.27
2119.00
1973.80
287.00
141.00
RI-DETR (1 2023a)
49.20
ERO-SCALE
49.10
52.82
171.70
56.18
Table
11:
PRO-SCALE for MaskDINO.
PRO-SCALA SCALE can easily work with MaskDINO models (Li et al., 2022a) and reduce the com- putations while having better Mask and Box AP performance. The PRO-SCALA configuration (p) P2, Pa) = (3,3,3), backbone is Res50, and training dataset COCO.
Table 9:
SCALE FPS comparison.
PRO-SCALE strikes a balance with high PQ score, and good FPS, offering a good trade-off between quality and speed. The PRO-SCALE
configuration is (Pi-P2, Ps)= (3,3,3), backbone is Res50, and dataset is COCO.
Performance (1)
GFLOPs (1)
Model
PQ
FPS (1)
Model
Mask AP
Box AP
Total
Encoder
M2F (Cheng et al., 2022)
51.73
4.91
MasDDINO + FRO- SCAL.P
46.00
50.50
302.10
204.00
PEM (Cavagnero et al., 2024)
46.38
7.31
45.66
50.83
212.40
108.00
PRO-SCALE
51.45
6.25
input image
M2F
SCALE
input image
M2F
PRO-SCALE
input image
M2F
PRO SCALE
input image
M2F
SCALE
Figure 6: Qualitative visualizations. We visualize examples of predicted panotic maps from M2F and PRO-SCALE. Even with 52% encoder GFLOPs reduction, PRO-SCALE shows better quality panoptic maps (backbone = SWIN-T, PRO-SCAIA config. (3, 3, 3)). Zoom-in for best view.
Qualitative analysis. We show some examples of predicted panoptic maps in Fig. 6 with SWIN- backbone. We set PRO-SCALE configuration to (3, 3, 3). Compared to M2F (Cheng al., 2022), PRO-SCALE consistently shows strong performance with drastically fewer GLOPs in everyday scenes (top row on COCO) as well as complex driving scenes (bottom row on Cityscapes).
CONCLUSION
In this paper, we propose an efficient transformer encoder PRO-SCALE for the Fask2Former uni- versal segmentation framework. It reduces the computational load by large margin with minimal degradation in performance. The core principle of PRO-SCALE length of the tokens with the layers of the encoder. Further, a light per-pixel embedding module introduced to alleviate the computational overhead arising from creating per-pixel embeddings without much sacrifice in performance. Our extensive experiments demonstrate that PRO-SCALE significantly lighter than prior prominent methods while maintaining competitive universal seg- mentation performance.
to progressively expand the
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Abdallah Ammar, Mahmoud I. Khalil, and Cherif Salama. Rt-yoso: Revisiting yoso for real-time panoptic segmentation. 2023 5th Novel Intelligent and Leading Emerging Sciences Conference (NILES), pp. 306-311, 2023. doi: 10. 100
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Niccolò Cavagnero, Gabriele Rosi, Claudia Cuttano, Francesca Pistilli, Marco Ciccone, Giuseppe Averta, and Fabio Cermelli. Pem: Prototype-based efficient maskformer for image segmentation. arXiv preprint arXiv22402.19422 2024.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-1788
2021.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Daan de Geus, Panagiotis Meletis, and Gijs Dubbelman. Fast panoptic segmentation network. IEEE Robotics and Automation Letters, 5(2): 1742-179,, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskelip. arXiv preprint arXiv:2208.0e8 08984, 2022.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98 166,-116.1110 2015.
Xiuye Gu, Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang, Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo, Huizhong Chen, Liang-Chie -Cich Chen, et al. Dataseg: Taming a universal multi- dataset multi-task segmentation model. Advances Neural Information Processing Systems, 36, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Weixiang Hong, Qingpei Guo, Wei Zhang. Jingdong Chen, and Wei Chu. Lpsnet: A lightweight so- lution for fast panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16746-16754, 2021.
Rui Hou, Jie Li, Arjun Bhargava, Allan Raventos, Vitor Guizilini, Chao Fang, Jerome Lynch, and Adrien Gaidon. Real-time panoptic segmentation from dense detections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8523-8532, 2020.
Jie Hu, Linyan Huang, Tianhe Ren, Shengchuan Zhang, Rongrong Ji, and Liujuan Cao. You only segment once: Towards real-time time panoptic segmentation. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Patlern Recognition, pp. 17819-17829, 2023.

---

Published as a conference paper at ICLR 2025
Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 2989-2998, 2023.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9404 0011-94133222 2019.
Feng Li, Hao Zhang, Huaizhe xu, Shilong Liu, Lei Zhang, Lionel M. Ni, and Heung- Yeung Shum. Mask dino: Towards a unified transformer-basee framework for object detection and segmenta- tion, 2022a.
Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lei Zhang, and Lionel Lite detr: An interleaved multi-scale encoder for efficient detr. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1558-115677, 2023a.
Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung- Yeung Shum. Mask DINO: Towards a unified transformer based framework for object detection and segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3041-3050 2023b.
Yanghao Li, Chao Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong. Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.51001, 2017.
Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, and Yi Liu. Detrs beat yolos real-time object detection. arXiv preprint arXiv:2304.08069 2023.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-571. leee, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Abdullah Rashwan, Jiageng Zhang, Ali Taalimi, Fan Yang, Xingyi Zhou, Chaochao Yan, Liang- Chich Chen, and Yeqing Li. Maskconver: Revisiting pure convolution model for panoptic seg- mentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 851-8 861- 2024.
Gabriele Rosi, Claudia Cuttano, Niccolò Cavagnero, Giuseppe Averta, and Fabio Cermelli. The revenge of bisenet: Efficient multi-task image segmentation. arXiv preprint arXiv:2404.09570, 2024.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
Josip Šarié, Marin Oršić, and Siniša Šegvić. Panoptic swiftnet: Pyramidal fusion for real-time panoptic segmentation. Remote Sensing, 1568).1968 2023.
12

---

Published as a conference paper at ICLR 2025
Shuyang Sun, Weijun Wang, Qihang Yu, Andrew Howard, Philip Torr, and Liang-Chiel Chen. Remax: Relaxing for better training on efficient panoptic segmentation. arXiv preprint orXIv:2306.17319, 2023.
Zhuowen Tu. Auto-context and its application high-level vision tasks. In 2008 IEEE Conference on Computer Vision and Paltern Recognition, pp. 1-8. IEEE, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. TPAMI, 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https: : /gttttb........ wwtp htt////ecooooeerrrreeeee
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self- supervised learning with swin transformers. arXiv preprint arXiv:2105.04553 2021.
Shilin Xu, Haobo Yuan, Qingyu Shi, Lu Qi, Jingbo Wang, Yibo Yang, Yining Li, Kai Chen, Yunhai Tong. Bernard Ghanem, et al. Rap-sam: Towards real-time all all-purpose segment anything. arXiv preprint (0l::2400..0224 2024.
Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318. 2021.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chiel Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. In NeurIPS, 2023.
H Zhang, Li, $ Liu, L Zhang, H Su, Zhu, LM Ni, and HY Shum. Dino: Detr with im- proved denoising anchor boxes for end-to-end object detection, arxiv 2022. arXiv preprint MoXi::2003.03605. 2022.
Zhuowen Tu Zheng Ding, Jicke Wang. Open-vocabulary universal image segmentation with maskelip. In International Conference on Machine Learning, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 633-641, 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENT DETAILS
Datasets. We examine PRO- SCALE with two image segmentation datasets: COCO (Lin al., 2014) (80 "things", 53 "stuff" categories) and Cityscapes (Cordts et al., 2016) (8 "things". 11 "stuff" categories). COCO has 118K training and 5K validation images. Cityscapes has 2975 training and 500 validation images.
Training details. All our training details and underlying strategy follow (Cheng et al., 2022), including for our baselines. We use Detectron2 (Wu et al., 2019) and PyTorch (Paszke et al., 2019) for implementation. We employ the AdamW optimizer (Loshchilov & Hutter, 2017) with step learning rate schedule. The initial learning rate is 0.0001. The backbone learning rate is multiplied with 0.1 and its weight decay set 0.05. We decay the learning rate 0.9 and 0.95 fractions of the total training steps by a factor of 10. For COCO, our models are trained for 50 epochs. For Cityscapes, we use 90k iterations. Batch size is set as 16. For data augmentation and calculating GFLOPs, we follow the strategies exactly as M2F. We report the average GFLOPs for all cases. We use distributed training with 8 A6000 GPUs.
Loss functions. We use the exact same loss functions and weights as (Cheng al., 2022). In particular, we use the binary gross-entropy loss and the dice loss (Milletari et al., 2016) for our mask loss. Both loss functions use a weight of 5.0. The final loss is a combination of mask loss and classification loss (cross-entropy loss). We set the classification loss weight as 2.0 for all classes, except 0.1 for the 'no object' class. Finally, we apply the identical post-processing methodology as described in (Cheng et al., 2022) to obtain the desired output formats for panoptic, semantic, and instance segmentation predictions.
B ADDITIONAL RESULTS
LPE pooling analysis. We analyzed the LPE module with 'average pooling' in place of 'max- pooling' on the Cityscapes dataset with Res50 backbone and (P1, P2, P3) = (3.3.3). We observed that the performance of the LPE module performs better with max-pooling (61.87% PQ) than average pooling (61.47% PQ).
Token redundancy visualization. We provide an explicit qualitative visualization that proves the token redundancy in early stages the transformer encoder in Fig. This analysis is on 100 randomly chosen images from COCO val set. In this figure, the r-axis represents the distance of neighborhood tokens from the candidate token (along the token dimension) and the y-axis represents the (average across the number of images) cosine similarity between the two. It can be observed that larger scale tokens, the token similarity higher than smaller token as the neighborhood distance increases. This elearly suggests that the information redundancy of larger scale tokens is comparatively higher than smaller scale tokens.
Efficiency vs Performance overview. Fig. 10 illustrates the trade-off between efficiency (GFLOPS the x-axis) and performance (PQ the y-axis) for different configurations of (p1, P2, P3) on the COCO dataset, with each point representing specific configuration. The red dashed line indicates the original model's performance (PQ 52.03), highlighting how the configurations compare in terms of maintaining performance while reducing computational costs.
Additional qualitative visualization. We show some examples of predicted panoptic maps in Fig. 11 with Res50 backbone on the COCO dataset. We set PRO-SCALE configuration to (3, 3, 3). Compared to Lite-M2F (Li et al., 2023a) and PEM (Cavagnero et al., 2024), PRO-SCALE consistently shows strong performance with drastically fewer GLOPs.
FPS analysis. The following analysis with A6000 GPUs. We analyzed the gain in speed when PRO-SCALE is integrate with different backbones in Tab. 16. Further, Tab. 15 provides overview of FPS with different baselines. PRO-SCALE provides good trade-off between performance and speed in different settings.
14

---

Published as a conference paper at ICLR 2025
CxHXW
CxH
C
2
flatten
Φ: c
sigmoid
1xHW;
i
CXH x W
CxH,W
flatten
S
product
CxHwW,
concat + addition
54
PRO-SCALE (Sec 3.1)
TRC module
$3
$
$2
$
S"
P1 x
P2
P3 x
ReLU
Deformable Attention Layers
Layer Norm
upsample
MaxPool2D
LPE module (Sec 3.1.1)
Figure 7: Token Re-Calibration (TRC) module. This unit first interpolates to the size of the target smaller-scale feature S, Then, the spatially flattened and resized 82 is passed through channel reduction layer (followed by sigmoid function) to produce attention map. This map is imposed on the spatially flattened S; to create S,.
C PRO-SCALE ADDENDUM: TOKEN RE-CALIBRATION (TRC) MODULE
Skipping the propagation of s₂ in the initial layers may lead to errors in predicted panoptic maps due to missing information from this feature scale. To avoid such map errors caused by imperfect token representations, we propose to calibrate the and using 82 in our TRC module. This module aims to enhance small-scale features s, E {sg, $1} using large-scale feature $2 without increasing the computations in PROOSCCLE transformer layers.
Intuition.
TRC structure. To utilize strengths of 82 in smaller scales, employ contrastive attention. In via an attention map. This
particular, we propose to enrich the tokens of s; by projecting attention map is obtained by using channel reduction layer ø followed by sigmoid function. As illustrated in Fig. 7, the final tokens š; are obtained as (0 represents elementwise multiplication):
8 S; 3 a sigmmiidiii (mmói((c((s)))
(4)
With above, Eq. 2, translate to P1 C(Aa) and P2 C(s', C(s',8g). We choose Φ(-) to be linear layer and set temperature 0.1, keeping computations negligible.
Impact of TRC module. Our TRC module is designed to incorporate the benefits of large scale feature 82 without increasing computational overhead in PRO-SCALE, In Tab. 12, we can see that the TRC module consistently improves the performance the proposed framework demonstrating the efficacy of this unit. Moreover, this unit only costs 0.006 GFLOPs for the COCO dataset. We also analyze the impact of c on the TRC module Tab. 13 and find that overall improves the performance of segmentation with best performance at 0.1. Finally, we provide visualizations of impact of TRC module two examples shown in Fig. 8. can be observed that the TRC module introduces stronger focus parts of images that need to be segmented by the model.
15

---

Published as a conference paper at ICLR 2025
$1.00
51.00
Table 13: Impact of on TRC. Tem- perature scaling of 0.1 shows the best performance. Here, backbone SWIN- T, dataset Cityscapes.
PQ (5G w/ TTCC unit
PQ (30) w/o TRC unc
30.00
30:40
50.40
30.20
50.20
PQ(%)
50.00
50.00
0.01
60.05
0.10
60.58
1.00
60.32
Impact of TRC module. Use of S2 via TRC to enhance {s3, improves the overall perfor- mance with light computational load. Here, backbone Res50, dataset COCO, config.=(p1:
Table 12:
5.00
60.41
10.00
60.34
input image
before TRC module
after TRC module
input image
before TRC module
after TRC module
J.P.Morgan
POLO
Figure 8: Effeec of TRC module. It can be observed that the TRC module introduces stronger focus on parts of images that need to be segmented by the model.
16

---

Published as a conference paper at ICLR 2025
1.00
Table 14: LPE pooling analysis. We ob- serve that max-pooling provides better results with PRO-SCALE than average pooling. The PRO-SCALE configuration is (P1, P2, Pa)
Coelne Sumilerity
0.80
0.00
(3,3,3,, backbone is Res50, training dataset is Cityscapes.
0.40
0.00
Pooling PQ (1)
10
100
200
300
400
500
700 800
avg avg-pooling
61.47
Neighborhood Distancc
max-pooling
61.87
Figure 9: Token redundancy visualization.
Efficiency Performance (COCO)
12.4
PROSCALE CJ. 3.
5L.
PROSCALE 11.3
2
518
PROSCALE 1.
GFICPS
Figure 10: Efficiency Performance for different PRO-SCAN SCALE (P1, P2. Ps) configurations COCO. The red lines indicate M2F's PQ and GFLOPs for reference.
Table 16: PRO-SCALE FPS comparison on dif- Table 15: PRO-SCALE FPS comparison with ferent backbones. PRO-SCALE strikes bal- baselines. PRO-SCALE strikes a balance with ance with high PQ score, and good FPS, offer- high PQ score, and good FPS, offering good ing a good trade-off between quality and speed. trade-off between quality and speed. Backbone is Here, (p1, (p,,P..P3) P3) (1, 1, 3), = SWIN-T, and dataset Cityscapes.
Backbone type
Performance (t)
Model
PQ (f)
FPS (?)
PQ
mIOu,
AP,
FPS (f)
L2te-M2F (Cheng et al., 2022)
62.29
6.01
SWIN-T
52.03
62.49
42.17
RT-M2F (Cavagnero
2024)
59.73
6.59
52.40
62.66
41.62
5.28
PRO-SCKLE (1.1.1)
PRO- SCALE (2.2.2)
62.37
60.58
6.47
6.07
Res50
51.73
51.35
61.53
61.94
40.82
41.72
5.71
PRO-SCALA S-LLECALELCCE (3,3.3)
63.06
54.11
64.39
44.54
MVTF-T
53.70
64.17
43.53
4.85
17

---

Published as a conference paper at ICLR 2025
input image
Lite-M2F
PEM
PRO-SCALE
Qualitative visualizations. We visualize few examples of predicted panotic maps from Lite-M2F (Li et al., 2023a) and PEM (Cavagnero al., 2024) and PRC- SCALE. Even with 52% transformer encoder GFLOPs reduction, PRO- SCALE shows better quality panoptic maps. Here, backbone Res50, PRO-SCALE configuration (3, 3, 3). Zoom-in for best view.
Figure 11:
18


Reviews:
[{"review_id": "A4BPmooAwx", "reviewer": "Reviewer_GmJD", "summary": "This paper proposes to Progressive Token Length Scaling, a strategy designed to enhance the efficiency of transformer encoders in universal segmentation models. The authors identify that the state-of-the-art model, Mask2Former, devotes over 50% of its computational resources to the transformer encoder, largely because it processes a full-length token representation from all backbone feature scales at every encoder layer. This paper addresses this inefficiency by progressively scaling the token length with each encoder layer, thereby significantly reducing computational demands—specifically, achieving around 52% reduction in GFLOPs—while maintaining performance levels on the COCO dataset.", "questions": "What distinguishes the LPE module from traditional fully convolutional network (FCN) style upsampling methods?\n\nIf I understand correctly, the computational costs in P1 and P2 are quadratically lower compared to P3. Would it be advantageous to use a different number of layers for each split instead of maintaining the same number of layers across all splits?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The progressive integration of finer-grain information within the transformer encoder is an intuitive and straightforward approach that proves to be both simple and effective.\n\nA comprehensive set of experiments on segmentation and detection tasks validate the design choices of the PRO-SCALE architecture, demonstrating its effectiveness.\n\nThe paper is clearly written and easy to follow, enhancing its accessibility and understanding.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i3VNnrjOs8", "reviewer": "Reviewer_g4Mz", "summary": "The paper introduces an efficient transformer encoder architecture that progressively increases token length and feature scale, coupled with a LPE module. This approach achieves an effective balance between computational efficiency and segmentation quality.", "questions": "For Figure6, could you provide more visualization comparisons with also some baselines like Lite-M2F, RT-M2F, or PEM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents its ideas with clarity and good technical writing.\n2. The experimental validation is comprehensive, featuring:\n    - Thorough comparative analyses\n    - Well-structured ablation studies\n    - Clear demonstration of each component's contribution to overall performance", "weaknesses": "1. In Table9 for FPS comparison, while Lite-M2F and RT-M2F are used as baselines in other evaluations, a complete comparison with all baseline models on FPS would strengthen the efficiency claims\n2. The paper states that ReMaX is orthogonal to the approach and ineffective on larger models, but could you provide quantitative evidence? Currently, the table results only demonstrate ReMaX has a good performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M8izWxDzIx", "reviewer": "Reviewer_1znQ", "summary": "This paper proposes the “PRO-SCALE” strategy to reduce the computational cost of Mask2Former encoders. The proposed approach can significantly reduce computations with minimal sacrifice in performance.", "questions": "Why not use depth-wise 3x3 convolutions for the LPE module?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing and motivation of this paper is very clear. \n2. The proposed method seems to achieve a better trade-off than several recent related approaches.", "weaknesses": "1. Although the authors emphasize the computational burden of the transformer encoder and propose dedicated methods to reduce computations, the overall reduction in proportion is not very obvious. The descriptions in the abstract seem to overstate the effect. It is better to clarify that 52% GFLOPs reduction in computation is for the encoder.\n\n2. A figure to intuitively show the comparisons in trade-off (speed vs performance or FLOPs vs performance) is necessary for understanding the practical value of the approach. \n\n3. The LPE module also contributes a lot to the reduction in computation. However, it is mainly because the computational efficiency of vanilla convolutions is too low. This further indicates that the proposed PRO-SCALE is not so important for the reduction of computational complexity of the entire system. Strictly speaking, the pixel embeddings generation module is not part of the transformer encoder. Compared to the LPE module, the TRC module in the appendix seems more interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a0YTr0nayt", "reviewer": "Reviewer_zhiy", "summary": "This paper presents *PRO-SCALE*, a method to improve the efficiency of the “transformer encoder” component of the state-of-the-art Mask2Former model for universal image segmentation. In Mask2Former, this transformer encoder (also called ‘pixel decoder’ [a]) takes the tokens of multi-resolution features from the backbone, applies multiple layers of deformable attention to all tokens, and then feeds them to the segmentation decoder. In contrast, PRO-SCALE applies the first deformable attention layers only to a small number of low-resolution feature tokens, and then gradually adds higher-resolution features in subsequent layers. By doing so, the number of operations is lower than in the original transformer encoder. Additionally, to efficiently generate the highest-resolution features that the decoder needs, PRO-SCALE contains a *Light Pixel Embedding* (LPE) module that applies max-pooling to high-resolution features from the backbone. With experiments on multiple datasets and with multiple backbones, PRO-SCALE is shown to reduce the total number of FLOPs by up to 27%, while keeping the segmentation quality (PQ, mIoU, AP) roughly the same as for Mask2Former. Additional experiments show that PRO-SCALE is also effective in combination with open-vocabulary and object detection models.\n\n[a] Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation”, CVPR 2022.", "questions": "I would like to ask the authors to address my concerns as formulated in the “weaknesses” section, to answer the questions posed there, and to revise the manuscript accordingly.\n\nOne additional question:\n\n1.\tHow does the presence of redundancy in the feature tokens relate to the presented PRO-SCALE method? Appendix B shows that higher-resolution features have a higher cosine similarity, but it is not clear to me how this observation motivates the PRO-SCALE design, where fewer deformable attention layers are applied to high-resolution features. Could the authors clarify this relation?\n\n---\n\n**Update after author discussion.** After reading the different reviews, the authors' response, and the revised manuscript, and asking some follow-up questions, I decide to keep my original rating. \n\nWhile I still believe that the idea of improving the efficiency of the decoder of Mask2Former can be valuable, I mainly have concerns about the actual efficiency of the proposed method. Importantly, the newly provided *prediction speed* results show that the actual efficiency improvement obtained by PRO-SCALE is limited. While PRO-SCALE obtains high FLOPs reductions, the impact on the prediction speed in terms of *frames per second* (FPS) in considerably lower. As a result, PRO-SCALE does not obtain a better *prediction speed vs. segmentation performance* balance than existing method Lite-DETR/Lite-M2F. Furthermore, it turns out that the impact of the LPE module on the prediction speed is almost negligible, while it also causes a segmentation performance drop.\n\nBecause the main purpose of the paper is to improve efficiency while keeping the segmentation performance as high as possible, I believe the value of the paper is considerably limited when (a) the proposed method is not shown to obtain a better *prediction speed vs. segmentation performance* balance than a highly related existing method (Lite-M2F/Lite-DETR), and (b) the actual efficiency improvement (in terms of FPS) of one of the main contributions (LPE module) is almost negligible while causing a drop in segmentation performance.\n\nIn their response, the authors mention that other benefits of reduced FLOPs are (i) improved energy efficiency because FLOPs correlate with energy use, and (ii) better compatibility and easier deployment on edge devices. However, Henderson et al. [a] experimentally show that there is little correlation between FLOPs and energy usage when comparing across different model architectures. As PRO-SCALE has a different architecture than the default Mask2Former, there is no guarantee that the reduced FLOPs of PRO-SCALE will translate to reduced energy use. As reduced energy usage is also not shown experimentally, this benefit cannot be verified. As for compatibility and deployment on edge devices, the paper provides no evidence or references for these benefits, so these benefits can also not be verified.\n\nOverall, while also considering the strengths of the work and the other reviews, these weaknesses cause me to keep my original rating.\n\n[a] Henderson et al., \"Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning,\" JMLR 2020.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe proposed method is simple but effective. It is simple because there are only two relatively small changes with respect to the original transformer encoder of Mask2Former: (a) the first deformable attention layers are applied to only a subset of the feature tokens, and (b) the model applies max-pooling on the highest-resolution features. By doing so, PRO-SCALE can reduce the total number of FLOPs of Mask2Former by up to 27%, without causing a drop in Panoptic Quality (PQ), mean IoU or Average Precision (AP). Although the adjustments to Mask2Former are minimal and the technical innovation of PRO-SCALE is limited, I believe it is valuable because the paper critically examines an inefficient component of a frequently used model, and finds that it can be made considerably more efficient while keeping the performance roughly the same, with only minor changes to the design.\n2.\tThrough experiments, PRO-SCALE is not only compared to Mask2Former with its default transformer encoder, but also to existing efficient versions of the transformer encoder that were proposed for object detection method DETR, i.e., Lite-M2F and RT-M2F in Tab. 1 and Tab. 2. The results of these comparisons show that PRO-SCALE also achieves a better efficiency-accuracy balance than these existing methods. This demonstrates the value of PRO-SCALE compared to these existing methods. \n3.\tThe paper contains many ablations and additional experiments (Tab. 3 to Tab. 7), which properly show the impact of different design choices and demonstrate the effectiveness of PRO-SCALE in different experimental settings, e.g., with different backbones or pre-training. \n4.\tThrough experiments, the paper shows that PRO-SCALE is not only effective on Mask2Former, but also on other models for other tasks. The results in Tab. 8, Tab. 10 and Tab. 11 show that PRO-SCALE achieves similar FLOPs reductions in the encoder when combined with object detection model DETR, two open-vocabulary segmentation models, and instance segmentation model Mask-DINO, while obtaining similar or better segmentation performance. This shows that the proposed PRO-SCALE method is more generally applicable than just on Mask2Former.", "weaknesses": "1.\tThe paper does not clearly explain how the original Mask2Former model generates the high-resolution *per-pixel embedding map* ${\\mathcal{E}_{emb}}$, and why this is less efficient than how PRO-SCALE does it with LPE. L242-L243 states that $\\mathbf{s}_1$ serves the purpose of creating the per-pixel embedding map, and that it uses a convolutional layer, but there is no clear description or depiction of the exact manner in which this is done. As a result, it is also not clear why the newly proposed LPE module is more efficient.\n2.\tRelated to this, some details are missing about the exact operation of the LPE method. Concretely, what is the stride of the MaxPool2D operation? Is it the same as the kernel size? If the stride is >1, then what is the impact of changing the stride (and therefore the resolution of ${\\mathcal{E}_{emb}}$) on the results, both qualitatively and quantitatively? This information is currently not available.\n3.\tSec. 3.1 (L233-L236) briefly mentions that PRO-SCALE additionally uses a so-called *token recalibration* operation, which enriches small-scale features with high-scale features to further enhance the segmentation accuracy. However, despite being part of the PRO-SCALE method, this operation is not visualized in Fig. 3, and not explained properly in Sec. 3.1. As a result, (a) Fig. 3 makes it seem like this operation does not exist at all, and (b) it is unclear from the main paper how a part of the method works. In other words, this *token recalibration* operation should be explained and visualized in the main paper.\n4.\tRelated to the previous point, the efficiency of the *token recalibration* operation is not evaluated, neither in the main paper nor in the appendix. As a result, it is not clear if it is actually efficient.\n5.\tFor most experiments, the paper reports the FLOPs but not the latency/FPS/throughput of the model. The FPS is only reported for the overall model in Tab. 9. As a result, for most configurations, it is not clear how/if the reduction in FLOPs translates to a speedup when the model is run on a GPU. The results of the paper would be stronger if the FPS/latency of the model was also reported for other experiments, i.e., at least for the main results in Tab. 1 and Tab. 2, and the ablations in Tab. 3, Tab. 5 and Fig. 4, but ideally even more.\n6.\tPRO-SCALE does not yield a significant efficiency improvement when used in combination with large backbones, e.g., Swin-L (see Tab. 7). Of course, this is expected, as the Swin-L backbone accounts for a much larger portion of the overall number of FLOPs than Swin-T, but it is still a weakness because PRO-SCALE’s value in larger models is limited.\n7.\tThe abstract contains a misleading statement. L023 states that PRO-SCALE can achieve a ~52% GFLOPs reduction with no drop in performance on the COCO dataset. This statement implies that, compared to the original Mask2Fomer, the overall PRO-SCALE model requires ~52% fewer GFLOPs. However, Tab. 1 shows that this 52% GFLOPs reduction concerns the encoder only, and that the overall GFLOPs reduction of the model is ~27%. The statement in the abstract should be altered by either specifying that the 52% reduction concerns the encoder, or changing the number from 52% to 27%.\n8.\tTab. 8, Tab. 10 and Tab. 11 only contain GFLOPs results for the encoder, not for the entire model. As a result, it is not clear what the actual overall improvement is of PRO-SCALE.\n9.\tL323 & L358 state that ReMaX is limited by the inherent efficiency of the model, but the efficiency for ReMaX is not provided in Tab. 1 or Tab. 2. In other words, this claim is not substantiated.\n10.\tL454-L455 states that, on average, the MoBY pre-trained backbone causes lower performance degradation than SL pre-trained weights, especially for instance segmentation. However, per Fig. 5 (right), compared to the Mask2Former baseline, the average drop for MoBY over all 4 settings is -2.35 AP, while it is -1.49 for supervised learning. Therefore, this statement is incorrect. This statement is not very important for the main message of the paper, so it doesn’t impact the value of the proposed method, but it should be altered nevertheless.\n\nSome minor points, which do not significantly affect my overall rating:\n\n11.\tThe text contains several mistakes/errors. Some examples:\n\n    a.\tL039 – “framework exhibit exceptional performance” => “framework exhibits exceptional performance”\n\n    b.\tL136 – “making Mask2Former universal segmentation model” => “making the Mask2Former universal segmentation model”\n\n    c.\tL153 – “map and class prediction heads” => “mask and class prediction heads”\n\n    d.\tL358 – Not clear what is meant by “becomes ineffective efficient”\n\n    e.\tL377 – \"effieiciency\" => \"efficiency\"\n\n    f.\tL376-L377 – “strong … than” => “stronger … than”\n\n    g.\tL436 – “$p_2$ vs. ($p_2$ + 3)” should be “$p_2$ vs. ($p_2$ + 2)”, as the baseline is $p_2 = 1$ and the comparison is “1 vs. 3”.\n\n    h.\tL437 – Likewise: “$p_3$ vs. ($p_3$ + 3)” => “$p_3$ vs. ($p_3$ + 2)”.\n\n    i.\tL503 – The caption of Tab. 11 states that PRO-SCALE achieves a better PQ, but the PQ is not reported in Tab. 11.\n\n    j.\tL534-L535 – “for Mask2Former universal segmentation framework” => “for the Mask2Former universal segmentation framework”\n\n12.\tIn Tab. 3, the experimental setting is not explicitly indicated. The numbers correspond with Swin-T on Cityscapes, but this is not explicitly mentioned in the caption. This should be mentioned, so that the table can be understood without having to check other tables.\n\n13.\tThe paper uses an inconsistent number of decimals for results on the same metrics. For instance, in Tab. 3: 132 GFLOPs for $C_1$ and 73.49 for $C_2$. It would be better if the number of decimals was consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)