Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation

## Abstract

Abstract A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions. With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses > > > 50% of its compute only on the transformer encoder. This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer. With this observation, we propose a strategy termed PRO gressive Token Length SCAL ing for E fficient transformer encoders ( PRO-SCALE ) that can be plugged-in to the Mask2Former-style segmentation architectures to significantly reduce the computational cost. The underlying principle of PRO-SCALE is: progressively scale the length of the tokens with the layers of the encoder. This allows PRO-SCALE to reduce computations by a large margin with minimal sacrifice in performance ( ∼ similar-to \sim ∼ 52% GFLOPs reduction with no drop in performance on COCO dataset). We validate our framework on multiple public benchmarks.

## 1 Introduction

The tasks of image segmentation (instance [], semantic [], and panoptic []) are recently being addressed together under the paradigm of “universal” image segmentation [, , , , , ]. This is due to the evolution of transformer-based [] approaches that can represent both stuff and things categories []) using general tokens, leading to a diminished distinction among the tasks of semantic, instance, and panoptic segmentation. Success of the state-of-the-art universal segmentation framework Mask2Former (M2F) [] can be attributed to its DEtection TRansformer [] or DETR-style architecture. This DETR-style segmentation (henceforth termed as M2F-style) framework exhibit exceptional performance across various segmentation tasks without the need for task-specific design choices, setting them apart from preceding modern panoptic segmentation frameworks [, , , ].

The strong performance of M2F-style architecture, however, incurs significant computational overhead hindering their scalability. In this paper, we address this important problem of designing an efficient M2F-style architecture for universal segmentation model. In particular, we present PROgressive Token Length SCALing for Efficient transformer encoders or PRO-SCALE that reduces the computational load occurring in the transformer encoder of such models with surprisingly low performance deterioration.

Figure: Figure 1: Compute distribution. (a)) Mask2Former-style segmentation model (b) In the Mask2Former model using Res50 [] and SWIN-T [] backbones, the transformer encoder contributes the most to the overall computation cost, accounting for 54.04% and 50.38%, respectively.
Refer to caption: extracted/2404.14657v1/figures/images/m2f_arch.png

M2F-style architectures contain a backbone (for multi-scale feature extraction from input images), a transformer encoder (to capture long-range dependencies and contextual relationships across the multi-scale backbone features), and a transformer decoder (for predicting the masks and labels) along with a segmentation module (see Fig. 1). In efficient segmentation models, where the backbones are traditionally lightweight (SWIN-T [] and Res50 []), cross-scale feature attention has been shown as a potent way to achieve high segmentation performance [, ]. However, it has been shown in [, , ] that due to a large number of query tokens introduced from multi-scale features of light backbones, the encoder incurs the highest computational cost. We make a similar observation in Fig. 1 for M2F where more than 50% of the compute comes from the encoder, given the standard backbones SWIN-T and Res50.

Existing solutions [, ] for making DETR-style architectures efficient are only designed to handle detection tasks. For example, Lite-DETR [] proposed to update larger and smaller scale features in different frequencies for efficient computation. RT-DETR [] on the other hand, proposed a hybrid encoder that transforms multi-scale features into a sequence of image features through intra-scale interaction and cross-scale fusion. Such detection-based strategies do not suit the segmentation task: either they propose to discard the use of multi-scale features (as in []) or lack the ability to reduce computations induced due to constructing a pixel embedding map (as in []). This significantly reduces their impact on performance-efficiency trade-off for segmentation models (*e.g*. $\sim$33% GFLOPs reduction but 11% segmentation accuracy drop).

To this end, we introduce PRO-SCALE, an efficient transformer encoder that has three key properties: (1) progressively increase the token length or input size at each encoder layer by introducing larger scale features in the deeper layers, (2) enhance small-scale features by explicitly modeling their inter-dependencies with the channels of higher scale features using a Token Re-Calibration (TRC) module, (3) simplifying the pixel embedding layer by replacing it with a Light Pixel Embedding (LPE) module.
The fundamental idea of PRO-SCALE is to address the possible redundancy that arises from consistently maintaining a full-length token sequence across all layers in the encoder. For example as shown in Fig. 2, M2F uses tokens from multi-scale features across all encoder layers resulting in expensive computations. With PRO-SCALE’s progressively expanding tokens, the reduction in sequence length leads to significant FLOPs savings with minimal to negligible degradation in segmentation performance (see Fig. 2). Extensive experiments show that PRO-SCALE based M2F-style architecture achieves a $\sim$52% reduction in transformer encoder GFLOPs while maintaining the same segmentation performance. In particular, PRO-SCALE-M2F with SWIN-T backbone achieves a 52.82% PQ with 171.7 GFLOPs (*vs*. original performance of 52.03% PQ with 234.5 GFLOPs) on the COCO [] dataset. To summarize, we make the following contributions:

- •
We present an efficient transformer encoder PRO-SCALE for M2F-style universal segmentation.
- •
PRO-SCALE operates on the fundamental idea of progressively expanding tokens along the encoder depth to address the computation redundancy. It is further assisted by a token re-calibration module and light pixel embedding module that effectively target performance improvement and computational reduction, respectively.
- •
Extensive experiments show that PRO-SCALE achieves the best performance-efficiency trade-off on two public benchmarks across diverse settings.

## 2 Related Works

Figure: Figure 2: Key idea and performance comparison of PRO-SCALE w.r.t. prior works. (a) illustrates the key idea of PRO-SCALE to progressively extend the token length in the transformer encoder. $\{{\bm{s}}_{2},{\bm{s}}_{3},{\bm{s}}_{4}\}$ represent different resolutions. In (b), we show two instantiates of our proposed transformer encoder PRO-SCALE, compared with Mask2Former (M2F) [] and RT-M2F (an adaptation of []). PRO-SCALE eliminates 80.43% and 52% of encoder GFLOPs from M2F while maintaining the competitive performance. Results are computed on the COCO [] dataset.
Refer to caption: extracted/2404.14657v1/figures/images/teaser_part_a.png

DETR [] embraces an end-to-end object detection approach with a set-prediction objective, discarding the need for manually crafted modules like anchor design and non-maximum suppression. Using the set prediction mechanism introduced in DETR, MaskFormer [] proposed an approach that converts any existing per-pixel classification model into a mask classification model. This resulted in a universal segmentation architecture that demonstrated state-of-the-art performance across all segmentation tasks on public benchmarks in diverse settings [, , , , ] without task-specific design choices. In particular, it employs a transformer decoder to predict a set of pairs (a class prediction, a binary mask). Mask2Former [] improved MaskFormer by using masked attention in the transformer decoder to restrict the attention to localized features centered around predicted segments. Recently, [] presented a DETR-style multi-task architecture by extending DINO [] for both detection and segmentation tasks. PEM [] introduced a MaskFormer based model that includes a multi-scale feature pyramid network to extract high-semantic-content features with context-based self-modulation to ensure efficiency. PRO-SCALE focuses on making Mask2Former-style universal segmentation models efficient.

In recent times, there have been some interesting efforts [, , , , , , , , ] to make task-specific or non-M2F-style panoptic segmentation models efficient. For example, YOSO [] introduced a lightweight panoptic segmentation model by utilizing a feature pyramid aggregator and separable dynamic decoders. ReMax [] proposed a training pipeline to address the dominant impact of false positive mask assignments in panoptic segmentation by utilizing a separate semantic prediction head. Above methods contain specific model design choices or training strategies suited for the panoptic segmentation task. Unlike these, PRO-SCALE does not conform to one segmentation task and still shows competitive performances.

## 3 Methodology

##### Framework Overview.

Following Mask2Former (M2F), PRO-SCALE is incorporated in a M2F-style framework. It is composed of a lightweight backbone, our novel transformer encoder design PRO-SCALE,
and a transformer decoder with map and class prediction heads (each described next). The overall model framework is shown in Fig. 3. In particular, the input image is fed to the backbone to create multi-scale features. These features are flattened and updated by PRO-SCALE (introduced in Sec. 3.1) with gradual expansion of input tokens along with depth of intermediate encoder layers, allowing a better performance-efficiency trade-off. We further introduce a Token Re-calibration module (described in Sec. 3.1.1) within PRO-SCALE that enriches small-scale features with high-scale features to gain segmentation performance without significant computational overhead. The transformer decoder uses the representations from PRO-SCALE and learnable object queries to compute mask embeddings. Finally, a segmentation module uses the decoder output and per-pixel embeddings from PRO-SCALE for mask predictions. PRO-SCALE provides these per-pixel embeddings required for mask prediction using a parameter-free Light-Pixel Embedding module (described in Sec. 3.1.2). We start by introducing notations and model components, before diving into the details.

- •
*A backbone network* that extracts features from an input image ${\bm{I}}\in\mathbb{R}^{H\times W\times 3}$. The backbone network can provide multi-scale feature maps $\{\bar{{\bm{s}}}_{1},\bar{{\bm{s}}}_{2},\bar{{\bm{s}}}_{3},\bar{{\bm{s}}}_{4}\}$. The spatial resolutions are typically $\nicefrac{{1}}{{4^{2}}}$, $\nicefrac{{1}}{{8^{2}}}$, $\nicefrac{{1}}{{16^{2}}}$, and $\nicefrac{{1}}{{32^{2}}}$ of the input image, respectively. We will denote the token form of these features as ${\bm{s}}_{1},{\bm{s}}_{2},{\bm{s}}_{3}$, and ${\bm{s}}_{4}$, respectively.
- •
*A Transformer encoder*, that enhances the image features $\{{\bm{s}}_{2},{\bm{s}}_{3},{\bm{s}}_{4}\}$ for the transformer decoder and creates a per-pixel embedding map from ${\bm{s}}_{1}$ for the final segmentation module. Feature enhancement is performed on a token representation ${\bm{P}}\in\mathbb{R}^{K\times C}$, where $K=\sum(\nicefrac{{HW}}{{32^{2}}}+\nicefrac{{HW}}{{16^{2}}}+\nicefrac{{HW}}{{8^
{2}}})$ and $C=256$ []. This ${\bm{P}}$ is composed of concatenated $\{{\bm{s}}_{2},{\bm{s}}_{3},{\bm{s}}_{4}\}$ obtained via flattening the spatial dimensions. The transformer encoder usually consists of several stacked transformer blocks. Following M2F, our framework also consists of six deformable attention transformer [] layers that includes a self-attention block and a feed-forward block.
- •
*A Transformer decoder along with segmentation module* that decodes binary masks from the modulated image features using a set of randomly initialized object queries. The transformer decoder consists of three stages, with each stage consisting of three transformer layers []. Each layer includes a self-attention block, a cross-attention block, and a feed-forward block. Each layer only handles tokens of one scale of features for efficiency [].

As described above, the original transformer encoder results in significantly expensive computations as it maintains the complete token length of ${\bm{P}}$ for all layers. Simply dropping larger scale features results in poor localization, especially in small objects []. For example, removing features ${\bm{s}}_{1}$ and ${\bm{s}}_{2}$ results in a degradation of $\sim 5\%$ AP compared to original model. Moreover, large-scale features are required to produce pixel embedding maps for strong segmentation performance. We will now explain how our approach tackles these obstacles.

Figure: Figure 3: Proposed framework. Our model includes our transformer encoder PRO-SCALE (Sec. 3.1), designed to reduce the computational load. $\{{\bm{s}}_{i}\}$s represent the multi-scale backbone features. PRO-SCALE uses a Token Re-Calibration (TRC) module (Sec. 3.1.1) to enhance $s_{4}$ and $s_{3}$ using $s_{2}$ and outputs $\widehat{s}_{4}$ and $\widehat{s}_{3}$, respectively. Further, $s_{1}$ goes through our efficient Light-Pixel Embedding (LPE) module (Sec. 3.1.2) to create pixel embeddings for mask prediction. $p_{1}$, $p_{2}$ and $p_{3}$ represent encoder layer frequency. $\{s^{\prime},s^{\prime\prime},s^{\prime\prime\prime}\}$ represent the outputs of respective layers.
Refer to caption: extracted/2404.14657v1/figures/images/main_framework.png

### 3.1 PRO-SCALE : Proposed Transformer Encoder

##### Intuition.

The bottleneck towards an efficient encoder are excessive large-scale features, where most of which are not informative but contain local details for different objects []. In order to better handle their usage in the encoder, we can leverage the composition of multi-scale features from the backbone:
(1) The limited quantity of small-scale features (*e.g*. ${\bm{s}}_{3}$, ${\bm{s}}_{4}$) captures abundant semantics. (2) The larger quantity of large-scale features (*e.g*. ${\bm{s}}_{1}$, ${\bm{s}}_{2}$) captures crucial local features important for segmenting varying scale of objects. For example, assuming a SWIN-T backbone based M2F, {${\bm{s}}_{2},{\bm{s}}_{3}$, ${\bm{s}}_{4}\}$ result in {76.19%, 19.04%, 4.76%} token contributions, respectively. Therefore, we propose to prioritize enrichment of tokens from small scale features earlier than the large scale features.

##### Encoder structure.

We build upon deformable attention as it exhibits linear complexity with the number of feature queries (in our case, represented by ${\bm{P}}$). We create three splits of ${\bm{P}}$ (following the three scales):

$$ $\displaystyle{\bm{P}}_{1}$ $\displaystyle=\mathcal{C}({\bm{s}}_{4})\in\mathbb{R}^{K_{1}\times C},$ (1) $\displaystyle{\bm{P}}_{2}$ $\displaystyle=\mathcal{C}({\bm{s}}^{\prime},{\bm{s}}_{3})\in\mathbb{R}^{K_{2} \times C}$ (2) $\displaystyle{\bm{P}}_{3}$ $\displaystyle=\mathcal{C}({\bm{s}}^{\prime\prime},{\bm{s}}_{2})\in\mathbb{R}^{ K_{3}\times C}$ (3) $$

Here, $\mathcal{C}(\cdot)$ denotes the concatenation operation along the token size dimension, $K_{3}=\sum(\nicefrac{{HW}}{{32^{2}}}+\nicefrac{{HW}}{{16^{2}}}+\nicefrac{{HW}}
{{8^{2}}}),K_{2}=\sum(\nicefrac{{HW}}{{32^{2}}}+\nicefrac{{HW}}{{16^{2}}})$, and $K_{1}=\nicefrac{{HW}}{{32^{2}}}$. These splits are then fed to three stages of the deformable attention transformer layers sequentially. The output of these stages are ${\bm{s}}^{\prime},{\bm{s}}^{\prime\prime}$, and ${\bm{s}}^{\prime\prime\prime}$, respectively. ${\bm{s}}^{\prime\prime\prime}$ is subsequently fed to the decoder. Each stage is repeated $p_{1},p_{2}$ and $p_{3}$ times before propagating the tokens to the next stage. This results in updation of ${\bm{s}}_{4}$ for $(p_{1}+p_{2}+p_{3})$, ${\bm{s}}_{3}$ for $(p_{2}+p_{3})$, and ${\bm{s}}_{2}$ for $p_{1}$ times, gradually expanding the token length of inputs to intermediate layers in PRO-SCALE. As we show in Sec. 4, this strategy reduces computing load by significant margins while maintaining performance.

Figure: Figure 4: Token Re-Calibration (TRC) module. This unit first interpolates ${\bm{s}}_{2}$ to the size of the target smaller-scale feature ${\bm{s}}_{i}$. Then, the spatially flattened and resized ${\bm{s}}_{2}$ is passed through a channel reduction layer $\phi$ (followed by sigmoid function) to produce an attention map. This map is imposed on the spatially flattened ${\bm{s}}_{i}$ to create $\widehat{{\bm{s}}}_{i}$.
Refer to caption: extracted/2404.14657v1/figures/images/trc.png

#### 3.1.1 Token Re-Calibration (TRC) Module

##### Intuition.

Skipping the propagation of ${\bm{s}}_{2}$ in the initial layers may lead to errors in predicted panoptic maps due to missing information from this feature scale. To avoid such map errors caused by imperfect token representations, we propose to calibrate the ${\bm{s}}_{3}$ and ${\bm{s}}_{4}$ using ${\bm{s}}_{2}$ in our TRC module. This module aims to enhance small-scale features ${\bm{s}}_{i}\in\{{\bm{s}}_{3},{\bm{s}}_{4}\}$ using large-scale feature ${\bm{s}}_{2}$ without increasing the computations in PRO-SCALE transformer layers.

##### TRC structure.

To utilize strengths of ${\bm{s}}_{2}$ in smaller scales, we employ contrastive attention. In particular, we propose to enrich the tokens of ${\bm{s}}_{i}$ by projecting ${\bm{s}}_{2}$ on ${\bm{s}}_{i}$ via an attention map. This attention map is obtained by using a channel reduction layer $\phi$ followed by a sigmoid function. As illustrated in Fig. 4, the final tokens $\widehat{{\bm{s}}}_{i}$ are obtained as ($\otimes$ represents elementwise multiplication):

$$ $\displaystyle\widehat{{\bm{s}}}_{i}={\bm{s}}_{i}\otimes{\texttt{sigmoid}( \epsilon\phi({\bm{s}}_{2}))}$ (4) $$

With above, Eq. 2, 3 translate to ${\bm{P}}_{1}=\mathcal{C}(\widehat{{\bm{s}}}_{4})$ and ${\bm{P}}_{2}=\mathcal{C}({\bm{s}}^{\prime},\widehat{{\bm{s}}}_{3})$. We choose $\phi(\cdot)$ to be a linear layer and set temperature $\epsilon=0.1$, keeping computations negligible.

#### 3.1.2 Light-Pixel Embedding (LPE) Module

##### Intuition.

Strong performance of M2F depends on multi-scale features computed from the backbone []. Tokens $\{{\bm{s}}_{2},{\bm{s}}_{3},{\bm{s}}_{4}\}$ are fed to the encoder layers to compute ${\bm{s}}^{\prime\prime\prime}$ in order to produce per-segment embeddings in the transformer decoder. ${\bm{s}}_{1}$, on the other hand, serves the purpose of creating the per-pixel embedding map $\mathcal{E}_{emb}$. However, it results in high computational load due to use of convolutional layer in original M2F. Hence rather than dropping ${\bm{s}}_{1}$, we assess the existing learnable module with a simpler module in our design.

##### LPE structure.

We propose to use a simple maxpooling layer followed by normalization and non-linearity to compute $\mathcal{E}_{emb}$. The goal of our LPE module is to mitigate this overhead while keeping the advantages of ${\bm{s}}_{1}$ in producing $\mathcal{E}_{emb}$. We observe it reduces almost $\sim$45% GFLOPs in the encoder architecture, but doesn’t significantly harm the overall model’s segmentation performance. In our implementation, we use a pooling kernel size of 3.

## 4 Experiments

In this section, we evaluate PRO-SCALE based M2F architecture on two benchmarks on all segmentation tasks. First, we observe that PRO-SCALE is extremely effective in reducing computational load while providing best trade-offs compared to baselines (Tab. 1, 2). Second, we analyze each additional component of PRO-SCALE (TRC and LPE units) in Tab. 4, 5, and 8, and Fig. 6. Third, we analyze the impact of different backbones and their pre-training strategies in Tab. 6 and Fig. 8. Finally, we visualize some segmentation examples in Fig. 9. In our Supplementary material, we provide our training, dataset (for COCO [] and Cityscapes []), and baseline details. We also provide additional results on larger backbones, analysis with object detection as well as open-vocab segmentation which further underscore the strengths of our encoder. For brevity, we will denote our overall framework as PRO-SCALE.

##### Evaluation metrics.

We evaluate PRO-SCALE and baselines in the universal segmentation setting. This means, following [], PRO-SCALE’s evaluation is performed using a model trained exclusively with panoptic segmentation annotations. For panoptic segmentation, the conventional PQ (Panoptic Quality []) metric is used. Following [], we report APp (Average-Precision []) metric for instance segmentation. This is computed on the ‘thing’
categories from the instance segmentation annotations. For semantic segmentation, we report mIoUp (mean Intersection-over-Union []) by merging instance masks from the same category. Here, subscript $p$ denotes evaluation after training with panoptic segmentation annotations. For computing the GFLOPs, we use image scale of (800, 1333) for COCO dataset, and (1024, 2048) for Cityscapes dataset. All models are trained and evaluated on the train and validation split, respectively.

##### Architecture Details.

We focus on standard lightweight backbones Res50 [], SWIN-Tiny [], and MViT2-Tiny [], pre-trained on ImageNet-1K []. We use PRO-SCALE as the transformer encoder. As per Sec. 3, we use different integer values of ($p_{1},p_{2}$, $p_{3}$) to instantiate different depths of the stages in PRO-SCALE. We directly adopt the M2F’s decoder that consists of masked attention [] with 9 layers in total and 100 learnable queries. Similar to M2F’s round robin design, PRO-SCALE feeds $\{{\bm{s}}_{2},{\bm{s}}_{3},{\bm{s}}_{4}\}$ into successive transformer decoder layers.

**Table 1: COCO evaluation. PRO-SCALE is extremely competitive against the baselines on COCO with at least 51.99% GFLOPs reduction compared to M2F with no performance drop. Non-M2F-style architectures are colored in gray.**
|  | Performance ($\uparrow$) |  | GFLOPs ($\downarrow$), (% decrement) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | PQ | mIOUp | APp |  | Total | Tx. Enc. |
| Backbone: SWIN-T |  |  |  |  |  |  |
| M2F [] | 52.03 | 62.49 | 42.17 |  | 234.50 | 117.00 |
| Lite-M2F [] | 52.70 | 63.08 | 41.10 |  | 188.00 (-19.83) | 79.78 (-31.81) |
| RT-M2F [] | 41.36 | 61.54 | 24.68 |  | 158.30 (-32.49) | 59.66 (-49.01) |
| PRO-SCALE (1, 1, 1) | 50.79 | 62.39 | 39.57 |  | 137.10 (-41.54) | 22.89 (-80.44) |
| PRO-SCALE (2, 2, 2) | 52.12 | 63.21 | 41.58 |  | 154.40 (-34.16) | 39.53 (-66.21) |
| PRO-SCALE (3, 3 ,3) | 52.82 | 63.49 | 42.60 |  | 171.70 (-26.78) | 56.18 (-51.99) |
| Backbone: Res50 |  |  |  |  |  |  |
| M2F [] | 51.73 | 61.94 | 41.72 |  | 229.10 | 135.00 |
| MF [] | 46.50 | 57.80 | 33.00 |  | 181.00 (-20.99) | – (–) |
| PEM [] | 46.38 | 55.95 | 34.25 |  | 110.90 (-51.59) | – (–) |
| YOSO [] | 48.40 | 58.74 | 36.87 |  | 114.50 (-50.02) | – (–) |
| RAP-SAM [] | 46.90 | – | – |  | 123.00 (-46.31) | – (–) |
| ReMax [] | 53.50 | – | – |  | – (–) | – (–) |
| PRO-SCALE (1, 1, 1) | 50.31 | 60.66 | 39.56 |  | 131.40 (-42.65) | 30.25 (-77.59) |
| PRO-SCALE (2, 2, 2) | 51.21 | 61.53 | 40.66 |  | 148.8 (-35.05) | 48.85 (-63.48) |
| PRO-SCALE (3, 3, 3) | 51.45 | 61.58 | 41.45 |  | 166.1 (-27.50) | 67.45 (-50.03) |

**Table 2: Cityscapes evaluation. PRO-SCALE shows strong efficiency trade-off compared to the baselines on Cityscapes, *e.g*. 51.96% (SWIN-T) and 50.17% (Res50) GFLOPs reduction and little-to-no accuracy drop. Non-M2F-style models are colored in gray. $\dagger$ trained for 200K iterations.**
|  | Performance ($\uparrow$) |  | GFLOPs ($\downarrow$), (% decrement) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | PQ | IoUp | APp |  | Total | Tx. Enc. |
| Backbone: SWIN-T |  |  |  |  |  |  |
| M2F [] | 64.00 | 80.77 | 39.26 |  | 537.8 | 281.00 |
| Lite-M2F [] | 62.29 | 79.43 | 36.57 |  | 428.7 (-20.29) | 172.00 (-38.79) |
| RT-M2F [] | 59.73 | 77.89 | 31.35 |  | 361.1 (-32.86) | 130.00 (-53.74) |
| PRO-SCALE (1, 1, 1) | 60.58 | 78.29 | 32.97 |  | 311.9 (-42.00) | 055.14 (-80.38) |
| PRO-SCALE (2, 2, 2) | 62.37 | 78.62 | 35.97 |  | 352.0 (-34.55) | 095.24 (-66.11) |
| PRO-SCALE (3, 3 ,3) | 63.06 | 77.94 | 37.81 |  | 392.1 (-27.09) | 135.00 (-51.96) |
| Backbone: Res50 |  |  |  |  |  |  |
| M2F [] | 61.86 | 76.94 | 37.35 |  | 524.1 | 291.0 |
| PEM [] | 61.10 | – | – |  | 236.6 (-51.59) | – (–) |
| YOSO [] | 59.70 | 76.05 | 33.76 |  | 265.1 (-49.42) | – (–) |
| ReMax^† [] | 65.40 | – | – |  | 294.7 (-43.77) | – (–) |
| PRO-SCALE (1, 1, 1) | 59.70 | 77.19 | 32.77 |  | 298.1 (-43.12) | 65.21 (-77.59) |
| PRO-SCALE (2, 2, 2) | 60.89 | 76.61 | 34.95 |  | 338.2 (-35.47) | 105.0 (-63.92) |
| PRO-SCALE (3, 3, 3) | 61.87 | 78.44 | 37.33 |  | 378.3 (-27.82) | 145.0 (-50.17) |

**Table 3: Impact of TRC module. Use of ${\bm{s}}_{2}$ via TRC to enhance $\{{\bm{s}}_{3},{\bm{s}}_{4}\}$ improves the overall performance with light computational load. Here, backbone = Res50, dataset = COCO.**
| $\epsilon$ |  | PQ(%) |
| --- | --- | --- |
| 0.01 |  | 60.05 |
| 0.10 |  | 60.58 |
| 1.00 |  | 60.32 |
| 5.00 |  | 60.41 |
| 10.00 |  | 60.34 |

### 4.1 Main Results

The analysis on the COCO dataset is presented in Tab. 1. We use M2F [] as reference for performance and present the following insights. First, PRO-SCALE is significantly computationally cheaper than baseline efficient M2F-style models such as Lite-M2F and RT-M2F. For example with SWIN-T backbone, PRO-SCALE achieves a PQ of 52.82 and with a computational cost of 171.70 GFLOPs. This PQ is 11.46 points better than that of RT-M2F, while enabling a $\sim$52% lighter transformer encoder. Compared to Lite-M2F, PRO-SCALE shows a better overall segmentation while decreasing approximately $\sim$52% of GFLOPs compared to $\sim$32% of Lite-M2F. Similar performance improvements can be observed over M2F and MaskFormer [] with Res50 backbone. Second, PRO-SCALE outperforms non-M2F-style models YOSO [] in panoptic segmentation by at least $\sim$2 points. YOSO has a slightly better computational load than PRO-SCALE, while ReMax [] has competitive accuracy. Note that, ReMax focuses on the training pipeline of panoptic segmentation [], which is orthogonal to our approach to design efficient segmentation architectures. Similarly, in comparison with a recent universal segmentation architecture PEM [] with Res50 backbone, PRO-SCALE shows stronger results in severe GFLOPs reduction settings ($p_{1},p_{2}$, $p_{3}$ = 1,1,1).
The analysis on Cityscapes dataset segmentation is shown in Tab. 2. Similar to COCO, PRO-SCALE is computationally most efficient compared to both M2F-style models (Lite-M2F and RT-M2F) and at-par with non-M2F-style models (YOSO and ReMax []) with extremely competitive performance. For example, PRO-SCALE with SWIN-T backbone achieves 63.06 PQ *vs*. Lite-M2F’s 62.29 PQ while having comparatively $\sim$27% less GLOPS than M2F. Similarly, compared to non-M2F-style models in Res50 backbone, PRO-SCALE demonstrates competitive performance with no specific panoptic segmentation design like YOSO [], or much fewer training iterations with specific panoptic segmentation training strategies like ReMax [] (200K vs 90K iterations).

### 4.2 Ablation Study

Figure: Figure 5: Impact of LPE module. Per-pixel embeddings produced by LPE do not significantly harm the performance but demonstrate a strong impact on the computational reduction. Here, backbone= SWIN-T, PRO-SCALE configuration: $c_{1}=(1,1,1)$, $c_{2}=(2,2,2)$, $c_{3}=(3,3,3)$, models = w/o LPE and w/ LPE), dataset = COCO.

##### Impact of TRC module.

Our TRC module is designed to incorporate the benefits of large scale feature ${\bm{s}}_{2}$ without increasing computational overhead in PRO-SCALE. In Tab. 4, we can see that the TRC module consistently improves the performance of the proposed framework demonstrating the efficacy of this unit. Moreover, this unit only costs 0.006 GFLOPs for the COCO dataset. Finally, we also analyze the impact of $\epsilon$ on the TRC module in Tab. 4 and find that it overall improves the performance of segmentation with best performance at $\epsilon=0.1$.

##### Impact of LPE module.

Our LPE module is aimed to drastically decrease the computational overhead without impairing the segmentation performance in PRO-SCALE. To analyze this, we compare LPE with the M2F’s convolutional layer based embedding layer module in Fig. 6. Here, we can observe that compared to this convolutional unit with learnable parameters, LPE is extremely effective. For example, for PRO-SCALE configuration $c_{3}=(3,3,3)$, LPE reduces the computations by $\sim$40% with performance degradation of 0.12 PQ points. Similarly, $c_{2}=(1,1,1)$ and $c_{2}=(2,2,2)$ provide similar computational benefits with 0.43 PQ points degradation. This demonstrates the favorable performance-computation trade-off LPE provides in PRO-SCALE.

**Table 5: LPE module with Lite-M2F. LPE can be flexibly integrated to improve the computational load of Lite-M2F [] with minimal performance degradation. Backbone = SWIN-T.**
|  |  | Performance |  | GFLOPs (% decrement) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Model | PQ | mIOUp | APp |  | Total | Tx. Enc. |
|  | Lite-M2F | 52.70 | 63.52 | 42.26 |  | 188.00 | 79.78 |
| COCO | + LPE | 52.32 | 63.08 | 41.10 |  | 154.60 (-17.77) | 43.92 (-44.95) |
|  | Lite-M2F | 62.29 | 79.43 | 36.57 |  | 428.7 | 172.00 |
| Cityscapes | + LPE | 61.35 | 79.05 | 35.91 |  | 351.4 (-18.03) | 94.69 (-44.95) |

To further demonstrate the potency of our proposed LPE module, we incorporate LPE in Lite-M2F and analyze the results in Tab. 5. We can observe that LPE shows similar impact on Lite-M2F: $\sim$45% less transformer encoder GLOPS with only $\sim$0.5-1 PQ point trade-off.

Figure: Figure 7: Analysis of PRO-SCALE configurations ($p_{1},p_{2},p_{3}$). Increments in different $p_{i}$ shows different performance-computational trade-off. Backbone = SWIN-T.

##### Variations in encoder configuration.

Here, we analyze the impact of PRO-SCALE configuration on model performance in Tab. 8. We can make some key observations. First, we observe that PRO-SCALE shows stronger performance if any $p_{i}>1$. This is trivial as more computations are allowed for the corresponding layers. Second, increasing $p_{3}$ and $p_{2}$ have different performance-efficiency trade-off: for COCO, $p_{2}$ *vs*. ($p_{2}+3$) results in only increment of $\sim 6.5$ GFLOPs with $\sim 1.0\%$ PQ improvement whereas $p_{3}$ *vs*. ($p_{3}+3$) results in a higher computational load of $\sim 32$ GFLOPs increase with $\sim 1.6\%$ PQ performance improvement. We make similar observations for the Cityscapes dataset for ($p_{1}+2$) *vs*. ($p_{2}+2$).

##### Variations in backbone architectures.

We analyze the performance of PRO-SCALE with different backbones in Tab. 6. Specifically, we observe the impacts of SWIN-T, Res50, and MViT-T in the segmentation performance. PRO-SCALE is versatile in providing strong performance-efficiency across diverse backbone types.

##### Variations in backbone pre-trained weights.

We analyze the performance of PRO-SCALE with different pre-training strategies of the backbones in Fig. 8. Specifically, we initialize SWIN-T with supervised learning (SL) and self-supervised learning (SSL) based ImageNet-1K weights. For SSL pre-training, we employ MoBY []. We can see that PRO-SCALE can gain performance robustness of SSL weights while providing the same trade-offs as SL weights. For example, when $c_{1}=(p_{1},p_{2},p_{3})=(1,1,1)$, MoBY shows an improvement of $\sim$1.6% APp with the same efficiency gains. On average, MoBY pre-trained backbone provides lower performance degradation compared to SL weights, especially in instance segmentation (Fig. 8, right).

Figure: Figure 9: Qualitative visualizations. We visualize few examples of predicted panotic maps from M2F [] and PRO-SCALE. Even with $52\%$ encoder GFLOPs reduction, PRO-SCALE shows better quality panoptic maps (backbone = SWIN-T, PRO-SCALE config. = (3, 3, 3)). Zoom-in for best view.
Refer to caption: extracted/2404.14657v1/figures/qual_result_images/coco/ex_1/000000001503.jpg

##### Qualitative analysis.

We show some examples of predicted panoptic maps in Fig. 9 with SWIN-T backbone. We set PRO-SCALE configuration to (3, 3, 3). Compared to M2F [], PRO-SCALE consistently shows strong performance with drastically fewer GLOPs in everyday scenes (top row of Fig. 9 on COCO) as well as complex driving scenes (bottom row of Fig. 9 on Cityscapes).

## 5 Conclusion

In this paper, we propose an efficient transformer encoder PRO-SCALE for Mask2Former-style universal segmentation framework. It reduces the computational load by a large margin with minimal degradation in performance. The core principle of PRO-SCALE is to progressively ‘expand’ the length of the tokens with the layers of the encoder. Since we skip using tokens from large scale features in the initial layers of PRO-SCALE, we further employ a token re-calibration module to leverage its benefits for performance improvement without having its computational overhead. Similarly, a light per-pixel embedding module to alleviate the computational overhead arising from creating per-pixel embeddings without much sacrifice in performance. Our extensive experiments demonstrate that PRO-SCALE is significantly lighter than prior prominent methods while maintaining competitive universal segmentation performance.

Supplementary Material for “Progressive Token Length Scaling in Transformer Encoders for
Efficient Universal Segmentation”

Abhishek Aich^†, Yumin Suh^†, Samuel Schulter^†, Manmohan Chandraker^†,‡

^†NEC Laboratories, America, USA, ^‡University of California, San Diego, USA

## A Experiment Details

##### Datasets.

We examine PRO-SCALE with two image segmentation datasets: COCO [] (80 “things”, 53 “stuff” categories) and Cityscapes [] (8 “things”, 11 “stuff” categories). COCO has 118K training and 5K validation images. Cityscapes has 2975 training and 500 validation images.

##### Baselines.

We compare our proposed framework with the original M2F, along with some recently proposed efficient transformer encoders for detection [, , ]. Specifically, we replace the transformer encoder of M2F with encoder architectures proposed in Lite-DETR [] and RT-DETR []. We call these Lite-M2F and RT-M2F, respectively. Note that, we opted for the “Lite-DETR H3L1-(6+1)$\times$1” setup without its key-aware deformable attention []. However, we modified this setup to (5+1) when implementing Lite-M2F. For completeness, we also compare with recent non-M2F-style panoptic segmentation models YOSO [], RAP-SAM [] and ReMax [].

##### Training details.

All our training details and underlying strategy follow [], including for our baselines. We use Detectron2 [] and PyTorch [] for implementation. We employ the AdamW optimizer [] with a step learning rate schedule. The initial learning rate is 0.0001. The backbone learning rate is multiplied with 0.1 and its weight decay is set as 0.05. We decay the learning rate at 0.9 and 0.95 fractions of the total training steps by a factor of 10. For COCO, our models are trained for 50 epochs. For Cityscapes, we use 90k iterations. Batch size is set as 16. For data augmentation and calculating GFLOPs, we follow the strategies exactly as M2F. We report the average GFLOPs for all cases. We use distributed training with 8 A6000 GPUs.

##### Loss functions.

We use the exact same loss functions and weights as []. In particular, we use the binary cross-entropy loss and the dice loss [] for our mask loss. Both loss functions use a weight of 5.0. The final loss is a combination of mask loss and classification loss (cross-entropy loss). We set the classification loss weight as 2.0 for all classes, except 0.1 for the ‘no object’ class. Finally, we apply the identical post-processing methodology as described in [] to obtain the desired output formats for panoptic, semantic, and instance segmentation predictions.

## B Additional Results

##### Object detection.

We applied PRO-SCALE to object detection on DINO []. Clearly, PRO-SCALE can integrate with object detection models and improve their accuracy and efficiency trade-off. All the models are trained with COCO dataset. The PRO-SCALE configuration is ($p_{1}$, $p_{2}$, $p_{3}$) = (3,3,3). The backbone is Res50.

##### Open-vocab universal segmentation.

We used a recent state-of-the-art open-vocabulary image segmentation method MackCLIP [] . Our method PRO-SCALE can easily work with M2F based open-vocab segmentation and reduce the computations while having better performance. In the table below, we followed training and testing protocols of [] and trained our model with COCO train set. We perform evaluation on COCO val set and cross-evaluation on ADE20K [] val set. The PRO-SCALE configuration is ($p_{1}$, $p_{2}$, $p_{3}$) = (3,3,3) and the backbone is Res50. Note that we can employ a different PRO-SCALE configuration to further reduce the computations.

**Table T1: PRO-SCALE for object detection. PRO-SCALE can easily work with DETR [] based object detection models (here, DINO []) and reduce the computations while having better performance. The PRO-SCALE configuration is ($p_{1}$, $p_{2}$, $p_{3}$, $p_{4}$) = (3,3,3,3), backbone is Res50, training dataset is COCO, and trained for 12 epochs.**
|  | Performance |  | GFLOPs (% decrement) |  |
| --- | --- | --- | --- | --- |
| Model | AP | AP50 |  | Encoder |
| DINO [] | 49.0 | 66.60 |  | 179.00 |
| + PRO-SCALE | 49.4 | 66.67 |  | 126.00 (-29.61) |

**Figure F1: Token redundancy visualization.**
| Pooling | PQ ($\uparrow$) |
| --- | --- |
| avg-pooling | 61.47 |
| max-pooling | 61.87 |

##### LPE pooling analysis.

We analyzed the LPE module with ‘average pooling’ in place of ‘max-pooling’ on the Cityscapes dataset with Res50 backbone and ($p_{1}$, $p_{2}$, $p_{3}$) = (3,3,3). We observed that the performance of the LPE module performs better with max-pooling (61.87% PQ) than average pooling (61.47% PQ).

##### Token redundancy visualization.

We provide an explicit qualitative visualization that proves the token redundancy in early stages of the transformer encoder in Fig. F1. This analysis is on 100 randomly chosen images from COCO val set. In this figure, the $x$-axis represents the distance of neighborhood tokens from the candidate token (along the token dimension) and the $y$-axis represents the (average across the number of images) cosine similarity between the two. It can be observed that in larger scale tokens, the token similarity is higher than smaller token as the neighborhood distance increases. This clearly suggests that the information redundancy of larger scale tokens is comparatively higher than smaller scale tokens.

##### TRC effect visualization.

We provide visualizations of impact of our token re-calibration (TRC) module in two examples shown in Fig. F2. It can be observed that the TRC module introduces stronger focus on parts of images that need to be segmented by the model.

Figure: Figure F2: Effect of TRC module. It can be observed that the TRC module introduces stronger focus on parts of images that need to be segmented by the model.
Refer to caption: extracted/2404.14657v1/figures/supp_images/TRC_impact_example_1.jpg

##### Larger backbone analysis.

We present some additional results with larger backbones namely ResNet101, SWIN-Small, SWIN-Base, and SWIN-Large on Cityscapes dataset in Fig. T4. PRO-SCALE reduces GFLOPs in all cases while providing a strong accuracy-efficiency trade-off. The PRO-SCALE configuration is ($p_{1}$, $p_{2}$, $p_{3}$) = (3,3,3). Note that, we can also employ a different PRO-SCALE configuration ($p_{1}$, $p_{2}$, $p_{3}$) to further reduce the computations.

**Table T4: Impact of large backbones. PRO-SCALE can provide significant efficiency boosts across various backbones. Here, $(p_{1},p_{2},p_{3})$ = (3, 3, 3), dataset = Cityscapes.**
|  | Performance ($\uparrow$) |  | GFLOPs ($\downarrow$), (% decrement) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Backbone<br>type | PQ | mIOUp | APp |  | Total | Tx. Enc. |
|  | 64.80 | 81.80 | 40.70 |  | 724.3 | 281 |
| SWIN-S | 64.41 | 80.19 | 39.90 |  | 578.5 (-20.13) | 135 (-51.96) |
|  | 66.10 | 82.70 | 42.80 |  | 1051.2 | 283 |
| SWIN-B | 65.27 | 81.97 | 41.12 |  | 905.4 (-13.87) | 137 (-51.59) |
|  | 66.60 | 82.90 | 43.60 |  | 1949.7 | 287 |
| SWIN-L | 65.93 | 82.77 | 41.80 |  | 1803.9 (-7.48) | 141 (-50.87) |
|  | 62.40 | 78.60 | 37.70 |  | 679.7 | 291 |
| Res101 | 61.28 | 76.50 | 36.49 |  | 533.9 (-21.45) | 145 (-50.17) |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: What distinguishes the LPE module from traditional fully convolutional network (FCN) style upsampling methods?

If I understand correctly, the computational costs in P1 and P2 are quadratically lower compared to P3. Would it be advantageous to use a different number of layers for each split instead of maintaining the same number of layers across all splits?
soundness: 3
strengths: The progressive integration of finer-grain information within the transformer encoder is an intuitive and straightforward approach that proves to be both simple and effective.

A comprehensive set of experiments on segmentation and detection tasks validate the design choices of the PRO-SCALE architecture, demonstrating its effectiveness.

The paper is clearly written and easy to follow, enhancing its accessibility and understanding.
summary: This paper proposes to Progressive Token Length Scaling, a strategy designed to enhance the efficiency of transformer encoders in universal segmentation models. The authors identify that the state-of-the-art model, Mask2Former, devotes over 50% of its computational resources to the transformer encoder, largely because it processes a full-length token representation from all backbone feature scales at every encoder layer. This paper addresses this inefficiency by progressively scaling the token length with each encoder layer, thereby significantly reducing computational demands—specifically, achieving around 52% reduction in GFLOPs—while maintaining performance levels on the COCO dataset.
weaknesses: None

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: For Figure6, could you provide more visualization comparisons with also some baselines like Lite-M2F, RT-M2F, or PEM?
soundness: 3
strengths: 1. The paper presents its ideas with clarity and good technical writing.
2. The experimental validation is comprehensive, featuring:
    - Thorough comparative analyses
    - Well-structured ablation studies
    - Clear demonstration of each component's contribution to overall performance
summary: The paper introduces an efficient transformer encoder architecture that progressively increases token length and feature scale, coupled with a LPE module. This approach achieves an effective balance between computational efficiency and segmentation quality.
weaknesses: 1. In Table9 for FPS comparison, while Lite-M2F and RT-M2F are used as baselines in other evaluations, a complete comparison with all baseline models on FPS would strengthen the efficiency claims
2. The paper states that ReMaX is orthogonal to the approach and ineffective on larger models, but could you provide quantitative evidence? Currently, the table results only demonstrate ReMaX has a good performance.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Why not use depth-wise 3x3 convolutions for the LPE module?
soundness: 3
strengths: 1. The writing and motivation of this paper is very clear. 
2. The proposed method seems to achieve a better trade-off than several recent related approaches.
summary: This paper proposes the “PRO-SCALE” strategy to reduce the computational cost of Mask2Former encoders. The proposed approach can significantly reduce computations with minimal sacrifice in performance.
weaknesses: 1. Although the authors emphasize the computational burden of the transformer encoder and propose dedicated methods to reduce computations, the overall reduction in proportion is not very obvious. The descriptions in the abstract seem to overstate the effect. It is better to clarify that 52% GFLOPs reduction in computation is for the encoder.

2. A figure to intuitively show the comparisons in trade-off (speed vs performance or FLOPs vs performance) is necessary for understanding the practical value of the approach. 

3. The LPE module also contributes a lot to the reduction in computation. However, it is mainly because the computational efficiency of vanilla convolutions is too low. This further indicates that the proposed PRO-SCALE is not so important for the reduction of computational complexity of the entire system. Strictly speaking, the pixel embeddings generation module is not part of the transformer encoder. Compared to the LPE module, the TRC module in the appendix seems more interesting.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: I would like to ask the authors to address my concerns as formulated in the “weaknesses” section, to answer the questions posed there, and to revise the manuscript accordingly.

One additional question:

1.	How does the presence of redundancy in the feature tokens relate to the presented PRO-SCALE method? Appendix B shows that higher-resolution features have a higher cosine similarity, but it is not clear to me how this observation motivates the PRO-SCALE design, where fewer deformable attention layers are applied to high-resolution features. Could the authors clarify this relation?

---

**Update after author discussion.** After reading the different reviews, the authors' response, and the revised manuscript, and asking some follow-up questions, I decide to keep my original rating. 

While I still believe that the idea of improving the efficiency of the decoder of Mask2Former can be valuable, I mainly have concerns about the actual efficiency of the proposed method. Importantly, the newly provided *prediction speed* results show that the actual efficiency improvement obtained by PRO-SCALE is limited. While PRO-SCALE obtains high FLOPs reductions, the impact on the prediction speed in terms of *frames per second* (FPS) in considerably lower. As a result, PRO-SCALE does not obtain a better *prediction speed vs. segmentation performance* balance than existing method Lite-DETR/Lite-M2F. Furthermore, it turns out that the impact of the LPE module on the prediction speed is almost negligible, while it also causes a segmentation performance drop.

Because the main purpose of the paper is to improve efficiency while keeping the segmentation performance as high as possible, I believe the value of the paper is considerably limited when (a) the proposed method is not shown to obtain a better *prediction speed vs. segmentation performance* balance than a highly related existing method (Lite-M2F/Lite-DETR), and (b) the actual efficiency improvement (in terms of FPS) of one of the main contributions (LPE module) is almost negligible while causing a drop in segmentation performance.

In their response, the authors mention that other benefits of reduced FLOPs are (i) improved energy efficiency because FLOPs correlate with energy use, and (ii) better compatibility and easier deployment on edge devices. However, Henderson et al. [a] experimentally show that there is little correlation between FLOPs and energy usage when comparing across different model architectures. As PRO-SCALE has a different architecture than the default Mask2Former, there is no guarantee that the reduced FLOPs of PRO-SCALE will translate to reduced energy use. As reduced energy usage is also not shown experimentally, this benefit cannot be verified. As for compatibility and deployment on edge devices, the paper provides no evidence or references for these benefits, so these benefits can also not be verified.

Overall, while also considering the strengths of the work and the other reviews, these weaknesses cause me to keep my original rating.

[a] Henderson et al., "Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning," JMLR 2020.
soundness: 2
strengths: 1.	The proposed method is simple but effective. It is simple because there are only two relatively small changes with respect to the original transformer encoder of Mask2Former: (a) the first deformable attention layers are applied to only a subset of the feature tokens, and (b) the model applies max-pooling on the highest-resolution features. By doing so, PRO-SCALE can reduce the total number of FLOPs of Mask2Former by up to 27%, without causing a drop in Panoptic Quality (PQ), mean IoU or Average Precision (AP). Although the adjustments to Mask2Former are minimal and the technical innovation of PRO-SCALE is limited, I believe it is valuable because the paper critically examines an inefficient component of a frequently used model, and finds that it can be made considerably more efficient while keeping the performance roughly the same, with only minor changes to the design.
2.	Through experiments, PRO-SCALE is not only compared to Mask2Former with its default transformer encoder, but also to existing efficient versions of the transformer encoder that were proposed for object detection method DETR, i.e., Lite-M2F and RT-M2F in Tab. 1 and Tab. 2. The results of these comparisons show that PRO-SCALE also achieves a better efficiency-accuracy balance than these existing methods. This demonstrates the value of PRO-SCALE compared to these existing methods. 
3.	The paper contains many ablations and additional experiments (Tab. 3 to Tab. 7), which properly show the impact of different design choices and demonstrate the effectiveness of PRO-SCALE in different experimental settings, e.g., with different backbones or pre-training. 
4.	Through experiments, the paper shows that PRO-SCALE is not only effective on Mask2Former, but also on other models for other tasks. The results in Tab. 8, Tab. 10 and Tab. 11 show that PRO-SCALE achieves similar FLOPs reductions in the encoder when combined with object detection model DETR, two open-vocabulary segmentation models, and instance segmentation model Mask-DINO, while obtaining similar or better segmentation performance. This shows that the proposed PRO-SCALE method is more generally applicable than just on Mask2Former.
summary: This paper presents *PRO-SCALE*, a method to improve the efficiency of the “transformer encoder” component of the state-of-the-art Mask2Former model for universal image segmentation. In Mask2Former, this transformer encoder (also called ‘pixel decoder’ [a]) takes the tokens of multi-resolution features from the backbone, applies multiple layers of deformable attention to all tokens, and then feeds them to the segmentation decoder. In contrast, PRO-SCALE applies the first deformable attention layers only to a small number of low-resolution feature tokens, and then gradually adds higher-resolution features in subsequent layers. By doing so, the number of operations is lower than in the original transformer encoder. Additionally, to efficiently generate the highest-resolution features that the decoder needs, PRO-SCALE contains a *Light Pixel Embedding* (LPE) module that applies max-pooling to high-resolution features from the backbone. With experiments on multiple datasets and with multiple backbones, PRO-SCALE is shown to reduce the total number of FLOPs by up to 27%, while keeping the segmentation quality (PQ, mIoU, AP) roughly the same as for Mask2Former. Additional experiments show that PRO-SCALE is also effective in combination with open-vocabulary and object detection models.

[a] Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation”, CVPR 2022.
weaknesses: 1.	The paper does not clearly explain how the original Mask2Former model generates the high-resolution *per-pixel embedding map* ${\mathcal{E}_{emb}}$, and why this is less efficient than how PRO-SCALE does it with LPE. L242-L243 states that $\mathbf{s}_1$ serves the purpose of creating the per-pixel embedding map, and that it uses a convolutional layer, but there is no clear description or depiction of the exact manner in which this is done. As a result, it is also not clear why the newly proposed LPE module is more efficient.
2.	Related to this, some details are missing about the exact operation of the LPE method. Concretely, what is the stride of the MaxPool2D operation? Is it the same as the kernel size? If the stride is >1, then what is the impact of changing the stride (and therefore the resolution of ${\mathcal{E}_{emb}}$) on the results, both qualitatively and quantitatively? This information is currently not available.
3.	Sec. 3.1 (L233-L236) briefly mentions that PRO-SCALE additionally uses a so-called *token recalibration* operation, which enriches small-scale features with high-scale features to further enhance the segmentation accuracy. However, despite being part of the PRO-SCALE method, this operation is not visualized in Fig. 3, and not explained properly in Sec. 3.1. As a result, (a) Fig. 3 makes it seem like this operation does not exist at all, and (b) it is unclear from the main paper how a part of the method works. In other words, this *token recalibration* operation should be explained and visualized in the main paper.
4.	Related to the previous point, the efficiency of the *token recalibration* operation is not evaluated, neither in the main paper nor in the appendix. As a result, it is not clear if it is actually efficient.
5.	For most experiments, the paper reports the FLOPs but not the latency/FPS/throughput of the model. The FPS is only reported for the overall model in Tab. 9. As a result, for most configurations, it is not clear how/if the reduction in FLOPs translates to a speedup when the model is run on a GPU. The results of the paper would be stronger if the FPS/latency of the model was also reported for other experiments, i.e., at least for the main results in Tab. 1 and Tab. 2, and the ablations in Tab. 3, Tab. 5 and Fig. 4, but ideally even more.
6.	PRO-SCALE does not yield a significant efficiency improvement when used in combination with large backbones, e.g., Swin-L (see Tab. 7). Of course, this is expected, as the Swin-L backbone accounts for a much larger portion of the overall number of FLOPs than Swin-T, but it is still a weakness because PRO-SCALE’s value in larger models is limited.
7.	The abstract contains a misleading statement. L023 states that PRO-SCALE can achieve a ~52% GFLOPs reduction with no drop in performance on the COCO dataset. This statement implies that, compared to the original Mask2Fomer, the overall PRO-SCALE model requires ~52% fewer GFLOPs. However, Tab. 1 shows that this 52% GFLOPs reduction concerns the encoder only, and that the overall GFLOPs reduction of the model is ~27%. The statement in the abstract should be altered by either specifying that the 52% reduction concerns the encoder, or changing the number from 52% to 27%.
8.	Tab. 8, Tab. 10 and Tab. 11 only contain GFLOPs results for the encoder, not for the entire model. As a result, it is not clear what the actual overall improvement is of PRO-SCALE.
9.	L323 & L358 state that ReMaX is limited by the inherent efficiency of the model, but the efficiency for ReMaX is not provided in Tab. 1 or Tab. 2. In other words, this claim is not substantiated.
10.	L454-L455 states that, on average, the MoBY pre-trained backbone causes lower performance degradation than SL pre-trained weights, especially for instance segmentation. However, per Fig. 5 (right), compared to the Mask2Former baseline, the average drop for MoBY over all 4 settings is -2.35 AP, while it is -1.49 for supervised learning. Therefore, this statement is incorrect. This statement is not very important for the main message of the paper, so it doesn’t impact the value of the proposed method, but it should be altered nevertheless.

Some minor points, which do not significantly affect my overall rating:

11.	The text contains several mistakes/errors. Some examples:

    a.	L039 – “framework exhibit exceptional performance” => “framework exhibits exceptional performance”

    b.	L136 – “making Mask2Former universal segmentation model” => “making the Mask2Former universal segmentation model”

    c.	L153 – “map and class prediction heads” => “mask and class prediction heads”

    d.	L358 – Not clear what is meant by “becomes ineffective efficient”

    e.	L377 – "effieiciency" => "efficiency"

    f.	L376-L377 – “strong … than” => “stronger … than”

    g.	L436 – “$p_2$ vs. ($p_2$ + 3)” should be “$p_2$ vs. ($p_2$ + 2)”, as the baseline is $p_2 = 1$ and the comparison is “1 vs. 3”.

    h.	L437 – Likewise: “$p_3$ vs. ($p_3$ + 3)” => “$p_3$ vs. ($p_3$ + 2)”.

    i.	L503 – The caption of Tab. 11 states that PRO-SCALE achieves a better PQ, but the PQ is not reported in Tab. 11.

    j.	L534-L535 – “for Mask2Former universal segmentation framework” => “for the Mask2Former universal segmentation framework”

12.	In Tab. 3, the experimental setting is not explicitly indicated. The numbers correspond with Swin-T on Cityscapes, but this is not explicitly mentioned in the caption. This should be mentioned, so that the table can be understood without having to check other tables.

13.	The paper uses an inconsistent number of decimals for results on the same metrics. For instance, in Tab. 3: 132 GFLOPs for $C_1$ and 73.49 for $C_2$. It would be better if the number of decimals was consistent.

Meta-review: The paper proposes PRO-SCALE, an efficient transformer encoder for universal segmentation that significantly reduces computational cost while maintaining performance, and introduces the Light Pixel Embedding (LPE) module to optimize the pixel embedding process within the Mask2Former architecture.

During rebuttal and discussions, three reviewers keep the original ratings and one reviewer slightly adjusts the score downward to 6, leading to the final scores of 6,6,6,5. The remaining biggest concern is limited inference speed improvement despite of significant FLOPs reduction.

After carefully  reviewing the paper and all the reviewers' comments and discussions, The AC agrees that the actual inference speed improvement is crucial for practical usage. However, Considering that this paper presents a solid method  with extensive evaluations and clear explanations, the AC recognizes its significant value in the compression for segmentation models. With an average score of 5.75, the paper is competitive for this year's ICLR conference. The AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.

Decision: Accept (Poster)