SegRet: An Efficient Design for Semantic Segmentation with Retentive Network

## Abstract

Abstract With the rapid evolution of autonomous driving technology and intelligent transportation systems, semantic segmentation has become increasingly critical. Precise interpretation and analysis of real-world environments are indispensable for these advanced applications. However, traditional semantic segmentation approaches frequently face challenges in balancing model performance with computational efficiency, especially regarding the volume of model parameters. To address these constraints, we propose SegRet, a novel model employing the Retentive Network (RetNet) architecture coupled with a lightweight residual decoder that integrates zero-initialization. SegRet offers three distinctive advantages: (1) Lightweight Residual Decoder: by embedding a zero-initialization layer within the residual network structure, the decoder remains computationally streamlined without sacrificing essential information propagation; (2) Robust Feature Extraction: adopting RetNet as its backbone enables SegRet to effectively capture hierarchical image features, thereby enriching the representation quality of extracted features; (3) Parameter Efficiency: SegRet attains state-of-the-art (SOTA) segmentation performance while markedly decreasing the number of parameters, ensuring high accuracy without imposing additional computational burdens. Comprehensive empirical evaluations on prominent benchmarks, such as ADE20K, Citycapes, and COCO-Stuff, highlight the effectiveness and superiority of our method.

## 1 Introduction

Semantic segmentation involves assigning semantic labels to every pixel within an image, thereby enabling models to comprehend and interpret scenes at the pixel level [21]. Tasks such as obstacle detection, lane tracking, and scene understanding critically depend on accurately identifying and characterizing elements like roads, vehicles, pedestrians, and other pertinent objects, particularly crucial for the safe and efficient functioning of autonomous vehicles [15, 45].

In recent years, deep learning-based methods have dominated semantic segmentation research. Classical segmentation frameworks, including Fully Convolutional Networks (FCN) [28], Mask R-CNN [18], and Pyramid Scene Parsing Network (PSPNet) [44], have significantly advanced this field. Additionally, the enormous successes of transformer-based models, such as Vision Transformer (ViT) [12], Swin Transformer [27], and Detection Transformer (DETR) [4], in Computer Vision (CV) have led to their widespread adoption as backbone architectures in semantic segmentation. However, the inherent computational complexity associated with the self-attention mechanism within Transformers poses substantial challenges. To address these complexities, several novel network architectures have emerged. ReViT [11] introduces hierarchical recursive attention, confining token interactions to local groups, thereby reducing computational overhead by eliminating global self-attention. ExMobileViT [37] combines depth-separable convolutions with adaptive token aggregation to effectively separate spatial from channel attention and significantly simplify the attention computation pipeline. EfficientFormer [23] refines model efficiency by optimizing network depth, width, and attention heads, leveraging hierarchical feature extraction to prioritize salient regions and substantially minimize redundant computations.

Figure: Figure 1: Performance comparison on ADE20K. Multi-scale inference is used for all results.
Refer to caption: x1.png

Recently, several network architectures have emerged to compete with Transformers, notably Mamba [14], RWKV [24], and RetNet [34], each demonstrating significant advantages. Mamba enhances efficiency in handling long sequences by utilizing a selective state-space model, which has inspired various extensions such as VMamba [26], VM-UNet [31], and RSMamba [5]. RWKV uniquely integrates the sequential recurrence of recurrent neural networks with the parallelized attention of transformers, employing a novel recurrent weighted key-value mechanism that efficiently processes sequential data, gaining broad adoption in both Natural Language Processing (NLP) and CV applications [24]. RetNet further improves inference efficiency and performance by streamlining the self-attention mechanism and incorporating hidden fusion modules [34].
Semantic segmentation architectures typically adopt an encoder-decoder framework, where the encoder extracts high-level semantic features from input images, and the decoder transforms these features into pixel-wise semantic labels. However, decoders often have extensive parameter counts, exemplified by models such as UperNet [35] and MaskFormer [8], presenting significant challenges in applications demanding real-time performance. Consequently, substantial research efforts focus on reducing model complexity and enhancing inference speed without sacrificing segmentation accuracy.

To overcome the aforementioned limitations, we introduce SegRet, an innovative approach featuring Vision RetNet, pre-trained on ImageNet1K, as a hierarchical feature extractor in its encoder. This encoder efficiently captures multi-scale hierarchical features ranging from lower to higher dimensional representations. Complementing this, our decoder is designed as a lightweight residual connection module containing just 2.6M parameters in its largest variant, utilizing zero-initialized layers to facilitate hierarchical feature fusion. As illustrated in Figure [1](https://arxiv.org/html/2502.14014v2#S1.F1), SegRet effectively balances accuracy with computational efficiency by capturing robust hierarchical features while significantly reducing model complexity. Empirical evaluations on benchmark datasets, including ADE20K [48], Cityscapes [9], and COCO-Stuff [3], confirm our method’s efficacy. Specifically, our Tiny variant attains mIoU scores of 49.39 on ADE20K, 81.75 on Cityscapes, and 43.32 on COCO-Stuff with merely 14.01M parameters, whereas our Large model, with 95.81M parameters, achieves superior mIoU scores of 52.23, 83.36, and 46.63, respectively.

## 2 Related Work

### 2.1 Semantic Segmentation.

Traditional semantic segmentation approaches primarily depend on manually engineered features and classifiers for pixel-level classification. These methods typically employ low-level features such as color, texture, and shape, coupled with algorithms like graph cuts and random forests, to perform segmentation tasks [46, 1, 40]. However, due to the inherent complexity and variability of real-world scenarios, traditional approaches often fail to adequately manage challenges such as occlusion and varying illumination, resulting in inaccurate segmentation outcomes.

The advent of deep learning has notably enhanced the performance of semantic segmentation. Deep learning models facilitate mapping from pixel-level features to semantic labels through upsampling and skip connections, markedly improving both accuracy and computational efficiency [30, 38, 16, 17]. Contemporary models predominantly utilize Transformer-based architectures, incorporating self-attention mechanisms to effectively capture global pixel dependencies, thereby improving the understanding and representation of semantic information within images [47, 36, 7, 39]. Recent advancements integrate zero-shot learning and prompt learning strategies into semantic segmentation, enabling the identification of new object classes without relying on annotated datasets and offering additional contextual guidance. These innovative approaches exhibit significant advantages in addressing challenges associated with data scarcity and further enhancing segmentation performance [20, 41, 43].

### 2.2 State Space Model

Given the limitations of the self-attention mechanism in effectively handling long text sequences, researchers have turned their attention toward alternative architectures to address this challenge. In particular, state-space models such as Mamba [14] and RetNet [34] have attracted considerable interest. Unlike Transformers, these architectures employ state-space modeling mechanisms tailored explicitly for sequence processing. Specifically, Mamba utilizes a selective state-space approach that achieves linear computational complexity, while RetNet employs recurrent state aggregation combined with three computational paradigms to robustly capture long-range dependencies. These advancements offer significant improvements in both efficiency and performance for long-sequence processing tasks compared to Transformer-based models.

Extending the proven efficacy of state-space models into visual domains, researchers have introduced several visual architectures based on these principles. These visual models not only inherit the capacity of state-space methods to manage extended sequences but also integrate distinctive aspects of visual data processing, enabling them to effectively handle complex visual information from images and videos. Prominent among these are Vmamba [26], Vision Mamba [50], and Vision RetNet [13]. These models have demonstrated notable success across various tasks, including image classification and object detection. Additionally, they offer innovative approaches for addressing intricate visual tasks such as video comprehension in dynamic contexts, medical image analysis involving multi-modal data fusion, and the detection of small infrared targets in low-contrast scenarios [22, 25, 6].

## 3 Method

This section presents the SegRet model, initially providing an overview of its foundational architecture. Following this, we delve into a comprehensive discussion of Vision RetNet, highlighting its effectiveness as a hierarchical feature extractor for robust feature representation. We then thoroughly examine the lightweight residual decoder, underscoring its crucial role in maintaining model accuracy while significantly reducing the number of parameters.

Figure: Figure 2: An overview of the proposed SegRet model. We use Vision RetNet (RMT) as a hierarchical feature extractor to introduce RetNet into semantic segmentation (Section [3.1](https://arxiv.org/html/2502.14014v2#S3.SS1)). For more efficient feature fusion, a zero-initialization residual decoder is applied to predict semantic segmentation masks (Section [3.2](https://arxiv.org/html/2502.14014v2#S3.SS2)).
Refer to caption: x2.png

As illustrated in Figure [2](https://arxiv.org/html/2502.14014v2#S3.F2), the SegRet model adopts an encoder-decoder structure. The encoder employs Vision RetNet to extract hierarchical features at four distinct resolutions. These multi-scale features are subsequently processed by the residual decoder, which comprises a linear mapping block and a zero-initialized residual block. After feature fusion within this decoder, the output is transformed into a semantic segmentation mask.

### 3.1 Vision RetNet Backbone

#### 3.1.1 RetNet

Firstly, we revisit the self-attention mechanism of the Transformer. For each input vector $X$, by multiplying matrices $W_{Q}$, $W_{K}$ and $W_{V}$ with $X$, we obtain the $Q$ (Query), $K$ (Key) and $V$ (Value) vectors respectively. Therefore, the self-attention is defined as:

$$ $\text{Attention}(Q,K,V)=Softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V,$ (1) $$

where $d_{k}$ is the dimension of the $K$ vectors.

To overcome challenges associated with training parallelism, efficient inference, and performance optimization commonly encountered by Transformers in complex task scenarios, the Retentive Network (RetNet) architecture has been introduced. RetNet establishes a novel foundational framework that theoretically elucidates the relationship between recurrence and attention mechanisms. Central to RetNet is the innovative retention mechanism tailored specifically for sequence modeling, facilitating three distinct computational modes: parallel, recurrent, and block-recurrent.

The retention mechanism is the core of RetNet, with the following fundamental principles:

$$ $o_{n}=\sum_{m=1}^{n}\,\gamma^{n-m}\left(Q_{n}e^{in\theta}\right)\left(K_{m}e^{ im\theta}\right)^{\dagger}v_{m},$ (2) $$

where $Q$ and $K$ are input-derived vectors generated through affine transformations. $e^{in\theta}$ and $e^{im\theta}$ serve as rotational factors that encode positional information using complex exponential forms, where $n$ and $m$ represent the positional indices within the sequence, and $\theta$ denotes the learnable parameters employed to model relative phase differences for the purpose of capturing sequential dependencies. $v_{m}$ stands for the value vector at position $m$, $\gamma$ represents an exponential decay factor, and ^† signifies the conjugate transpose.

Building upon this foundation, the parallel form is deduced as:

$$ $\displaystyle Q=(XW_{Q})\odot\Theta,K=(XW_{K})\odot\bar{\Theta},V=XW_{V},$ (3) $\displaystyle\Theta_{n}=e^{in\theta},D_{nm}=\begin{cases}\gamma^{n-m},&n\geq m \\ 0,&n<m\end{cases},$ (4) $\displaystyle Retention(X)=(QK^{\top}\odot D)V,$ (5) $$

where $\bar{\Theta}$ is the complex conjugate of $\Theta$, $\odot$ denotes the Hadamard product, $D\in\mathbb{R}^{|x|\times|x|}$ represents causal masking and exponential decay, indicating relative distances within a one-dimensional sequence.
RetNet’s exceptional performance, training parallelism, cost-effective deployment, and efficient inference collectively enable it to successfully mitigate the challenge of excessive computational complexity associated with Transformer.

#### 3.1.2 Vision RetNet

Vision RetNet [13] is among the pioneering visual state-space models introduced specifically for CV tasks. It enhances the original RetNet architecture by modifying the matrix $D$, adapting it from a unidirectional form utilized in NLP to a bidirectional form tailored for CV applications. This adaptation effectively manages causal masking and exponential decay, aligning the model’s design with the distinct requirements of CV tasks.

Formally, the BiRetention operator is defined as follows:

$$ $\displaystyle\operatorname{BiRetention}(X)$ $\displaystyle=\left(QK^{\top}\odot D^{Bi}\right)V,$ (6) $\displaystyle D_{nm}^{Bi}$ $\displaystyle=\gamma^{|n-m|},$ (7) $$

where BiRetention denotes the retention with bidirectional modeling ability.

Specifically, for the two-dimensional spatial attributes of images, the matrix D is modified to its two-dimensional version to better capture spatial relationships, expressed as:

$$ $D_{nm}^{2d}=\gamma^{|x_{n}-x_{m}|+|y_{n}-y_{m}|},$ (8) $$

where $x$ and $y$ are two-dimensional coordinates in the image.

Moreover, the high resolution typical of visual images generates an extensive number of tokens, significantly increasing computational complexity. To effectively mitigate this issue, Vision RetNet introduces a computational strategy that decomposes processing along both horizontal and vertical image axes. Specifically, it calculates attention mechanisms and distance matrices independently in these two directions, thereby considerably reducing computational overhead. The detailed computational procedure is described as follows:

$$ $\displaystyle Q_{H},K_{H}=(Q,K)^{B,L,C\rightarrow B,W,H,C},$ (9) $\displaystyle Q_{W},K_{W}=(Q,K)^{B,L,C\rightarrow B,H,W,C},$ (10) $\displaystyle Attn_{H}=Softmax(Q_{H}K_{H}{}^{T})\odot D^{H},$ (11) $\displaystyle Attn_{W}=Softmax(Q_{W}K_{W}{}^{T})\odot D^{W},$ (12) $\displaystyle D_{nm}^{H}=\gamma^{|y_{n}-y_{m}|},\quad D_{nm}^{W}=\gamma^{|x_{n }-x_{m}|},$ (13) $\displaystyle ReSA_{dec}(X)=Attn_{H}(Attn_{W}V)^{T},$ (14) $$

where $B$ denotes the batch size, $H$ and $W$ indicate the number of non-overlapping patches along the height and width of the input image, respectively, $C$ is the feature dimensionality of each token, and $L=H\times W$ corresponds to the total number of visual tokens obtained by partitioning the image and flattening the patches into one-dimensional embeddings.

As shown in Fig [2](https://arxiv.org/html/2502.14014v2#S3.F2), the proposed SegRet model consists of Vision RetNet, which combines the output features of Vision RetNet blocks after each downsampling step into hierarchical feature matrices and feeds them into the decoder for further processing.

### 3.2 Lighting residual decoder

Currently, most widely adopted decoders rely heavily on intricate CNN or Transformer architectures, leading to substantial parameter counts and compromised real-time performance. To overcome these limitations, we introduce a lightweight residual decoder. As depicted in Figure [2](https://arxiv.org/html/2502.14014v2#S3.F2), the decoder comprises two main components: a linear mapping block and a zero-initialized residual structure. The detailed architecture of our decoder is presented as follows:

The decoder accepts as input a set of features derived from various layers, represented by $\tilde{F}=[f_{1},f_{2},...,f_{n}]$, where each $f_{i}$ corresponds to features extracted from the $i$-th layer and exhibits different channel dimensions. The primary goal of the decoder is to consolidate these multi-scale features into a coherent output with dimensions $n_{cls}\times H\times W$, where $n_{cls}$ denotes the total number of classes, and $H$ and $W$ specify the output image’s height and width, respectively. The decoding procedure is formally described as follows:

Initially, each input feature $f_{i}$ is subjected to a linear transformation to unify its channel dimension to a common size $C$, yielding the transformed feature $F_{i}$. This step can be formulated as:

$$ $F_{i}=Linear(f_{i}).$ (15) $$

To preserve the original information while enhancing feature representation, we apply a residual connection between the original feature $f_{i}$ and its linearly transformed counterpart $F_{i}$. This residual connection is realized through a $1\times 1$ convolutional layer initialized with zeros, ensuring that the output feature map maintains the same channel dimension as $f_{i}$. Formally, the output of the residual connection layer is defined as:

$$ $F^{\prime}_{i}=f_{i}+\text{Zero-initialized Conv}(F_{i}),$ (16) $$

where zero-initialized Conv denotes a $1\times 1$ convolution operation with zero initialization. All features $F^{\prime}_{i}$ are upsampled to $1/4$ of the image size. The upsampled feature can be represented as $\hat{F}_{i}=\text{Upsample}(F^{\prime}_{i})$. The upsampled features $\hat{F}_{i}$ are then concatenated to obtain the output image. Assuming the merged feature is $M$, the concatenation operation can be represented as:

$$ $M=Concat(\hat{F}_{1},...,\hat{F}_{n}).$ (17) $$

The dimensions of $M$ are $[4C,H,W]$.
Finally, the merged feature $M$ undergoes a convolution mapping to adjust the channel size to the required number of classes $n_{cls}$. This process can be expressed as:

$$ $Output=Conv(M).$ (18) $$

## 4 Experiments

We conducted comprehensive comparisons with recent SOTA methods on the ADE20K, Cityscapes, and COCO-Stuff datasets, demonstrating the effective integration of RetNet into the semantic segmentation domain through our SegRet model. By evaluating model parameters alongside mean Intersection over Union (mIoU) scores, our experiments substantiate that SegRet is a robust and competitive solution for semantic segmentation tasks.

##### Datasets

ADE20K [48], Cityscapes [9], and COCO-Stuff [3] are widely recognized semantic segmentation benchmarks. ADE20K is a large-scale scene parsing dataset comprising over 20,000 high-resolution images that span a diverse range of scenes and environments. Each image is densely annotated with 150 distinct semantic categories, encompassing objects, environments, and parts. Cityscapes focuses on the analysis and understanding of urban scenes. It includes 5,000 finely annotated high-resolution images captured from 50 cities across Germany. Each image provides pixel-level annotations across 19 semantic categories, covering typical urban elements such as roads, vehicles, and pedestrians. COCO-Stuff is a curated subset of the Microsoft COCO dataset, containing more than 10,000 images. Each image is annotated with 171 categories, including common objects, scenes, and background elements. Distinguished by its rich and fine-grained annotations, COCO-Stuff serves as a robust benchmark for evaluating the performance of semantic segmentation models.

### 4.1 Implementation details

To enhance the model’s adaptability across diverse application scenarios, we introduce four SegRet variants of varying capacities: Tiny, Small, Base, and Large. The corresponding decoder parameter counts are 0.814M, 0.814M, 0.871M, and 2.607M, respectively. Each variant employs the same backbone as Vision RetNet [13], pre-trained on the ImageNet1K dataset [10]. The decoder’s channel dimensions $C$ for the four configurations are set to [256, 256, 256, 512], respectively. The SegRet model is implemented based on the MMSegmentation framework [29] and trained using four NVIDIA A40 GPUs. For data preprocessing, images from the ADE20K and COCO-Stuff datasets are randomly flipped and cropped to a resolution of $512\times 512$, while Cityscapes images are processed to $512\times 1024$. Specifically, for the ADE20K dataset, the inputs of the Large variant are randomly cropped to $640\times 640$. To ensure a fair comparison, we refrain from employing advanced training strategies such as OHEM [32]. Instead, we adopt the cross-entropy loss function in conjunction with the AdamW optimizer, using a learning rate of 0.0001 and a weight decay of 0.01. Batch sizes are configured as 16 for ADE20K and COCO-Stuff, and 8 for Cityscapes. Training is conducted for 160,000 iterations on ADE20K and Cityscapes, and 80,000 iterations on COCO-Stuff. Evaluation protocols are aligned with those used in Mask2Former[7] to ensure consistency and comparability.

### 4.2 Main results

We quantitatively analyze SegRet’s results on ADE20K, CityScape, and COCO-stuff, showcasing its remarkable performance in semantic segmentation tasks. It should be noted that all FLOPs are calculated at an input resolution of $512\times 2048$.

##### ADE20K

Table [1](https://arxiv.org/html/2502.14014v2#S4.T1) presents a comparative analysis of SegRet-Tiny against recent SOTA methods in terms of parameter count and mIoU. The results demonstrate that SegRet achieves superior performance among models with comparable parameter scales, attaining an mIoU of 49.39 with only 14.01M parameters. For example, SegRet matches the performance of Mask2Former (Swin-T) while using less than one-third of the parameters. Moreover, when compared to Vision Mamba (Vim-Ti), which has a similar parameter count, SegRet surpasses it by 9.7% in mIoU. Additional experimental results across various model scales are included in the Appendix for further reference.

**Table 1: Comparison of the proposed SegRet-Tiny model on the ADE20K validation dataset. In comparison to SOTA methods, our Tiny variant exhibits notable advantages in both parameter efficiency and mIoU performance. "SS" and "MS" denote single-scale and multi-scale inference, respectively. The best-performing results are highlighted in bold for clarity.**
| Method | Backbone | Decoder head | Image Size | #params | mIoU(SS) | mIoU(MS) | FLOPs |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SenFormer [2] | R50 |  | 512*512 | 55M | 44.4 | 45.2 | 179G |
| SenFormer [2] | R101 |  | 512*512 | 79M | 46.9 | 47.9 | 199G |
| SegFormer [36] | MiT-B1 |  | 512*512 | 13.7M | 42.21 | 43.1 | 15.9G |
| SegFormer [36] | MiT-B2 |  | 512*512 | 27.5M | 46.5 | 47.5 | 62.4G |
| Vision Mamba [50] | Vim-Ti | UperNet | 512*512 | 13M | - | 40.2 | - |
| Vision Mamba [50] | Vim-S | UperNet | 512*512 | 46M | - | 44.9 | - |
| SeMask [19] | SeMask Swin-T | FPN | 512*512 | 35M | 42.06 | 43.36 | 40G |
| SeMask [19] | SeMask Swin-S | FPN | 512*512 | 56M | 45.92 | 47.63 | 63G |
| Swin [27] | Swin-T | UperNet | 512*512 | 60M | - | 46.1 | 236G |
| Swin [27] | Swin-S | UperNet | 512*512 | 81M | - | 49.3 | 259G |
| RMT [13] | RMT-T | FPN | 512*512 | 17M | - | 46.4 | 33.7G |
| RMT [13] | RMT-S | FPN | 512*512 | 30M | - | 49.4 | 180G |
| SenFormer [2] | Swin-T |  | 512*512 | 59M | 46 | - | 179G |
| MaskFormer [8] | Swin-T |  | 512*512 | 42M | 46.7±0.7 | 48.8±0.6 | 55G |
| Vmamba [26] | VMamba-T | UperNet | 512*512 | 55M | 47.3 | 48.3 | 939G |
| Mask2Fromer [7] | Swin-T |  | 512*512 | 47M | 47.7 | 49.6 | 74G |
| SegRet-Tiny | RMT-T |  | 512*512 | 14.01M | 48.76 | 49.39 | 72.28G |

##### Cityscapes

As presented in Table [2](https://arxiv.org/html/2502.14014v2#S4.T2), our SegRet-Tiny model demonstrates strong performance on the Cityscapes dataset. In comparison with methods such as EFCD-Small (R101) and SegFormer (MiT-B1), SegRet achieves notable improvements in both SS and MS inference, attaining 81.75% and 82.17%, respectively, while maintaining a comparable parameter count. Furthermore, when contrasted with larger models such as SeMask and Segmenter(ViT-L/16 with Seg-L-Mask/16), SegRet delivers superior mIoU(MS) scores, highlighting its robust generalization capability in MS settings. Additional performance metrics for other model scales are provided in the Appendix.·

**Table 2: Comparison of the proposed SegRet-Tiny model on Cityscapes dataset.**
| Method | Backbone | Decoder head | Image Size | #params | mIoU(SS) | mIoU(MS) | FLOPs |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SenFormer [2] | R50 |  | 512*1024 | 55M | 78.8 | 80.1 | 179G |
| SenFormer [2] | R101 |  | 512*1024 | 79M | 79.9 | 81.4 | 199G |
| ECFD-tiny [42] | R50 |  | 512*1024 | 41M | 79.91 | 81.18 | 206G |
| ECFD-small [42] | R50 |  | 512*1024 | 51M | 80.14 | 81.32 | 222G |
| ECFD-tiny [42] | R101 |  | 512*1024 | 60M | 80.5 | 81.48 | 245G |
| ECFD-small [42] | R101 |  | 512*1024 | 70M | 80.74 | 82 | 261G |
| SeMask [19] | SeMask Swin-T | FPN | 768*768 | 34M | 74.92 | 76.56 | 84G |
| SegFormer [36] | MiT-B1 |  | 1024*1024 | 13.7M | 78.5 | 80 | 243.7G |
| Segmenter [33] | DeiT-B/16 | Seg-B*/16 | 768*768 | - | - | 80.5 | - |
| Segmenter [33] | DeiT-B/16 | Seg-B*-Mask/16 | 768*768 | - | - | 80.6 | - |
| Segmenter [33] | ViT-L/16 | Seg-L/16 | 768*768 | - | - | 80.7 | - |
| Segmenter [33] | ViT-L/16 | Seg-L-Mask/16 | 768*768 | - | 79.1 | 81.3 | - |
| SegRet-Tiny | RMT-T |  | 512*1024 | 14.01M | 81.75 | 82.17 | 72.28G |

##### COCO-Stuff

The proposed SegRet-Tiny model exhibits substantial performance advantages on the COCO-Stuff dataset. As shown in Table [3](https://arxiv.org/html/2502.14014v2#S4.T3), SegRet-Tiny outperforms several SOTA methods, including MaskFormer, SenFormer, SeMask, and APPNet, achieving higher mIoU scores. Specifically, the model attains mIoU scores of 42.22 and 43.32 under SS and MS inference settings, respectively. These results underscore SegRet-Tiny’s strong performance and parameter efficiency. Additional performance comparisons across different model scales are provided in the Appendix.

**Table 3: Comparison of the proposed SegRet-Tiny model on COCO-Stuff dataset.**
| Method | Backbone | Decoder head | Image Size | #params | mIoU(SS) | mIoU(MS) |
| --- | --- | --- | --- | --- | --- | --- |
| MaskFormer [8] | R50 |  | 640*640 | - | 37.1±0.4 | 38.9±0.2 |
| SenFormer [2] | R50 |  | 512*512 | 55M | 40 | 41.3 |
| SeMask [19] | SeMask Swin-T | FPN | 512*512 | 35M | 37.53 | 38.88 |
| APPNet [49] | HRNet-W48 | APPNet+HRNet | 520*520 | 69.7M | 36.9 | - |
| APPNet [49] | HRNet-W48 | APPNet+OCR | 520*520 | 72.3M | 40.3 | - |
| SegRet-Tiny | RMT-T |  | 512*512 | 14.01M | 42.22 | 43.32 |

##### Qualitative analysis

As illustrated in Figure [3](https://arxiv.org/html/2502.14014v2#S4.F3), we performed a qualitative analysis on the ADE20K validation set, conducting a detailed comparison between our SegRet-Tiny model and the MaskFormer (Swin-T) model. The results reveal that SegRet-Tiny consistently outperforms MaskFormer, particularly in capturing fine details and reducing classification errors. This performance advantage can be largely attributed to the robust feature extraction capabilities of Vision RetNet, combined with the simplicity and effectiveness of our proposed lightweight decoder architecture.

Figure: Figure 3: Qualitative analysis on the ADE20K. The first column displays the ground truth values, while the outputs of MaskFormer and our proposed SegRet model are presented in the second and third columns, respectively.
Refer to caption: extracted/6489056/fig/test_ADE_val_00000350.png

### 4.3 Ablation Studies

In this section, we present a series of ablation studies designed to evaluate the effectiveness of the proposed residual structure in the decoder, with particular emphasis on the role of zero-initialized residual layers. Furthermore, we investigate the influence of the decoder’s channel size $C$ on model performance. All experiments were conducted using the ADE20K dataset.

##### Zero-initialized residual layer

Table [4](https://arxiv.org/html/2502.14014v2#S4.T4) presents an investigation into the effect of incorporating zero-initialized residual (ZIR) layers within the proposed decoder architecture. The experiments were conducted using the SegRet-Small variant, with all configurations kept consistent with the standard training setup. Results indicate that the inclusion of the ZIR layer yields a 0.79% improvement in mIoU, while introducing only a modest parameter increase of 0.25M These findings underscore the efficacy of the SegRet decoder design.

**Table 4: Influence of zero-initialized residual layers on SegRet**
| Method | ZIR Layer | #params | mIoU(SS) |
| --- | --- | --- | --- |
| SegRet-Small |  | 26.27M | 49.9 |
| ✓ | 26.52M | 50.69 |  |

##### The impact of the decoder channel size C 𝐶 C italic_C

We examined the influence of the decoder channel size $C$ on the performance of the proposed model. As shown in Table [5](https://arxiv.org/html/2502.14014v2#S4.T5), increasing $C$ generally leads to improved model performance, with the highest mIoU achieved when $C=512$. However, further increasing $C$ to 768 results in both increased model complexity and a decline in mIoU, suggesting diminishing returns beyond a certain capacity. Based on empirical validations, we adopted $C=256$ for the Tiny, Small, and Base variants, and $C=512$ for the Large model in our formal experimental settings.

**Table 5: The Impact of Decoder Channel Size $C$ on SegRet**
| Method | $C$ | #params | mIoU(SS) |
| --- | --- | --- | --- |
| SegRet-Large | 256 | 94.26M | 50.9 |
| 512 | 95.81M | 52 |  |
| 768 | 98.54M | 51.57 |  |

## 5 Conclusion

We introduce SegRet, a semantic segmentation model that integrates Vision RetNet as the encoder and employs a zero-initialized residual decoder. In this architecture, RetNet is leveraged for hierarchical feature extraction, while the decoder incorporates zero-initialized layers within its residual connections to enhance efficiency. Experimental results reveal that SegRet achieves remarkable performance across four variants and two benchmarks, maintaining or even improving segmentation accuracy despite a substantial reduction in parameter count. Nonetheless, a current limitation of SegRet lies in its underperformance on more specialized tasks such as medical image segmentation and remote sensing image analysis.

## References

- [1]
Arbelaez, P., Maire, M., Fowlkes, C., and Malik, J.
Contour detection and hierarchical image segmentation.
IEEE transactions on pattern analysis and machine intelligence 33, 5 (2010), 898–916.
- [2]
Bousselham, W., Thibault, G., Pagano, L., Machireddy, A., Gray, J., Chang, Y. H., and Song, X.
Efficient self-ensemble for semantic segmentation.
arXiv preprint arXiv:2111.13280 (2021).
- [3]
Caesar, H., Uijlings, J., and Ferrari, V.
Coco-stuff: Thing and stuff classes in context.
In Proceedings of the IEEE conference on computer vision and pattern recognition (2018), pp. 1209–1218.
- [4]
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S.
End-to-end object detection with transformers.
In European conference on computer vision (2020), Springer, pp. 213–229.
- [5]
Chen, K., Chen, B., Liu, C., Li, W., Zou, Z., and Shi, Z.
Rsmamba: Remote sensing image classification with state space model.
arXiv preprint arXiv:2403.19654 (2024).
- [6]
Chen, T., Tan, Z., Gong, T., Chu, Q., Wu, Y., Liu, B., Ye, J., and Yu, N.
Mim-istd: Mamba-in-mamba for efficient infrared small target detection.
arXiv preprint arXiv:2403.02148 (2024).
- [7]
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Girdhar, R.
Masked-attention mask transformer for universal image segmentation.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022), pp. 1290–1299.
- [8]
Cheng, B., Schwing, A., and Kirillov, A.
Per-pixel classification is not all you need for semantic segmentation.
Advances in neural information processing systems 34 (2021), 17864–17875.
- [9]
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B.
The cityscapes dataset for semantic urban scene understanding.
In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 3213–3223.
- [10]
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.
Imagenet: A large-scale hierarchical image database.
In 2009 IEEE conference on computer vision and pattern recognition (2009), Ieee, pp. 248–255.
- [11]
Diko, A., Avola, D., Cascio, M., and Cinque, L.
Revit: Enhancing vision transformers with attention residual connections for visual recognition.
arXiv preprint arXiv:2402.11301 (2024).
- [12]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S.
An image is worth 16x16 words: Transformers for image recognition at scale.
In International Conference on Learning Representations (2021).
- [13]
Fan, Q., Huang, H., Chen, M., Liu, H., and He, R.
Rmt: Retentive networks meet vision transformers.
arXiv preprint arXiv:2309.11523 (2023).
- [14]
Gu, A., and Dao, T.
Mamba: Linear-time sequence modeling with selective state spaces.
arXiv preprint arXiv:2312.00752 (2023).
- [15]
Hao, S., Zhou, Y., and Guo, Y.
A brief survey on semantic segmentation with deep learning.
Neurocomputing 406 (2020), 302–321.
- [16]
He, J., Deng, Z., and Qiao, Y.
Dynamic multi-scale filters for semantic segmentation.
In Proceedings of the IEEE/CVF international conference on computer vision (2019), pp. 3562–3572.
- [17]
He, J., Deng, Z., Zhou, L., Wang, Y., and Qiao, Y.
Adaptive pyramid context network for semantic segmentation.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2019), pp. 7519–7528.
- [18]
He, K., Gkioxari, G., Dollár, P., and Girshick, R.
Mask r-cnn.
IEEE Transactions on Pattern Analysis & Machine Intelligence (2017).
- [19]
Jain, J., Singh, A., Orlov, N., Huang, Z., Li, J., Walton, S., and Shi, H.
Semask: Semantically masked transformers for semantic segmentation.
In Proceedings of the IEEE/CVF International Conference on Computer Vision (2023), pp. 752–761.
- [20]
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al.
Segment anything.
In Proceedings of the IEEE/CVF International Conference on Computer Vision (2023), pp. 4015–4026.
- [21]
Lateef, F., and Ruichek, Y.
Survey on semantic segmentation using deep learning techniques.
Neurocomputing 338 (2019), 321–348.
- [22]
Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., and Qiao, Y.
Videomamba: State space model for efficient video understanding.
arXiv preprint arXiv:2403.06977 (2024).
- [23]
Li, Y., Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., Wang, Y., and Ren, J.
Efficientformer: Vision transformers at mobilenet speed.
Advances in Neural Information Processing Systems 35 (2022), 12934–12949.
- [24]
Li, Z., Xia, T., Chang, Y., and Wu, Y.
A survey of rwkv.
arXiv preprint arXiv:2412.14847 (2024).
- [25]
Liu, J., Yang, H., Zhou, H.-Y., Xi, Y., Yu, L., Yu, Y., Liang, Y., Shi, G., Zhang, S., Zheng, H., et al.
Swin-umamba: Mamba-based unet with imagenet-based pretraining.
arXiv preprint arXiv:2402.03302 (2024).
- [26]
Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y.
Vmamba: Visual state space model.
arXiv preprint arXiv:2401.10166 (2024).
- [27]
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.
Swin transformer: Hierarchical vision transformer using shifted windows.
In Proceedings of the IEEE/CVF international conference on computer vision (2021), pp. 10012–10022.
- [28]
Long, J., Shelhamer, E., and Darrell, T.
Fully convolutional networks for semantic segmentation.
CoRR abs/1411.4038 (2014).
- [29]
MMSegmentation Contributors.
OpenMMLab Semantic Segmentation Toolbox and Benchmark, July 2020.
- [30]
Paszke, A., Chaurasia, A., Kim, S., and Culurciello, E.
Enet: A deep neural network architecture for real-time semantic segmentation.
arXiv preprint arXiv:1606.02147 (2016).
- [31]
Ruan, J., and Xiang, S.
Vm-unet: Vision mamba unet for medical image segmentation.
arXiv preprint arXiv:2402.02491 (2024).
- [32]
Shrivastava, A., Gupta, A., and Girshick, R.
Training region-based object detectors with online hard example mining.
In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 761–769.
- [33]
Strudel, R., Garcia, R., Laptev, I., and Schmid, C.
Segmenter: Transformer for semantic segmentation.
In Proceedings of the IEEE/CVF international conference on computer vision (2021), pp. 7262–7272.
- [34]
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F.
Retentive network: A successor to transformer for large language models.
arXiv preprint arXiv:2307.08621 (2023).
- [35]
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J.
Unified perceptual parsing for scene understanding.
In Proceedings of the European conference on computer vision (ECCV) (2018), pp. 418–434.
- [36]
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., and Luo, P.
Segformer: Simple and efficient design for semantic segmentation with transformers.
Advances in neural information processing systems 34 (2021), 12077–12090.
- [37]
Yang, G., Kwon, Y., and Kim, H.
Exmobilevit: Lightweight classifier extension for mobile vision transformer.
arXiv preprint arXiv:2309.01310 (2023).
- [38]
Yu, F., Koltun, V., and Funkhouser, T.
Dilated residual networks.
In Proceedings of the IEEE conference on computer vision and pattern recognition (2017), pp. 472–480.
- [39]
Zhang, B., Tian, Z., Tang, Q., Chu, X., Wei, X., Shen, C., et al.
Segvit: Semantic segmentation with plain vision transformers.
Advances in Neural Information Processing Systems 35 (2022), 4971–4982.
- [40]
Zhang, C., Xue, Z., Zhu, X., Wang, H., Huang, Q., and Tian, Q.
Boosted random contextual semantic space based representation for visual recognition.
Information Sciences 369 (2016), 160–170.
- [41]
Zhang, H., Su, Y., Xu, X., and Jia, K.
Improving the generalization of segmentation foundation model under distribution shift via weakly supervised adaptation.
arXiv preprint arXiv:2312.03502 (2023).
- [42]
Zhang, S., Ren, G., Zeng, X., Zhang, L., Du, K., Liu, G., and Lin, H.
Efficient cross-information fusion decoder for semantic segmentation.
Computer Vision and Image Understanding 240 (2024), 103918.
- [43]
Zhang, Z., Cai, H., and Han, S.
Efficientvit-sam: Accelerated segment anything model without performance loss.
arXiv preprint arXiv:2402.05008 (2024).
- [44]
Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J.
Pyramid scene parsing network.
In Proceedings of the IEEE conference on computer vision and pattern recognition (2017), pp. 2881–2890.
- [45]
Zhao, J., Wu, Y., Deng, R., Xu, S., Gao, J., and Burke, A.
A survey of autonomous driving from a deep learning perspective.
ACM Computing Surveys.
- [46]
Zheng, Q., Li, W., Hu, W., and Wu, G.
An interactive image segmentation algorithm based on graph cut.
Procedia Engineering 29 (2012), 1420–1424.
- [47]
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H., et al.
Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2021), pp. 6881–6890.
- [48]
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A.
Scene parsing through ade20k dataset.
In Proceedings of the IEEE conference on computer vision and pattern recognition (2017), pp. 633–641.
- [49]
Zhu, G., Wang, R., Liu, Y., Zhu, Z., Gao, C., Liu, L., and Sang, N.
An adaptive post-processing network with the global-local aggregation for semantic segmentation.
IEEE Transactions on Circuits and Systems for Video Technology (2023).
- [50]
Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., and Wang, X.
Vision mamba: Efficient visual representation learning with bidirectional state space model.
arXiv preprint arXiv:2401.09417 (2024).

## Appendix A Additional results

In this section, we present a comprehensive analysis of the experimental results achieved by the SegRet-Small, Base, and Large variants on the ADE20K, Cityscapes, and COCO-Stuff datasets. Furthermore, we examine the effect of input resolution scaling on overall model performance.

### A.1 ADE20K

Table [6](https://arxiv.org/html/2502.14014v2#A1.T6) summarizes the performance of models across different scales and architectural designs on the ADE20K validation set. Among smaller models, SenFormer and SegFormer deliver competitive results; however, our SegRet-Small variant surpasses both, achieving an mIoU of 50.7. In the mid-sized category, Swin and Mask2Former perform strongly, yet our SegRet-Base model attains a higher mIoU of 51.63. For large-scale models, MaskFormer and VMamba demonstrate commendable performance, with our SegRet-Large variant delivering results on par with these leading approaches.

### A.2 Cityscapes

Table [7](https://arxiv.org/html/2502.14014v2#A1.T7) provides a detailed comparison of model performance across various configurations on the Cityscapes dataset. In the small model category, our SegRet-Small model achieves mIoU scores of 82.59% (SS) and 83.26% (MS), outperforming other small-scale counterparts. For the base model group, SegRet-Base records 83.17% in SS and 83.8% in MS, delivering superior accuracy while maintaining moderate computational complexity. In the large model category, SegRet-Large attains an mIoU of 83.36% (SS), marginally outperforming comparable models. While it slightly trails Mask2Former (Swin-B*) in MS performance (84.5%), SegRet-Large maintains a parameter count of just 95.81M, offering a favorable balance between performance and computational efficiency.

### A.3 COCO-Stuff

As shown in Table [8](https://arxiv.org/html/2502.14014v2#A1.T8), we further evaluated the performance of SegRet models with different sizes on the COCO-Stuff dataset. The results highlight the clear advantages of our proposed models, which employ RMT-S, RMT-B, and RMT-L as backbone networks. Specifically, our models achieve high average IoU scores of 44.32, 45.92, and 45.78 under single-scale (SS) inference, and 45.48, 46.06, and 46.63 under multi-scale (MS) inference, respectively. In contrast, competing models demonstrate comparatively lower performance across both evaluation settings. These results underscore the strong segmentation accuracy and generalization capabilities of our SegRet models on the COCO-Stuff dataset.

**Table 6: The results of various model sizes on the ADE20K dataset. * Indicates pretraining on ImageNet22K.**
| Method | Backbone | Decoder head | Image Size | #params | mIoU(SS) | mIoU(MS) | FLOPs |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SenFormer | Swin-S |  | 512*512 | 81M | 49.2 | - | 202G |
| RMT | RMT-S | UperNet | 512*512 | 56M | - | 49.8 | 937G |
| RMT | RMT-B | FPN | 512*512 | 57M | - | 50.4 | 294G |
| SegFormer | MiT-B3 |  | 512*512 | 47.3M | 49.4 | 50 | 79G |
| VMamba | VMamba-S | UperNet | 512*512 | 76M | 49.5 | 50.5 | 1037G |
| VMamba | VMamba-S | UperNet | 640*640 | 76M | 50.8 | 50.8 | 1620G |
| MaskFormer | Swin-S |  | 512*512 | 63M | 49.8±0.4 | 51.0±0.4 | 79G |
| SegFormer | MiT-B4 |  | 512*512 | 64.1M | 50.31 | 51.1 | 95.7G |
| SegRet-Small | RMT-S |  | 512*512 | 26.52M | 50.7 | 51.29 | 117.1G |
| Swin | Swin-B* | UperNet | 640*640 | 121M | - | 51.6 | 471G |
| SegFormer | MiT-B5 |  | 640*640 | 84.7M | 51 | 51.8 | 183.3G |
| RMT | RMT-B | UperNet | 512*512 | 83M | - | 52 | 1051G |
| Mask2Former | Swin-S |  | 512*512 | 69M | 51.3 | 52.4 | 98G |
| SegRet-Base | RMT-B |  | 512*512 | 53.05M | 51.63 | 52.13 | 229.66G |
| SeMask | SeMask Swin-B* | FPN | 512*512 | 96M | 49.35 | 50.98 | 107G |
| VMamba | VMamba-B | UperNet | 512*512 | 110M | 50 | 51.3 | 1167G |
| RMT | RMT-L | FPN | 512*512 | 98M | - | 51.4 | 482G |
| MaskFormer | Swin-B |  | 640*640 | 102M | 51.1±0.2 | 52.3±0.4 | 195G |
| SegRet-Large | RMT-L |  | 640*640 | 95.81M | 52 | 52.23 | 478.54G |

**Table 7: The results of various model sizes on the Cityscapes dataset. * Indicates pretraining on ImageNet22K.**
| Method | Backbone | Decoder head | Image Size | #params | mIoU(SS) | mIoU(MS) | FLOPs |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mask2Former | R50 |  | 512*1024 | 44M | 79.4 | 82.2 | 293G |
| Mask2Former | R101 |  | 512*1024 | 63M | 80.1 | 81.9 | 226G |
| SeMask | SeMask Swin-S | FPN | 768*768 | 56M | 77.13 | 79.14 | 134G |
| Mask2Former | Swin-T |  | 512*1024 | 47M | 82.1 | 83 | 232G |
| SegFormer | MiT-B2 |  | 1024*1024 | 27.5M | 81 | 82.2 | 717.1G |
| SegRet-Small | RMT-S |  | 512*1024 | 26.52M | 82.93 | 83.52 | 117.1G |
| SeMask | SeMask Swin-B* | FPN | 768*768 | 96M | 77.7 | 79.73 | 217G |
| SegFormer | MiT-B3 |  | 1024*1024 | 47.3M | 81.7 | 83.3 | 962.9G |
| Mask2Former | Swin-S |  | 512*1024 | 69M | 82.6 | 83.6 | 313G |
| SegRet-Base | RMT-B |  | 512*1024 | 53.05M | 83.28 | 83.87 | 229.66G |
| SeMask | SeMask Swin-L* | FPN | 768*768 | 211M | 78.53 | 80.39 | 455G |
| SegFormer | MiT-B5 |  | 1024*1024 | 84.7M | 82.4 | 84 | 1460.4G |
| Mask2Former | Swin-B* |  | 512*1024 | 107M | 83.3 | 84.5 | 466G |
| Mask2Former | Swin-L* |  | 512*1024 | 215M | 83.3 | 84.3 | 868G |
| ECFD-tiny | Swin-Large |  | 512*1024 | 209M | 82.67 | 83.41 | 473G |
| ECFD-small | Swin-Large |  | 512*1024 | 218M | 83.1 | 83.61 | 488G |
| SegRet-Large | RMT-L |  | 512*1024 | 95.81M | 83.36 | 83.91 | 478.54G |

**Table 8: The results of various model sizes on the COCO-Stuff dataset. * Indicates pretraining on ImageNet22K.**
| Method | Backbone | Decoder head | Image Size | #params | mIoU(SS) | mIoU(MS) |
| --- | --- | --- | --- | --- | --- | --- |
| MaskFormer | R101 |  | 640*640 | - | 38.1±0.3 | 39.8±0.6 |
| MaskFormer | R101c |  | 640*640 | - | 38.0±0.3 | 39.3±0.4 |
| SenFormer | R101 |  | 512*512 | 79M | 41 | 42.1 |
| SegRet-Small | RMT-S |  | 512*512 | 26.52M | 44.32 | 45.48 |
| SeMask | SeMask Swin-S | FPN | 512*512 | 56M | 40.72 | 42.27 |
| SegRet-Base | RMT-B |  | 512*512 | 53.05M | 45.92 | 46.06 |
| SeMask | SeMask Swin-B* | FPN | 512*512 | 96M | 44.68 | 46.3 |
| SegRet-Large | RMT-L |  | 512*512 | 95.81M | 45.78 | 46.63 |

### A.4 Input Scaling

We also conducted input scaling experiments using the Cityscapes dataset to examine the effect of varying image resolutions on model performance. As illustrated in Figure [4](https://arxiv.org/html/2502.14014v2#A1.F4), our model consistently outperformed alternative methods across four different input sizes: $512\times 1024$, $768\times 768$, $640\times 1280$, and $1024\times 1024$. Notably, the model achieved its best performance with an input size of $1024\times 1024$, reaching a peak mIoU of 82.02. Despite its competitive accuracy, our model maintains a compact design with only 14.01M parameters, highlighting its effectiveness in balancing performance and parameter efficiency.

Figure: Figure 4: A Comparative Study of Scaling Input Sizes on Cityscapes. We examined the impact of four different input dimensions ($512\times 1024$, $768\times 768$, $640\times 1280$, and $1024\times 1024$) on the model performance.
Refer to caption: x3.png

### A.5 The effectiveness of the decoder

As illustrated in Figure [5](https://arxiv.org/html/2502.14014v2#A1.F5), we conducted experiments evaluating the performance of different decoder architectures when integrated with the RMT-T backbone. The results reveal that the RMT-T+UperNet achieves mIoU scores of 47.51 (SS) and 48.84 (MS), while the RMT-T+FPN combination yields 47.20 and 48.13, respectively. Notably, the integration of RMT-T with our proposed decoder achieves the highest performance, attaining mIoU scores of 48.76 (SS) and 49.39 (MS). These findings underscore the pivotal role of the decoder in enhancing the overall segmentation performance.

Figure: Figure 5: Performance Insights of Vision RetNet with Different Decoders.
Refer to caption: extracted/6489056/fig/decoder_ablation.png

### A.6 Limitations

Although SegRet demonstrates robust performance in semantic segmentation of road scenes for autonomous driving, its generalizability to other domains like medical imaging and remote sensing remains limited. In medical image analysis, the model’s ability to segment small structures is hindered by inadequate modeling of fine-grained features and cross-modal relationships. For remote sensing applications, substantial scale variations and complex scene compositions further challenge its adaptability. Future research directions may include developing domain-adaptive modules to enhance feature distribution alignment across different domains, as well as incorporating attention-guided upsampling mechanisms in the decoder to better capture small-object characteristics.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The authors should provide FLOPs and latency comparisons with Mask2Former or SegFormer at the same input size to validate the efficiency claims.

2. Results on small-object segmentation (e.g., Cityscapes fine classes) are needed to evaluate if the lightweight decoder compromises fine detail.
soundness: 2
strengths: 1. This paper achieves a strong efficiency trade-off, particularly at smaller model sizes, with documented improvements over comparable baseline methods.

2. This paper provides transparent ablation studies on decoder design and input scaling, along with clear training details and code to ensure reproducibility.

3.  This paper honestly addresses limitations and suggests plausible next steps, such as domain adaptation and attention-guided upsampling.
summary: SegRet couples Vision RetNet as a hierarchical encoder with a lightweight zero-initialized residual decoder. It targets parameter efficiency while maintaining competitive mIoU on ADE20K, Cityscapes, and COCO-Stuff. Ablations show a modest but consistent gain from the zero-initialized residual (≈+0.8 mIoU) and comparisons demonstrate favorable accuracy/params trade-offs (e.g., SegRet-Tiny ~14M params reaching ~49.4 mIoU on ADE20K and ~42.2/43.3 mIoU on COCO-Stuff). Code link (anonymous) is provided.
weaknesses: 1. The novelty lies mainly in the minimalist decoder design; the use of RetNet as the encoder and residual fusion represents an incremental improvement rather than a conceptual breakthrough.

2. While results are competitive, they do not clearly establish state-of-the-art performance under similar computational constraints in the most challenging benchmarks. Stronger comparisons using identical training configurations would be beneficial.

---

confidence: 5
contribution: 1
initial_rating: 2
presentation: 3
questions: See weakness.
soundness: 2
strengths: The paper is well organized. The proposed methods achieves SOTA performance with low computational resources.
summary: This paper proposed a semantic segmentation network that leverages retentive network. The network consists of a encoder backbone and a decoder. The encoder backbone contains 4 consecutive RMT blocks. The decoder utilizes a zero initialized convolution layer in parallel with a linear layer. Experiments on 3 benchmark datasets ADE20K, Cityscapes and COCO-Stuff and demonstrate the effectiveness of the proposed method.
weaknesses: 1. Lack of contribution. The proposed method's backbone is almost the same as the compared method RMT, which cannot be viewed as contribution. The improvement in the decoder is also minimum.

2. The compared methods contains mostly general vision backbones, and the result on many other vision tasks, such as object classification and detection, are available. But the proposed method is not comparing with them.

3. The comparison in Tab. 2, 3, 6, 7, 8 is not fair. The image size should keep the same, because the number of parameters, latency and losses are calculated depends on the image size.

4. More qualitative results should be provided, such as more segmentation results from compared methods, feature analysis, etc.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. The “zero-initialized residual” decoder seems to differ from prior FPN/UPerNet-style decoders only by a zero-initialized 1×1 residual branch. Could you provide a formal motivation or derivation—for instance, how zero initialization affects gradient flow, optimization stability, or representational bias compared to standard residual or skip connections?

2. How is SegRet conceptually distinct from known efficient decoders such as SegFormer’s MLP-based fusion, LiteSeg, or MobileViT decoders? Please clarify what architectural element or training principle is genuinely new, rather than a simplified re-implementation.

3. Since The SegRet model is implemented based on the MMSegmentation framework could you highlight key difference between both of them. 

4. You state the method underperforms for medical/remote sensing (p. 9; A.6, p. 15). Can you include one focused study (e.g., small-lesion segmentation) with failure analysis to clarify whether the limitation is encoder scale, decoder capacity, or upsampling choice?
soundness: 2
strengths: Clean encoder–decoder story using Vision RetNet as a hierarchical backbone (Fig. 2, p. 4; Sec. 3.1) and a small decoder (Sec. 3.2, p. 5–6).

Competitive tiny/small regime: SegRet-Tiny (≈14 M params) is strong vs. other “tiny” setups across datasets (Tables 1–3, pp. 7–8).

Readable presentation and sensible training protocol (Sec. 4.1, p. 7), with an (anonymous) code link in the Reproducibility Statement (p. 10).

The principal strength of SegRet lies in its computational efficiency and architectural simplicity—it successfully integrates the Vision RetNet encoder, which captures long-range dependencies with linear complexity, with a lightweight zero-initialized residual decoder that minimizes parameters without degrading accuracy.
summary: The paper proposes SegRet, a new semantic segmentation framework that integrates the recently developed Retentive Network (RetNet) — a state-space model designed to replace self-attention — with a lightweight, zero-initialized residual decoder, while significantly reducing parameter count, making it a compact, efficiency-oriented yet not fully state-of-the-art semantic segmentation framework.
This design enables SegRet to achieve competitive segmentation performance across ADE20K, Cityscapes, and COCO-Stuff using significantly fewer parameters and computational resources than traditional Transformer-based models like Mask2Former or Swin, making it particularly well-suited for real-time or resource-constrained vision applications such as autonomous driving and embedded systems.
weaknesses: Vision RetNet is adopted largely as-is; the paper does not contribute new retention variants for vision, nor new theory atop RetNet. The retention machinery and bidirectional vision adaptation (BiRetention, horizontal/vertical decomposition) are recaps of prior work (Sec. 3.1; Eqs. 6–14).

The “zero-initialized residual decoder” amounts to a zero-init 1×1 residual branch after channel unification, followed by standard upsample-concat-conv (Eqs. 15–18). This design is extremely close to well-known lightweight decoders (FPN-style merges, 1×1 residual adapters), and the paper does not show conceptual novelty beyond the initialization trick.

Impact of the “novelty” is marginal. The only ablation directly tied to the claimed novelty (adding ZIR) shows +0.79 mIoU with +0.25M params on ADE20K for the Small variant (Table 4), which is a small incremental gain that does not justify SOTA claims by itself.

The core claim is “parameter/efficiency superiority,” yet there is no latency (ms), throughput (FPS), or peak memory on any GPU/CPU, nor profile at multiple resolutions (Sec. 4, pp. 7–9). FLOPs alone do not predict wall-clock; kernel efficiency (e.g., attention vs. state-space primitives) and cache behavior matter. Absent these, “efficient” remains asserted, not demonstrated.


Provide a theoretical or empirical rationale for the zero-init residual path: gradients at init, linearization analysis, effect on optimization dynamics; compare with ResNet-style identity and LayerScale variants.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: 1.The main text does not reference Figure 1. In addition, the meaning of multi-scale inference is unclear, and it should be explained why each method in the comparison has multiple parameter counts.
2.The Vision RetNet Backbone is an existing work, yet a disproportionate amount of space is devoted to describing it.
3.In Figure 2, the MASA diagram in the upper-left corner shows only one input, whereas the lower diagram depicts six inputs—this inconsistency should be clarified.
4.Equations (9–14) are densely stacked, which reduces readability and should be reformatted for clarity.
5.In Section 4.1, both Tiny and Small variants are reported to have identical parameter counts (0.814 M), which seems implausible and requires verification.
6.In Sections 4.2 and A.1–A.3, using dataset names as sub-section titles is inappropriate; more descriptive methodological headings are recommended.
7.In Table 3, the comparative methods are too few, especially recent ones from the past two years.
8.In Figure 3, the qualitative comparison on the ADE20K dataset includes only one baseline, and noticeable differences remain between the proposed results and the ground truth. Moreover, qualitative comparisons on the other two datasets are missing.
9.The description of the zero-initialized component is vague—does it consist of only a single convolution operation?
10.The overall contribution lacks substantial innovation beyond architectural integration.
soundness: 2
strengths: Balanced accuracy and efficiency: SegRet achieves strong segmentation performance on major benchmarks (ADE20K, Cityscapes, COCO-Stuff) while significantly reducing parameter count and computational cost, demonstrating excellent scalability for real-time or embedded applications.
Strong feature representation capability: By leveraging RetNet’s long-range dependency modeling, SegRet enhances multi-level feature extraction, resulting in richer contextual understanding and improved segmentation quality.
summary: The paper introduces SegRet, a semantic segmentation model that pairs a RetNet backbone with a lightweight residual decoder using zero-initialization. The design targets the accuracy–efficiency trade-off common in autonomous driving and intelligent transportation. Specifically, the zero-initialized residual decoder keeps computation low while stabilizing information flow, and RetNet provides robust hierarchical feature extraction. The approach emphasizes parameter efficiency, claiming SOTA-level performance with markedly fewer parameters. Experiments on ADE20K, Cityscapes, and COCO-Stuff demonstrate strong accuracy under tight computational budgets, indicating SegRet’s suitability for real-time or resource-constrained deployments.
weaknesses: The contribution hinges on pairing RetNet with a zero-initialized lightweight residual decoder, but the paper doesn’t explain why this pairing is fundamentally new vs. existing efficient backbones + residual decoders, or how it differs from prior zero-init/ResNet-style stabilizers.


Decision: Reject