SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation
Authors: Xiaowen Ma
Zhen-Liang Ni
Xinghao Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Pixel-level classifier
multi-domain distillation
semantic adaptive
spatial adaptive
Primary area: machine_vision

Abstract: Vanilla pixel-level classifiers for semantic segmentation are based on a certain paradigm, involving the inner product of fixed prototypes obtained from the training set and pixel features in the test image.  This approach, however, encounters significant limitations, i.e., feature deviation in the semantic domain and information loss in the spatial domain. The former struggles with large intra-class variance among pixel features from different images, while the latter fails to utilize the structured information of semantic objects effectively.  This leads to blurred mask boundaries as well as a deficiency of fine-grained recognition capability.  In this paper, we propose a novel Semantic and Spatial Adaptive Classifier (SSA-Seg) to address the above challenges.  Specifically, we employ the coarse masks obtained from the fixed prototypes as a guide to adjust the fixed prototype towards the center of the semantic and spatial domains in the test image.  The adapted prototypes in semantic and spatial domains are then simultaneously considered to accomplish classification decisions.  In addition, we propose an online multi-domain distillation learning strategy to improve the adaption process.  Experimental results on three publicly available benchmarks show that the proposed SSA-Seg significantly improves the segmentation performance of the baseline models with only a minimal increase in computational cost.

SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation
Xiaowen Ma Ma,
Zhenliang Ni
Xinghao Chen
Huawei Noah's Ark Lab
'Zhejiang University
https://githul.
com/xwmaxwma//SAA
Abstract
Vanilla pixel-level classifiers for semantic segmentation based certain paradigm, involving the inner product of fixed prototypes obtained from the train- ing set and pixel features in the test image. This approach, however, encounters significant limitations, i.e., feature deviation in the semantic domain and informa- tion loss in the spatial domain. The former struggles with large intra-class variance among pixel features from different images, while the latter fails to utilize the structured information of semantic objects effectively. This leads to blurred mask boundaries as well as deficiency of fine-grained recognition capability. In this paper, we propose a novel Semantic and Spatial Adaptive Classifier (SSA-Seg) to address the above challenges. Specifically, we employ the coarse masks obtained from the fixed prototypes as guide to adjust the fixed prototype towards the center of the semantic and spatial domains in the test image. The adapted prototypes in semantic and spatial domains are then simultancously considered to accomplish classification decisions. In addition, propose online multi-domain distillation learning strategy to improve the adaption process. Experimental results on three publicly available benchmarks show that the proposed SSA-Seg significantly im- proves the segmentation performance the baseline models with only a minimal increase in computational cost.
Introduction
Semantic segmentation, fundamental task in computer vision, aims assigning a category label to each pixel in a given image and is widely used in various domains such as autonomous driving [25],
industrial detection [13], satellite image analysis [60, 27] and smart city [26]. Mainstream methods, such as FCN [34], DeepLab [5, 6). PSPNet [59], and SegNeXt [17] mainly use parametric softmax to classify each pixel. In recent developments, transformer- based approaches like MaskFormer [10], Mask2Former [9], and SegVit ViT [55] classify masks by directly learning query vectors, termed as mask- level classification. However, these mask-level classification models often require heavyweight cross- attention decoders, which limit their deployment in resource-constraing
scenarios [45, 57, 14, 35, 7J.
Vanilla parametric softmax uses the convolutional kernel weights as the fixed semantic prototypes and obtains segmentation masks by computing the inner product the pixel features and the prototypes. However, this pixel-level classification method has two obvious drawbacks: (1) feature deviation in the semantic domain. Due to complex backgrounds and varying object distributions, pixel features in the test images tend to have large intra-class variance with pixel features in the training set. However, the fixed semantic prototypes, representing the semantic feature distribution on the training set, will be far away from the pixel features of the corresponding class when applied to the test image, as shown in Fig. 1(b). (2) information loss in the spatial domain. The vanilla pixel-level classifiers only perform
Equal contribution,
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Adaytation
Adaptation
Position basis
SSA-Seg
Fixed Protefype
(o) Test Image
(b) Feature distribation
(6) Semantic prototype sdaptation
(d) Spatial protatype adaptation
Figure 1: A example of vanilla pixel-level classifiers, where the ScaFormer-L. [45] is the haseline and the feature distribution is visualized with t-SNE. (a) is a test image of the ADE20K dataset, and (b) denotes the feature distributions in the semantic domain of (a). with purple and gray dots denoting the pixel features the test image of the door and other categories, respectively. Blue star denotes the fixed prototype trained on training set of the door category. It shows that vanilla pixel-level classifiers directly interact pixel features with the fixed semantic prototypes, which leads to feature deviation in the semantic domain and information loss in the spatial domain problems. In contrast, SSA-Seg makes classification decisions based adaptive semantic and spatial prototypes by prompting the prototypes offiet toward the center of the semantic domain and the spatial domain, as shown in (c) and (d). Visual comparison the baseline and SSA-Seg can be found in Fig. 5.
inner products of fixed prototypes and pixel features in the semantic domain, and lack modeling of the relationship between prototypes and pixel features in the spatial domain. Therefore, the explicit structural information of target objects is not fully utilized, leading to suboptimal segmentation of border regions and small targets.
Recent studies [62, 28, 46] have improved the parametric softmax classifier. ProtoSeg [62] proposes a nonparametric prototype to replace the standard classifier. Through prototype- metric learning, they enhanced the construction of pixel embedding space. GMMSeg [28] models the joint distribution of pixels and categories and learns the Gaussian mixture classifier in the pixel feature space by using the EM algorithm. However, these methods still rely on fixed prototype classifiers, when confronted with varying data distributions on test images, the problem of feature deviation in the semantic domain still exists. Besides, the Context-aware Classifier (CAC) [44] is proposed, which a dynamic classifier by utilizes contextual cues. It adaptively adjusts classification prototypes based on feature content and thus can alleviate the feature deviation in the semantic domain problem to some extent. However, due to the lack of constraints not controllable. In extreme cases, the offset will even be the direction away from the semantic features of the image. Besides, these works ignore modeling the relationship between prototype and pixel features in the spatial domain, which limits further improvements in model performance.
semantic prototypes, the offset of the fixed prototype is
To address the above issues, a Semantic and Spatial Adaptive Classifier (SSA-Seg) is proposed, which solves the above two problems by facilitating the offset of the fixed prototype towards the center of the semantic domain and the center of the spatial domain of the test image, as shown in Fig. 1(c) and (d). Note that the semantic domain center of each class is the mean semantic features belonging to this class, and similar concepts such as class center also appear in [56, 21J. Similarly, the center of the spatial domain is the mean spatial features belonging to the class. Specifically, we retain the original 1 x 1 convolution to obtain coarse mask. Besides, position embedding is applied to the output features to obtain the spatial information. Then, the coarse mask is applied as the guide to obtain the semantic domain center and spatial domain center in each sample, which is then processed obtain adaptive semantic and spatial prototypes. Finally, classification decisions are made by simultaneously considering prototypes in the semantic and spatial domains.
In addition, ground truth can be used to improve classifier performance [44]. Therefore, a training- only teacher classifier is designed to introduce ground truth information for calibrating the primary classifier. Specifically, we design multi-domain knowledge distillation to enhance the primary classifier from different domains. First, the response domain distillation method distills the outputs of the two classifiers based on a boundary-aware and category-awar distillation loss, which conveys accurate semantic and structural information in the ground truth. Then, semantic domain distillation and spatial domain distillation are used to constrain the offset of the prototype. In this way, multi- domain distillation improves the feature representation of the primary classifier, which significantly improves the test performance.
The proposed method significantly improves the segmentation performance of the baseline model with a very small increase in computational cost on three commonly used datasets: ADE20k, PASCAL-Context, and COCO-stuff-IOK, Furthermore, compared to other advanced classifiers

---

such as GMMSeg [28] and CAC [44], our SSA-Seg achieves significant performance improvement on several baseline methods. In particular. by applying SSA-Seg, we achieve the state-of-the art lightweight segmentation performance. Specially, SeaFormer-I. [45] obtain 45.36% mloU on ADE20k with only IG more FL.OPs and 0.5ms more latency. Simarlily, SegNext-T [17] obtain 38.91% mloU on COCO-stufi 10K and 52.58% mloU on PASCAL-Context dataset.
2 Related Work
2.1 Semantic Segmentation
Semantic segmentation one of the basic tasks in computer vision, whose goal is to assign a category each pixel in image. The existing semantic segmentation methods be divided into two categories: pixel-level classification model [34, 54, 45, 57, 14, 17] and mask-level classification model [10, 9, 55, 18, 31, 51 Since Fully Convolutional Networks (FCNs) [34], pixel-level classifica- tion has been the mainstream semantic segmentation method. Subsequent works focus on optimizing backbone extraction features [17, 45, 8, 52, 33, 53, 50], or improving the decode head for context modeling [37, 36, 59, 6, 15, 22, 54, 24, 29].
Recently, mask-level classification models have become popular, which learn object queries to classify masks without classifying each pixel. MaskFormer [10] first proposes to use mask clas- sification for semantic segmentation tasks, which is inspired by [3]. Mask2Former [9] optimizes MaskFormer [10] by constraining cross- cross-attention within the predicted masked region to extract local features. Seg
ViT ViT [55] proposes apply the mask classifier to plain vision transformers. However, mask-level classification models often have high computational complexity due to the need for multiple cross-attention to update the query, and their application fields limited. Pixel-level classification model, as the mainstream method, has wide range of application fields. Therefore, we rethink pixel-level classifiers improve the performance of pixel-level classification.
2.2 Pixel-level Classifier
The current mainstream Pixel-level classifier essentially discriminant classification model based on softmax. The potential data distribution is completely ignored, which limits the expressive ability of the model. Recently, some works [62, 28, 44] have improved the softmax classifier to improve the segmentation performance. ProtoSeg [62] proposes non-parametric prototype to replace the standard classifier. Through prototype-basee
metric learning, the construction of pixel embedding space is better. GMMSeg [28] models the joint distribution of pixels and categories and learns the Gaussian mixture classifier the pixel feature space by using the EM algorithm. It can capture the pixel feature distribution of each category in fine detail by using the generator pattern. Context-aware classifer [44] extracts the context based on the input to generate a sample adaptive classifier. However, none of these methods position embedding improve the classifier. Our classifier can sense both semantic and spatial information, to make more accurate classification decisions.
Method
3.1 Preliminary: Vanilla Pixel-level Classifier
We first review the architecture of the vanilla pixel-level classifier for semantic segmentation. For an input image X, the features are first extracted by backbone and processed by decode head to obtain the output features Sy E IR #xxx x where H. W, D denote the height, width, and channel of the feature map, respectively. The current mainstream strategy uses a fixed prototype classifier, i.e., simple convolution maps feature to the category space:
M. S, @ S".
(1)
where Me E H denotes the segmentation mask, E IR K denotes the weights of x 1 convolution kernel, @ denotes matrix multiplication and K is the number of category.
The above scheme essentially treats the convolutional kernel weights as fixed semantic prototypes and obtains segmentation masks based on the inner product with pixel features in the semantic domain. Therefore, the two questions of feature deviation in the semantic domain and information loss the spatial domain, which have been described before, to be solved.
3

---

SPPA
SSA-Seg
Online Multi-domain Distillation
8/
88A-Seg
Spetial
$
SSA-Seg
5/
Primary Classifier
Responso Aerg
Teacher
5
Clussifier
M
Semantic
Eys
34
8 Fixed prototype
Position basis
GT Nok
Semantic feature
Spatial feature
8 Semantic center
3: Spatial center
6p
Semantic prototype
2 Spatial prototype
8,
PE: Position encode
X
Matrix multiplication
Sy BP
PB: Position bucket
Element- summation
Figure 2: SSA-Sez Seg overview. For the semantic features S, output from the backbone and decode head, we first generate spatial features P by position encode. Then we retain the original 1 x 1 convolution to generate the coarse mask Mc. Guided by MM. we generate the center of the semantic domain and spatial domain in the pre-classified representations and fused them with the fixed semantic prototypes S and the prototype position basis P to generate the semantic prototypes Sp and the spatial prototype Pp. Finally, we consider simultaneously semantic and spatial prototypes to perform classification decisions. The right figure shows an online teacher classifier only for training. where the coarse mask is replaced with ground truth mask to participate in model training, and constrains the prototype adaption and transfer accurate semantic and spatial knowledge the primary classifier based on multi-domain distillation learning.
3.2 Semantic and Spatial Adaptive Classifier
Based on the above analysis in Section 3.1, we propose a novel Semantic and Spatial Adaptive Classifier (SSA-Seg), shown in Fig. 2. The proposed SSA-Seg consists of three parts: semantic prototype adaptation, spatial prototype adaptation, and online multi-domain distillation.
Specifically, for the semantic features Sy output from the backbone and the decode head, we retain the original X 1 convolution operation to generate the coarse mask MM. Next, we obtain the spatial features Py from the semantic features Sy. Guided by MM. we integrate the semantic and spatial features of each category to obtain the centers of the corresponding semantic and spatial domains, i.e., S. and Pe. Then, we fuse Se, Pa with the fixed semantic prototypes S and prototype position basis P. respectively, to generate adaptive semantic prototypes Sp and spatial prototypes Pp. Finally, modify Eq. to make more accurate classification decisions,
O₂ (S, @ Pf) @ (Sp @ P₂)
(2)
where @ denotes wiement-wise summation. In addition, we introduce teacher classifier and online multi-domain distillation strategy to improve the performance of SSA-Seg. The teacher classifier has the same structure the primary classifier, except that the coarse mask M. is replaced by the ground-trumt truth mask Mg. as shown in Fig. 2. Therefore, the teacher classifier can output more accurate semantic prototypes Sp. spatial prototypes Pp and segmentation masks Ô These outputs can be used as soft labels to constrain the adaptation process of prototypes through online multi-domain distillation, which consists of response domain distillation loss Lyd, semantic domain distillation loss Led, and spatial domain distillation loss Lpd- Therefore, the training loss C is,
C Arhred + hohad + Aphpd:
(3)
where = + Latcer Lo= = Lb + Ca Le and L9 denote cross-entropy loss for and Ó, respectively. Similarly, L and L denote dice loss [38] for O and Ó, respectively.
3.2.1 Semantic Prototype Adaptation.
Previous work [44] utilizes additional contextual cues to offset the fixed prototypes and thus adapt to the semantic feature distributions of different images, alleviating the feature deviation in the semantic domain problem some extent. However, due to the lack of constraints, the offset of the fixed prototype is not controllable. In extreme cases, the offset will even be in the direction away from semantic features (An example visualization can be found in Fig. 6.). Therefore, we propose SEmantic Prototype Adaptation (SEPA), which offsets fixed semantic prototypes based on coarse mask-guided semantic feature distributions, and constrains the adaptive semantic prototype S, with semantic domain distillation.

---

Specifically, we first multiply the semantic features Sy and the coarse mask Me to obtain the semantic domain center Sc, i.e., S. = Softmax (Mc) @ Sy, where Se E R denotes the average features of all pixel features belonging to different classes of the coarse mask-guided pre-classified representations. We then concatenate S. with the fixed prototype S, and fuse them through a 1 x convolution layer Φx to obtain the adaptive semantic prototype Sp E Rx
S = 6, O),
(4)
where @ denotes channel concatenation operation. With the above operation and the constraint of semantic domain distillation loss, the fixed prototype offsets towards the semantic feature distribution of the test image. As result, the pixel features possess higher similarity to the semantic prototypes of the corresponding categories, which promotes more pixels to be correctly categorized.
3.2.2 Spatial Prototype Adaptation.
Previous classifiers [34, 28, 62] are mainly based on the inner product of pixel features and prototypes in the semantic domain, without utilizing the rich spatial information of the image. However, most of the target objects in semantic segmentation tasks possess regular shapes, such as doors, windows, and roads. Modeling the spatial relations of pixel and prototype can introduce structured information about the target objects, thus improving the segmentation performance for boundary regions and small targets. Therefore, we introduce spatial prototype adaptation (SPPA), which aims to make classification decisions with additional consideration of the spatial relation between pixel features and prototypes.
We first obtain the spatial features Py E HxWxD with position encoding of the pixel features Sy, i.e., P, = PB(S)). Here we choose conditional position encoding [11], which useful for encoding neighborhood information to further localize the mask region. Similarly, we perform matrix multiplication based on the coarse mask with the feature position coding to obtain Pa.
Pe Softmax (Me) & Py.
(5)
where Pa E R denotes the spatial domain center of the coarse mask-guided pre-classified representations. Note that here we implement the softmax function for M, in spatial dimensions, which facilitates modeling the spatial distribution of the different categories and thus the spatial location of the category prototypes on the image.
In addition, only a few categories appear on the image in most cases. Therefore, only the positions of the corresponding categories in the Pa have practical significance. In order to maintain the stability of training, we define a randomly initialized position basis P e R2 xxD and concatenate it with P. to obtain the spatial prototype P, E IR IgXxD after mapping with 1 convolution layer op.
P, = (P @ P).
(6)
Similarly, based on the above operation and the constraint of spatial domain distillation loss, the prototype position basis is offset towards the center of the mask region for each class of the test image, thus generating the spatial prototype Pp, Therefore, the spatial relationship of the pixel features with the prototype can be taken into account when making the classification decision, which improves the segmentation performance of the boundaries and the small target regions. This is verified by the qualitative and quantitative analysis in Section 4.2.
3.2.3
Online
Distillation Learning.
Although semantic and spatial domain adaptations can motivate better interaction of prototypes with pixel features, the offset of semantic and spatial prototypes is not controllable due to the lack of constraint, which affects the segmentation performance of the model. In this paper, we propose an online multi-domain distillation learning to optimize the process of feature generation and constrain the adaptation of the semantic and spatial prototype.
Specifically, different from the previous widely adopted offline distillation learning method [30, 42, 47, 58, 23], we incorporate ground truth directly into the model training process [44] and construct soft labels, which can convey useful information to the model and do not require the additional process of training teacher models. As shown in Fig. 2, we first create a new branch with pixel features S, as inputs and use the ground truth Mg to guide the adaption process of semantic and spatial prototypes, which generates the semantic prototype S,,. spatial prototype P,,. and segmentation mask Ô Pp. Sp. and Ô are used as the online teachers to distillate Pp, Sp, and O, respectively.
5

---

09 80
100 34
80
(a) Sn- distillation
(b) Sy. distillation
(c) Sy- Ss-diistillation
(d) Sp-dittillttion
Figure 3: Visualization the inter-class relation matrix the semantic prototypes S, and Sy, and the latter possesses better inter-class separability. This motivates us to add semantic domain distillation loss to constrain the adaption of the semantic prototypes. The results show that after semantic domain distillation, the semantic prototypes have better separability, which facilitates category recognition.
Response Domain Distillation. The ground truth guided segmentation mask Ô has higher entropy value compared to the one-hot label, which can provide more information to the model. We first give the expression for the original response domain distillation learning,
K
HW
E Σ w(O (og(+(()))) L
C.d.
(7)
HW
where sb denotes the Softmax function along the channel dimension. However, the original expression averages the distillation loss for pixels at all spatial locations, which results in the spatial structural information of the object and the semantic information of a few sample categories being masked by pixels at other locations. Therefore, design boundary- to induce the transfer of semantic and spatial information from Ô to the prediction masks O.
and category- aware distillation loss
Specifically, we first obtain the semantic mask via My. We then use the Canny operation to extract the boundary of & and obtain the boundary mask B. Based on & and B, we modify Eq. 7,
HW
B B
C. B 2i
HW
Lrd = 2K
(8)
Σ B
where B3 = E B denotes the boundary mask of class k and B3 denotes the non-boundary mask of class k. Finally, we retain the entropy aware of [44] in order to adjust the contribution of each element according to the level of information,
Cnd -1 2K
HW B.H 21-1
ENW Σ B
HW C. B
)
H' = Σ MOO llog( ((t())(())) (9)
JW B
where Hi denotes the entropy of the i-th pixel prediction of Ó.
Semantic Domain Distillation. We propose the semantic domain distillation loss Lad to guide the offset process of semantic prototypes. As shown in Fig. 3 (a) and (b), the ground truth guided semantic prototype S, exhibits better inter-class separation properties compared to the Sp. In fact, the inter-class separation properties are critical for optimizing the feature embedding space and making classification decisions. Previous work [58] focuses on constraining inter-class relationships to be identical, which is difficult to train and leads to poor generalization. Intuitively, for inter-class relationships, we only need to constrain the closer classes in the student model to be farther away. For those classes that further away compared to the teacher model, we do not need to negatively supervise them. Specifically, we first compute the inter-class similarity matrix of semantic prototypes,
M = (( M ((S,, Then, the difference between M RKKK and M E RKXK calculated as,
(10)
Ma= AMMA
(11)
where A is mask operation that sets the value of both the diagonal position and the position less than zero to zero. Finally, we compute the semantic domain distillation loss C
K
1
Lad
M
(12)
K

---

Table I: Performance comparison of SSA-Seg state-of-the-art general (top) and light weight (bottom) methods.
The numher of FLOPSs (G) calculated on the input size of 512 512 for ADE20K and COCO-Stuff-100 10K,, and 480 PAStAL-Comtex.. The latency (ms) calculated on the input size 512 512 on V100 GPU. The green number indicates the increase from the baseline.
ADE20K
COCO-Sufff 10K
MASCAL-Conteer
Method
Backhone
Latency
Params
FL.PPs
OCRNet [54]
67.2
8.6
164.8
43.30
164.8
36,16
143.2
48.22
HRNer-W48
+SSA-Seg
69.3
8.7
165.0
47.47
165.0
37,94
143.3
50.21
UperNet [48]
236.1
236.1
38,93
207.5
51.93
Swin-T
+S8A-Seg
54.3
61.1
236.3
47,56 pue
236.3
42.30 purr
207.7
54.91 12.3
SegFormer [49]
52.5
52.5
MIT-B5
+SSA-Seg
70.1
82.3
52.6
50.74 pm
52.6
45.55 .ra
45.8
59.14 (a.
UperNet [48]
105.5
233.8
404.9
51.68
404.9
46.85
362.9
60.50
Swin-I.
+SSA-Seg
107.3
234.9
405.2
$2.69 yue
405.2
48.94 12.0
363.2
61.83 11.30
ViT-Adapter [8]
283.3
363.8
616.1
54.40
616.1
50.16
541.5
65.77
ViT-Adapter-1
+SSA-Seg
616.3
55.39 tace
616.3
51.18
AFFommer-Bi [14]
25.1
3.0
4.3
39.94
4.3
33.22
3.7
48.57
AFFommer
4SSA-Seg
26.0
3.3
4.4
41.92
4,4
36,40
3.7
49.72
SeaFormer [45]
1.8
40.05
1.8
ScaFormer-B B
SSSA-Seg
27.3
8.8
1.8
42.46 12.4
1.8
35.92 1500
1.6
47.00 r1.2
SogNeXt-T
22.8
6.2
41.04
6.2
36.39
5.4
50.35
MSCAN-T
+SSA-Seg
23.3
6.3
43.90 ya.
6.3
38.91 -1.
5,4
52.58 12.23
SeaFormer- [45]
29.4
14.0
6.4
42.36
6.4
35,99
5.6
49.14
SeaFormer-L
+SSA-Seg
29.9
6.4
49.66 pu:s
Note that Sp. in Eq. 10 is obtained based on the average semantic domain center of all training images in batch. can be used to minimize the negative impact of noisy images and accelerate model convergence.
With semantic domain distillation, the semantic prototype Sp exhibits similar separability to S,, which facilitates the recognition of categories. as shown Fig. 3 (c) and (d).
Spatial Domain Distillation. Unlike semantic domain distillation, since spatial domain distillation focuses the spatial structure information of an object, we tend to constrain the spatial prototypes guided by the rough mask to be equal to the spatial prototypes guided by the ground-truth mask,
K
D
Pd
* (Pp) (og())())))).
(13)
K
E
Experiments
We perform experiments on the ADE20K [61], PASCAL-Context [39] and COCO-Stuff 10K [i] datasets. We MMSegmennatiio [12] and follow the common training settings. Please refer to the Appendix for more details.
4.1 Main Results
Efficiency Comparison. We first focus the efficiency changes introduced by SSA-Seg, as shown in Table can be observed that the additional FLOPS and latency introduced by SSA-Seg negligible for methods of different sizes. For example, for general semantic segmentation methods such as ViT-Adapter-L, we only increase latency by 0.56% and flops by 0.03%, while for lightweight methods such as SeaFormer-L. we only increase latency by 1.70% and flops by 0.78%. It can be attributed to the fact that we add only a depth-wise convolution and some 1 convolutions to the primary classifier without changing the backbone and segmentation head. Therefore, the increased memory consumption, FLOPs and latency of SSA-Seg are negligible compared to the original model.
Performance on state-of-the-ar general methods. As shown in Table 1, the proposed SSA-Seg can significantly improve the segmentation performance of various general models with negligi- ble decrease in efficiency. For example, the application of SSA-Seg on UperNet-Swin-TT ViT-Adapter-L. bring 3.42% and 1.01% mloU performance improvements on ADE20K dataset, respectively.
and
7

---

Performance on state-of-the-art lightweight methods. SSA-Seg helps methods such as SeaFormer and SegNeXt achieve more superior performance without compromising efficiency. As shown in Table 1, by applying SSA-Seg. SeaFormer-L. achieves 45.36% mloU on ADE20K, while SegNext- T achieves 38.91 and 52.58% mloU on COCO-Stuff-100 10K and PASCAL-Comin datasets, re- spectively. This is the state-of-the-arr performance obtained in real-time segmentation tasks.
Comparison with state-of- the-art classifiers. To fur-
Table 2: Comparison with other state-of of-thee-rt art classifiers. FLOPs (G) are
calculated using the input size of 512 512 on the ADE20K dataset.
ther prove its superior perfor- mance, the SSA-Seg is com- pared with other state-of-the- art classifiers, as shown in Table 2. When UperNet is used as the baseline, the pro- posed SSA-Seg can improve the mloU by 1.20%, while
Metbod
Backbone
FLOPs
ADE20K
COCO.
PCN [34]
275.7
39,9
32.5
+ProtoSeg [62]
278.5
4
34.0
ResNet101 [19]
[466
278.5
41.1
33.0
SSA-Seg
275.9
44.3
36.6
UperNet
297.2
48.0
42.8
+GMMSeg [28]
302.3
44.3
Swin-B [32]
+DNC [6]]
308.6
48.6 tm
43.1 t13
+SSA-Seg
297.5
49,2
45,2
GMMSeg and DNC only in- crease mloU by 1.00% and 0.60%, respectively. When OCRNet is used as the base- line, SSA-Seg exceeds base- line by 4.2% mloU, which far exceeds GMMSeg by 2.7% mloU. Also, SSA-Seg ex- ceeds CAC 1.8% mloU. For SegNeXt-T, SSA-Seg exceeds CAC by 0.9% mloU. Moreover, for the lightweight model SeaFormer, CAC cannot bring growth. SSA-Seg can still achieve 2.5% mloU growth. The above experimental results demonstrate that the proposed SSA-Seg achieves state-of-the-eee performance.
OCRNet [54]
164.8
43.3
36.2
+GMMSeg [28]
HRNeVV2-W48 [43]
169.8
44.8
+CAC [+]]
164.9
45.7
+SSA-Seg
165.0
47.5 142
37,9
6.2
41.0
36.4
+CAC [44]
(ICAN-T[[I]]
6.2
43.
37.5
+SSA-Seg
6.3
43.9
38,9
SeaFarmer-B B [45]
1.8
40.0
33.3
+CAC|44] [44]
SeaFomer-B
[45]
1.8
40.1 tu
35.5
+SSA-Seg
1.8
42.5 ya.
35,9
Comparison with state-of-the-art mask classification models. In order to validate the effectiveness of SSA- Seg, we further perform a comprehensive comparison with the ofateoof-the-ar mask classification methods, shown in Table 3. It should be noted that the DPG Head in CGRSeg [40] confficts with SSA-Seg, and we remove the DPG Head before adding SSA-Seg. Therefore, compared with CGRSeg, CGRSeg+SSA- ficiency metrics such as parameters, FLOPs and latency. The results show that by combining SSA-Seg, the existing pixel-level segmentation baselines achieve a better balance between efficiency and performance compared to mask classification methods.
Table 3: Comparison with state-of-the-art mask classification models. FLOPs (G) calculated using the input size The latency (ms) is measured on a single V100 GPU with input size 5 512x112.
x 512.
Method
Params
FLOPs
Latency
mloU
MaskFormer [10]
41.3
55.1
31.0
44.5
rather reduces the ef-
Musmerr [9]
44.0
70.1
55.2
47.2
YOSO [20]
42.0
37.3
28.3
44.7
PEM [4]
35.6
46.9
CGRSeg-B [H0]
19.1
7.7
36.0
45.5
+SSA-See
19.3
7.6
36.0
47.1
CGRSeg-L [40]
35.7
14.9
43.3
48.3
+SSA-Seg
35.8
14.8
4.2 Ablation Study
Ablation of the position basis. We first explore the ne-
Table Ablation of the position basis.
cessity of spatial prototype adaptation. As shown in Table 4, Baseline+basi only means only randomly initialized position basis is used and the baseline is SeaFormer-L. It can be observed that when only the positional basis is retained, the performance of the model degraded due to the lack of a spatial prototype adaptation process and the corresponding spatial domain distillation.
Method
Params FL.PPs
Latency
mloU
SSA-Seg
14.2
6.4
29.9
45,4
Baseline=basis only
14.2
6.4
29.8
44,3
Ablation of Spatial domain center. We also conduct an ablation experiment the spatial domain center, as shown in Table 6. The results show that applying softmax in the spatial dimension gives better performance. This can be interpreted as the model learns the relative spatial distribution of each category and thus models a more accurate spatial domain center.
Ablation of PE. We explore the effect of different position encoding methods on the ADE20K dataset in Table 7. We chose three widely used methods, namely Sinusoidal position coding [3], learnable absolute position coding [16], and feature-dependent conditional position coding, i.e., CPVT [11].
8

---

Table 5: Alation experiments of the response domain distillation loss Lrd.
Table 6: Ablation experiments of the generation of Pa. Softmax srw denotes apply softmax operation on spatial dimension.
Table 7: Ablation experiments of po-
sition encoding (PE) methods on the ADE20K dataset.
Method
mloU
PE
mloU
SSA-Seg
45,36
Method
mloU
Sinusoidal [3]
44.18
-H.
44.82
Softmax 44.86
Leamable [16]
44.02
-B 44.77
Softmax 45.36
CPVT
45.36
-E
45.11
The results show that CPVT is particularly effective in improving the results. This can be explained by the fact that CPVT is able to encode neighborhood information and preserve the implicit positional priors to locate the region of interest the center of the spatial domain for the semantic mask. In contrast, sinusoidal and learnable absolute position encoding can only describe one anchor point, which is not conducive to the localization of segmented fragments.
Ablation of structure. A series of ablation experiments are performed verify the validity of the SEmantic Prototype Adaption (SEPA) and the SPatial Prototype Adaption (SPPA), respectively. The experimental results shown in Table 8.
Table 8: Ablation experiment of SEPA
and SPPA.
Method
FLOPs
mloU
Baseline
6.4
42.36
The 1x1 convolution with softmax is used as the baseline. It can be found that the baseline achieves only 42.36% mloU. By applying SEPA, mloU increases by 0.52% with growth of less than 0.1 GFlops. When combined with Lad and Cg. mloU increases by 1. 1.90%. This validates the necessity of semantic domain distillation to constrain the semantic prototype adap- tation process, i.e., they are structurally inseparable. Similarly,
SEPA
6.4
42.88
SPPA
6.4
42.76
SEPA SPPA
6.4
43.27
SEPA Lg + Ltd
6.4
44.26
SPPA Lg + Led
6.4
44.48
SSA-Seg
6.4
45.36
By applying SPPA and C, + Ln. mIoU increases by 2.12% and Flops increases by 0.1 The experimental results show that SEPA and SPPA can improve the segmentation performance.
Effect of multi-domain knowledge distillation. Multi-domain knowledge distillation can significantly improve the feature representation of SSg-Seg. To verify its performance, we con- duct related ablation experiments. The experimental results are shown in Table 9. In the baseline, only the primary classifier is used, i.e., without teacher classifier. Lg can increase mIoU by 0.60%. By applying response domain distillation Lrd, semantic domain distillation Ltd. and spatial domain distillation Lnd the mloU can be further increased by 0.67%, 1.09% and 0.84%, respectively. Multi-domain distillation as a whole can achieve a 2.09% mloU improvement. The above experimental results show that each domain distillation can improve the accuracy, and multi multi-domain distillation can further significantly improve the model performance. Furthermore, perform ablation analysis of Cyd design, as shown in Table 5. The results show that the intro- duction of entropy H. category mask E, and boundary mask B has a role to play in the improvement of the model performance.
Table 9: Ablation experiment of online
multi-domain knowledge distillation.
Method
mloU
Baseline SEPA SPPA
43.27
Ly
43.87
Ln + Ltd
44.54
Lg + Led
44.96
Ly + Lyd
44.71
Ly + Lust End
45.17
Ln + List Lest Led
45.36
Excluding the effect of model size. To demonstrate that the performance improvement of the baselines are due to the effective design of the SSA-Seg rather than the introduction of additional parameters and computational consumption, we carry out the exper- iments shown in Table 10. Specially, Baseline+ Conv means we add several convolutional layers to the orig- inal SeaFormer-L decoder in order to have the same model size GeaFoormer-LSSSA-SSS Note that although SSA-Seg has more parameters, it exhibits better computational efficiency and latency, which is more important for the efficiency of the model. The experimental results show that such a significant performance improvement (i.e +3.0%) cannot be obtained by simply increasing the baseline parameters and computational consumption.
Table 10: Experiments on whether performance growth is due model size increase.
Method
Params FLOPs Latency
mlol
SSA-Seg Basclinc+Conv
14.2
6.4
29.9
45.4
14.0
6.5
30.4
42.9
Effect of SSA-Seg on training. In addition, to further validate the impact of SSA-Seg on model training, we use SeaFormer-L as the baseline model to evaluate the mloU boost of SSA-Seg on the

---

SuaveRR S3A
19020001
124 90407
(a) Baseline roulls
(b) 1-SNE 5SA--Sa
(c) S5A-Sag resuls
Figure 5: Comparison of SSA-Seeg and Baseline (SeaFormer-L) results. Purple and gray indicate pixel features the door and other categories, re- spectively. Orange star indicates the initial fixed prototype of wall category, and red star indicates the adapted semantic prototype.
.200
24%
Figure 4: mlaU of the validation set on (a) ADE20K and (b) COCO-Stufl 10K with iterations.
validation set with different iterations, as shown in Fig. It can be observed that compared to the baseline model, mIoU increases more significantly and only requires 1/3 of the iterations to achieve the same segmentation performance as the baseline model. This indicates that the SSA-Seg accelcrates model learning and improves model performance. Note that due to the introduction of the teacher classifier, we need an average of 1.1 times the training time compared to the original model. However, the teacher classifier is removed when deploying the model, thus does not affect the efficiency.
Visual comparison of SSA-Seg and Baseline results. We provide the visualization results of SSA-Seg, which serves as complementary illustration to Fig. 1, further demonstrate that the SSA-Seg facilitates the adaptation of the fixed prototypes. As shown in Fig. 5, SSA-Seg effectively facilitates the offset of the prototype towards the center of the semantic domain (Fig. (b)). Since the adapted prototype is closer to the pixel features of the corresponding category, model can perform classification decisions more efficiently. As result, SSA-Seg obtains more accurate segmentation masks compared to the baseline (Fig. (c) Fig. (a)).
Examples of extreme offsets without distillation. We provide visual sample to demonstrate our claim in 3.2.1 i.e., the offset will even be in the direction away from the semantic features. As shown in Fig. 6, purple and gray indicate pixel features in the wall and other categories, respectively. Orange star indi- cates the initial fixed prototype of wall category, and red star indicates the adapted semantic prototype. It can be observed that without distillation, the offset of the prototype is uncontrollable and in extreme cases moves away from the semantic features the corre- sponding image, resulting in more pixels belonging to the wall category being misclassified as curtains. Therefore, we introduce online multi-domain distil- lation to constrain the adaptation of the semantic and spatial prototype.
(a) Image
(b) Result without distillation
Result with distillation
L-SNE without distillation
(0) I-SNE distillation
Figure Examples extreme offsets without distillation.
5 Conclusion
In this paper, we analyze that current pixel-level classifiers for semantic segmentation suffers limita- tions such as feature deviation in the semantic domain and information loss in the spatial domain. To this end, we propose novel Semantic and Spatial Adaptive Classifier (SSA-Seg). Specifically, we employ the coarse masks obtained from the fixed prototypes as guide adjust the fixed prototype towards the center of the semantic and spatial domains in the test image. In addition, we propose an online
distillation learning strategy guide the adaption process. Experimental results on three publicly available benchmarks show that the proposed SSA-Seg significantly improves the segmentation performance of the baseline models with only a minimal increase computational cost. In particular, SSA-Seg boosts the lightweight model to achieve state-of-the-art performance. The superior performance proves that SSA-Seg can replace the vanilla pixel-level classifiers and thus contribute the semantic segmentation research.
10

---

References
[1] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1209-1218, 2018. 7, 15
[2] Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. Proceedings of the IEEE/CVF
International Conference Computer Vision, pages 17302-17313, 2023. 17
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zogoruyko. End-to-ent object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020. 3, 8.9
[4] Niccolò Cavagnero, Gabriele Rosi, Claudia Cuttano, Francesca Pistilli, Marco Ciccone, Giuseppe Averta, and Fabio Cermelli. Pem: Prototype-based efficient maskformer for im- age segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 15804-15813, 2024. 8
[5] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706. 05587, 2017.
[6] Liang-Chin Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018. 1,
3
[7] Xinghao Chen, Yiman Zhang, and Yunhe Wang. Mtp: multi-task pruning for efficient semantic segmentation networks. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2022.
[8] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In The Eleventh International Conference Learning Representations, 2023. 3,7
[9] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-12999 2022. 1, 3, 3, 18, 18
[10] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34: 17864- 17875, 2021. 3, 8, 17, 18
[11] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional
encodings for vision transformers. In ICLR 2023, 2023. 5, 8, 18
[12] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox 2020.
and benchmark.
com/ open momeen-mmmlle aal hwwlllaannnneeeeneen
15
[13] Tamás Czimmermann, Gastone Ciuti, Mario Milazzo, Marcello Chiurazzi, Stefano Roccella, Calogero Maria Oddo, and Paolo Dario. Visual-based defect detection and classification approaches for industrial applications-a survey. Sensors, 20(5):1459, 2020.
[14] Bo Dong, Pichao Wang, and Fan Wang. Head-free lightweight semantic segmentation with linear transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 516-524, 2023. 1,3,7
[15] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3146-3154. 2019.
[16] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International conference on machine learning, pages 1243- 1252. PMLR, 2017. 8,9
[17] Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning L Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in Neural Information Processing Systems, 35:1140-118 2022. 1, 3, 7, 8, 17
11

---

[18] Haoyu He, Jianfei Cai, Zizheng Pan, Jing Liu, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Dynamic aocus-aware positional queries for semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11299-1 1 1308, 2023.
3
[19] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Jie Hu, Linyan Huang, Tianhe Ren, Shengchuan Zhang, Rongrong Ji, and Liujuan Cao. You only segment once: Towards real-time panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17819-17829, 2023. 8
[21] Ye Huang, Di Kang, Liang Chen, Xuefei Zhe, Wenjing Jia, Linchao Bao, and Xiangjian He. Car: Class-aware regularizations for semantic segmentation. In European Conference on Computer Vision, pages 518-534. Springer, 2022.
[22] Zilong Huang, Xinggang Wang, Lichao Huang. Chang Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 603-612, 2019.
[23] Deyi Ji, Haoran Wang, Mingyuan Tao, Jianqiang Huang, Xian- SannSheng Hua, and Hongtao Lu. Structural and statistical texture knowledge distillation for semantic segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16876-16885, 2022. 5
[24] Zhenchao Jin, Bin Liu, Qi Chu, and Nenghai Yu. Isnet: Integrate image-level and semantic-level context for semantic segmentation. In Proceedings of the IEEE/CVE international conference on computer vision, pages 7189-7198, 2021. 3, 18
[25] Çağnı Kaymak and Ayşegül Uçar. A brief survey and an application of semantic image segmentation for autonomous driving. Handbook of Deep Learning Applications, pages 161- 200, 2019. 1
[26] Julian Laufs, Hervé Borrion, and Ben Bradford. Security and the smart city: A systematic review, Sustainable cities and society, 55:102023, 2020. 1
[27] Xiangtai Li, Hao He, Xia Li, Duo Li, Guangliang Cheng, Jianping Shi, Lubin Weng. Yunhai Tong, and Zhouchen Lin. Pointflow: Flowing semantics through points for aerial image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4217-4226, 2021. 1
[28] Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Gmmseg: Gaussian mixture based generative semantic segmentation models. Advances in Neural Information Processing Systems, 35:31360-31333 2022. 2, 3, 8, 18
[29] Sun-Ao Liu, Hongtao Xie, Hai Xu, Yongdong Zhang, and Qi Tian. Partial class activation attention for semantic segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16836-16845, 2022. 3
[30] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, pages 2604-2613, 2019. 5
[31] Yuhe Liu, Chuanjian Liu, Kai Han, Quan Tang, and Zengchang Qin. Boosting semantic segmentation from the perspective of explicit class embeddings. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 821-831, 2023. 3, 17
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021. 8
[33] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Pruceedings of the IEEF/CVE conference on computer vision and patlern recognition, pages 11976-11 1986, 2022. 3
[34] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015. 3, 5,8
12

---

[35] Chengtao Lv, Hong Chen, Jinyang Guo, Yifu Ding, and Xianglong Liu. Ptg4sam: Post- Post-training quantization for segment anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15941-15951, 2024.
[36]
Xiaowen Ma, Rui Che. Xinyu Wang, Mengting Ma. Sensen Wu, Tian Feng, and Wei Zhang. Docnet: Dual-domain optimized class-aware network for remote sensing image segmentation. IEEE Geoscience and Remote Sensing Letters, 21:1-5, 2024. 3
[37] Xiaowen Ma, Rongrong Lian, Zhenkai Wu, Hongbo Guo, Mengting Ma, Sensen Wu, Zhenhong Du, Siyang Song, and Wei Zhang. Logean++: Adaptive local-global class-aware network for semantic segmentation of remote sensing imagery, 2024. 3
[38] Fausto Milletari, Nassir Navab, and Beyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages
leee, 2016. 4
[39] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 891-898, 2014. 7, 15
[40] Zhenliang Ni, Xinghao Chen, Yingjie Zhai, Yehui Tang, and Yunhe Wang. Context-guided spa- tial feature reconstruction for efficient semantic segmentation. arXiv preprint arXiv:2405.06228, 2024.
[41] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017. 17
[42] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5311-5320, 2021. 5
[43] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693-5703, 2019. 8
[44] Zhuotao Tian, Jiequan Cui, Li Jiang, Xiaojuan Qi, Xin Lai, Yixin Chen, Shu Liu, and Jiaya Jia. Learning context-aware classifier for semantic segmentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 2438-2446 2023. 2, 3, 4, 6, 8
[45] Qiang Wan, Zilong Huang. Jiachen Lu, Gang YU, and Li Zhang. Seaformer: Squeere-enhanced axial transformer for mobile semantic segmentation. The Eleventh International Conference on Learning Representations, 2023. 1, 2, 7,
[46] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu, Visual recognition with deep nearest centroids. arXiv preprint arXiv:2209.0733: 2022. 2, 8, 18
[47] Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16, pages 346-362.
Springer, 2020. 5
[48] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018. 7
[49] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 14:12077-12090, 2021. 7
[50] Yixing Xu, Chao Li, Dong Li, Xiao Sheng. Fan Jiang, Lu Tian, and Ashish Sirasao. Fdyit: Improve the hierarchical architecture of vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 5950-5960, 2023. 3
[51] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means mask transformer. In European Conference on Computer Vision, pages 288-307. Springer, 2022. 3
13

---

[52] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si. Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of the IEEE/CVE conference computer vision and paltern recognition, pages 10819-10829, 2022.
3
[53] Weihao Yu, Pan Zhou, Shuicheng Yan, and Xinchao Wang. Inceptionnext: When inception meets convnext. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5672-5683, 2024. 3
[54] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object- representations for semantic segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 173-190. Springer, 2020. 3, 7, 8, 17, 18
[55] Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit:
Semantic segmentation with plain vision transformers. Advances in Neural Information Processing Systems, 19:4971-49822 2022. 1,3, 17
[56] Fan Zhang, Yanqin Chen, Zhihang Li, Zhibin Hong, Jingtuo Liu, Feifei Ma. Junyu Han, and Errui Ding. Acfnet: Attentional class feature network for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6798-6698900 68806687,22 2019. 2, 18
[57] Wenqiang Zhang, Zilong Huang, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, and Chunhua Shen. Topformer: Token pyramid transformer for mobile semantic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12083-12093, 2022. 1,3
[58] Zhengbo Zhang, Chunluan Zhou, and Zhigang Tu. Distilling inter-class distance for semantic segmentation. arXiv preprint arXiv:2205. 03650, 2022. 5, 6
[59] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2881-2890, 2017. 1,3
[60] Zhuo Zheng, Yanfei Zhong, Junjue Wang. and Ailong Ma. Foreground-aware relation network for geospatial object segmentation in high spatial resolution remote sensing imagery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages -196,4105,,0 2020.
[61] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 633-641, 2017. 7, 15
[62] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Lue Van Gool. Rethinking semantic segmentation: A prototype view. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2582-2593, 2022. 2, 3, 5, 8, 18
14

---

SUMMARY OF THE APPENDIX
In the appendix, we provide the following items that shed deeper insight contributions:
S A: Dataset and Implementation Details.
B More experimental results.
SC: More qualitative visualization.
SD: Extra analysis
A Dataset and Implementation Details.
A.1 Datasets
ADE20K [61] a challenging segmentation dataset, which contains about 20,000 images and covers 150 categories. All images annotated with pixel-Ievel objects and object parts labels. The training set, validation set, and the test set contain 20210, 2000, and 3352 images respectively.
PASCAL-Context [39] is common semantic segmentation dataset, which contains 10100 images. The train dataset contains 4996 images and test set contains 5104 images. 59 categories are labeled in this dataset.
COCO-Stuff-10k [1] an extension of the coco dataset, which labels pixel-level objects. It contains 172 categories with 80 things, 91 stuff and unlabeled class. There are 9K/1K images for training and testing, respectively.
A.2 Implementation details
We use MMSegmentation [12] and follow the common training settings. During training, we apply data enhancement sequentially by random horizontal flipping, random resizing with a scale between 0.5 and 2.0, and random cropping. For ADE20K and COCO-Stufi 10K, we have a cropping size of 512 x 512, while for PASCAL-Context we have a cropping size of 480 x 480. In addition, the batch size of all datasets is 16, and the total iterations for ADE20K, COCO-Stuff-10R 10K and PASCAL-Context number are 160k, 80k and 80k, respectively. When inference is performed, we use whole-image inference mode. We use the mean intersection and merger ratio (mloU) as main metric to evaluate the performance. For the chosen baseline model, we fully follow the experimental configurations such learning rate and optimizer of the original model. During testing, test strategies for fair comparison. In the implementation of this paper, Xr, Ap. A are all set to I, and the edge size of boundary is set to 4. Note that for SSA-Scg-related experiments, choose different random seed replications for three times and select the corresponding intermediate values as the final evaluation metric values. The experimental error is less than 0.4%.
single-scale (SS)
B More experimental results.
Ablation of edge size of boundary. The size of the edges of the boundary masks affects the performance of the response domain distillation. We explore this shown in Table 11. The results show that the optimal segmentation results are achieved when the size of the edges is set to 4. In particular, there is a slight degradation the performance of the model when either the edge size of the boundary is increased (i.e., edge size is 5) or decreased (i.e., edge size 3). We argue that plausible explanation is that small edge size of the boundary is
Table 11: Ablation experiment of the edge size of boundary the ADE20K datasct.
not sufficient to retain enough boundary information, while large edge size of the boundary does not decouple the knowledge transfer between boundary and non- non-boundary regions, which results in the pixel feature information of boundary regions being interfered with by non-boundary regions.
Edge size
mloU
3
44.74
5
44.98
4
45.36
Ablation of different loss combinations. We conduct experiments on the ade20k dataset to explore the segmentation performance of different loss combinations, as shown in Table 13. We observe that only when both X, and Ap are 1, the corresponding segmentation
15

---

(a) Image
Image
(c) Image
(d) t-SNE of images
Figure 7: t-SNE of example images, which randomly selected from the ADE20K dataset. The first represents distribution of pixel features in the door class, and the second row represents table class. can be observed that due to the complex scenarios and varying object distributions, pixel features of the same class tend to exhibit larger intra-class variance when the trained model on the training set is applied to the test set.
results are relatively high, i.e., 44.67%, 44.82%, and 44.86%. Furthermore, we achieve the most superior segmentation performance when A, is also 1.
Ablation of the distillation method. An
Table 12: Ablation of the distillation method.
alternative distillation strategy is to self-
distill instead of using ground-truth, self- distillation means we use the output of the main classifier O distill the coarse mask Mr instead of the truth mask Mg. The results in Table 12 show that the self-
Method
Params
FLOPs
Latency
mloU
SSA-Seg self-distillation
14.2
6.4
29.9
45.4
14.2
6.4
29.9
43.5
distillation scheme performs poorly. We argue that the ground-truth can provide more accurate semantic information, while the segmentation results of output often have some misclassified pixels, which can mislead the model's learning process from the teacher classifiers. Therefore, the prototype will offset inaccurately and impairs the segmentation performance.
C More qualitative visualization.
Visual analysis of segmentation output. We SeaFormer-L as the baseline to visualize the segmentation mask, as shown in Fig. 8. SeaFormerr and CAC misclassify the door as the wall, and the sofas the armchair the second and fourth row of images, respectively. Our method has high accuracy for these confusing classes, which verifies that semantic prototype adaptation can mitigate the intra-class variance problem for correct recognition. In addition, the segmentation masks of SeaFormer and CAC for the curtain show fragmentation in the first row of images. While in the third row of images, the masks for the sidewalk show the same situation. In contrast, the shape of our segmentation masks is close to ground truth, especially in the boundary region. This validates the effectiveness of our introduction of spatial prototype adaption to model the structure of semantic objects.
Table 13: Ablation experiment of differ- ent loss combinations on the ADE20K dataset. The baseline is SeaFormer-L.
A
A
Ap
mioU
0.5
1.0
1.0
43.91
2.0
1.0
1.0
43.14
4.0
1.0
1.0
43.62
1.0
0.5
1.0
44.67
2.0
1.0
44.82
1.0
4.0
1.0
44.86
Visual analysis of CAM. In addition, for the features output from the backbone, from top to bottom, we carry out CAM for the class of curtain, door, sidewalk, and sofa, respectively. As shown in Fig. 8, it can be seen that compared to SeaFormer and CAC, SSA-Seg is able to present stronger activation values the center region of the mask and does not show too much activation in irrelevant regions. These qualitative analyses re- flect the effectiveness SSA-Seg in performing classification decisions.
1.0
1.0
0.5
44.05
1.0
1.0
2.0
44.48
1.0
1.0
4.0
43.90
2.0
2.0
2.0
44.03
4.0
4.0
4.0
43.02
1.0
1.0
1.0
45,36
16

---

1943
Image
GT
SeaFormer
CAC
SSA-Seg SeaFormer
CAC
SSA-Seg
Figure 8: Visualization of segmentation predictions and class activation maps [41] for features output from the backbone on the ADE20K dataset. SeaFormer-L. is the baseline.
Visualization example of intra-class variance for different test images. Compared to Fig. We provide additional photographic illustrations in Fig. 7 to support our statements, i.e., Due complex backgrounds and varying object distributions, pixel features the test images tend to have a large intra-class variance with pixel features in the training set.
D Extra analysis.
D.1 Limitation analysis.
As novel pixel-level classifier, SSA-Seg is able to significantly improve the performance of various baselines with minor increase in computational cost. However, it is not compatible with mask classification methods [10, 9] due to differences in classification paradigms. Therefore, SSA-Seg cannot be applied improve the performance of segmentation networks such Mask2Former [9], Seg [55], and ECENet [31]. This limits the application scope of SSA-Seg to some extent. However, when SSA-Seg is combined with efficient segmentation models such SegNeXt [17], EfficientViT [2], it can achieve even better performance with the same or less computation compared state-of-the-art methods. Therefore, SSA-Seg needs rely
advanced pixel-level semantic segmentation methods to boost its performance on general semantic segmentation tasks.
Note that in the field of mobile segmentation (or real-time segmentation), the pixel-level classification paradigm is still absolutely dominant, due to the fact that masked classification methods require multiple cross-attention latency. Whereas SSA-Seg is able to integrate directly into existing mobile segmentation models, thus significantly improving segmentation performance with essentially compromise model efficiency. This validates the applicability and deployability of SSA-Seg.
update the query, resulting in higher computational complexity and
D.2 Comparison with OCRNet.
SSA-Seg is obviously different from some class-level context modeling approaches such as OCRNet [54]. First, OCRNet is a context aggregation module that enhances the representation of features based on context-aggregated class representations, but still requires a Softmax classifier (i.c., fixed semantic prototype) perform classification decisions. SSA-Seg, on the other hand, classifier whose semantic prototype is dynamic and can be offset towards the center of the semantic domain of the image. addition, SSA-Seg can be combined with OCRNet as a classifier enhance the performance of OCRNet, shown in Table 1. Finally, SSA-Seg additionally introduces spatial prototype adaptation and online multi-domain distillation, which significantly different from the work of OCRNet. Overall, SSA-Seg aims to adapt the fixed prototype towards the center of the semantic and spatial domains in the test image to solve the problem of feature deviation in the semantic domain and loss of information in the spatial domain that exists in fixed prototype classifiers.
17

---

Therefore, -Sg is significantly different from existing context modeling methods [54, 56, 24], mask classifiers [9, 10], and pixel-level classifiers [28, 46].
D.3 Structural Effectiveness Analysis.
An interesting phenomenon as shown in Table is that the SSA-Seg architecture (i.c., Base- line+SEPA+SPPA) achieves only marginal improvements (+0.91%), which is not indication that the design of SSA-Seg is ineffective. Note that both the semantic prototype adaptation and spatial prototype adaptation processes involved in SSA-Seg need be constrained by the corresponding distillation loss. Otherwise, the offset of the corresponding prototype will be uncontrollable. In extreme cases, the offset will even be in the direction away from the semantic/spatial features, as shown in Fig. 6. Therefore, the adaptation process of SSA-Seg in the semantic or spatial domain needs to be combined with the corresponding distillation loss, and the two are closely linked and inseparable. As shown in Table 8, both the adaptation of semantic prototypes (+1.9%) and the adaptation of spatial prototypes (+2. 12%) significantly improve the segmentation performance the baseline model, which validates the effectiveness of the SSA-Seg architecture design.
D.4
Complementary analysis of SPPA.
Please note that the SPPA does not assume that objects of the same class should all in a specific region of the image. Although SPPA proposes the concept of spatial domain centers, our spatial domain centers are obtained based on the Conditional Positional Vocoding (CPVT) [11], which represents for relative segments rather than absolute anchor points. Therefore, our spatial domain center models the collection of information about neighboring segments belonging to the same object, rather than absolute one point. In other words, it enables the model to indirectly take into account semantic features of pixels at neighboring locations when classifying them. Second, many images in ADE20k and COCO-stuff have the same category scattered in different areas. Nevertheless, our SSA-Seg can still significantly improve the performance of the model on these datasets. This is because SPPA takes into account the spatial structure information. We have conducted ablation experiments on SPPA in the paper, as shown in Table 8. When SPPA and distillation loss are applied, the model accuracy increased by 2.12% mloU. This validates the effectiveness of spatial domain adaptation.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to abstract and the lines 74-81 of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the Sec. D in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the implementation details, please refer to Sec. in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: We provide all the code and configuration files in order to reproduce the experiments in the paper. The datasets are publicly available and can be downloaded.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niit cc/
public/ guides CodeSubmi as i cassoooooicyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: i ps cc/ hcc/pbll c/ des CodeeS Summi i ss onPol 1 ayy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec. A in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec. A in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification:
We did not conduct statistics on this topic.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification: After review, the research conducted in the paper complied with the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the public datasets and code repositories involved in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' raw w thcode com/ data setss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "HZ4qPxrkm1", "reviewer": "Reviewer_ra2D", "summary": "This paper proposes an adaptive method to improve the semantic segmentation quality. The main idea is to adaptively update the prototypes by using the coarse segmentation masks predicted by the baseline method. Both semantic and spatial prototypes are employed to achieve complementary improvement. Extensive experimental evaluation has been conducted on three public benchmark datasets,  showing consistent improvement over different baseline methods.", "questions": "See weakness", "limitations": "Limitations are discussed in the appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method seems to be novel. The computation overhead of the method is negligible. \n\n2. The results show that the proposed method can consistently improve many segmentation methods, in particular, the methods with lower latencies. \n\n3. Extensive ablation studies are included to show the contributions of different components.", "weaknesses": "1. Not every loss term in the Equation 3 is ablated, for example L_c and L_dice\n\n2. Since mask-level classifiers become more and more popular, how could be proposed method be adapted to mask-level methods such as mask2former?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their time and efforts in reviewing our work and providing valuable feedback that can further strengthen our manuscript. Below please find our detailed responses:\n\n#### **Ablation of loss**\n\n------\n\nWe retain by default the cross-entropy losses $L_{ce}$ and $L_{dice}$, the former is an basic loss function for semantic segmentation and the latter commonly used in mask classification models for mask generation. We focus on ablation analysis of online multi-domain distillation loss in the paper, which is one of the important contributions of the paper. We have also done the ablation as shown below: \n\n| Method        | mIoU  |\n| ------------- | ----- |\n| SSA           | 45.36 |\n| -$L_{dice}^c$ | 44.91 |\n| -$L_{dice}^g$ | 45.06 |\n\nwe observe that when removing $L_{dice}^c$ or $L_{dice}^g$ (the cross-entropy loss cannot be removed), the mIoU slightly decreases, i.e., from 45.36 to 44.91 and 45.06 respectively. \n\n#### **Adapt to mask-level methods**\n\n------\n\nAs pixel-level classifiers, SSA and mask classification methods belong to two different technical routes. However, SSA is lightweight enough, thus has the potential to be combined with mask classification methods. For example, SSA can be embedded behind a mask classifier to further refine the pre-classification results obtained by Mask2Former. We will explore better adaptation method in our future work to further improve the range of SSA applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1BS5xRLwwS", "reviewer": "Reviewer_bEdU", "summary": "This work primarily focuses on semantic segmentation. Specifically, a semantic and spatial adaptive (SSA) classifier is proposed to address  the \"feature deviation in the semantic domain and information loss in the spatial domain\" issues. The proposed classifier mainly consists of \n\n- Semantic Prototype Adaptation,\n- Spatial Prototype Adaptation,\n- Online Multi-Domain Distillation Learning.\n\nThe authors claim that \"experimental results on three publicly available benchmarks show that the proposed SSA significantly improves the segmentation performance of the baseline models with only a minimal increase in computational cost.\"", "questions": "The main issues with this work have already been listed in the weaknesses box. The presence of numerous unreliable contents and my belief that the premise of the proposed algorithm is unreasonable are the main reasons I am inclined to reject this paper.", "limitations": "The author does not discuss the limitations of their algorithm in the paper. I have provided several suggestions for improvement in the weaknesses box, which I hope will help the authors enhance the quality of this work.", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strengths of this paper can be summarized as follows,\n\n- Overall, the author clearly introduces the research motivation and the proposed method.\n\n- The author validates the effectiveness of the proposed algorithm across different datasets and segmentation frameworks.", "weaknesses": "The weaknesses of the paper are as follows,\n\n- The authors claim in the abstract that the code for the proposed algorithm is included in the supplementary materials, but I did not find any related code there.\n\n- Some expressions in the paper feel redundant, e.g., \"directly classify masks by directly learning query vectors\".\n\n- In the introduction section, the authors argue that previous mask-level classification models have cumbersome decoders. Therefore, the authors should conduct a comprehensive comparison (params, fps, flops, miou, etc) between their method and these previous methods in the experiments section.\n\n- Tables 1 to 3 only provide the FLOPs metric but do not include FPS and parameters, which are also crucial for deploying the model in resource-constrained environments.\n\n- Can the authors provide any experiments or references to support the statement \"Due to complex backgrounds and varying object distributions, pixel features in the test images tend to have a large intra-class variance with pixel features in the training set\"? I think that a single image in Figure 1 can not prove this point.\n\n- To be honest, Figure 1 is very confusing to me. For example, what do the gray dots represent? (pixel features belonging to other categories?) Can you provide the model's prediction results for this image? If your visualization is accurate, the prediction should include many pixels of other categories being classified as “door”. In other words, I believe there is a high likelihood that your visualization is unreliable, namely, in the t-SNE visualization results, being close to a semantic prototype does not necessarily mean that the model classifies it as the corresponding semantic category in the feature space.\n\n- If the proposed SSA strategy truly facilitates “the offset of the fixed prototype towards the center of the semantic domain and the center of the spatial domain of the test image,” why don't the authors directly visualize this result using t-SNE like Fig. 1(b) instead of presenting a less credible schematic diagram?\n\n- I disagree with the content in lines 36 to 40 of the paper. Modeling the relationship between prototypes and pixel features in the spatial domain is incorrect. This means that when classifying pixels, models need to consider which types of objects generally appear at that pixel location. This is obviously an overfitting to the current dataset and has no practical application value.\n\n- The authors argue that \"pixel features in the test images tend to have a large intra-class variance with pixel features in the training set\" and their proposed method can address this issue. If this is indeed the case, the author should apply their method to cross-domain segmentation tasks to better demonstrate its effectiveness.\n\n- The content in the Related Works section is too brief.\n\n- In line 126, the authors mention that \"Based on the above analysis\", where is the analysis?\n\n- Can the authors provide any experiments or references to support the statement between line 150-151?\n\n- The content stated in lines 155 to 157 does not correspond with the formula.\n\n- Section 3.2.1 essentially just integrates features similar to class prototypes and does not achieve the author's claimed “facilitating the offset of the fixed prototype towards the center of the semantic domain.”\n\n- “utilizing the rich spatial information of the image” should be the responsibility of context modules (such as aspp and ppm) in the decoder head. The premise of section 3.2.2 is fundamentally flawed.\n\n- Would the model's performance improve if only “randomly initialized position basis” is used? Can the authors provide related ablation experiments?\n\n- If I understand correctly, section 3.2.2 essentially assumes that objects of the same class should all be in a specific region of the image. However, this assumption is unreasonable because, in semantic segmentation, different instances of the same class can be far apart.\n\n- To be honest, I believe the methods in sections 3.2.1 and 3.2.2 are not highly related to the problem analyzed by the authors or the expected solutions to that problem.\n\n- Would the model's performance improve if section 3.2.3 was replaced with self-distillation? \n\n- Due to the existence of $\\mathcal{M}_c$, the problem mentioned by the authors in the introduction section still persists in the proposed algorithm. In my opinion, the performance improvement is mainly due to the addition of more parameters and the introduction of more loss functions. While these changes have brought about performance gains, they do not align with the authors' motivation.\n\nDue to these weaknesses, I believe this paper should be rejected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Code of SSA**. We download the supplementary material and confirm that the code is included. \n\n**Redundant expressions.** We will streamline the expression in the revised version with your comments.\n\n**Comparison with mask-level classification models** and **parameters comparison**. please refer to ***global rebuttal***.\n\n**Experiments or references to support the statement.** We provide more graphic illustrations in Fig.1 of rebuttal. In addation, a simple hypothesis can help you to solve this confusion. If pixel features of the same class do not exhibit large intra-class variance in the training and test sets, then the model should achieve very high accuracy on the test set. However, experimental results show that the baseline only has an mIoU of 42.4, and the IoU of door category is only 46.9. SSA can somewhat target this problem to achieve improved segmentation performance (mIou 45.4, IoU 49.7).\n\n**Explanation of the Fig.1.** Your understanding is accurate.  We provide a prediction result of the baseline as shown in Fig.2 (a) of rebuttal,which shows that many pixels belonging to door are misclassified. This is consistent with the visualization results of t-SNE. Your statement that the prediction should include many pixels of other categories classified as \"doors\" is not accurate. This is because they may be closer to the prototypes of other categories than to the prototypes of doors. In particular, due to the lack of constraints, the prototypes of some categories are more similar, as shown in Fig.3 of the paper.\n\n**t-SNE of the result.** The schematic diagram facilitates us to describe the technical route of the whole approach, i.e., semantic prototype adaptation (Fig. 1(c)) and spatial prototype adaptation (Fig. 1(d)).We also provide t-SNE and prediction for SSA in Fig.2 of rebuttal, which show that SSA indeed facilitates the offset of the fixed prototype towards the semantic domain center and the spatial domain center of the test image.\n\n**Overfitting to the current dataset.** I'm in disagreement with you. As far as I know, with the exception of multimodal segmentation work (e.g., SAM), current semantic segmentation methods (such as PEM (CVPR2024), and CGRSeg (ECCV2024)) are generally trained and tested on a single dataset. They also essentially fit the current dataset. Note that we significantly improve the performance of various state-of-the-art baseline models on all three commonly used semantic segmentation benchmarks, which validate the practical application value.\n\n**Apply method to cross-domain segmentation tasks.** It is not the focus of this article, but we will apply it to cross-domain segmentation tasks in future work.\n\n**Related works is brief.** We will add more descriptions of related work in the revised version.\n\n**where is the analysis.** The analysis is in Sec 3.1, i.e., the inadequacy of the vanilla pixel-level classifier (feature deviation in the semantic domain and information loss in the spatial domain) .\n\n**Experiments to support line 150-151.** We provide an example as shown in Figure 3 of rebuttal.\n\n**Content does not match the formula.** We have verified that there is no mismatch.\n\n**Explanation of the prototype offset.** I think you're misunderstanding the offset. In your terms, Sec 3.2.1 essentially just integrates features similar to class prototypes. however, the weights of the original fixed prototype S do change through this integration of features, and there is no problem with interpreting this as an offset of the prototype. In particular, we constrain this offset process through semantic domain distillation, which ensures that the prototypes are shifted towards the center of the semantic domain.\n\n**Premise of section 3.2.2.** Contextual modules such as ppm and aspp utilize spatial information to enhance the feature representation of pixels through contextual modeling. And SSA models the spatial information of semantic objects through spatial prototype adaptation and in turn influences the similarity between pixels and prototypes to optimize the classification decision process. They are different technical routes and can gain from each other, as shown in Table 1 of the paper.\n\n**Ablation of position basis.** Please refer to Table 2 of rebuttal. \n\n**Different instances can be far apart.** We have considered this before, but three aspects ensure that SSA proposes the design of SPPA.1, in the semantic segmentation dataset, only one instance for a category occurs in most of the pictures (especially the less-sample categories, such as door and sidewalk in Fig. 5 of the paper).SSA is better able to bring in the structured information of these objects.2, considering the occurrence of the same category in the pictures of different instances, we implement SPPA based on CPVT. it models spatial prototypes as relative segments rather than absolute anchors. The detailed analysis can be found in lines 314-321 of the paper.3, Ablation experiments verify the effectiveness of SPPA.\n\n**Relation of the methods and problem.** I'm confused about this. The problem we are posing is that vanilla pixel-level classifiers suffer from feature deviation in the semantic domain and information loss in the spatial domain. And we mitigated that problem with semantic prototype adaptation and spatial prototype adaptation. We believe that the relation between the problem and the method is highly relevant.\n\n**Performance with self-distillation.** Please refer to Table 3 of rebuttal. We argue that the prediction mask is not accurate, and using it as an input to the teacher classifier will introduce incorrect category features and noise.\n\n**performance gains not align with motivation.** Please refer to Table 5 of rebuttal, which shows that simply increasing the size of the model does not result in such a large performance gain. As for the loss function, it was proposed to serve the structure of the model, which is an inseparable part of SSA and one of the contributions of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uROXfsNFDK", "reviewer": "Reviewer_Dw1G", "summary": "In this paper, it analyzes that current pixel-level classifiers for semantic segmentation suffers limitations such as feature deviation in the semantic domain and information loss in the spatial domain. To this end, the authors propose a novel Semantic and Spatial Adaptive (SSA) classifier. Specifically, the authors employ the coarse masks obtained from the fixed prototypes as a guide to adjust the fixed prototype towards the center of the semantic and spatial domains in the test image. In addition, the authors propose an online multi-domain distillation learning strategy to guide the adaption process. Experimental results on three publicly available benchmarks show that the proposed SSA significantly improves the segmentation performance of the baseline models with only a minimal increase in computational cost.", "questions": "No. please see weakness part.", "limitations": "No. please see weakness part.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths are as follows:\n1. A semantic and spatial adaptive (SSA) classifier is proposed, which facilitates the offset of the fixed prototype towards the center of the semantic domain and the center of the spatial domain of the test image. \n2. This paper designs multi-domain knowledge distillation to enhance the primary classifier from different domains. First, the response domain distillation distills the outputs of the two classifiers based on a boundary-aware and category-aware distillation loss, which conveys accurate semantic and structural information in the ground truth.\n3. The proposed method significantly improves the segmentation performance of the baseline model  with a very small increase in computational cost on three commonly used datasets: ADE20k, PASCAL-Context, and COCO-stuff-10K.", "weaknesses": "The main weaknesses are as follows:\n1. Why not evaluate the proposed methods on high resolution dataset, like Cityscape. How about the memory usage comparison? what's the input image resolution for training and inference? These are key information need to put in this paper.\n2. For the online multi-domain Distillation, in line 199 \"we first create a new branch with pixel\", what does the \"new branch\" mean? There is no detail for the \"Teacher classifier\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their time and efforts in reviewing our work and providing valuable feedback that can further strengthen our manuscript. Below please find our detailed responses:\n\n#### **Results on high resolution dataset.** \n\n------\n\nDue to page limits, we chose three widely used benchmark datasets for our experiments, i.e., ADE20K, COCO-Stuff-10K and PASCAL-Context. Following your comments, we have conducted several experiments on the CityScape dataset. FLOPs and Latency are calculated with the input resolution of 1024x2048 . We will add more baselines in the revised version.\n\n|                 | Params (M) | FLOPs (G) | Latency (ms) | mIoU |\n| --------------- | ---------- | --------- | ------------ | ---- |\n| SegNext-T       | 4.2        | 48.5      | 143.5        | 79.8 |\n| SegNext-T+SSA   | 4.5        | 48.6      | 145.4        | 81.2 |\n| SeaFormer-B     | 8.6        | 14.1      | 33.0         | 77.7 |\n| SeaFormer-B+SSA | 8.8        | 14.2      | 34.3         | 79.5 |\n\nAs shown in the table, SSA also has a significant improvement in mIoU on the Cityscape dataset. For example, when SeaFormer-B is used as the baseline, SSA possesses an mIoU boost of 1.8 and does not affect the efficiency of the model. This is due to the fact that SSA mainly performs convolutional operations on the prototype and is less affected by the image resolution. Therefore, the efficiency and performance is more prominent when applied to high-resolution images.\n\n#### **memory usage comparison**\n\n------\n\nMemory usage metrics have received little attention on segmentation tasks. For example, recent segmentation methods including SegViT (NeurIPS2022), CGRSeg ( ECCV2024), PEM (CVPR2024), SeaFormer (ICLR2023), and EfficientViT (ICCV2023) all do not give memory usage.  Therefore, for ease of comparison with the previous method, we do not give the memory usage. In addition, to answer your question, we have done a simple test using SeaFormer-L as the baseline, and the required memory is 4479M and 4511M respectively. This further verifies the relevant content of lines 247-250 of the paper. We will subsequently count the memory usage of the model under more baselines and add them to the revised version.\n\nIn addition we have also added the comparison of parameters as shown in table 4 of rebuttal. In conclusion, the extra parameters, FLOPs, Memory and Latency brought by SSA are all negligible.\n\n#### **Input image resolution for training and inference**\n\n------\n\nThe resolution of the input images and the experimental settings are described in lines 523-535 of the Appendix. Specifically, when training, for ADE20K and COCO-Stuff-10K, we have a cropping size of 512 × 512, while for PASCAL-Context, we have a cropping size of 480 × 480. When inference is performed, we use whole-image inference mode. We'll add it to the main paper later.\n\n\n\n#### **Detail of teacher classifier** \n\n------\n\nThe architecture of the teacher classifier is shown in Fig. 2 and details can be found in the caption of Fig. 2. Specially, the architecture of the teacher classifier is identical to the main classifier, with the difference that it takes the truth mask $M_g$ as input, while the main classifier takes the coarse mask $M_c$ as input. We will add a more detailed description for lines 198-202 of the paper in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for their thoughtful and constructive feedback. We are pleased that they recognized the novelty of the methodology, the completeness of the experiments and the excellent segmentation performance. In addition, each reviewer individually made some very valuable comments, which are conducive to the quality of the manuscript. Therefore, in addition to this global response, we have responded to each reviewer's comments separately and made the following changes to our manuscript based on the reviewers' suggestions.

- additional quantitative analyses, including comparisons of the number of parameters under various baselines and memory usage, and comparison with state-of-the-art mask classification methods.

- Additional ablation experiments, including dice loss, positional basis, optional self-distillation, and the exclusion of the hypothesis that the performance improvement is due to model size changes.

- Additional qualitative analyses, including visualization of intra-class feature distributions for more images, segmentation masks and feature visualizations for example images, and analysis of extreme offset examples.
- Other issues will be revised as per specific comments and responses.

Note that please refer to the supplementary pdf for some of the visualization and quantitative experimental results.


## Complementary Comparison Experiments

------

**Comparison with mask-level classification models.** We add some experiments with mask classification methods as shown in Table 1 of rebuttal. It should be noted that since the DPG module of CGRSeg is slightly conflicting with SSA, we remove it before adding SSA. Therefore, there is a slight decrease in the FLOPs and Latency of the model compared to the baseline. It can be observed that SSA improves CGRSeg-B and CGRSeg-L by 1.6 and 0.7, respectively. In particular, CGRSeg-L+SSA significantly outperforms recent mask classification methods such as YOSO and PEM at the same model size, with an improvement of 4.3 and 3.5, respectively. These experiment results validate that SSA exhibits a better balance of model efficiency and performance compared to masked classification methods on semantic segmentation tasks.

**FPS and parameters comparison.** As the main comparison experiment, we have provided the comparison of FLOPs, Latency (i.e., 1/fps) and mIoU in Table 1 of paper, which demonstrate the satisfactory performance and efficiency of SSA. In addition, we have further provide the comparison of parameters in Table 4 of rebuttal. There is a slight increase in the parameters of the model due to the introduction of a position basis and several 1x1 convolutional layers. In conclusion, the increase in parameters, FLOPs and latency due to SSA is negligible.

Meta-review: This paper received split recommendations after the post-rebuttal discussion. The rebuttal adequately addressed most reviewer concerns about technical correctness and experimental validation. During the author-reviewer discussion, the authors provided extensive clarifications that led two reviewers to recommend acceptance. Reviewer bEdU chose not to engage in the discussion or to comment on the specific author’s responses, leaving their rejection recommendation unchanged. Overall, the paper presents a relatively novel and effective semantic segmentation technique that would interest the broad NeurIPS community. Reviewer comments and corrections, particularly those expressed by Reviewer bEdU, should be carefully included in the final version.
Decision: Accept (poster)