ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentation
Authors: Shuyang Sun
Weijun Wang
Andrew G. Howard
Qihang Yu
Philip Torr
Liang-Chieh Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Panoptic segmentation
efficient models
TL;DR: Simple relaxations that help improve training of efficient panoptic segmentation.

Abstract: This paper presents a new mechanism to facilitate the training of mask transformers for efficient panoptic segmentation, democratizing its deployment. We observe that due to the high complexity in the training objective of panoptic segmentation, it will inevitably lead to much higher penalization on false positive. Such unbalanced loss makes the training process of the end-to-end mask-transformer based architectures difficult, especially for efficient models. In this paper, we present ReMaX that adds relaxation to mask predictions and class predictions during the training phase for panoptic segmentation. We demonstrate that via these simple relaxation techniques during training, our model can be consistently improved by a clear margin without any extra computational cost on inference. By combining our method with efficient backbones like MobileNetV3-Small, our method achieves new state-of-the-art results for efficient panoptic segmentation on COCO, ADE20K and Cityscapes. Code and pre-trained checkpoints will be available at https://github.com/google-research/deeplab2.

ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentation
Shuyang Sun''Wijuu Wang Qihang Yu' Andrew Howard Philip Torr Liang-Chieh Chen University of Oxford "Google Research
Abstract
This paper presents new mechanism to facilitate the training of mask transformers for efficient panoptic segmentation, democratizing its deployment. We observe that due the high complexity in the training objective of panoptic segmentation, it will inevitably lead to much higher penalization on false positive. Such unbalanced loss transfrranformer based architectures
makes the training process of the end-to-end difficult, especially for efficient models. In this paper, we present ReMaX that adds relaxation to mask predictions and class predictions during the training phase for panoptic segmentation. We demonstrate that via these simple relaxation techniques during training, model can be consistently improved by clear margin without any extra computational cost on inference. By combining our method with efficient backbones like MobileNerV3-Small our method achieves new state-of-the-an results for efficient panoptic segmentation on COCO, ADE20K and Cityscapes. Code and pre-trained checkpoints will be available https: //ttt/gb..tt//t///g com/ Google-reseane deeplaab..
Introduction
Panoptic segmentation [36] aims provide holistic scene understanding [63] by unifying instance segmentation [21] and semantic segmentation [24]. The comprehensive understanding of the scene is obtained by assigning each pixel label, encoding both semantic class and instance identity. Prior works adopt separate segmentation modules, specific to instance and semantic segmentation, followed by another fusion module resolve the discrepancy [71, 12, 35, 70, 53, 42]. More recently, thanks to the transformer architecture [64, 4], mask transformers [66, 13, 76, 43, 72. 14, 73] are proposed for end-to-end panoptic segmentation by directly predicting class-labeled masks.
Although the definition of panoptic segmentation only permits each pixel to be associated with just mask entity, some recent mask transformer-based methods [13, 76, 14, 40] apply sigmoid cross-entropy loss (i.e,, not enforcing a single prediction via softmax cross cross-entropy loss) for mask supervision. This allows each pixel to be associated with multiple mask predictions, leading to an extremely unbalanced loss during training. As shown in Figure 1, when using the sigmoid cross- entropy loss supervise the mask branch, the false-positive (FP) loss can be even 10 x larger than the false-negative (FN) loss. Surprisingly, such unbalanced loss leads to better results than using softmax cross-entropy, which indicates that the gradients produced by the FP loss are still helpful for better performance.
However, the radical imbalance in the losses makes it difficult for the network to produce confident predictions, especially for efficient backbones [28, 57, 27], as they tend to make more mistakes given the smaller model size. Meanwhile, the training process will also become unstable due to the large scale loss fluctuation. To address this issue, recent approaches [4, 13, 14, 40] need to carefully clip
Work done during internship at Google Research. Correspondence to: kevinsun@robote. ox.ac........................... ac.uk Work done while Google.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Softmax
Pixel
Pixel Decoder
mesic
125
Encoder
mazic
maskc
100
semy
75
dlss
50
cless
25
clocs
Jan
Xaa
Xpa
Figure 1: The histogram shows the ratio of false positives to false negatives for the cross- entropy loss, using sigmoid as the activation function, the false positive loss is always over 100x greater than the false negative, making the total loss to be extremely unbalanced.
Figure 2: The overall pipeline for mask- transformers. @ represents the matrix multipli- cation. Here Xpan and x represent the input features of the 7th stage for panoptic and seman- tic heads respectively.
a logarithmic scale. When
the gradients during training to a very small value like 0.01; otherwise, the loss would explode and the training would collapse. In this way, the convergence of the network will also be slower. A natural question thus emerges: Is there a way to keep those positive gradients, while better stabilizing the training of the network?
To deal with the aforementioned conflicts in the learning objectives, one naive solution is to apply weighted sigmoid cross entropy loss during training. However, simply applying the hand-crafted weights would cquivalently scale the losses for all data points, which means those positive and helpful gradients will be also scaled down. Therefore, in this paper, we present way that can adaptively adjust the loss weights by only adding training-ting time relaxation to mask transformers [73, 66, 13, 14, 43, 76]. In particular, propose types of relaxation: Relaxation on Masks (ReMask) and Relaxation Classes (ReClass).
The proposed ReMask is motivated by the observation that semantic segmentation is relatively easier task than panoptic segmentation, where only the predicted semantic class required for each pixel without distinguishing between multiple instances of the same class. As a result, semantic segmentation prediction could serve as a coarse-grained task and guide the semantic learning of panoptic segmentation. Specifically, instead of directly learning to predict the panoptic masks, add another auxiliary branch during training to predict the semantic segmentation outputs for the corresponding image. The panoptic prediction is then calibrated by the semantic segmentation outputs to avoid producing too many false positive predictions. In this way, the network can be penalized less by false positive losses.
The proposed ReClass is motivated by the observation that each predicted mask may potentially contain regions involving multiple classes, especially during the early training stage, although each ground-truth mask and final predicted mask should only contain one target in the mask transformer framework [66]. To account for this discrepancy, mask with a softened label, allowing the ground-trutt labels to have multiple classes. The weights of each class is determined by the overlap of each predicted mask with all ground-trute masks.
replace the original one-hot class label for each
By applying such simple techniques for relaxation to the state-of-the-art kMaX- <MaX-DeepLab [73], our method, called ReMaX, can train the network stably without any gradient-clipping operation with over 10x greater learning rate than the baseline. Experimental results have shown that our method not only speeds up the training by 3x, but also leads to much better results for panoptic segmentation. Overall, ReMaX sets new state-of-the-art record for efficient panoptic segmentation. Notably, for efficient backbones like MobileNerV3--Smal and MobileNetV3-LLarr outperform the strong baseline by 4.9 and 5.2 in PQ on COCO panoptic for short schedule training: while achieves 2.9 and 2.1 improvement in PQ for the final results (i.e., long schedules). Mcanwhile,
[27]. our method can

---

our model with a Seial-ResNet50 (MaX-S) [65] backbone outperforms all state-of-the art methods with 3x larger backbones like ConvNeXt-L [47] on Cityscapes [17]. Our model can also achieve the state-of-the-art performance when compared with the other state-of-the-ar efficient panoptic segmentation architectures like YOSO [29] and MaskConver [55] on COCO [44], ADE20K [77] and Cityscapes [17] for efficient panoptic segmentation.
2
Related Work
Mask Transformers for image segmentation. Recent advancements in image segmentation has proven that Mask Transformers [66], which predict class-labeled object masks through the Hungarian matching of predicted and ground truth masks using Transformers as task decoders [64, 4], outperform box-based methods [35, 70, 54] that decompose panoptic segmentation into multiple surrogate tasks, such predicting masks for detected object bounding boxes [23] and fusing instance and semantic segmentation [48, 10] with merging modules [42, 53, 45, 71, 12, 41]. The Mask Transformer based methods rely on converting object queries to mask embedding vectors [32, 62, 67], which are then multiplied with pixel features to generate predicted masks. Other approaches such as Segmenter [59] and MaskFormer [14] have also used mask transformers for semantic segmentation. K-Net [76] proposes dynamic kernels for generating masks. CMTT-Deepl.ab [72] suggests an additional clustering update term to improve transformer's cross-attention. Panoptic Segformer [43] enhances mask transformers with deformable attention [79]. Mask2Former [14] adopts masked-attention, along with other technical improvements such as cascaded transformer decoders [4], deformable attention [79], and uncertainty-basee point supervision [37], while
[73] employs k-means cross- attention. OneFormer [31] extends Mask2Former with a multi-task train-once design. Our work builds on top of the modern mask transformer, kMaX- -exx-eeeLLabb [73], and adopts novel relaxation methods to improve model capacity.
The proposed Relaxation on Masks (ReMask) is similar to the masked-attention in Mask2Former [14] and the k-means attention in kMaX-Deepl.ab [73] in the sense that we also apply pixel-fittrrngg operations to the predicted masks. However, our ReMask operation is fundamentally distinct from
theirs in several ways: (1) we learn the threshold used to filter pixels in panoptic mask predictions through semantic head during training, while both masked- use either hard thresholding or argmax operation on pixel-wise confidence for filtering; (2) our approach relaxes the training objective by applying pixel-wise semantic loss the semantic mask for ReMask, while they do not have explicit supervision for that purpose; and (3) we demonstrate that ReMask can complement k-means attention in Section 4.
[14] and k-means attention [73]
Acceleration for Mask Transformers for efficient panoptic segmentation. DETR [4] success- fully proves that Transformer-basee
approaches can be used as decoders for panoptic segmentation, however, still suffer from the slow training problem which requires over 300 epochs for just one go. Recent works [14, 73, 79, 50] have found that applying locality eenhant--nhanned attention mechanism can help to boost the speed of training for instance and panoptic segmentation. Meanwhile, some other works [76, 43, 33] found that by removing the bi-partite matching for stuff classes and applying separate group of mask queries for stuff classes can also help to speed up the convergence. Unlike them, which apply architectural level changes to the network, our method only applies training-time relaxation to the framework, which do not introduce any extra cost during testing. Apart from the training acceleration, recent works [26, 29, 12. 55, 51] focus on how to make the system for panoptic segmentation more efficient. However, all these works focus on the modulated architecutural design while our approach focus on the training pipeline, which should be two orthogonal directions.
Coarse-to-fine refinement for image segmentation. In the field of computer vision, it is com- mon practice learn representations from coarse to fine, particularly in image segmentation. For instance, DeepLab [6. 8] proposes a graph-based approach [38, 7] that gradually refines seg- mentation results. Recently, transformer-basee
methods for image and video segmentation such as [66, 14, 76, 69, 43, 20, 74, 78] have also adopted a multi--tage strategy to iteratively improve predicted segmentation outcomes in transformer decoders. The concept of using coarse-grained features (e.g., semantic segmentation) to adjust fine-grained predictions (e.g., instance segmentation) present certain existing works, including [11, 2, 3]. However, these approaches can lead to substantial increase in model size and number of parameters during both training and inference.
3

---

Xsem
pan
Image
Ground Truth
ReClass
Semantic head
Panoptic head
basebill glover 1/0
lanveball glaver o.8
2erront
sem
Meem
pan
stop grad
Figure 4: Demonstration on how ReClass works. We utilize the mask rendered in blue as example. Our ReClass operation aims to soften the class ground truth by considering the degree of overlap between the prediction and ground-truth mask. The blue mask intersects with both masks of "baseball glove" and "person", so the final class weights contain both and the activation of "person" in the prediction will no longer be regarded as a false positive case during training.
Lan
M pan
Figure 3: ReMask Operation. Modules, rep- resentations and operations rendered in gray are not used in testing. & and @ represent the matrix multiplication and Hadamard multipli- cation and + means clement-wise sum. The x symbol and "stop grad" mean that no gradient is flown to mgen from Lpan during training.
By contrast, our ReMaX focuses solely on utilizing the coarse-fine hierarchy for relaxation without introducing any additional parameters computational costs during inference.
Regularization and relaxation techniques. The proposed Relaxation on Classes (ReClass) in- volves adjusting label weights based on the prior knowledge of mask overlaps, which is analogous to the re-labeling strategy employed in CutMix based methods such as [75, 5, 60], as well as label smoothing [61] used in image classification. However, the problem that we are tackling is substantially different from the above label smoothing related methods in image classification. In image classifica- tion, especially for large-scale single-class image recognition benchmarks like ImageNet [56], is unavoidable for images to cover some of the content for other similar classes, and label smoothing proposed alleviate such labelling noise into the training process. However, since our approach is designed for Mask Transformers [66, 13, 14, 73, 72] for panoptic segmentation, each image is precisely labelled to pixel-level, there no such label noise in our dataset. We observe that other than the class prediction, the Mask Transformer approaches also introduce primary class identification task. The proposal of ReClass operation reduces the complexity for the classification task in Mask Transformers. Prior to the emergence of Mask Transformers, earlier approaches did not encounter this issue as they predicted class labels directly on pixels instead masks.
3 Method
Before delving into the details of our method, we briefly recap the framework of mask transform- ers [66] for end-to-end panoptic segmentation. Mask Transformers like [66, 14, 76, 69, 43] perform both semantic and instance segmentation on the entire image using a single Transformer-basee model. These approaches basically divide the entire framework into 3 parts: a backbone for feature extraction, a pixel decoder with feature pyramid that fuses the feature generated by the backbone, and transformer mask decoder that translates features from the pixel decoder into panoptic masks and their corresponding class categories.
In the transformer decoder, set of mask queries is learnt to segment the image into a set of masks by mask head and their corresponding categories by classification head. These queries are updated within each transformer decoder (typically, there are at least 6 transformer decoders) by the cross-attention mechanism [64] so that the mask and class predictions are gradually refined. The set of predictions are matched with the ground truth via bipartite matching during training; while these queries will be filtered with different thresholds as post-processing during inference. We follow the same post-prrcessing kMaX- KMaX-DeepLad DeepLab [73].

---

3.1 Relaxation on Masks (ReMask)
The proposed Relaxation on Masks (ReMask) aims to ease the training of panoptic segmentation models. Panoptic segmentation is commonly viewed more intricate task than semantic seg- mentation, since it requires the model to undertake two types of segmentation (namely, instance segmentation and semantic segmentation). In semantic segmentation, all pixels in image labeled with their respective class, without distinguishing between multiple instances (things) of the same class. As a result, semantic segmentation is regarded as more coarse-grained task when compared to panoptic segmentation. Current trend in panoptic segmentation is to model things and stuff in unified framework and resorts to train both the searse-grained segmentation task on stuff and the more fine aeeeggraind segmentation task on things together using stricter composite objective on things, which makes the model training more difficult. We thus propose ReMask to exploit an auxiliary semantic segmentation branch to facilitate the training.
Definition. Here we first define H, W as the height and width of the feature, .NQ the number of mask queries. Ne denotes the number of semantic classes for the target dataset, dy is the number of channels for the query representation, and dzan is the number of channels for the input of semantic head. As shown in Figure 2 and 3, given a mask representation Xyan E RWW x xNa, we apply a panoptic mask head to generate panoptic mask logits mpan E <HWXNo, A mask classification head to generate the corresponding classification result p E NoxNe is applied for each query representation q E R A semantic head is applied after the semantic feature Xsem E RR TgHWxdl Xd from the pixel decoder to produces a pixel-wise semantic segmentation map maen E #WWNNe assigning a class label each pixel. As for the structure for semantic head, we apply an ASPP module [9] and a 1 X 1 convolution layer afterwards to transform daen channels into No channels as the semantic prediction. Note that the whole auxiliary semantic branch will be skipped during inference as shown in Figure 3. Since the channel dimensionality between Meen and mpan is different, we map the semantic masks into the panoptic space by:
Mgan a (o
(1)
where (-) function represents the sigmoid function that normalizes the logits into interval [0, 1]. Then we can generate the relaxed panoptic outputs mpan in the semantic masking process as follows:
mpan
mpan + (meen ② mpan);
where the @ represents the Hadamard product operation. Through the ReMask operation, the false positive predictions in Myan can be suppressed by myen. so that during training each relaxed mask query can quickly focus on areas of their corresponding classes. Here we apply identity mapping to keep the original magnitude of mpan so that we can remove the semantic branch during testing. This makes ReMask as a complete relaxation technique that does not incur any overhead cost during testing. The re-scaled panoptic outputs mpan will be supervised by the losses Lpan-
Stop gradient for a simpler objective to Mzan- In order to prevent the losses designed for panoptic segmentation from affecting the parameters in the semantic head, we halt the gradient flow to Maen- as illustrated in Figure 3. This means that the semantic head is solely supervised by a semantic loss Lzen: so that it can focus the objective of semantic segmentation, which is a less complex task.
How does ReMask work? As defined above, there two factors that ReMask operation helps training, (1) the Hadamard product operation between the semantic outputs and the panoptic outputs that helps to suppress the false positive loss; and (2) the relaxation on training objectives that trains the entire network simultaneously with consistent (coarse-grained) semantic predictions. Since the semantic masking can also enhance the locality of the transformer decoder like [14, 73], we conducted experiments by replacing Mşen with ground truth semantic masks to determine whether it the training relaxation or the local enhancement that improves the training. When Myen is assigned with ground truth, there will be no Leen applied to each stage, so that Mpan is applied with the most accurate local enhancement. In this way, there are large amount of false positive predictions masked by the ground truth semantic masks, so that the false positive gradient will be greatly reduced. The results will be reported Section 4. The semantic masking can be viewed as local enhancement as would suppress the extreme false-positive predictions via simple masking operation.
5

---

52.5
54.0
Method
Backbone
Resolation
FFS
PQ
51.0
4
2x fastr
52.2
Panoptic-DeepLab [12]
Panoptic- Panopti-DDeeL.ab
MNV3-L
R50 [22]
641x641
641x641
20.0
26.3
30.0
35.1
48.0
Real-time [26]
R50 [22]
600x1333
15.9
37.1
44.6
450
MaskConver [55]
MN-MH [16]
640x640
40.2
37.2
45.
13.2
MaskFormer [14]
R50 [22)
800x
17.6
46,5
42.
3x lasver
YOSO [29]
R50 [22]
800x 1333
23.6
48.4
YOSO [29]
R50 [22]
512x800
45.6
46,4
39.0
38.0
4033
404
AMaX-Deephe [73]
R50 [22
1281x1281
16.3
53.0
36.0
3W laster
ReMaX-T
MNV3-S [27]
641x641
108.7
40.4
33.0
32.5
ReMaX-S'
MNV3-L [27]
641
80,9
44,6
ReMaX-M
R50 [22]
641
51.9
49.1
33.
Debaxd
ReMaX-B
R50 [2 1281 x 1281
16.3
54.2
10CK
Table Comparison with other sate-of-the-are ef- ficient models (2 15 FPS) on COCO val set. The Pareto curve is shown in Figure 6 (b). The FPS of all models are evaluated on a NVIDIA V100 GPU with batch size 1. 14 represent the application of efficient pixel and transformer decoders. Please check the appendix for details.
Figure 5: Performance on COCO val compared the baseline XMaX-DeepLan [73]. ReMaX can lead to 3x faster convergence compared to the baseline, and can improve the baselines by clear margin. The performance of ResNet-50 can be further improved to 54.2 PQ when the model is trained for 200K iterations.
3.2 Relaxation on Classes (ReClass)
Mask Transformers [66, 14, 73, 43] operate under the assumption that each mask prediction corre- sponds to a single class, and therefore, the ground truth for the classification head are one-hot vectors. However, in practice, each imperfect mask predicted by the model during the training process may intersect with multiple ground truth masks, especially during the early stage of training. As shown in Figure 4, the blue mask, which is the mask prediction, actually covers two classes ("baseball glove" and "person") defined in the ground truth. the class-wise ground truth only contains the class "baseball glove", the prediction for "person" will be regarded false positive case. However, the existence of features of other entities would bring over verr-ennalization that makes the network predictions to be under-confident.
To resolve the above problem, we introduce another relaxation strategy on class logits, namely Class- wise Relaxation (ReClass), that re-assigns the class confidence for the label of each predicted mask according to the overlap between the predicted and ground truth semantic masks. We denote the one- hot class labels as y, the ground truth binary semantic masks as S = [So...,.Sww' sHw] E {0, MW XNe. the supplement class weights is calculated by:
(mmpn) TS ym
(3)
Σ HW
S;
where ym denotes the label weighted by the normalized intersections between the predicted and the ground truth masks. With ym. we further define the final class weight ý E [0, 1|.V follows:
y nym + (1 - mymmy,
(4)
where the denotes the smooth factor for ReClass that controls the degree of the relaxation applying to the classification head.
4 Experimental Results
4.1 Datasets and Evaluation Metric
Our study of ReMaX involves analyzing its performance on three commonly used image segmentation
datasets. COCO [44] supports semantic, instance, and panoptic segmentation with 80 "things" and 53 "stuff" categories; Cityscapes [17] consists of "things" and 11 "stuff" categories; and ADE20K [77] contains 100 "things" and 50 "stuff" categories. We evaluate our method using the Panoptic Quality (PQ) metric defined in [36] (for panoptic segmentation), the Average Precision defined in [44] (for instance segmentation), and the mloU [19] metric (for semantic segmentation).

---

PQ
COCO
PQ
Cityscapes
54.0
69.0
88.4
65.7
$1,0
530
66.29
66.2
49.1
48.0
45.0
465
484
46.4
51.6
63.0
62.1
64.3
62.5
19 /39./
42.9
42.59
40.4
60.0
39.0
570
58.8 8
57.7
33.0
36,0
3/.1
35.1
ManiFormet
RaMaX
KAXX Coeplab 37.1
Y060
54.0
HeMox
YCSC
Mox Deeptab
+56.7
Paroptic Geeplit
MaskConver
Reakine
PShas
52.5
30,0
Reuitane
51.0
HaskaForm
10
25
40
55
70
85
100
FPS
0
5
10
15
20 25 TPS
(a)
(b)
Figure 6: FPS-PQ Pareto curve on (a) COCO Panoptic val set and (b) Cityscapes val set. Details
of the corresponding data points can be found in Table and 9. We compare method with
other sfate-of-the-aa the-art efficient pipelines for panoptic segmentation including kMaX- -Mee-eDepLab [73],
Mask2Former [14], YOSO [29], -eeepaa [12], Real-time Panoptic Segmentation [26],
UPSNet [70], LPSNet [25], MaskFormer [13], and MaskConver [55].
4.2 Results on COCO Panoptic
Implementation details. The macro-architecture of ReMaX basically follows kMaX-
DeepLab [73], while we incorporate our modules introduced in Section 3 into the corresponding heads. Concretely, we use the key in each k-means cross-attention operation Xzen defined in Figure 3. The semantic head introduced during training consists of an ASPP module [8] and a x 1 convolution that outputs Ne number of channels. The specification of models with different size is introduced in the appendix. For other details like post-processing and data preparation, we strictly follow kMaX-Deepl.ab [73].
Training details. We basically follow the training recipe proposed in kMaX- DeepLab [73] but make some changes to the hyper-parameters since we add more relaxation to the network. Here we high-light the necessary and the full training details and specification of our models can be also found in the appendix. The learning rate for the ImageNet-pretranne [56] backbone is multiplied with smaller learning rate factor 0.1. For training augmentations, we adopt multi-scale training by randomly scaling the input images with a scaling ratio from 0.3 to 1.7 and then cropping it into resolution 1281 x 1281. Following [66, 72, 73], we further apply random color jittering [18], and panoptic copy-paste augmentation [33, 58] to train the network. DropPath [30, 39] is applied to the backbone, the transformer decoder. Adam W [34, 49] optimizer is used with weight decay 0.005 for short schedule 50K and 100K with a batch size 64. For long schedule, we set the weight decay to 0.02. The initial learning rate is set to 0.006, which is multiplied by a decay factor of 0.1 when the training reaches 85% and 95% of the total iterations. The entire framework is implemented with Dee [68] in TensorFlow [1]. Following [66], we apply PQ-style loss, a Mask-ID cross-entropy loss, and the instance discrimination loss to better learn the feature extracted from the backhone.
For all experiments if not specified, we default to use ResNet-50 as the backbone and apply ReMask to the first 4 stages of transformer decoder. The 77 for ReClass operation is set to 0.1. All models are trained for 27 epochs (i.e., 50K iterations). The loss weight for the auxiliary semantic loss Zzem applied to each stage in the transformer decoder is set to 0.5 and the weights for those loss terms in Epan are set the same as <Max-Deeplable
ReMaX significantly improves the training convergence and outperforms the baseline by large margin. As shown in Figure 5, we can see that when training the model under different training schedules 50K, 100K and 150K, our method outperform the baselines by clear margin for all different schedules. Concretely, ReMaX can outperform the state-of-the-an baseline kMaX- DeepLab by significant 3.6 PQ when trained under a short-term schedule 50K iterations (27 epochs) for backbone ResNet-50. Notably, our model trained with only 50K iterations performs even better than kMaX-Deepl ab [73] trained for the 100K iterations (54 epochs), which means that our model can speed up the training process by approximately 2x. We kindly note that the performance of ResNet-50 can be further improved to 54.2 PQ for 200K iterations. ReMaX works very well with
7

---

w/ grad-
Activation
ReMaX?
clip?
PQ
#ReMasks
2
4
6
1j
0.01 0.05 0.1 0.2
softmax
48.8
PQ
50.4 51.9
52.4
51.5
PQ 51.7 51.7 51.9 52.4 51.5 Table 4: The impact of dif- fernt nj defined in Eq. for ReClass. Here we observe that the result reaches its peak when 0.1.
sef
49.5
si sigmoid
50.4
Table 3: The effect of number of
sigmoid
51.2
ReMask applied. ReMaX per-
sigmoid
52.4
forms the best when ReMask is ap-
Table 2: The impact of ac- tivation function and gradi- ent clipping.
plied the first 4 stages of the trans-
former decoder.
w/ identity w/ ReMask mapping?
Method
Backbone
FPS PQ
test?
PQ
MaskI [13]
17.6 46.5
w/ stop- grad? w/ gt?
PQ
x
52.4
K-Net [76]
PanSerFormer [43]
7.8 49.6
47.1
x
52.4
52.4
Mask2Former [14]
R50 [22]
8.6 51.9
N/A
<
45.1
52.1
51.9
MaskDINO [40]
KMaX- Ma [73]
16.8 53.0
26.3
53.0
X
x
36.6*
Table 7: The effect of stop gradient and gt-masking.
Table 5: Effect of applying identity mapping and auxil- lary head for ReMask dur- ing testing. Removing the auxiliary semantic head will not lead to performance drop when mpan is applied with identity mapping.
RcMaX
26.31 54.2
Table 6: Comparison on COCO
The denotation w/ gt? means whether we use ground-truth semantic masks for mgea- The result without the stop-gradien operation does not well converge in training.
val with other models using
ResNet-50 as the backbone. 'The FPS here is evaluated under resolu- tion 1200 x 800 on V100 and the model is trained for 200K iterations. is evaluated using a A100 GPU.
efficient backbones including MobileNerV3-Smal [27] and MobileNerV3-Large [27], which surpass the baseline performance by 4.9 and 5.2 PQ for 50K iterations, and 3.3 and 2.5 PQ respectively for 150K iterations. These results demonstrate that the proposed relaxation can significantly boost the convergence speed, yet can lead to better results when the network is trained under longer schedule.
ReMaX vs. other sfate-of-the-art models for eflicient panoptic segmentation. Table and Figure 6 (a) compares our method with other state-of-the-art methods for efficient panoptic segmentation on COCO Panoptic. We present models with different resolution and model capacity, namely ReMaX-Tiny (T), ReMaa--Saaaa (S), ReMaX-
(M) and ReMaX-Bass Base (B). Due to the limit of space, the detailed specification of these models included in the appendix. According to the Pareto curve shown in Figure 6 (a), our approach outperforms the previous state-of-the-art efficient models by clear margin. Specifically, on COCO Panoptic val set, our models achieve 40.4, 44.6, 6, 49.1 and 54.2 PQ with 109, 81, 52 and 16 FPS for ReMaX-T, ReMaX-S, ReMaX-M MM and ReMaX-B B respectively. The speed of these models evaluated under the resolution 641 x 641 except for ReMaa-Base, which is evaluated under resolution 1281 1281. Meanwhile, shown in Table 6, our largest model with the backbone ResNet-50 also achieves better performance than the other non-efficient state-of-the-an methods with the same backbone.
Effect of different activation, and the use of gradient clipping. Table presents the effect of using different activation function (si (sigmoid d vs. sof tmax) for the Mask-ID cross-entropy loss and the (-) defined in Eq (1). From the table we observe that ReMask performs better when using si sigmoid as the activation function, but our method can get rid of gradient clipping and still get better result.
Can we use the ground-truth masks for local enhancement instead of ReMask? As discussed in Section 3, to figure out whether it is the loss relaxation or the pixel filtering that improves the training, we propose experiments to replace mşen with the ground truth semantic masks during training. When Myen is changed into the ground truth, all positive predictions outside the ground-trutt masks will be removed, which means that the false positive loss would be significantly scaled down. The huge drop (52.4 vs. 45.1 PQ in Table 7) indicates that the gradients from false positive losses can benefit the final performance. Table 7 also shows that when enabling the gradient flow from the panoptic loss to the semantic predictions, the whole framework cannot converge well and lead to drastic drop in performance (36.6 PQ). The semantic masks Mşan faces a simpler objective (i.e. only semantic segmentation) if the gradient flow is halted.
8

---

w/semantic masking
w/ Leem
w/ ReClass
Iterations
PQ
(local enhancement) (loss relaxation)
?
X
X
x
50K
50.4
150K
53.0
X
X
50K
51.3
X
X
150K
53.0
50K
51.7
50K
52.4
150K
54.0
Table 8: The relative impact of loss relaxation and semantic masking (local enhancement) on COCO Panoptic val set under short (50K) and long (150K) training schedule.
The relative contribution of loss relaxation and local enhancement in ReMask. The auxiliary semantic loss term can be viewed as loss relaxation, while the semantic masking branch can be viewed as local enhancement. To disentangle the relative contribution of loss relaxation and local enhancement, we conducted another ablation study that removes the semantic masking branch (the concrete grey arrow right under "stop grad" in Figure 3), which would remove the local enhancement (semantic masking) but keep the auxiliary semantic loss term for loss relaxation. The results are reported in Table relaxation leads to 0.9 increase in PQ: while the semantic masking contributes to an additional 0.4 gain in PQ. The long-schedute training (i.e. 150K iterations) demonstrates that semantic masking is critical in ReMask because applying semantic loss relaxation alone without semantic masking does not result in any improvement. In other words, only using semantic loss relaxation may expedite the early stage of training (e.g, 50K iterations), but it fails to improve the ultimate convergence quality.
The short training schedule of 50K iterations shows that the semantic loss
The number of mask relaxation. Table 3 shows the effect of the number of ReMask applied to each stage, from which we can observe that the performance gradually increases and reaches its peak 52.4 PQ when the number of ReMask is 4, which is also our final setting for all other ablation studies. Using too many ReMask (> 4) operations in the network may add too many relaxation to the framework, that it cannot fit well the final complex goal for panoptic segmentation.
ReClass can also help improve the performance for ReMaX. We investigate ReClass and its apper-paramete 7) in this part and report the results in Table 4. In Table 4, we ablate 5 different from 0 to 0.2 and find that ReClass performs the best when 1) = 0.1, leading to a 0.5 gain compared to the strong baseline. The efficacy of ReClass validates our assumption that each mask may cover regions of multiple classes.
Effect of the removing auxiliary semantic head for ReMask during testing. The ReMask op- eration can be both applied and removed during testing. In Table 5, shows that the accuracy comparable under the two settings. In Table 5 we also show the necessity of applying identity mapping to Mpan during training in order to remove the auxiliary semantic head during testing. Without the identity mapping training, removing semantic head during testing would lead 0.5 drop from 52.4 (the first row in Table 5) to 51.9.
4.3 Results on Cityscapes
Implementation details. Our models are trained using batch size of 32 on 32 TPUv3 cores, with a total of 60K iterations. The first 5K iterations constitute the warm-up stage, where the learning rate gradually increases from 0 to 3 10-3 During training, the input images are padded to 1025 x 2049 pixels. In addition, we employ multi-task loss function that includes four loss components with different weights. Specifically, the weights for the PQ-style loss (part of Cpan), auxiliary semantic loss Leen, mask-id cross-entrop loss (part of Epan). and instance discrimination loss are set to 3.0, 1.0. 0.3 and 1.0, respectively. To generate feature representations for our model, we use 256 cluster centers and incorporate an extra bottleneck block in the pixel decoder, which produces features with an output stride of These design are basically proposed in kMaX-DeepLab [73] and we simply follow here for fair comparison.
Results on Cityscapes. As shown in Table 9 and Figure (b), it shows that our method can achieve even better performance when using smaller backbone MobileNeeV3--arrg (62.5 PQ) while the

---

Method
Backbone
FPS PQ
Method
Backbone
FPS #params
PQ
Fask2Former [14]
R50 [22]
4.1 62.1
Former [73]
Swin-L [46]
216M
66.6
Panoptic- DeeplLab [12]
Xception-71 [15]
5.7 63.0
kMuX-DeepLab [73]
MaX-S [66]
6.5
74M
66.4
LPSNet [25]
R50 [22]
7.7 59.7
RMaX-DeepLab [73]
ConwNeXi-L" [47]
3.1 232M
68.4
Dempptic [12]
R50 [22]
8.5 59.7
OneFormer [31]
ConvNeXt-1.7 [47]
220M
68.5
kMaX-DeepLab [73]
R50 [22]
9.0 64.3
ReMaX
MaX-S
[27]
6.5
74M
68.7
Real-time [26]
R50 [22]
10.1 58.8
Table 10: Cityscapes val set results for larger
YOSO [29]
R50 [22]
11.1 59.7
backbones. PPre-trained ImageNet-22k.
AMaX-DeepLad [73]
MNV3-L [27]
22.8 60.2
ReMaX
R50 [22]
9.0 65.4
Method
Backbone Resolution
FPS
PQ mloU
ReMaX
MNV3-L [27]
22.8 62.5
MaskFormer [13]
640-2560
34.7
ReMaX
MNV3-S [27]
25.6 57.7
Mask2Tormer [14]
640-2560
39.7
Table Cityscapes val set results for lightweight backbones. We consider methods without pre- training on extra data like COCO [44] and Map- illary Vistas [52] and test-time augmentation for fair comparison. We evaluate our FPS with resolu- tion 1025 x 2049 and V100 GPU. The FPS for other methods are evaluated using the resolution reported in their original papers.
YOSO [29]
R50 [22)
640-2560
35.4.8.0
kMaX- DeepLab [73]
641x641
38.7 45.0
kMaX- Daeppa [73]
x281x
14.4/2.3
ReMaX
641
41.9 45.7
R50 [22]
ReMuX
1281x1281
14.4 46.9
Table I1: ADE20K val set results. Our FPS is
evaluated on NVIDIA V100 GPU under the corresponding resolution reported in the table.
other methods are based on ResNet-50. Meanwhile, our model with Axial-ResNet-50 (i.e., MaX- S. 74M parameters) as the backbone can outperform the state-of-the-art models [31, 73] with a ConvNeXt-L backbone 220M parameters). The Pareto curve in Figure 6 (b) clearly demonstrates the efficacy of our method in terms of speed-accuracy trade-off.
4.4 Results on ADE20K
Implementation details. We basically follow the same experimental setup as the COCO dataset, with the exception that train our model for 100K iterations (54 epochs). In addition, conduct experiments using input resolutions of 1281 x 1281 pixels and 641 x 641 respectively. During inference, we process the entire input image as whole and resize longer side to target size then pad the shorter side. Previous approaches use a sliding window approach, which may require more computational resources, but expected yield better performance in terms of accuracy and detection quality. As for the hyper-parameter for ReMask and ReClass, we used the same setting as what we propose on COCO.
Results on ADE20K. In Table 11, compared the performance of ReMaX with other methods, using Re as the backbone, and found that our model outperforms the baseline model by 1.6 in terms of mIOU, which is clear margin compared the baseline, since do not require any additional computational cost but only the relaxation during training. We also find that our model surpass the baseline model kMaX-Deepl.al by 1.1 in terms of PQ. When comparing with other frameworks that also incorporate ResNet-50 as the backbone, we show that our model is significantly better than Mask2Former and MaskFormer by 3.7 and 8.7 PQ respectively.
Conclusion
This paper presents a novel approach called ReMaX, comprising two components, ReMask and ReClass, that leads better training for panoptic segmentation with Mask Transformers. The proposed method is shown to have significant impact on training speed and final performance, especially for efficient models. In principle, ReMaX has the potential to be generalized other non- transformer-basee panoptic segmentation frameworks as long as it has a panoptic mask representation and a semantic mask representation. In this paper, we mainly verify our method based on state-of- the-art mask transformers [73]. We will further validate the generalization capability of our method in future work. We hope that our work will inspire further investigation in this direction, leading to more efficient and accurate panoptic segmentation models.
10

---

Acknowledgement. We would like to thank Yukun Zhu, Xuan Yang at Google Research for their kind help and discussion. Shuyang Sun and Philip Torr are supported by the UKRI grant: Turing Al Fellowship EP/W002981// 1/1 and EPSRC/MURI grant: EP/N019474/1. We would also like to thank the Royal Academy of Engineering and FiveAl.

---

References
[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Lovenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, 2016. 7
[2] Anurag Amab and Philip HS Torr. Bottom-up instance segmentation using deep higher-order crfs. In BMVC, 2016. 3
[3] Anurag Amah, Sadeep Jayasumana, Shuai Zheng, and Philip HS Torr. Higher order conditional random fields in deep neural networks. In ECCV, 2016. 3
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 1,3
[5] Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. CVPR, 2022.
[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille, Semantic image segmentation with deep convolutional nets and fully connected erfs. ICLR, 2015.
[7] Liang- Chich Chen, Alexander Schwing,, Alan Yuille, and Raquel Urtasun. Learning deep structured models. In ICML., 2015. 3
[8] Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE TPAMI, 2017. 3,7
[9] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. 17X.v:/706.05587 2017.
[10] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decode with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 3
[11] Bowen Cheng, Hlang-Chich Chen, Yunchao Wei, Yukun Zhu, Zilong Huang, Jinjun Xiong, Thomas S Huang, Wen-Mei Hwu, and Honghui Shi. Spgnet: Semantic prediction guidanee for seene parsing. In ICCV, 2019. 3
[12] Bowen Cheng, Maxwell D Collins, Yukun Zhu. Ting Liu. Thomas S Huang, Hartwig Adam, and Liang- Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmenta-
Chieh Chen. Panoptic-Deeple 7, 10 tion. CVPR, 2020.
[13] Bowen Cheng, Alexander G Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurlPS, 2021. 1, 7, 10
[14] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 1, 1, 3, 4, 5, 6, 7, 8, 10, 16,
[15] François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 10
[16] Grace Chu, Okan Arikan, Gabriel Bender, Weijun Wang, Achille Brighton, Pieter-Jan Kindermans, Hanxiao Liu, Berkin Akin, Suyog Gupta, and Andrew Howard. Discovering multi hardw-hardwwre mobile models via architecture search. In CVPR workshop. 2021.
[17] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 3,6
[18] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Lc. Autoaugment: Learning augmentation policies from data. In CVPR, 2019. 7
[19] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88:303-338, 2010. 6
[20] Xiuye Gu, Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang. Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo, Huizhong Chen, Liang- Chich Chen, et al. Dataseg: Taming a universal multi-dataset multi-tas task segmentation model. NewrIPS, 2023.
12

---

[21] Bharath Hariharan. Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV, 2014.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6, 8, 10, 16
[23] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 3
[24] Xuming He, Richard Zemel, and Miguel Á Carreira-Perpiñann Multiscale conditional random fields for image labeling. In CVPR, 2004.
[25] Weixiang Hong, Qingpei Guo, Wei Zhang, Jingdong Chen, and Wei Chu. Lpsnet: lightweight solution for fast panoptic segmentation. In CVPR, 2021. 7,
[26] Rui Hou, Jie Li, Arjun Bhargava, Allan Raventos, Vitor Guizilini, Chao Fang. Jerome Lynch, Adrien Gaidon. Real-time time panoptic segmentation from dense detections. In CVPR, 2020. 3, 6, 7, 10
[27] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019. 1, 2, 6, 8,
10, 16
[28] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. wwXiv:/704.0400 2017.
[29] Jie Hu, Linyan Huang, Tianhe Ren, Shengchuan Zhang. Rongrong Ji, and Liujuan Cao. You only segment once: Towards real-time panoptic segmentation. In CVPR, 2023. 3, 6, 7, 10
[30] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In ECCV, 2016.7
[31] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. CVPR, 2023. 3,
[32] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. NeurIPS, 2016.
[33] Dahun Kim, Jun Xie, Huiyu Wang, Siyuan Qiao, Qihang Yu, Hong-Scok Kim, Hartwig Adam, In So Kweon, and Liang-Chiel Chieh Chen. SeaeFoomer--eeep... Video Mask Transformer. CVPR, 2022. 3,
[34] Diederik Kingma and Jimmy Ba. Adam: method stochastic optimization. ICLR, 2015.
[35] Alexander Kirillov. Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019. 3.3
[36] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019. 1,6
[37] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR, 2020.
[38] Philipp Krähenbühl and Vladlen Koltun. Efficient inference fully connected erfs with gaussian edge potentials. NeurlPS, 2011.
[39] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint 1rXiv:1605.00666 2016.
[40] Feng Li, Hao Zhang, Shilong Liu, Lei Zhang. Lionel Ni, Heung Yeung Shum, et al. Mask dino: Towards unified transfommer-asedd framework for object detection and segmentation. In CVPR, 2023, 1, 8
[41] Qizhu Li, Xiaojuan Qi, and Philip Torr. Unifying training and inference for panoptic segmentation. In CVPR, 2020.
[42] Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, and Xingang Wang. Attention- guided unified network for panoptic segmentation. In CVPR, 2019. 1,3
[43] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Tong Lu, and Ping Luo. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In CVPR, 2022.
1, 2, 3, 4, 6,
13

---

[44] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 3, 6, 10
[45] Huanyu Liu, Chao Peng, Changqian Yu, Jingho Wang, Xu Liu, Gang Yu, and Wei Jiang. An end-to-end network for panoptic segmentation. In CVPR, 2019. 3
[46] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 10, 16
[47] Zhuang Liu, Hanzi Mao. Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In CVPR, 2022. 3, 10
[48] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. CVPR, 2015.
[49] Tyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[50] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Hougiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr fast training convergence. In ICCV, 2021. 3
[51] Rohit Mohan and Abhinav Valada. Efficientps: Efficient panoptic segmentation. IJC,, 129(5):1551-1557, 2021. 3
[52] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017. 10
[53] Lorenzo Porzi, Samuel Rota Bulò, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmenta- tion. In CVPR, 2019. 1,3
[54] Siyuan Qiao, Liang-Chicl Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. In CVPR, 2021. 3
[55] Abdullah Rashwan, Yeqing Li, Xingyi Zhou, Jiageng Zhang, and Fan Yang. Maskconver: A universal panoptic and semantic segmentation model with pure convolutions. OpenReview, 2023. 3, 6,7
[56] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Audrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 115:211 252, 2015, 4,7
[57] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018
[58] Inkyu Shin, Dahun Kim, Qihang Yu, Jun Xie, Hong-Seok Kim, Bradley Green, In So Kweon, Kuk-Jin Yoon, and Liang-Chiel Chen. Video-kmax: A simple unified approach for online and near-online video panoptic segmentation. In WACV. 2024.
[59] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
[60] Shuyang Sun, Jie-Neng Chen, Ruifei He, Alan Yuille, Philip Torr, and Song Bai. Lumix: Improving mixup by better modelling label uncertainty, arXiv preprint arXiv:2211.15846. 2022,
[61] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
[62] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV, 2020. 3
[63] Zhuowen Tu, Xiangrong Chen, Alan L Yuille, and Song-Chun Chun Zhu. Image parsing: Unifying segmentation, detection, and recognition. IJC,, 63:1 13-140, 2005.
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017. 1,
[65] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang- Chich Chen. Axial- DeepLab: Alone-Alone Axial-Attention for Panoptic Segmentation. In ECCV, 2020. 3, 16
[66] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang Chich Chen, Max Max-deeplab; End-to-end panoptic segmentation with mask transformers. In CVPR, 2021. 1, 2, 4, 6, 7, 10, 16
14

---

[67] Xinlong Wang, Rufeng Zhang. Tao Kong, Lei Li, and Chunhua Shen. SOLOv2: Dynamic and fast instance segmentation. NeurlPS, 2020.
[68] Mark Weber, Huiyu Wang, Siyuan Qiao, Jun Xie, Maxwell D. Collins, Yukun Zhu, Liangzhe Yuan, Dahun Kim. Qihang Yu, Daniel Cremers, Laura Leal-Taixe, Alan L. Yuille, Florian Schroff. Hartwig Adam, and Liang- Ching Chen. DeepLab2; A TensorFlow Library for Deep Labeling. arXiv: 2106.09748, 2021.
[69] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 2021. 3,4
[70] Yuwen Xiong. Renjie Liao, Hengshuang Zhao, Rui Hu. Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: unified panoptic segmentation network. CVPR, 2019. 1, 3,3,7 7
[71] Tien-Ju Yang, Maxwell D Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang. Vivienne Sze, George Papandreou, and Liang-Chiin Chich Chen. Deeperlab: Single-shot image parser. arXiv:1902.05093; 2019. 3,3
[72] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022. 1, 3,
[73] Qihang Yu. Huiyu Wang. Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. In ECCV, 2022. 2, 3, 4, 5, 6, 7, 8, 9, 10. 16
[74] Qihang Yu, Ju Hc, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. NeurIPS, 2023. 3
[75] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[76] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NeurlPS, 2021. 1, 3,
[77] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 3,
[78] Yi Zhou, Hui Zhang, Hana Lee, Shuyang Sun, Pingjum Li, Yangguang Zhu, ByungIn Yoo, Xiaojuan Qi, and Jae-Joon Han. Slot-vps: Object-centric representation learning for video panoptic segmentation. In CVPR, 2022.
[79] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-ena object detection. In ICLR, 2021. 3
15

---

Appendix
A Loss Visualization of ReMaX
Spenzid Nataes, at5-00500
Method
Backhone
#Params
FLOPs FPS
PQ
KMaX KMaX-DeepLab [73]
ConvNeXi-TT [66]
6IM
172G
21.8
Ouur
ReMaX
Con-NeXXi-1 [66]
Mask2Former [14]
Swin-B [46]
107M
406G
56.4
75-
kMaX Deepl [73]
Con'NeXXiS St [66]
83M
251G
16.5
56.3
50
ReMaX
ConyNeXt-S [66]
83M
251G
16.5
S6.6
25
Table 12: Results for larger models on COCO val set. FLOPs and FPS evaluated with the input size 1200 x 800 and a V100 GPU. f: ImageNet-22K pretraining.
1.75
2.00
2.25
2.50
2.75
3.03
3.25
3.10
FP Fh/ loos. log scale
Figure 7: The histogram shows the ratio of false positives to false negatives for the cross-entropy loss, logarithmic scale.
We visualize the loss applied with ReMask and the loss applied without ReMask in Figure 7, from which observe that ReMask effectively reduce extremely high false positive losses; therefore, our method can stabilize the training of the framework.
B
Model Specification
#Pixel
#Transformer Decoders
Model
Backbone
Resolution
#FLOPs #Params FPS
Decoders
ReMaX-T T
MNV3-S [27]
641 x 641 1,
[1,
18.8G
18.6M
109
ReMaX-S
MNV3-L [27]
641 x 641
1,
1,
20.9G
22.0M
ReMaX-M
R50 [22]
641 x 611
1,
[,
67.8G
50.8M
52
ReMaX-B
R50 [22]
1281 x 1281
1,
[2,
294.7G
56.6M
26
Table 13: Specification of different models in ReMaX family.
We provide the specification of our models and their corresponding number parameters and FLOPs in Table 13. We kindly note that the numbers of pixel decoders with the format ( represent the
numbers for features with 32 16 8 11 times of the input size. We use Axial attention [65] for all feature maps with resolution 32 16 the rest. The denotation -, for the transformer decoders represents the numbers for resolution of To 1] times the input size.
1 of the input size, and regular bottleneck residual blocks [22] for
C Performance for Larger Models
We also validate the performance of ReMaX for larger models e.g. ConyNeXt-Tiny (T) and ConvNeXt- Small (S). From Table 12 we can find that ReMaX can achieve better results compared to the baseline KMaX-DeepLab [73] and Mask2Former [14]. However, the improvement ReMaX gets saturated when the numbers become high. Notably, when using ConvNeXt-T backbone, ReMaX can lead to 0.6 PQ increase over kMaX kMaX-DeepLate La,, while incurring no extra computational cost during inference. The improvement is noticeable, kMaX- MaX-DeepLab only further improves 1.0 PQ by using ConvNeXt-S backbone, at the cost of extra 36% more parameters (22M) and 46% more FLOPs (79G).
D Limitations
Since we implement our method in TensorFlow, the baselines we can build upon limited. We have re-implemented ReMaX in PyTorch and applied it with Mask2former [14]. The result is
16

---

Methods
Epochs
PQ
Mask2former
24
48.36
Mask2former + ReMaX
24
50.24
Table 14: ReMaX is effective on Mask2Former [14] on COCO Panoptic val set.
reported in table 14. Due to the time limit, we did not reproduce the originally reported Mask2Former results by fully exploring all the hyper-parameters the same Mask2Former baseline ReMaX boost the overall accuracy. This could demonstrate that ReMaX also effective for other segmentation frameworks like Mask2former. need to validate ReMaX based on more frameworks such as YOSO in PyTorch to demonstrate its effectiveness. Meanwhile, ReClass measures the weight of each class according to the size of each mask, which may not be accurate and can be further improved in the future.
However, the table above shows that based on
future work, we
E
Boarder Impact
Our method can help better train models for efficient panoptic segmentation. It can also be used to develop new applications in areas such autonomous driving, robotics, and augmented reality. For example, in autonomous driving, efficient panoptic segmentation can be used to identify and track other vehicles, pedestrians, and obstacles the road. This information can be used to help the navigate safely. In robotics, efficient panoptic segmentation can be used to help robots understand their surroundings and avoid obstacles. This information can be used to help robots perform tasks such as picking and placing objects or navigating through cluttered environments. In augmented reality, efficient panoptic segmentation can be used overlay digital information on top of the real world. This information can be used to provide users with information about their surroundings or to help them with tasks such as finding their way around new city. Overall, our method can be used boost a variety of applications in the field of computer vision and robotics.
17


Reviews:
[{"review_id": "cYLEY4URMq", "reviewer": "Reviewer_5TWL", "summary": "This paper introduces a novel mechanism called ReMaX to enhance the training of mask transformers for efficient panoptic segmentation, making it more accessible and practical. The authors observe that the high complexity of panoptic segmentation training objectives often results in imbalanced loss, leading to challenges in training end-to-end mask-transformer based architectures, particularly for efficient models.\nIn response to this challenge, the authors propose ReMaX, which incorporates relaxation techniques for mask predictions and class predictions during training for panoptic segmentation. Through these simple relaxation strategies, the model consistently improves its performance without incurring any additional computational cost during inference.\nThe effectiveness of ReMaX is demonstrated by integrating it into efficient backbones like MobileNetV3-Small. The proposed method achieves a new state-of-the-art record for efficient panoptic segmentation on benchmark datasets such as COCO, ADE20K, and Cityscapes.\nThe results showcase the significance of ReMaX in improving the performance of mask transformers and its potential for advancing the field of efficient panoptic segmentation.", "questions": "I'm positive about this paper. However I still have some concerns in the Weaknesses. I'll make the final decision after I see the response from the authors for those questions in Weaknesses.", "limitations": "I cannot find any limitations or potential negative societal impact which the authors list.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "By applying such simple techniques for relaxation to the state-of-the-art kMaX-DeepLab, ReMaX can train the network stably without any gradient-clipping operation under a learning rate that is over 10× greater than the baseline. Experimental results have show that the proposed method both boosts the training speed by 3×, and also leads to much better results for panoptic segmentation. ReMaX sets a new state-of-the-art record for efficient panoptic segmentation.", "weaknesses": "(1) Do you have the experimental results on test set for COCO, CityScapes and ADE20K?\n(2) How do you balance Lpan and Lose? Are there any weights for these two losses?\n(3) In Table 2, what is the result of using softmax as activation and with grad-clip?\n(4) In Table 5, I'm wondering why removing the auxiliary semantic head will not lead to performance drop when using identity mapping.\n(5) In Table 7, why does PQ drop a lot when you use ground-truth semantic masks for m_sem? What is the result of using ground-truth semantic masks with the stop-gradient operation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback. We hope that the subsequent response will clarify the issues highlighted and contribute to an improved rating.\n\n*Q1. COCO/CityScapes/ADE20K test set*\n\n   Thank you for the suggestion. We intend to include test set results for these datasets in the finalized version. Regrettably, due to the shortage of training resources (GPUs) and time constraints, we are unable to present these results in the current rebuttal.\n\n---\n\n*Q2. loss weights on $\\mathcal{L}\\_{pan}$ and $\\mathcal{L}\\_{sem}$.*\n   \n   We have mentioned it in L246-247, and L307-308, respectively. We kindly note that the panoptic losses $\\mathcal{L}_{pan}$ consist of multiple losses including PQ-style losses and mask-id loss.\n   We suppose the current inconsistency in loss names may lead to confusion for the audience to read, therefore, we would revise L246-247 to:\n\n   > The loss weight for $\\mathcal{L}\\_{sem}$ is 0.5 and that for $\\mathcal{L}\\_{pan}$ is set as the same with kMaX-DeepLab [64].\n\n   Then we would revise L307-308 to:\n\n   > The weights for the PQ-style loss (part of $\\mathcal{L}\\_{pan}$), auxiliary semantic loss ($\\mathcal{L}_{sem}$), mask-id cross-entropy loss (part of $\\mathcal{L}\\_{pan}$), and instance discrimination loss are set to 3.0, 1.0, 0.3 and 1.0.\n\n---   \n\n*Q3. In Table 2, what is the result of using softmax as activation and with grad-clip?*\n\n   Softmax tends to produce fewer False Positives than Sigmoid because each pixel is limited to a single positive prediction, as outlined in lines L24-26 of the manuscript.\n   By contrast, Sigmoid permits each pixel to correlate with several mask predictions, which can result in a highly unbalanced loss during training.\n\n   When applying Softmax with grad-clip, the network converged too slowly (10+ points lower than the baseline) because of insufficient positive gradients; so we did not report it in Table 2.\n\n---\n\n*Q4.  In Table 5, I'm wondering why removing the auxiliary semantic head will not lead to a performance drop when using identity mapping.*\n\n   We employed identity mapping to preserve the initial prediction, with the semantic branch serving solely as a relaxation mechanism. The overarching goal for panoptic segmentation remains unaltered. \n   It's important to highlight that only the **`first four stages`** utilize ReMask, not all mask decoders. Eliminating the semantic branch might influence the intermediate predictions, but it won't have a direct impact on the final prediction. \n   This may suggest that variations in intermediate predictions might not substantially influence the final performance.\n\n---\n\n*Q5. In Table 7, why does PQ drop a lot when you use ground-truth semantic masks for $m\\_{sem}$? What is the result of using ground-truth semantic masks with the stop-gradient operation?*\n\n   This is a good question. To make it clearer, using the ground-truth semantic masks (gt-masks) means that :\n   \n   1. There is no semantic loss during training, which does not provide any relaxation for the training objective as semantic segmentation is a sub-task of panoptic segmentation.\n  \n   2. All false-positive losses outside the gt-masks would be eliminated, indicating that the false-positive losses are still important. Removing most of them will lead the network to converge to sub-optimal. \n\n   3. ReMask here is to help eliminate the **`extreme`** false positive losses to prevent the loss distribution to be unbalanced (Figure A2 in supplementary material).\n\n   We would like to clarify that when using ground-truth masks for semantic masking, there would be no parameters in the semantic branch; therefore, there is no gradient to be stopped in this scenario."}, {"review_id": "6hgiWP4yAT", "reviewer": "Reviewer_8sND", "summary": "The manuscript presents two novel heuristics for training efficient panoptic models based on mask-level recognition and pixel-to-mask assignment. \n\nThe first heuristics affects pixel-to-mask assignment and is referred to as ReMask. ReMask has been designed to balance the overwhelming contribution of false positive mask assignments to the mask-assignment loss by leveraging an independent semantic prediction head. In particular, the authors suppress the pixel assignment towards masks that get recognized into classes that are inconsistent with local semantic predictions. However, Table 7 suggests that most of the improvement does not stem from training relaxation and that the benefits may be caused by enhanced locality of the recognition process through Lsem. \n\nThe second heuristics affects mask-level recognition and is referred to as ReClass. ReClass changes the classification targets of the predicted masks from one-hot winner-takes-all to mixtures of one-hot assignments of all incident ground-truth masks. It appears that the authors conjecture that ReClass contributes to the convergence speed by reducing the penalty of inaccurate masks during early training.       \n", "questions": "Q1. Can you disentangle the relative contribution of loss relaxation and local enhancement (cf Table 7)?\n", "limitations": "It would be interesting to discuss whether there is any benefit in conjunction with weaker (RN-18) and stronger (SWIN, ConvNext) backbones.\n\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "S1. Panoptic segmentation is an important computer vision task with many applications.\n\nS2. State-of-the-art performance among approaches based on mid-range backbones (RN50, MNV3).\n\nS3. Ablations and validations suggest that the proposed heuristics contribute significant performance improvements.\n\nS4. The proposed heuristics can be removed during inference; this results in competitive inference speeds.", "weaknesses": "W1. the manuscript requires non-linear reading effort: \n* lines 161-181 start to make sense only after reading the equations\n* equations for x_pan and x_sem are missing (they should start from shared features)\n* d_pan, d_sem, N_Q and N_C should be defined before use.\n\nW2. a bird's-eye view figure is missing (example: Fig.2 in [10]).\n\nW3. many small details:\n* l161: it appears that x_pan should be HWxd_pan?\n* l148: post-processing is unclear.\n\nSuggestions\n\nS1. it may be interesting to mention the following related work:\n* Fully Convolutional Networks for Panoptic Segmentation with Point-Based Supervision. TPAMI 2023.   \n* Panoptic SwiftNet: Pyramidal Fusion for Real-Time Panoptic Segmentation. Remote Sensing. 2023. \n* Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation. CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We will improve the writing of the final paper based on all reviewers' feedback. In the meantime, we hope the answer below can help improve the readability of the paper.\n\n*W1. (1) lines 161-181 requires non-linear reading efforts.*\n\nWe guess this might be due to the reviewer finding it hard to read the paper while referring to Figure 2 since it is separated apart into two pages. To make the paper easier to read, we will put the text of L161-181 close to Figure 2 together on the same page.\n\n*W1. (2) Equations for $x\\_{pan}$ and $x\\_{sem}$ are missing*\n\nThanks for the suggestion. $x\\_{pan}$ and $x\\_{sem}$ indeed start from the shared features. To make it clear, we plan to add a bird's-eye view figure which will better illustrate how $x\\_{pan}$ and $x\\_{sem}$ in Figure 2 are related to the overall architecture, e.g. kMaX-Deeplab or MaskFormer.\n\n*W1. (3) d_pan, d_sem, N_Q and N_C should be defined before use.*\n\nWe kindly argue that we did not define $d_{pan}$, but use $N_Q$ since it represents the number of queries for the transformer decoder. Finally, we thank the reviewer for the suggestion. We will move all these notations currently provided in L166-169 to an earlier part of the paper, closer to where they are first used, i.e. line 164.\n\n---\n\n*W2. A bird's-eye view figure is missing (example: Fig.2 in [10]).*\n\nThank you for the thoughtful suggestion. \nDue to the original space constraint, we didn't include a bird's eye view figure.\nWe plan to add a bird's-eye view figure which will better illustrate how $x\\_{sem}$ and $x\\_{pan}$ in Figure 2 are related to the overall architecture, e.g. kMaX-Deeplab or MaskFormer.\n\nMeanwhile, we would like to point out that Figure 2 showcases the ReMask process, while Figure 3 details ReClass. They are presented in separate figures because ReClass is solely related to the loss and does not alter the architecture. Regarding the entire process, it completely followed kMaX-DeepLab. We recognize that this may be challenging for those unfamiliar with mask transformers. \n\n---\n\n*W3. (1) it appears that $x\\_{pan}$ should be HWx $d\\_{pan}$?*\n\nWe thank the reviewer for the question. We kindly note that we have never used (or defined) $d\\_{pan}$ in our manuscript and $x\\_{pan} \\in \\mathbb{R}^{HW\\times N\\_Q}$ is defined in L161. This is due to the structure of masked transformers, where the number of panoptic masks is defined by the number of queries $N\\_Q$. We also have defined another term $d\\_{q}$ in L167-168.  I hope this will clarify this confusion.\n\n*W3. (2) Post-processing is unclear.*\n\nThanks for the question. We completely followed the post-processing in kMaX-DeepLab [64] without any change. We will mention this technical detail in the final paper and make it clear to readers.\n\n---\n\n*S1. Related work.*\n\nThanks for the supplement. We will add and discuss all three papers in the revised paper.\n\n---\n\n*Q1. Can you disentangle the relative contribution of loss relaxation and local enhancement (cf Table 7)?*\n\nThis is a good question! We added another ablation study that removes the semantic masking (the concrete grey arrow right under \"stop grad\" in Figure 2).\nThis would keep the semantic loss $\\mathcal{L}\\_{sem}$ for semantic relaxation but remove the local enhancement (semantic masking). The result is shown below and will be added to the final paper.\n\n| *w/* semantic masking?$^{[a]}$ | *w/* $\\mathcal{L}\\_{sem}$?$^{[b]}$ | *w/* ReClass? | PQ |\n| :--------------------: | :------------------------: | :-: | :- |\n| | | | 50.4 |\n| | &#x2611; | | 51.3 |\n| &#x2611; | &#x2611; | | 51.7 |\n| &#x2611; | &#x2611; | &#x2611; | 52.4 |\n\nThe table above shows that the semantic relaxation would lead to a 0.9 increase in PQ; while with the semantic masking it would lead to an additional 0.4 PQ gain. The semantic masking can be linked to local enhancement as it would suppress the extreme false-positive predictions via a simple masking operation.\n\n---\n\n*Q2. Results on R18 and stronger backbones (Swin/ConvNext)*\n\nWe thank the reviewer for the suggestion, and kindly remind the reviewer that we have evaluated our method on weaker backbones that are even smaller than ResNet-18 (MobileNetV3-Small and MobileNetV3-Large in Table 1 and 8). We also reported the result for the ConvNeXt model in Table A2 of the supplementary material.\n\n---\n\n$^{[a]}$: local enhancement\n\n$^{[b]}$: semantic loss relaxation\n"}, {"review_id": "FqcVKOBniS", "reviewer": "Reviewer_c7F1", "summary": "This paper shows that the existing sota panoptic segmentation method\u001ds have an unbalanced loss ( excessively large false-positive loss due to the use of the sigmoid function). The authors designed two relaxation mechanisms to relax the supervision at the mask and class levels, thereby improving training efficiency and improve accuracy.", "questions": "see weakness", "limitations": "No applicable", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper reveals the problem of excessive false-positive loss caused by the use of the sigmoid function in current transformer-based panoptic segmentation models and proves that false-positive loss is also helpful for training and proper relaxation of constraints can improve training efficiency, which is instructive for the community.\n2. Both approaches to relaxation in the paper (ReMask and ReClass) design are interesting and effective.\n3. The proposed ReMax is excellent in training efficiency, inference efficiency, and accuracy.", "weaknesses": "1. On the one hand, excessive false-postive loss affects training efficiency, but on the other hand, false-postive loss also benefits the final result. The author provides only an empirical solution, lacking more in-depth discussions. I wonder about the qualitative analysis of the impact of different scales of false-positive loss on the final result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the recognition of our paper.\n\n*Q. Qualitative analysis of the impact of different scales of false-positive loss on the final result.*\n\nWell spotted! We experimented with various methods to adjust the scales of FP/FN losses. \nPlease refer to the table below for these results, where the reported result for FP loss scaling represents the most optimal scaling factor we have tried.\nOverall, as summarized in line 39-44, we observed that equivalently scaling the magnitude of false-positive losses for all examples doesn't enhance performance.\nThe crux of ReMaX's effectiveness lies in its ability to dynamically filter out extreme losses, drawing parallels with gradient-clipping.\n\n\n| Loss-scale Method |   PQ  |\n| :---------------: |   :-  |\n| baseline          |  50.4 |\n| w/ FP loss scale $\\downarrow$  |  50.4 |\n| w/ FN loss scale $\\uparrow$  |  50.9 |\n| w/ Grad-clip  |  51.2 |\n| w/ ReMask  |  51.7 |\n| w/ ReMask+ReClass  |  52.4 |\n\nBased on the above table, it's evident that holistically scaling down the false-positive loss doesn't yield a performance boost.\nIn fact, we've tested various scaling factors to holistically scale down the false-positive loss, yet none contributed to better performance. \nThe result for FP loss scaling reported here represents the most optimal scaling factor we identified."}, {"review_id": "NAPWALvmyb", "reviewer": "Reviewer_Mpjf", "summary": "This paper presents a relaxation technique for training Efficient Panoptic Segmentation models called ReMaX. Based on the observation of much higher false positive penalisation in training panoptic segmentation models, it introduces two relaxation designs, ReMask and ReClass. Results are reported on COCO, ADE20K and Cityscapes datasets. ", "questions": "About the ReClass operation. What if the overlapped objects belong to the same category? For instance, two 'persons' in Fig. 3.  ", "limitations": "Error bars are not reported. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is clear, and the finding is interesting. \n2. The proposed method is reasonable and fits well with the motivation. \n3. Using soft semantic segmentation prediction for relaxing the training of mask transformers makes sense to me. \n4. Consistent improvements are obtained with the proposed method, and ablations are thorough. According to the experiments, the proposed method can work well with multiple existing approaches. ", "weaknesses": "1. The design seems coupled with mask transformers. It may not be generalised to all efficient panoptic segmentation models. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer for the recognition of our paper.\n\n*W1. The design seems coupled with mask transformers. It may not be generalised to all efficient panoptic segmentation models.*\n\nWe thank the reviewer for the suggestion and agree that it is interesting to further explore the effectiveness of our method with other segmentation frameworks. In principle, ReMaX has the potential to be generalized to other non-transformer-based panoptic segmentation frameworks with a panoptic mask representation $m\\_{pan}$ and a semantic mask representation $m\\_{sem}$. But it is beyond the scope of this work and rebuttal. We currently mainly explore our method in kMaX-DeepLab (most recent state-of-the-art), and have also quickly experimented with Mask2Former in this rebuttal. We will further validate the generalizability and effectiveness of our method in future work.\n\n---\n\n*Q1. About the ReClass operation. What if the overlapped objects belong to the same category? For instance, two 'persons' in Fig. 3.*\n\nThis is a good question. Since different instances with the same category belong to the same semantic mask, the class labels for their instance masks **will not change**.\n\n---\n\n*L1. Error bar not provided.*\n\nThanks for the suggestion. We will add it to the revised paper."}, {"review_id": "i66UV1Gcdh", "reviewer": "Reviewer_EbKy", "summary": "This paper presented a new mechanism to train efficient panoptic segmentation frameworks, which adds relaxation to mask predictions and class prediction for panoptic segmentation. In experiments, the authors demonstrated that the relaxation techniques can consistently improve panoptic segmentation frameworks.", "questions": "1. Why can the loss function with semantic masks be a relaxation that helps training? Training with semantic masks has been demonstrated to improve the instance segmentation in many frameworks such as CondInst. Therefore, what is the main contribution or the novelty of the proposed ReMask should be carefully discussed.\n2. In Table 6, the results of MaskDINO and ReMaX should be tested with same GPU. When the GPU is V100, ReMax achieves 16.3 FPS (from Table 1), but MaskDINO achieves 16.8 FPS. \n\nTypo:\n1. Line 71: \"... like YOSO [26] and MaskConver [26] ... \"", "limitations": "1. The contributions of this paper may be limited. For the segmentation framework, this paper directly uses the kMax-DeepLab. For the ReMask technique, it has been demonstrated in instance segmentation. For the ReClass technique, the effect of applying it or not is not shown in experiments.\n\n2. The generalization of the proposed method should be demonstrated based on more frameworks such as Mask2Former, YOSO, and Panoptic DeepLab, not only kMax-Deeplab.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The results are impressive, especially, with R50 backbone, ReMax-M achieves 49.1 PQ and 51.9 FPS on COCO dataset.\n2. This paper is well-written and the motivation is clear.\n", "weaknesses": "1. The generalization of the proposed training mechanism should be demonstrated.\n2. Some experimental results are unconvincing.\n\nPleas see questions and limitations for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the insightful feedback. We hope that the subsequent response will address the concerns voiced in the review. We thank the reviewer for pointing out the typo, which we will fix in the final revision.\n\n| *w/* semantic masking? | *w/* $\\mathcal{L}\\_{sem}$? | *w/* ReClass? | Iterations | PQ |\n| :--------------------: | :------------------------: | :-: | :-: |  :-: |\n| | | | 50K | 50.4 |\n| | | | 150K | 53.0 |\n| | &#x2611; | | 50K | 51.3 |\n| | &#x2611; | | 150K | 53.0 |\n|         &#x2611;       | &#x2611; | | 50K | 51.7 |\n|         &#x2611;       | &#x2611; | &#x2611; | 50K | 52.4 |\n|         &#x2611;       | &#x2611; | &#x2611; | 150K | 54.0 |\n\n*Q1. Training with semantic masks has been demonstrated to improve instance segmentation in many frameworks such as CondInst. Therefore, what is the main contribution or the novelty of the proposed ReMask should be carefully discussed.*\n\nWe appreciate the insightful suggestion. We echo that incorporating semantic loss such as CondInst [A1] and YOLACT [A2] can indeed enhance the performance of instance segmentation. This is also evident from our experiments, as demonstrated in the table above.\n\nHowever, we respectfully point out that our approach differs from the application of loss in [A1] and [A2] in the following aspects:\n\n1. **The key of ReMask is semantic masking instead of purely semantic loss.**\n   As demonstrated in the above table, the direct application of semantic loss, without semantic masking, does not result in any improvement for long-schedule training (i.e., 150K iterations). It's only when semantic masking is implemented that the network tends toward better convergence, a central aspect of our methodology. While the exclusive use of semantic loss may expedite the initial stages of the training process (e.g, 50K iterations), it fails to improve the ultimate convergence quality.\n   \n2. **The ReMask is applied along with mask transformers while CondInst and YOLACT are not.**\n   We kindly argue that our method is applied with mask transformers while CondInst and YOLACT are conventional segmentation frameworks.\n\n3. The meaning of relaxation is two-fold:\n\n   (1) We posit that compared to panoptic segmentation, semantic segmentation presents a less challenging task. As such, employing semantic prediction to fine-tune the results of panoptic segmentation can be viewed as a form of relaxation strategy.  \n\n   (2) The ReClass process alters the initial one-hot label into a softer tensor, thereby easing the intensity of the strict supervision and can be regarded as a way of relaxation.\n\nWe would add the above discussion in the revised paper and carefully clarify the difference between ours and the related papers.\n\n---\n\n*Q2. In Table 6, the results of MaskDINO and ReMaX should be tested with the same GPU. When the GPU is V100, ReMax achieves 16.3 FPS (from Table 1), but MaskDINO achieves 16.8 FPS.*\n\nWe kindly note that 16.8 FPS (typo, should be 14.8 in the original paper) of Mask DINO is evaluated on **`A100`**, while our 16.3 FPS is evaluated on **`V100`** and a higher resolution 1281x1281. Here we report the detailed FPS below:\n\n| Method | GPU | Resolution | FPS |\n| :----: | :----: | :-: | :-: |\n| Mask DINO | A100 | Not reported | 14.8 |\n| Mask DINO | V100 | 1200x800 $^\\dagger$ | 10.9 |\n| Ours | V100 | 1200x800 | 26.3 |\n| Ours | V100 | 1281x1281 | 16.3 |\n\nFor fair comparison, as shown in the table above, ours is about 2x faster than the Mask DINO model for 1200x800 resolution on V100. We will revise the manuscript accordingly to further clarify the confusion.\n$^\\dagger$ We recompute the FPS of Mask DINO on our own device.\n\n---\n\n*L1.  For the ReMask technique, it has been demonstrated in instance segmentation. For the ReClass technique, the effect of applying it or not is not shown in experiments.*\n\nWe kindly note that in Table 4 of the manuscript, we have reported that the application of ReClass results in a **`0.7`** PQ increase when comparing the second column ($\\eta=0$) with the fifth column ($\\eta=0.1$). For distinctions between ReMask and prior methods [A1] and [A2], please refer to the previous response.\n\n---\n\n*L2. The generalization of the proposed method should be demonstrated based on more frameworks such as Mask2Former, YOSO, and Panoptic DeepLab, not only kMax-Deeplab.*\n\nThis is a good suggestion. We have re-implemented ReMaX for Mask2former in PyTorch and reported the results in the table below.\n| Method | Epochs | PQ |\n| :----: | :----: | :-:|\n| Mask2former | 24 | 48.36 |\n| Mask2former + ReMaX | 24 | 50.24 |\n\nDue to the time limit, we did not reproduce the originally reported Mask2Former results by fully exploring all the hyper-parameters. However, the table above shows that based on the same Mask2Former baseline ReMaX boost its accuracy. This could demonstrate that ReMaX is also effective for other segmentation frameworks like Mask2former. We will add such results on more baseline models like Mask2Former and YOSO to the final paper.\n\n---\n\n### Reference\n[A1]: Tian Z, Shen C, Chen H. Conditional convolutions for instance segmentation. ECCV 2020.\n\n[A2]: Bolya D, Zhou C, Xiao F, et al. Yolact: Real-time instance segmentation. ICCV 2019.\n"}]

Author rebuttal:

Meta-review: In this work, the authors propose two training techniques for panoptic segmentation, namely ReMask and ReClass, to address issues in panoptic segmentation training. The proposed techniques are simple and sweet, and are validated to be effective with good results. The work received unanimous recommendations on acceptance from the reviewers. The AC agrees with the reviewers and recommends acceptance. The authors are highly encouraged to incorporate the suggestions/results from the rebuttal discussions in their final version and revise accordingly.
Decision: Accept (poster)