SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the  $\eta$  denotes the smooth factor for ReClass that controls the degree of the relaxation applying to the classification head.
Our study of ReMaX involves analyzing its performance on three commonly used image segmentation datasets. COCO [44] supports semantic, instance, and panoptic segmentation with 80 "things" and 53 "stuff" categories; Cityscapes [17] consists of 8 "things" and 11 "stuff" categories; and ADE20K [77] contains 100 "things" and 50 "stuff" categories. We evaluate our method using the Panoptic Quality (PQ) metric defined in [36] (for panoptic segmentation), the Average Precision defined in [44] (for instance segmentation), and the mIoU [19] metric (for semantic segmentation).
(b)
Figure 6: FPS-PQ Pareto curve on (a) COCO Panoptic val set and (b) Cityscapes val set. Details of the corresponding data points can be found in Table 1 and 9. We compare our method with other state-of-the-art efficient pipelines for panoptic segmentation including kMaX-DeepLab [73], Mask2Former [14], YOSO [29], Panoptic-DeepLab [12], Real-time Panoptic Segmentation [26], UPSNet [70], LPSNet [25], MaskFormer [13], and MaskConver [55].
4.2 Results on COCO Panoptic
Implementation details. The macro-architecture of ReMaX basically follows kMaX-DeepLab [73], while we incorporate our modules introduced in Section 3 into the corresponding heads. Concretely, we use the key in each  $\mathbf{k}$ -means cross-attention operation as  $\mathbf{x}_{\mathrm{sem}}$  defined in Figure 3. The semantic head introduced during training consists of an ASPP module [8] and a  $1\times 1$  convolution that outputs  $N_{C}$  number of channels. The specification of models with different size is introduced in the appendix. For other details like post-processing and data preparation, we strictly follow kMaX-DeepLab [73].
Training details. We basically follow the training recipe proposed in kMaX-DeepLab [73] but make some changes to the hyper-parameters since we add more relaxation to the network. Here we high-light the necessary and the full training details and specification of our models can be also found in the appendix. The learning rate for the ImageNet-pretrained [56] backbone is multiplied with a smaller learning rate factor 0.1. For training augmentations, we adopt multi-scale training by randomly scaling the input images with a scaling ratio from 0.3 to 1.7 and then cropping it into resolution  $1281 \times 1281$ . Following [66, 72, 73], we further apply random color jittering [18], and panoptic copy-paste augmentation [33, 58] to train the network. DropPath [30, 39] is applied to the backbone, the transformer decoder. AdamW [34, 49] optimizer is used with weight decay 0.005 for short schedule 50K and 100K with a batch size 64. For long schedule, we set the weight decay to 0.02. The initial learning rate is set to 0.006, which is multiplied by a decay factor of 0.1 when the training reaches  $85\%$  and  $95\%$  of the total iterations. The entire framework is implemented with DeepLab2 [68] in TensorFlow [1]. Following [66], we apply a PQ-style loss, a Mask-ID cross-entropy loss, and the instance discrimination loss to better learn the feature extracted from the backbone.
For all experiments if not specified, we default to use ResNet-50 as the backbone and apply ReMask to the first 4 stages of transformer decoder. The  $\eta$  for ReClass operation is set to 0.1. All models are trained for 27 epochs (i.e., 50K iterations). The loss weight for the auxiliary semantic loss  $\mathcal{L}_{\mathrm{sem}}$  applied to each stage in the transformer decoder is set to 0.5 and the weights for those loss terms in  $\mathcal{L}_{\mathrm{pan}}$  are set the same as kMaX-Deeplab[73].
ReMaX significantly improves the training convergence and outperforms the baseline by a large margin. As shown in Figure 5, we can see that when training the model under different training schedules 50K, 100K and 150K, our method outperforms the baselines by a clear margin for all different schedules. Concretely, ReMaX can outperform the state-of-the-art baseline kMaX-DeepLab by a significant 3.6 PQ when trained under a short-term schedule 50K iterations (27 epochs) for backbone ResNet-50. Notably, our model trained with only 50K iterations performs even better than kMaX-DeepLab [73] trained for the 100K iterations (54 epochs), which means that our model can speed up the training process by approximately  $2 \times$ . We kindly note that the performance of ResNet-50 can be further improved to 54.2 PQ for 200K iterations. ReMaX works very well with
Activation	w/
ReMaX?	w/
grad-
clip?	PQ
softmax	×	×	48.8
softmax	✓	×	49.5
sigmoid	×	×	50.4
sigmoid	×	✓	51.2
sigmoid	✓	×	52.4
Table 2: The impact of activation function and gradient clipping.
w/ identity mapping?	w/ ReMask in test?	PQ
✓	×	52.4
✓	✓	52.4
×	✓	52.1
×	×	51.9
Table 5: Effect of applying identity mapping and auxiliary head for ReMask during testing. Removing the auxiliary semantic head will not lead to performance drop when  $\widehat{\mathbf{m}}_{\mathrm{pan}}$  is applied with identity mapping.
#ReMasks	0	2	4	6
PQ	50.4	51.9	52.4	51.5
Table 3: The effect of number of ReMask applied. ReMaX performs the best when ReMask is applied to the first 4 stages of the transformer decoder.
Method	Backbone	FPS	PQ
MaskFormer [13]	17.6	46.5
K-Net [76]	-	47.1
PanSegFormer [43]	7.8	49.6
Mask2Former [14]	R50 [22]	8.6	51.9
kMaX-DeepLab [73]	26.3	53.0
MaskDINO [40]	16.8‡	53.0
ReMaX	26.3†	54.2
Table 6: Comparison on COCO val with other models using ResNet-50 as the backbone.  ${}^{ \dagger  }$  The FPS here is evaluated under resolution  ${1200} \times  {800}$  on V100 and the model is trained for  ${200}\mathrm{\;K}$  iterations.  ${}^{ \ddagger  }$  is evaluated using a A100 GPU.
η	0	0.01	0.05	0.1	0.2
PQ	51.7	51.7	51.9	52.4	51.5
Table 4: The impact of different  $\eta$  defined in Eq. 4 for ReClass. Here we observe that the result reaches its peak when  $\eta  = {0.1}$  .
w/ stop-grad?	w/ gt?	PQ
✓	×	52.4
N/A	✓	45.1
×	×	36.6*
Table 7: The effect of stop gradient and gt-masking. The denotation  $w / \mathbf{g}t?$  means whether we use ground-truth semantic masks for  $\mathbf{m}_{\mathrm{sem}}$ . * The result without the stop-gradient operation does not well converge in training.
efficient backbones including MobileNetV3-Small [27] and MobileNetV3-Large [27], which surpass the baseline performance by 4.9 and 5.2 PQ for 50K iterations, and 3.3 and 2.5 PQ respectively for 150K iterations. These results demonstrate that the proposed relaxation can significantly boost the convergence speed, yet can lead to better results when the network is trained under a longer schedule.
ReMaX vs. other state-of-the-art models for efficient panoptic segmentation. Table 1 and Figure 6 (a) compares our method with other state-of-the-art methods for efficient panoptic segmentation on COCO Panoptic. We present 4 models with different resolution and model capacity, namely ReMaX-Tiny (T), ReMaX-Small (S), ReMaX-Medium (M) and ReMaX-Base (B). Due to the limit of space, the detailed specification of these models is included in the appendix. According to the Pareto curve shown in Figure 6 (a), our approach outperforms the previous state-of-the-art efficient models by a clear margin. Specifically, on COCO Panoptic val set, our models achieve 40.4, 44.6, 49.1 and 54.2 PQ with 109, 81, 52 and 16 FPS for ReMaX-T, ReMaX-S, ReMaX-M and ReMaX-B respectively. The speed of these models is evaluated under the resolution  $641 \times 641$  except for ReMaX-Base, which is evaluated under resolution  $1281 \times 1281$ . Meanwhile, as shown in Table 6, our largest model with the backbone ResNet-50 also achieves better performance than the other non-efficient state-of-the-art methods with the same backbone.
Effect of different activation, and the use of gradient clipping. Table 2 presents the effect of using different activation function (sigmoid vs. softmax) for the Mask-ID cross-entropy loss and the  $\sigma(\cdot)$  defined in Eq (1). From the table we observe that ReMask performs better when using sigmoid as the activation function, but our method can get rid of gradient clipping and still get a better result.
Can we use the ground-truth masks for local enhancement instead of ReMask? As discussed in Section 3, to figure out whether it is the loss relaxation or the pixel filtering that improves the training, we propose experiments to replace  $\mathbf{m}_{\mathrm{sem}}$  with the ground truth semantic masks during training. When  $\mathbf{m}_{\mathrm{sem}}$  is changed into the ground truth, all positive predictions outside the ground-truth masks will be removed, which means that the false positive loss would be significantly scaled down. The huge drop (52.4 vs. 45.1 PQ in Table 7) indicates that the gradients from false positive losses can benefit the final performance. Table 7 also shows that when enabling the gradient flow from the panoptic loss to the semantic predictions, the whole framework cannot converge well and lead to a drastic drop in performance (36.6 PQ). The semantic masks  $\mathbf{m}_{\mathrm{sem}}$  faces a simpler objective (i.e. only semantic segmentation) if the gradient flow is halted.
w/semantic masking (local enhancement)	w/Lsem(loss relaxation)	w/ReClass?	Iterations	PQ
×	×	×	50K	50.4
×	×	×	150K	53.0
×	✓	×	50K	51.3
×	✓	×	150K	53.0
✓	✓	×	50K	51.7
✓	✓	✓	50K	52.4
✓	✓	✓	150K	54.0
Table 8: The relative impact of loss relaxation and semantic masking (local enhancement) on COCO Panoptic val set under short (50K) and long (150K) training schedule.
The relative contribution of loss relaxation and local enhancement in ReMask. The auxiliary semantic loss term can be viewed as loss relaxation, while the semantic masking branch can be viewed as local enhancement. To disentangle the relative contribution of loss relaxation and local enhancement, we conducted another ablation study that removes the semantic masking branch (the concrete grey arrow right under "stop grad" in Figure 3), which would remove the local enhancement (semantic masking) but keep the auxiliary semantic loss term for loss relaxation. The results are reported in Table 8. The short training schedule of 50K iterations shows that the semantic loss relaxation leads to a 0.9 increase in PQ; while the semantic masking contributes to an additional 0.4 gain in PQ. The long-schedule training (i.e. 150K iterations) demonstrates that semantic masking is critical in ReMask because applying semantic loss relaxation alone without semantic masking does not result in any improvement. In other words, only using semantic loss relaxation may expedite the early stage of training (e.g., 50K iterations), but it fails to improve the ultimate convergence quality.
The number of mask relaxation. Table 3 shows the effect of the number of ReMask applied to each stage, from which we can observe that the performance gradually increases and reaches its peak at 52.4 PQ when the number of ReMask is 4, which is also our final setting for all other ablation studies. Using too many ReMask ( $>4$ ) operations in the network may add too many relaxation to the framework, so that it cannot fit well to the final complex goal for panoptic segmentation.
ReClass can also help improve the performance for ReMaX. We investigate ReClass and its hyper-parameter  $\eta$  in this part and report the results in Table 4. In Table 4, we ablate 5 different  $\eta$  from 0 to 0.2 and find that ReClass performs the best when  $\eta = 0.1$ , leading to a 0.5 gain compared to the strong baseline. The efficacy of ReClass validates our assumption that each mask may cover regions of multiple classes.
Effect of the removing auxiliary semantic head for ReMask during testing. The ReMask operation can be both applied and removed during testing. In Table 5, it shows that the accuracy is comparable under the two settings. In Table 5 we also show the necessity of applying identity mapping to  $\mathbf{m}_{\mathrm{pan}}$  during training in order to remove the auxiliary semantic head during testing. Without the identity mapping at training, removing semantic head during testing would lead to 0.5 drop from 52.4 (the first row in Table 5) to 51.9.
4.3 Results on Cityscapes
Implementation details. Our models are trained using a batch size of 32 on 32 TPUv3 cores, with a total of 60K iterations. The first 5K iterations constitute the warm-up stage, where the learning rate gradually increases from 0 to  $3 \times 10^{-3}$ . During training, the input images are padded to  $1025 \times 2049$  pixels. In addition, we employ a multi-task loss function that includes four loss components with different weights. Specifically, the weights for the PQ-style loss (part of  $\mathcal{L}_{\mathrm{pan}}$ ), auxiliary semantic loss  $\mathcal{L}_{\mathrm{sem}}$ , mask-id cross-entropy loss (part of  $\mathcal{L}_{\mathrm{pan}}$ ), and instance discrimination loss are set to 3.0, 1.0, 0.3 and 1.0, respectively. To generate feature representations for our model, we use 256 cluster centers and incorporate an extra bottleneck block in the pixel decoder, which produces features with an output stride of 2. These design are basically proposed in kMaX-DeepLab [73] and we simply follow here for fair comparison.
Results on Cityscapes. As shown in Table 9 and Figure 6 (b), it shows that our method can achieve even better performance when using a smaller backbone MobileNetV3-Large (62.5 PQ) while the
Method	Backbone	FPS	PQ
Mask2Former [14]	R50 [22]	4.1	62.1
Panoptic-DeepLab [12]	Xception-71 [15]	5.7	63.0
LPSNet [25]	R50 [22]	7.7	59.7
Panoptic-DeepLab [12]	R50 [22]	8.5	59.7
kMaX-DeepLab [73]	R50 [22]	9.0	64.3
Real-time [26]	R50 [22]	10.1	58.8
YOso [29]	R50 [22]	11.1	59.7
kMaX-DeepLab [73]	MNV3-L [27]	22.8	60.2
ReMaX	R50 [22]	9.0	65.4
ReMaX	MNV3-L [27]	22.8	62.5
ReMaX	MNV3-S [27]	25.6	57.7
Table 9: Cityscapes val set results for lightweight backbones. We consider methods without pretraining on extra data like COCO [44] and Mapillary Vistas [52] and test-time augmentation for fair comparison. We evaluate our FPS with resolution  $1025 \times 2049$  and a V100 GPU. The FPS for other methods are evaluated using the resolution reported in their original papers.
Method	Backbone	FPS	#params	PQ
Mask2Former [73]	Swin-L†[46]	-	216M	66.6
kMaX-DeepLab [73]	MaX-S†[66]	6.5	74M	66.4
kMaX-DeepLab [73]	ConvNeXt-L†[47]	3.1	232M	68.4
OneFormer [31]	ConvNeXt-L†[47]	-	220M	68.5
ReMaX	MaX-S†[27]	6.5	74M	68.7
Table 10: Cityscapes val set results for larger backbones.  ${}^{ \dagger  }$  Pre-trained on ImageNet-22k.
Method	Backbone	Resolution	FPS	PQ mIoU
MaskFormer [13]	640-2560	-	34.7 -
Mask2Former [14]	640-2560	-	39.7 46.1
YOso [29]	R50 [22]	640-2560	35.4	38.0 -
kMaX-DeepLab [73]	641×641	38.7	41.5 45.0
kMaX-DeepLab [73]	1281×1281	14.4	42.3 45.3
ReMaX	R50 [22]	641×641	38.7	41.9 45.7
ReMaX	1281×1281	14.4	43.4 46.9
Table 11: ADE20K val set results. Our FPS is evaluated on a NVIDIA V100 GPU under the corresponding resolution reported in the table.
other methods are based on ResNet-50. Meanwhile, our model with Axial-ResNet-50 (i.e., MaXS, 74M parameters) as the backbone can outperform the state-of-the-art models [31, 73] with a ConvNeXt-L backbone ( $>220\mathrm{M}$  parameters). The Pareto curve in Figure 6 (b) clearly demonstrates the efficacy of our method in terms of speed-accuracy trade-off.
4.4 Results on ADE20K
Implementation details. We basically follow the same experimental setup as the COCO dataset, with the exception that we train our model for 100K iterations (54 epochs). In addition, we conduct experiments using input resolutions of  $1281 \times 1281$  pixels and  $641 \times 641$  respectively. During inference, we process the entire input image as a whole and resize longer side to target size then pad the shorter side. Previous approaches use a sliding window approach, which may require more computational resources, but it is expected to yield better performance in terms of accuracy and detection quality. As for the hyper-parameter for ReMask and ReClass, we used the same setting as what we propose on COCO.
Results on ADE20K. In Table 11, we compared the performance of ReMaX with other methods, using ResNet-50 as the backbone, and found that our model outperforms the baseline model by 1.6 in terms of mIOU, which is a clear margin compared to the baseline, since we do not require any additional computational cost but only the relaxation during training. We also find that our model can surpass the baseline model kMaX-DeepLab by 1.1 in terms of PQ. When comparing with other frameworks that also incorporate ResNet-50 as the backbone, we show that our model is significantly better than Mask2Former and MaskFormer by 3.7 and 8.7 PQ respectively.
This paper presents a novel approach called ReMaX, comprising two components, ReMask and ReClass, that leads to better training for panoptic segmentation with Mask Transformers. The proposed method is shown to have a significant impact on training speed and final performance, especially for efficient models. In principle, ReMaX has the potential to be generalized to other non-transformer-based panoptic segmentation frameworks as long as it has a panoptic mask representation and a semantic mask representation. In this paper, we mainly verify our method based on state-of-the-art mask transformers [73]. We will further validate the generalization capability of our method in future work. We hope that our work will inspire further investigation in this direction, leading to more efficient and accurate panoptic segmentation models.
Acknowledgement. We would like to thank Yukun Zhu, Xuan Yang at Google Research for their kind help and discussion. Shuyang Sun and Philip Torr are supported by the UKRI grant: Turing AI Fellowship EP/W002981/1 and EPSRC/MURI grant: EP/N019474/1. We would also like to thank the Royal Academy of Engineering and FiveAI.
[67] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. SOLOv2: Dynamic and fast instance segmentation. NeurIPS, 2020. 3
[68] Mark Weber, Huiyu Wang, Siyuan Qiao, Jun Xie, Maxwell D. Collins, Yukun Zhu, Liangzhe Yuan, Dahn Kim, Qihang Yu, Daniel Cremers, Laura Leal-Taixe, Alan L. Yuille, Florian Schroff, Hartwig Adam, and Liang-Chieh Chen. DeepLab2: A TensorFlow Library for Deep Labeling. arXiv: 2106.09748, 2021. 7
[69] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 2021. 3, 4
[70] Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In CVPR, 2019. 1, 3, 7
[71] Tien-Ju Yang, Maxwell D Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, and Liang-Chieh Chen. Deeperlab: Single-shot image parser. arXiv:1902.05093, 2019. 1, 3
[72] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022. 1, 3, 4, 7
[73] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. In ECCV, 2022. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 16
[74] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. NeurIPS, 2023. 3
[75] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019. 4
[76] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NeurIPS, 2021. 1, 2, 3, 4, 8
[77] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 3, 6
[78] Yi Zhou, Hui Zhang, Hana Lee, Shuyang Sun, Pingjun Li, Yangguang Zhu, ByungIn Yoo, Xiaojuan Qi, and Jae-Joon Han. Slot-vps: Object-centric representation learning for video panoptic segmentation. In CVPR, 2022. 3
[79] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021. 3
A Loss Visualization of ReMaX
Figure 7: The histogram shows the ratio of false positives to false negatives for the cross-entropy loss, on a logarithmic scale.
Method	Backbone	#Params	FLOPs	FPS	PQ
kMaX-DeepLab [73]	ConvNeXt-T†[66]	61M	172G	21.8	55.3
ReMaX	ConvNeXt-T†[66]	61M	172G	21.8	55.9
Mask2Former [14]	Swin-B†[46]	107M	466G	-	56.4
kMaX-DeepLab [73]	ConvNeXt-S†[66]	83M	251G	16.5	56.3
ReMaX	ConvNeXt-S†[66]	83M	251G	16.5	56.6
We visualize the loss applied with ReMask and the loss applied without ReMask in Figure 7, from which we can observe that ReMask can effectively reduce extremely high false positive losses; therefore, our method can stabilize the training of the framework.
B Model Specification
Table 12: Results for larger models on COCO val set. FLOPs and FPS are evaluated with the input size  ${1200} \times$  800 and a V100 GPU. †: ImageNet-22K pretraining.
Model	Backbone	Resolution	#Pixel
Decoders	#Transformer
Decoders	#FLOPs	#Params	FPS
ReMaX-T	MNV3-S [27]	641 × 641	[1, 1, 1, 1]	[1, 1, 1]	18.8G	18.6M	109
ReMaX-S	MNV3-L [27]	641 × 641	[1, 1, 1, 1]	[1, 1, 1]	20.9G	22.0M	81
ReMaX-M	R50 [22]	641 × 641	[1, 5, 1, 1]	[1, 1, 1]	67.8G	50.8M	52
ReMaX-B	R50 [22]	1281 × 1281	[1, 5, 1, 1]	[2, 2, 2]	294.7G	56.6M	26
Table 13: Specification of different models in ReMaX family.
We provide the specification of our models and their corresponding number of parameters and FLOPs in Table 13. We kindly note that the numbers of pixel decoders with the format  $\left[\cdot ,\cdot ,\cdot ,\cdot \right]$  represent the numbers for features with  $\left[\frac{1}{32},\frac{1}{16},\frac{1}{8},\frac{1}{4}\right]$  times of the input size. We use Axial attention [65] for all feature maps with resolution  $\frac{1}{32}$ ,  $\frac{1}{16}$  of the input size, and regular bottleneck residual blocks [22] for the rest. The denotation  $\left[\cdot ,\cdot ,\cdot \right]$  for the transformer decoders represents the numbers for resolution of  $\left[\frac{1}{16},\frac{1}{8},\frac{1}{4}\right]$  times of the input size.
C Performance for Larger Models
We also validate the performance of ReMaX for larger models e.g. ConvNeXt-Tiny (T) and ConvNeXt-Small (S). From Table 12 we can find that ReMaX can achieve better results compared to the baseline kMaX-DeepLab [73] and Mask2Former [14]. However, the improvement of ReMaX gets saturated when the numbers become high. Notably, when using ConvNeXt-T backbone, ReMaX can lead to 0.6 PQ increase over kMaX-DeepLab, while incurring no extra computational cost during inference. The improvement is noticeable, as kMaX-DeepLab only further improves 1.0 PQ by using ConvNeXt-S backbone, at the cost of extra  $36\%$  more parameters (22M) and  $46\%$  more FLOPs (79G).
Since we implement our method in TensorFlow, the baselines we can build upon is limited. We have re-implemented ReMaX in PyTorch and applied it with Mask2former [14]. The result is
Methods	Epochs	PQ
Mask2former	24	48.36
Mask2former + ReMaX	24	50.24
Table 14: ReMaX is effective on Mask2Former [14] on COCO Panoptic val set.
reported in table 14. Due to the time limit, we did not reproduce the originally reported Mask2Former results by fully exploring all the hyper-parameters. However, the table above shows that based on the same Mask2Former baseline ReMaX boost the overall accuracy. This could demonstrate that ReMaX is also effective for other segmentation frameworks like Mask2former. In future work, we need to validate ReMaX based on more frameworks such as YOSO in PyTorch to demonstrate its effectiveness. Meanwhile, ReClass measures the weight of each class according to the size of each mask, which may not be accurate and can be further improved in the future.
E Boarder Impact