RankSEG-RMA: An Efficient Segmentation Algorithm via Reciprocal Moment Approximation
Authors: Zixun Wang
Ben Dai
Venue: NeurIPS 2025 poster ()
Keywords: semantic segmentation
reciprocal moment approximation
Dice and IoU optimization
Primary area: general_machine_learning

Abstract: Semantic segmentation labels each pixel in an image with its corresponding class, and is typically evaluated using the Intersection over Union (IoU) and Dice metrics to quantify the overlap between predicted and ground-truth segmentation masks. In the literature, most existing methods estimate pixel-wise class probabilities, then apply argmax or thresholding to obtain the final prediction. These methods have been shown to generally lead to inconsistent or suboptimal results, as they do not directly maximize segmentation metrics. To address this issue, a novel consistent segmentation framework, RankSEG, has been proposed, which includes RankDice and RankIoU specifically designed to optimize the Dice and IoU metrics, respectively. Although RankSEG almost guarantees improved performance, it suffers from two major drawbacks. First, it is its computational expense—RankDice has a complexity of $\mathcal{O}(d \log d)$ with a substantial constant factor (where $d$ represents the number of pixels), while RankIoU exhibits even higher complexity $\mathcal{O}(d^2)$, thus limiting its practical application. For instance, in LiTS, prediction with RankSEG takes 16.33 seconds compared to just 0.01 seconds with the argmax rule. Second, RankSEG is only applicable to overlapping segmentation settings, where multiple classes can occupy the same pixel, which contrasts with standard benchmarks that typically assume non-overlapping segmentation. In this paper, we overcome these two drawbacks via a \textit{reciprocal moment approximation} (RMA) of RankSEG with the following contributions:  (i) we improve RankSEG using RMA, namely RankSEG-RMA, reduces the complexity of both algorithms to $\mathcal{O}(d)$ while maintaining comparable performance; (ii) inspired by RMA, we develop a pixel-wise score function that allows efficient implementation for non-overlapping segmentation settings. We illustrate the effectiveness of our method across various datasets and state-of-the-art models. The code of our method is available in: \url{https://github.com/ZixunWang/RankSEG-RMA}.

RankSEG-RMA: An Efficient Segmentation Algorithm via Reciprocal Moment Approximation
Zixun Wang
Ben Dai
Department of Statistics and Data Science The Chinese University of Hong Kong 115522501280 10t5500122inkkc cuhk edu. hk
Department of Statistics and Data Science The Chinese University of Hong Kong bendai@cus edu. hk
Abstract
Semantic segmentation labels each pixel in image with its corresponding class, and is typically evaluated using the Intersection over Union (IoU) and Dice metrics quantify the overlap between predicted and ground truth segmentation masks. In the literature, most existing methods estimate pixel-wise class probabilities, then apply argmax thresholding obtain the final prediction. These methods have been shown to generally lead to inconsistent or suboptimal results, as they do not directly maximize segmentation metrics. To address this issue, novel consistent segmentation framework, RankSEG, has been proposed, which includes RankDice and RankIoU specifically designed to optimize the Dice and IoU metries, respectively. Although RankSEG almost guarantees improved performance, it suf- fers from two major drawbacks. First, it its computational Aapense-RRanDice has a complexity of O(d d) with a substantial constant factor (where d rep- resents the number of pixels), while RankIoU exhibits even higher complexity Odd), thus limiting its practical application. For instance, in LiTS, prediction with RankSEG takes 16.33 seconds compared to just 0.01 seconds with the argmax rule. Second, RankSEG is only applicable to overlapping segmentation settings, where multiple classes can occupy the same pixel, which contrasts with standard benchmarks that typically assume non-overlapping segmentation. In this paper, we overcome these two drawbacks via a reciprocal moment approximation (RMA) of RankSEG with the following contributions: (i) we improve RankSEG using RMA, namely RankSEG-RMA, reduces the complexity both algorithms to (d) while maintaining comparable performance; (ii) inspired by RMA, develop pixel-wise score function that allows efficient implementation for oveellerlg segmentation settings. We illustrate the effectiveness of our method across vari- ous datasets and state-of-the-an models. The code of our method is available in: https: thub. com com/Ziiuuee /tt///iixuunnnnnnee htt.://wuunaaaa/nnaan-------- RanKSEG-RMA.
Introduction
Semantic segmentation is fundamental task in computer vision that assigns each pixel in image to a specific class, serving as a cornersoone for applications such as autonomous driving [Cordts et al., 2016, Feng et al., 2020], medical image analysis [Heller et al., 2019, Bilic al., 2023], and augmented reality [Ko and Lee, 2020].
Evaluating the performance of segmentation models naturally requires appropriate metrics that accurately reflect segmentation quality. Specifically, pixel-wise accuracy (Acc) is often biased toward classes that occupy large image regions and fails to account for false positives [Everingham et al., 2010, Wang et al., 2023a]. Consequently, the Intersection over Union (IoU) and Dice metrics have emerged as the standard evaluation measures for semantic segmentation [Cordts al., 2016,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Zhou et al., 2017]. However, regardless of the metries employed, most existing works adhere to a classification based segmentation procedure: (i) training-step: training a model to estimate pixel-wise class probabilities using strictly proper loss [Gneiting and Raftery, 2007] (e.g., cross-entropy loss [Mao et al., 2023]); (ii) prediction-step: followed by applying argmax or thresholding to these probabilities for the final prediction [Chen et al., 2017, Zhao et al., 2017, Xie et al., 2021]. J. Yet, as demonstrated by Dai and Li [2023], the prediction-step by argmax and thresholding are inconsistent, meaning that even with an infinite number of data and perfect probability estimation, those approaches still cannot achieve optimal performance in terms of IoU and Dice metrics. Therefore, these methods typically suboptimal in practical applications.
An alternative direction designing surrogate loss functions which attempt to optimize IoU or Dice directly, with the most popular approaches being soft-loU/Dice loss [Rahman and Wang, 2016, Sudre et al., 2017, Belbode et al., 2020] and Lovász extension loss [Yu and Blaschko, 2018, Berman et al., 2018). However, Lovász hinge loss has been shown to be inconsistent by Finocchiaro et al. [2022]. and consequently, its empirical performance improvements remain controversial [Ma et al., 2021, Dai and Li, 2023]. For soft- loss, the consistency remains unclear. Nevertheless, soft IoU/Dice loss functions are non-convex, making optimization challenging and unstable in practice. Perhaps for this reason, soft-loU/Dice loss is typically used in combination with cross-entropy through ad hoc training strategies, with final segmentation predictions made using argmax and thresholding operations. These approaches generally require tuning an additional hyperparameter- the weight between cross-entropy and soft-loU/Dice loss, resulting in high computational costs and inconvenience in practice.
To this end, a ranking- based consistent segmentation rule (RankSHG; Dai and Li [2023]) is specif- ically developed to directly optimize loU and Dice metrics. Unlike the argmax rule and surrogate loss functions, RankSEG offers provable consistency and practical performance improvement. Fur- thermore, compared to surrogate loss functions, RankSEG only modifies the prediction-step and can serve as a plug-and-play module by directly utilizing model trained with cross-entropy loss, simply replacing the argmax operation prediction-step.
While theoretically sound, their approach exhibits notable limitations: (1) the algorithms are com- putationally intensive for high dimensional data-with RankDice, the less demanding of the two, having time complexity of O lol d) with a large constant factor, where d is the number of pixels. For example, it requires 16.33 seconds on the LiTS [Bilic et al., 2023] dataset, compared to only 0.01 seconds by the argmax rule. (2) In multiclass segmentation, the algorithms are only applicable in overlapping settings where multiple classes can occupy the same pixel, which deviates from standard benchmarks [Everingham et al., 2010, Cordts et al., 2016], and also restricts the application of RankSEG in certain scenarios, such as panoptic segmentation [Kirillov et al., 2019].
Contribution. In this paper, we leverage reciprocal moment approximation (RMA) in segmenta- tion to address the aforementioned disadvantages with the following contributions:
We propose RankSEG-RMA, which reduces the computational complexity of RankSEG (both IoU and Dice) to (dd while preserving comparable performance.
We develop a pixel-wise score function based on RMA, enabling efficient adaptation to non- overlapping segmentation settings, in line with standard benchmarks.
We have theoretically established the quality of the proposed R.A (Theorem 2), and empirical evidence demonstrate that our method not only outperforms the conventional argmax rule but also significantly reduces computational costs compared to existing RankSEG algorithms.
2 Background
In this section, we begin by distinguishing between two different definitions of the IoU and Dice metrics: DDcee and IoU'/Dice' [Wang al., 2023a], advocating for the latter in practical applications. Building upon IoU'/Dice', we review RankSEG and its approximated algorithms.
For clarity, our discussion starts with binary segmentation, with extensions to multiclass segmentation presented in Section 3.2. Let X E Rd Y {0, 1)dd represent the random variables for an image and its corresponding segmentation mask respectively. Consider dataset {x), y;)) consisting of realizations. The segmentation function ò Rd - {0, 1)d produces predicted mask 8(x) e 10.

---

for a test image R.. We denote (x) P(Y,, = 1|x) as the conditional probability of pixel being a foreground pixel given the image x. Index set {1, d} denoted [d].
2.1 Dice/IoU metrics and its variations in implementation
Dice/IoU metrics are defined based on true positives (TP), false positives (FP), and false negatives (FN), However, in practical implementations, the calculation of these components (TP, FP, FN) can be specifically defined at either the dataset or image level, yielding two different metric implementations. For example, the dataset-level and image-level TP are computed as follows:
TP" = (yl.-.. . yH) (xT(x). ... .8T(x.) Σ v]((x)), =
Specifically, dataset-level TP aggregates values across the entire dataset, while image-level TP is computed separately for each image. This distinction leads to different averaging strategies when calculating Dice and IoU metrics. Furthermore, dataset-level and image-level Dice are defined as:
2TP (6) P (6) (D) FPD (8) + FND (8) Dice' (6) = 1
(T)
Dice
==1 TP! FP! + ())
where FPD, FP' and FND, FN' are defined analogously the dataset-level image-level.
Although Dice are more prevalent in the literature [Everingham et al., 2010, Cordts et al., 2016], a growing trend [Liu et al., 2023, Kirillov et al., 2023, Wang al., 2023a] recognizes loU'/Dice' as more favorable for two key reasons. Firstly, loU"/Dice" exhibit a bias toward large objects (Yang al., 2022], which dominate the confusion matrix. This is particularly concerning given the size imbalance in existing datasets [Wang et al., 2023a]. In safe-critical applications, such medical image analysis or autonomous driving, failing to detect small but critical objects can be catastrophic. Secondly, IoU'/Dice' offers statistical information at the image-level. For instance, the variance of IDU'' quantifies robustness, and the lower quantile measures worst-case performance [Wang et al., 2023a]. Consequently, we adopt IoU'/ as the focus in this paper. Notably, these two types of metrics differ significantly the population level. RankSEG-ben based methods are designed to optimize image-level metrics, which in turn may consequently result in decreased performance on dataset-level metries.
2.2 RankSEG and its blind approximation
For simplicity, we will omit the dependence on x hereafter, but it important to note that all following notations are conditional on X = x. RankSEG [Dai and Li, 2023] establishes a novel segmentation framework that directly (or consistently) maximizes Dice/IoU metrics. Specifically, it first ranks the pixel-wise class probabilities and then selects the top r" pixels as segmented pixels, where 7" is so-called the optimal volume. This framework is primarily motivated by the optimal rule outlined in the following theorem; similar result for IoU! is omitted for brevity.
Theorem (The Bayes rule for Dice' segmenntgtono [Dai and Li, 2023]). Assume that Y; 1 Y2JX. segmentation rule 8 a global maximizer of R(Dice' (8)) if and only if 8f (p; 2 Pir- ) where jr is the index with the 7-th largest probubility. The optimal volume T" is given by:
2p, argmax T(T)) with T(T))= = - ++ TE,0,1,-. .)
T" =
(1)
JEJ.
where Jr (j' Py Pj. is the index set of the top T conditional probabilities with To 0. and 1' Ei By is a Poisson-binomicl random variable with By being Bernoulli random variable with success probability Py-.
An intuitive interpretation of Theorem is that Pj.- serves as an adaptive threshold that varies across different input images, in contrast to the fixed threshold (0.5) commonly used in binary segmentation framework. This adaptation, in return, indicates that a fixed threshold framework leads to suboptimal performance in terms of Dice This is illustrated by the following example.
Example. Consider d = 2 with the conventional thresholding or argmax rule yields ă (1,0)T, Since Dic, 0.827 > 0.607 se Dice' ((1, ))), the thresholding or argmax rule is suboptimal.
0.7, 0.4. The Bayes rule produces 8' = (1,1)T, whereas
3

---

Blind approximation (BA), The primary computational bottleneck in RankDice is the optimiza- tion of the optimal volume. Specifically, computing (JT) for all T E 1.. d] in (1) has a complexity of (dd)). To mitigate this, Dai and Li [2023] proposed RankDice-BA, which replaces -- with I to make the expectation independent with index j. yielding an approximation for (J+):
BA (J+) =E
p,) ( JET.
2
Σ 2P(T = )
(En) p.). jEJ.
(2)
Fast Fourier transform (FFT) is then used to reduce the overall complexity in evaluating TBA (T,) for all T E {0, {0. 1...,,....., d} to OOd log d). While this achieves significant improvement, BA method still exhibits the following limitations: (1) the constant factor associated with FFT is generally non-negligible in practice; (2) it is challenging to apply in non-overlapping settings, as shown in Dai and Li [2023, Lemma 7]: and (3) BA is not readily applicable to RankloU due to the large approximation errors, which therefore remains (d²) time complexity. To address these limitations, we propose a reciprocal moment approximation that further reduces the complexity of both RankDice and RankIoU to (d) and enables efficient solution for non-overlapping segmentation.
3 RankSEG-RMA
3.1 Reciprocal moment approximation
We begin by introducing the reciprocal moment approximation, which is technique for approximat- ing the reciprocal moment (or negative first moment) of a Poisson-binomial random variable.
Theorem 2 (Reciprocal moment approximation RankSEG). Let be a Poisson-binomial random
variable, then for any T 2 I, we have
d d
[ + < E(T +T)
Er - 1)
(3)
Therefore, we propose the following TRMA to approximate T(T+) in (1):
TRMA (T+) T IET +
Epi).
Σ pj).
(4)
JEJ,
and its approximation error for any set I S [d] and r |Z] is bounded by:
|TTMA (I) - π (T) < 2(E + -
(5)
Theorem 2 provides two main results: (i) the RMA approximation form (4) for approximating π (TT)). inspired by the exchange of expectation and reciprocal in (3); and (ii) provable error bound that characterize the quality of the RMA approximation. The primary advantage of using RMA is that it avoids expanding the reciprocal moment (RM) into sum of terms, which is computationally expensive. Specifically, (RMA(())) converts such a nonlinear expectation into a linear one, allowing the evaluation of TRMA for all [d] to be performed in ((1) time, once ET and Σjer, Pj are precomputed. In a sharp contrast, evaluating (T,) for any т E [d] requires (dd) operations each time. Notably, the first result, (3) in Theorem 2, credited to Dai and Li [2023] and built upon more fundamental results of reciprocal moments Chao and Strawderman [1972], Wooff [1985], is quite general and may be of independent interest for other applications.
The approximation error bound (5), particularly when I Jy, decreases as the expected volume of predicted mask increases, which typically occurs when d is large. Even for small objects occupying only a 30 30 region in 256 256 image, with an expected volume E(T) T 1000, the approximation error remains below 0.1%, which is generally acceptable in practice.
We now summarize RankDice-RMA for binary segmentation in Algorithm 1. RankloU-RMA is developed analogously in Section B, with the same approximation. The first two steps prepare and store intermediate values for evaluating @RMA (F.) based on estimated probabilities P. After that, we identify the optimal volume 7" and make prediction by selecting the top 7" pixels. Neglecting the sorting operation, the time complexity of the BankDice-RMa. and RankioU-RMA is reduced to (d)) compared (dd log d) for RankDice-BA and (d2) ) for RankloU. For example, RankDice-RMa achieves 48x speedup for RankDice- BA in LiTS dataset [Bilic et al., 2023].

---

Algorithm RankDice-RMA--innry
Input: Estimated probability map P E [0. 1 for a given input image.
Output: The predicted segmentation mask (0.1]a
1:
Rank probabilities in descending order, yielding Ps: 2... pje. Prepare cumulative sum of top probabilities and mean of Poisson-binomia. binomial
2:
d
Gr
Pje k=1
for E [d]
Ā = Σ
pj.
j=1
3: Compute ARMA (J+) - h+++1 for [d]. according to (4),
4: Determine optimal volume F argmax tRMA (F+).
5: Make prediction by 8, (p, 2 Pj-- for E [d].
3.2 RMA-score for non-overlapping multiclass segmentation
To extend RankSEG to non-overlapping multiclass segmentation, a natural approach is first applying binary RankSEG to each class independently, and then address any overlaps. As discussed in the introduction, perfectly addressing overlaps is currently beyond the capabilities of RankSEG, as the non-overlapping constraint leads to nonlinear assignment problem [Kuhn, 1955], which is generally computationally intractable. Therefore, the focus of this section is on utilizing RMA to solve overlapping pixels provided by RankSEG, ultimately producing non-overlapping segmentation.
Angmox grab Fredictish
Closa
Cluss B
Argmax: prob
RankSEG RMM
+
i
RMA-scores
0.05 0.10 0.05
0.10 RoviSES RMA
0) Accest everlaps
(2) Discard everless
L17
NKS IUAN, 3
0.450 00 4.45
(linery)
Compulic
2.09
2.31
2,09
0.05 0.30 0.40 0.0 0.05
7.09
1.1
Probobility
Tunalau n
RankSEG-RMA Pradiction
(latt 68.04)
B
In
Ig Trrestop
Figure I: Comparison of Argmax-prob and RankSEG-RMA. (a) Argmax-prob: each pixel is predicted the class with the highest probability. (b) RankSEG-RMAA segmentation masks I for each class are obtained independently; non-overlapping parts Σ, are accepted, while overlaps Toverlap are resolved by applying argmax over RMA-scores.
Our solution draws inspiration from Argmax-prob method, which efficiently resolves non-overlapping constraint by assigning pixels to their highest-probability classes, that is,
35 argmmax Peuj cE(C]
Vj E [d].
where Pe.j is the estimated probability of pixel belonging to class c. This approach is computationally efficient but does not guarantee optimal performance. One reason is that merely examining probability values does not accurately reflect how individual pixel assignments contribute segmentation metrics. To address this issue, we extend the probability in the argmax framework to Dice/loU-related score. Unlike the probability score, our proposed score function grounded on the Bayes rule in Theorem and can be efficiently computed by RMA. We refer to the score function RMA-score.
5

---

Algorithm 2 RankDice-RMA-Muliddas
Input: Estimated probability map P E [0, 1/xxd
Output: The predicted segmentation mask å E [C]4.
1: /+ Obtain overlapping segmentation mask +/
2: for = to do
3: 3 = RankDice- Raaicee-MM--innaaa (i..ainaaaya.........1 {j Ves 1}.
4: end for
5: /+ Resolve overlapping by argmax over RMA-acores */
6: Identify overlapping indices, Toverlap = U (Iz ITt).
7: for == to C' do
8: Discard assignments for overlapping pixels, T = I \ Toverlap,
9: Accept prediction for not overlapping pixels, 5j for j E Ic-
10: end for
11: Compute RMA-scores, Âej via (7) for Tovorlap and [C].
12: Resolve overlapping by argmax, 8, = argmax,E[c] Do. for j E Toverlap,
Return 8
To proceed, let I denote the index set of pixels assigned to class by RankSEG, Toverlop Uofe (It n It) the index set of overlapping pixels, and I It_ Tovertup the non-overlapping
part, as illustrated in Figure 1. We resolve Toverlap to ensure segmentation masks are non-overlapping:
8y = argmax Segj Vj E Tovertup,
(6)
cE(C]
where Doy is increment of Dice-RMAD by adding pixel j for class c, which defined as:
A RRMA (I, (11) ARMA (I.) 2 (Pes + ΣiEt. Pe.s)
2 2 Po,s [Tc + The +1
(7)
Icl + The +2
where The =E 21== Pe.j represents the estimated mean volume of class c. Intuitively, (6) maximizes the immediate improvement by choosing the class that yields the highest marginal gain in the Dice- RMA objective. While this greedy solution does guarantee globally optimal assignment over all overlapping pixels simultaneously, it is computationally efficient and empirically effective for reducing overlap and improving final segmentation performance.
To summarize, the procedure RankDice-RMA for multiclass segmentation is presented in Algo- rithm 2. After applying binary RankDice-RMAA the predicted set I of each class e is obtained.
The overlapping pixels Toverlup are then identified, and the assignments for these pixels are discarded. The non-overlapping pixels Ig are assigned to their respective classes. Finally, we compute the RMA-scores for the overlapping pixels and resolve overlaps by selecting the class with the highest score. The complexity for addressing overlapping is OCd)), which is worse than Argmax- prob.
4 Experiments
4.1 Setup
Datasets. We conduct experiments on five datasets: (1) PASCAL VOC [Everingham et al., 2010]. (2) Cityscapes [Cordts et al., 2016].C0 (3) ADE20K [Zhou et al., 2017], (4) LiTS [Bilic et al., 2023], and (5) KiTS [Heller et al., 2021]. These datasets cover diverse range of scenarios, including urban scenes (Cityscapes), "thing" and "stuff" (PASCAL VOC and ADE20K), as well as medical images (LiTS and KiTS). The datasets contain between 200 images (LiTS) and 20,000 images (ADE20K), and the number of classes varies from binary segmentation (LiTS) to over a hundred (ADE20K). We only segment tumors in LiTS and KiTS, treating them as binary segmentation tasks.
Models. We employ following six segmentation models: (1) UNet [Ronneberger et al., 2015), (2) DeepLabV3+ [Chen et al., 2018), (3) PSPNet [Zhao et al., 2017], (4) UPerNet [Xiao al.,

---

2018], (5) SegFormer [Xie et al., 2021], and (6) CPT [Tang et al., 2025]. The first four models are CNN-based and utilize backbones such as ResNet [He et al., 2016] or ConvNeXt [Liu al.. 2022], whereas SegFormer and CPT are transformer-based models. The models are trained using the cross-entropy loss, and we compare the proposed RankSEG-RMA with the conventional argmax or thresholding rule for multiclass or binary segmentation, respectively, The training details can be found in Section D.
Evaluation. As discussed in Section 2.1, we evaluate the segmentation models using both mloU'/mDice' and mloUU/ which are straightforward extensions of binary metric Dice! to multiclass segmentation. The metrics with superscripts and C differ when not all classes are present in every image (see Wang et al. [2023a] for details).
4.2 Overall performance
Table 1: Performance for different prediction methods with various models on PASCAL VOC, Cityscapes, and ADE20K.
Model
Prediction
PASCAL VDC
Cityscapes
ADF20K
mioU!
mioU"
miDice!
milice
mioU
maloU"
milice'
miDice"
miou
mioU
miDice!
miDice
PSPNer
Argmaxx-prob
83.59
72.59
87.69
78.22
71.33
63.38
78.96
71.34
49.78
33.83
56. 86.89
(RosNe150)
RankDiasRMAA
84.21
79.75
72.28
50.70
56.52
43.67
PSPNet
Argmas prob
65,48
75.57
89.18
80.78
73.07
65.89
50.45
73.55
51.32
37.42
58.66
$4.44
(ResNet(D))
Ran
85.9%
76.64
89.74
N1.94
73.72
66.53
81.14
74.28
51.57
38.09
59.17
45.29
DeepLabV3+
Argmax--pon poob
84.19
73.96
88.11
79.31
73.55
65. 65.98
50.80
73.63
49.78
33.83
56.89
40.35
(ResNet50)
RankDice
84.79
74.49
34.28
57.19
Deeplabh'34 ah133
prgmax-piob
77.25
89.83
82.08
66.17
52.53
37.52
59.57
44,13
(ResNett01)
RankDiiee RMA
86,80
78.14
90.32
83.14
73.92
66.68
81.24
74.33
$2.64
38.14
59.95
44.85
SegFormer
Argmux- prob
85,40
75.85
89.21
81.13
73.24
65.57
50.19
73.16
53.03
38.19
60.06
44.83
(MITR2)
RankDis--RMA
85.85
76.01
W9.44
81.04
73.81
66.41
74.13
53.67
39.09
61.09
SegFormer
Argmaax-pood
86.86
77.57
90.11
82.15
73.32
66.13
73.65
54.09
40.00
61.03
(MiTB4)
RankDive RMA
87,28
78.59
90.56
83.22
74.10
67.14
81.33
74.74
54.72
40.82
61.92
47.57
UPPerNet
Argnax-prooh
87.82
79.52
91.01
84.11
75.66
68.83
82.61
76.08
56.94
42.86
63.98
49.61
(ConvNeXi)
RaniDice-RMA RMA
88.25
80.31
91.48
84.98
76.17
69.57
83.21
76.97
$7.67
43.84
64.93
50.85
CPT
Argmax- prob
88.56
80.74
91.62
85.18
75.33
68.39
82.25
75.74
57.85
41.59
64.75
51.27
(Swin-Jargo)
RenkDie--MMA
88.89
92.01
76.76
58.63
45.56
65.83
52.58
Table 2: Performance for different prediction methods with various models on LiTS and KiTS.
||
LiTS
KiTS
LITS
KiTS
Prediction
Model
Model
lou!
Dice!
lot!
Dice
loU!
Dice!
loU!
Dice!
Argmax-prob Rank BA
DeepLabV3+
34.31
42.81
54.61
47,20
UNet
36.40
45.18
56.03
49.28
(ResNet18)
36.11
45.04
58.00
50.57
(ResNet18)
38.34
47.54
59.10
52.08
Bane-DiceRMM
36.12
45.04
58.00
50.57
38.34
47.54
59.10
$2.08
Argmax-prob RankDicc-BA
Deepl a+
38.45
47.58
61.16
34.19
UNet
38.45
47.58
37.36
$1.00
(ResNNe500
40.09
49.50
63.56
56.22
(ResNet50)
40.71
50.08
60.07
50.34
SankDice-RMa
40.09
49.50
63,56
56.22
40.70
50.07
60.07
53,54
Table Time consumption (in seconds) of model forward and different prediction rules with single A100 GPU. DeepLabV3+ (ResNet50) is used for the medical datasets, while UPerNet (ConvNeXt) is
used for the others. The mean and standard deviation over 10 runs are reported. X indicates that the method is not applicable due to non non-overlapping benchmark setups.
Pascal VOC
Cityscapes
ADE20K
LITS
KITS
Argmax-prok prob RankDice-RMMA RankDice-BA
0.05 (+0.01)
0.22 (+0.11)
0.43 (+0.08)
0.01
(10.00)
0.01
(+0.00)
6.93 (+1.14)
10.15 (+1.77)
58.00 (+3.44)
0.34 (+1.19)
0.26 (+0.15)
X
X
X
16.33 (+1.19)
9.99
(+0.15)
Model forward
40.77 (+5.155
175.81 (+3.02)
324.59 (= -13.42)
14.15 (+0.64)
11.86
(+0.20)
Results for PASCAL VOC, Cityscapes, and ADE20k are presented Table 1, while those for LiTS and KITS are shown in Table 2. The best performance within each model is highlighted in bold, while the best across all models is highlighted in pink. If two performances are very close and both are the best, we highlight both. Three observations can be drawn from these results.
7

---

Our proposed method significantly outperforms the conventional Argmax-prod probb across all datasets and models, irrespective of light or heavy backbones, demonstrating effectivenesss and robustness. For instance, on Cityscapes with SegFormer (MiTB4), MankDice-RMA. im- proves mDice' and mDicc by 0.85% and 1.09%, respectively. addition, on LiTS with UNet (ResNer50), RankDice-RMAA outperforms Argmax-proh by 2.49% in Dice
As shown in Tables 2 and 3, RankDice-RMM achieves significant time efficiency improvements over RankDice-BA while maintaining similar performance the LiTS (48x speedup) and KiT'S datasets (38x speedup). Hence, we conclude that RankDice- RAM is a strict improvement over RankDice-BA. Although RankDice-RMM consumption is negligible compared to the model forward time. In contrast, such argument can not be applied to RankDice-BA whose time consumption is comparable to the model forward.
is slower than Argmax-prob, the absolute time
RankDice- RMA simultaneously boost IoU performance, even though it is originally moti- vated by the Bayes rule for Dice. Furthermore, RankDice-RMA and RankIoU-RM achieve nearly identical performance across all experiments (results for RankloU-RMA simplicity). suggesting that the two metrics are closely related and that either RankDice-RMi. or RankloU-RMA can serve unified prediction method for both metrics.
omitted for
4.3 Class-wise performance
To further evaluate the performance of RankDice- RMM,,, we report class-wise results on PASCAL VOC in Table 4. Improvements over regmax-prob are highlighted in green for positive changes and red for negative ones. The results indicate that RankDice-RMM consistently enhances performance across most classes. Two key observations can be made:
The performance gains are more pronounced for classes with lower baseline performance, suggesting that RaMA is particularly effective for difficult classes. For instance, the Chair class, which exhibits low IoU of 48.05% under Argmax-prob, enhancement of 2.47%; whereas the treoplane class, with high initial IoU of 90.39%, only sees a marginal improvement of 0.45%. This trend may result in negative changes for classes like Bird and Sheep, where Argmax-prol already performs well, leaving limited room for improvement with the Bayes rule.
boosted to 50.52%,
Although the error bound in Theorem 2 implies a larger approximation error for classes with smaller volume, the results show that RankDice-RMN RMA still achieves substantial performance gains for these small objects. For example, our analysis indicates that Bottle and Chair are among the smallest objects in the dataset. Nonetheless, these classes exhibit significant improvements, possibly because the benefits of the Bayes rule outweigh the approximation error.
Table 4: Class-wise IoU on PASCAL VOC with UPerNet (ConvNeXt).
Prediction
Aeroplane
Bicycle
Bird
Boat
Bottle
Bus
Car
Cat
Chair
Cow
Argmax-prob
90.39
50.33
91.18 90.86
81.19 81.70
69.21
89.55
78.78
92.24
48.05
92.47
RankDice-RMA
90.84
51.76
71.77
90.31
80.46
92.43
50.52
92.69
(Improvment)
+0.45
+..43
-0.32
+0.51
+2.56
+0.76
+1.68
+0.19
+2.47
+0.22
Prediction
Dining Table
Dog
Horse
Motorbike
Person
Potted Plant
Shoep
Sofa
Train
TV Monitor
Argmax-prob
$4.12
93.55
91.21
89.51
82.26
$7.11
92.59
62.09
91.97
77.25
Ran
55.28
93.56
91.34
89.80
83.55
59.20
92.54
62.88
92.17
78.05
(Improvmmen))
+1.16
+0.01
+0.13
+0.29
+1.29
+2.09
-0.05
+0.79
+0.20
+0.80
4.4 Worst-case analysis
For safe-critical applications, it crucial evaluate the worst-case performance of segmentation models. In this context, image-level metrics provide more detailed insights than dataset-level metrics for assessing worst-case scenarios [Wang al., 2023a]. Without loss of generality, consider mloUf mloU? v .... & mloU", denote the sorted image-level mloU values for images in test. set. We define the average mIoU over those below the lowest q-th quantile as:
[ng]
m
m
nqq
8

---

By definition, this metric quantifies performance of the worst q-th quantile images. Table 5 presents the mloU and 10 results, where UPerNet(ConvNeett) is used for PASCAL VOC, Cityscapes. and ADE20K, while Deepl ab ( is used for LITS and KiTS. The results demonstrate that our method also improves the worst-case performance across all datasets.
Table 5: m and mloU on PASCAL VOC, Cityscapes, ADE20K, LiTS, and KiTS.
m
mloU
Prediction
VOC
Cityscapes
ADE20K
LiTS
KITS
VOC
Cityscapes
ADE20K
LITS
KiTS
Argmax-prob
44.80
59.02
24.99
2.13
6.72
52.10
61.72
29.54
4.05
8.39
RankDice-RMA
46.21
59.96
25.56
2.70
8.49
53.17
62.51
30,35
4.81
10.38
(Improvement)
+1.41
+0.94
+0.57
+0.57
+1.77
+1.07
+0.79
+0.81
+0.76
+1.89
Imago-wise IoU
Imap
Grrund inch
Arynus-peob
Rank RM-A
Ioli
Image
Figure 2: Image-wise performance and an example of worst-case segmentation on KiTS. The left plot presents the IoU for each image, with indices sorted in ascending order according to IoU under Argmax-pros The right plot displays the segmentation results for slice of a worst-case image.
Figure shows image-level IoU for a KiTS validation fold and a worst-case segmentation example. The left plot shows loU values for each image. It is evident that method outperforms Argmax- prob across most images, especially on difficult cases. The right plot displays segmentation results for worst-case image. The tumor consists of two adjacent segments, but Argmax-prob captures only small portion of the larger segment and almost misses the smaller one. In contrast, our method not only produces more complete segmentation but also successfully identifies the smaller segment. This example highlights the potential of our method for challenging clinical scenarios.
4.5 Ablation studies
Effect of the RMA-score. We have introduced the RMA-score to address Toverlap that occurs when applying RankDice for each class independently. We now demonstrate that the scores are indeed crucial for improving performance by comparing them with two ad-hoc alternatives in (6):
Prob-scores. The predicted probability Pe., is directly used as score of pixel j for class c.
WProb-scores. As inspired by the RMA-scores or intuitive reasoning, classes with more already predicted pixels should have lower preference when resolving overlaps. Hence, weighted version of the predicted probabilities is considered, i.e., & Pe.j/ /Zal.
As shown in Table 6, WProb-scores outperform Prob-scores on Pascal VOC and Cityscapes, sup- porting the intuition to account for predicted volume. However, WProb-scores underperform on ADE20K. indicating that simple weighting fails when many classes are present. In contrast, RMA-scores consistently perform best, particularly ADE20K, where overlapping phenomena more complex due to the large number of classes. This superiority is due to that RMA-scores are derived from the Bayes rule, making them more principled than heuristic methods. These results support our claim that RMA-scores are essential for improved performance.
Table 6: mloU! of using different scores.
Pascal VOC
Cityscapes
ADE20K
Prob-scores
87.83
75.75
$6.96
WProb-scores
88.17
75.89
56.75
RMA-scores
88.25
76.17
57.67
Effect of different bounds in RMA. Recall that Theorem 2 provides both lower and upper bounds under RMA, with lower bound being preferred for simplicity. As complement, we further find that using the upper bound as an alternative approximation yields same performance. This suggests that the choice different bounds does not bother. More importantly, this confirms that the bounds are tight, aligning with theoretical analysis in Theorem 2.

---

Conclusion
In this paper, we propose RankSEG-RMA, a novel segmentation algorithm that grounds on the Bayes rule, and enjoys computational efficiency by using reciprocal moment approximation (RMA). Extensive experiments across various datasets and models demonstrate that RankSEG- performs the conventional Argmax-prob and significantly reduces computational cost compared to the existing RankSEG-BB B.. Nevertheless, two limitations are noteworthy for future improvement. First, the proposed overlap resolution method predicts each pixel independently. which may be optimal; future work could explore more global approaches while maintaining computational efficiency. Second, work builds upon the assumption of conditional independence in Bayes rule, which could be relaxed in subsequent research.
out-
Acknowledgments
We thank the anonymous Area Chair and reviewers for their valuable feedback, suggestions and support. This work was supported by the Hong Kong RGC-ECS Grant 24302422 and Hong Kong RGC Grant 14304823.
References
Peter L Bartlett, Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-55) 2006.
Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: tractable surrogate for the optimization of the intersection-over-uni measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4413- 4421, 2018
Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, al. The liver tumor segmentation benchmark (lits). Medical Image Analysis, 84:102680, 2023.
Min-Te Chao and WE Strawderman. Negative moments of positive random variables. Journal of the American Staristical Association, 67(338):429 431, 1972.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint wxXiv:/706.05887 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 801-814 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3213-3223, 2016.
Ben Dai and Chunlin Li. Rankseg: consistent ranking- based framework for segmentation. Journal of Machine Learning Research, 24(224):1-50, 2023.
Krzysztof Dembezynski, Arkadiusz Jachnik, Wojciech Kotlowski, Willem Waegeman, and Eyke Hüllermeier. Optimizing the f-measure in multi-label classification: Plug-in rule approach versus structured loss minimization. In International Conference on Machine Learning, pages 1 1130-1138. PMLR, 2013.
Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, and Matthew B Blaschko. Optimization for medical image segmentation: theory and practice when evaluating with dice score or jaceard index. IEFE Transactions on Medical Imaging, 39(11): 3679-3690, 2020.
10

---

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 303-338, 2010.
Di Feng, Christian Haase-Schütte Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic seg- mentation for autonomous driving: Datasets, methods, and challenges. IEEE Trunsactions Intelligent Transportation Systems, 22(3::331-1166) 2020.
Jessica Finocchiaro, Rafael Frongillo, and Enrique B Nueve. The structured abstain problem and the lovász hinge. In Conference on Learning Theory, pages 3718 37400. PMLR, 2022.
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-375 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
Nicholas Heller, Niranjan Sathianathen, Arveen Kalapara, Edward Walczak, Keenan Moore, Heather Kaluzniak, Joel Rosenberg, Paul Blake, Zachary Rengel, Makinna Oestreich, al. The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes. arXiv preprint arXiv: Ttt::/90: 004.00445, 2019.
Nicholas Heller, Fabian Isensee, Klaus Maier-Hein, Xiaoshuai Hou, Chunmei Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, et al. The state of the art in kidney and kidney tumor segmentation in contrast- contrast-enhanced ct imaging: Results of the kits challenge. Medical Image Analysis, 67:101821, 2021.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4404-9413, 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
Tae-young Ko and Seung-ho Lee, Novel method of semantic segmentation applicable augmented reality. Sensors, 20(6):1737, 2020.
Harold W Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83 1955.
Yi Lin. A note on margin-basee based loss functions classification. Statistics & Probability Letters, 68 (1):73-82, 2004.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23592-23601, 2023.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 1 11976-11986, 2022.
Jun Ma, Jianan Chen, Matthew Ng, Rui Huang, Yu Li, Chen Li, Xiaoping Yang, and Anne L Martel. Loss odyssey in medical image segmentation. Medical Image Analysis, 71:102035, 2021.
Anqi Mao, Mehryar Mohri, and Yutao Zhong, Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine learning, pages 23803-23825 PMLR, 2023.
Sebastian Nowozin. Optimal decisions from probabilistic models: the intersection-over-uiio case. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 548-555, 2014.

---

Dian Qin, Jia-Jun Bu, Zhe Liu, Xin Shen, Sheng Zhou, Jing-Jun Gu, Zui-Hua Wang, Lei Wu, and Hui-Fen Dai. Efficient medical image segmentation based on knowledge distillation. IEEE Transactions Medical Imaging, 40( 40(12):3800-3811 2021.
Md Atiqur Rahman and Yang Wang. Optimizing intersection-over-umee in deep neural networks for image segmentation. In International Symposium on Visual Computing, pages 234-244. Springer, 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister Intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin. and M Jorge Cardoso. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3, pages 240-248. Springer, 2017.
Quan Tang, Chuanjian Liu, Fagui Liu, Jun Jiang, Bowen Zhang, CL Philip Chen, Kai Han, and Yunhe Wang. Rethinking feature reconstruction via category prototype in semantic segmentation. IEFE Transactions on Image Processing, 2025.
Ambuj Tewari and Peter L Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research, 8(5), 2007.
Zifu Wang. Maxim Berman, Amal -rikin Philip Torr, Devis Tuia, Tinne Tuytelaars, Luc V Gool, Jiaqian Yu, and Matthew Blaschko. Revisiting evaluation metrics for semantic segmenta- tion: Optimization and evaluation of fine-grained intersection over union. Advances in Neural Information Processing Systems, 36:60144-6022. 2023a.
Zifu Wang. Teodora Popordanoska, Jeroen Bertels, Robin Lemmens, and Matthew B Blaschko. Dice semimetrie losses: Optimizing the dice score with soft labels. In International Conference Medical Image Computing and Computer-Assisted Intervention, pages 475-485. Springer, 2023b.
David A Wooff. Bounds on reciprocal moments with applications and developments in stein estima- tion and post-stratification. Journal of the Royal Statistical Society: Series B (Methodological), 47 (2):362-371, 1985.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418-434, 2018.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 12077-12090. 2021.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155-18165 2022.
Language-awe -ware
Jaqian Yu and Matthew B Blaschko. The lovász hinge: A novel convex surrogate for submodular losses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(3)::35-7448 2018.
Tong Zhang, Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research, 5(Oct): 1251, 2004.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
pages
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 633-641, 2017.
12

---

Table of Contents
Segmentation calibration and RankSEG Framework
B RankloU-RMA
14
C Proof of Theorem
15
Training Details
16
E Additional Results
16
E.1 Statistical Significance Test
16
More Qualitative Visualizations
A
Segmentation calibration and RankSEG Framework
Given training dataset (xx, where E X, y; E V, loss function & 3 -> R, and hypothesis class 71- = {h: }), the empirical risk and population risk are defined as:
Re(h) - Σ ((h(x:), and Re(h) Ex.v (e()(X), Y)].
The empirical risk minimizer Top = argmin (h) is used for making predictions. However, often the case that the target loss function is neither differentiable and nor convex, such as the zero-one loss classification or the negative of /Dic! in our case, making direct optimization infeasible. Therefore, surrogate loss function @ Z y + R, combined with surrogate hypothesis class [= X + Z} and a decoding function (also known as link function) d + Y, is typically employed:
f argmin R and h = do JEF
Note that the surrogate loss is designed to be easier to optimize than the original loss, the output space the surrogate hypothesis may not align with the label space, and the decoding function maps the surrogate prediction back to the original label space. The desired property of the surrogate loss is calibrated, as specified in Theorem A.I.
Definition A.1 (Calibration). A surrogate loss Φ associated with decoding function d, is calibrated with respect to a target loss if, for any distribution over X and any sequences {fmheN C F.
the following holds:
(Ro(fn) + inf Ro(l) fEF
= (Re(d fw) inf Re(h)) WER
as
n
For example, hinge loss with sign as decoding function and cross entropy loss with argmax as decoding function calibrated with respect zero-one loss in binary classification and multiclass classification, respectively [Lin, 2004, Zhang, 2004, Bartlett et al., 2006, Tewari and Bartlett, 2007, Mao al., 2023].
In general, there are two principled approaches to achieving calibration or consistency: (1) designing consistent surrogate loss function and making predictions via suitable decoding function [Bartlett et al., 2006, Tewari and Bartlett, 2007], and (2) directly deriving the Bayes rule for target metrics and plugging in the estimated probabilities for prediction [Nowozin, 2014, Dembezynski et al., 2013, Dai and Li, 2023].
RankSEG [Dai and Li, 2023] belongs to the latter category. It does not require a carefully designed surrogate loss function and can directly applied models trained with cross-entropy loss; however, the decoding step is more involved. Nevertheless, Dai and Li [2023] demonstrate ranking property of the Bayes rule, stating that the optimal prediction is to select the top r" pixels with the highest conditional probabilities, which significantly simplifies the decoding step.
13

---

B RankloU-RMA
Theorem B.1 (The Bayes rule for loU segmeentttnot [Dai and Li, 2023]). Assume that Y; 1 X
A segmentation rule 8' is global maximizer of (o))) if and only i (p; 2 Pj. and jr is the index with T-th largest probability. The optimal volume T" is given by:
argmax V(J+) with v(J+) rE(0,1,-..d .d]
Σ Pj JEJ,
1
(8)
T T+
where J+ (j' Py Pje is the index set of the top conditional probabilities with To 0. is boisson-binomial random variable with By being a Bernoulli random variable with success probability Pyt
and r -J, = Lj' By.
According to Theorem B.I, the Bayes rule for IoU' shares substantial similarity with that of Dice', both of which consist of two parts: (1) ranking the conditional probabilities and (2) selecting the top 7' pixels positives. The primary difference lies in the computation of score functions when determining of the optimal volume 7", which is tailored to the respective metric. The consistency of RankSEG [Dai and Li, 2023, Lemma 10] is established by plugging in the estimated probabilities (x; 0), where 0 is the model parameter trained by minimizing a strictly proper loss [Gneiting and Raftery, 2007].
Note that replacing 1'-7. with I' in Theorem B. leads to large approximation emoor, especially when is large. Therefore, Blind approximation is longer applicable in this case. However, RMA technique can still be employed to approximate v(J,):
PRMM (J+)
Σ JET,
Pj
T + E(T-5.)
Based on this, we develop RankIoU-RMM for binary segmentation, described in Algorithm 3. This algorithm is highly similar RankDice-RMA, with the only difference being the use of the target function ((T).
Algorithm 3 RankloU-RMA RMA-Bna
Input: Estimated probability map P E [0, 1jd.
Output: The predicted segmentation mask E 10.
1: Rank probabilities in descending order, yielding P,: 2 2... Pje:
2: Prepare cumulative sum of top probabilities and mean of Poisson-binomia. binomial
Tr
Pj.
for E [d],
Am
j=1
3: Compute TRMA
TRMA((7) (T.)
2. 7+ 4r) for [d], according to (9).
4: Determine optimal volume 7 = argmax (R,,((F))
5: Make prediction by 8, = 1pp 2 Pj-- for E [d].
In order to extend RankloU-RMM to non-overlapping multiclass segmentation, it suffices to use the following R.M--scrrss for IoU, followed by argmax resolve overlaps:
Paái + Σke 2. Pe.k
ΣxEr, Pack
i2y = u (1)) ((i.)= [nl + (Ni Pa.j Exer, Peck) [Tel + (Ti Ebe2, (10)
where I. is the index set of pixels assigned to class and füe EE= Peg. The second term in (10) approximates the loU when predicting mask by 2.,, while the first term approximates the loU when pixel further included. Similarly, Algorithm can obtained by simply replacing the RMA-scoree used in Miw.DiiccRMM-MMltiilll
14

---

Algorithm 4 RankloU-RMA-Multicaas
Input: Estimated probability map P E [0, 1[cxd
Output: The predicted segmentation mask E [C]4.
1: /+ Obtain overlapping segmentation mask +/
2: for = to do
3: 3 = RankIoU-RMA RMA--UU--MAAAA Rww.---MA---------- (p.).. {j Wes 1}.
4: end for
5: /+ Resolve overlapping by argmax over RMA-scores +/
6: Identify overlapping indices, Toverlap = Uete (Ix² 0 It)).
7: for c= 1 to C do
8: Discard assignments for overlapping pixels, Ie It. I
9: Accept prediction for not overlapping pixels, 8, = c for j E Ie.
10: end for
11: Compute RMA-scores, Das via (10) for E T and e E [C].
12: Resolve overlapping by argmax, 8, argmax,ejcy Des for j Toverlap
13: Return
C Proof of Theorem 2
The following two lemmas are used in the proof of (3) in Theorem 2.
Lemma C.1 (Chao and Strawderman [1972]). Let a E R and X be random variable such that X a > a.s. Define the probability generating function of X as x Bttx) for s s
Then,
L'' Gx Gx(u)a--dd
E
X+a
Lemma C.2 (Wooff [1985).. Let A ~ Bin(d, p) be binomial random variable. Then, for any >> the following inequalities hold:
E (At) (d + 1)p +a
Note that binomial random variable A Bin(d, p) and PB(pI,P2, Po, ... Pa) have probability generating functions:
random variable I
Gt(t) = (1 - pt and Gr(t)= = II
(1 P, + ptt).
j=1
Now are ready to prove Theorem 2.
Proof. We first prove (3). The lower bound that (ET +T)-1 & +T) follows from the Jensen's
inequality. Let A ~ Bin(d, p), where p = d=1 2j=1 Pý. To prove the upper bound, we have:
Gr(( = / r--1 II - Ps +p,t) dt
E
r+T
/ Ga( = E(
t 1-1((1 - 4+ b dt
A+7
(
The first and last equalities follow from Theorem C.I. The first inequality is due to the arithmetic and geometric means inequality, and the last inequality follows from Theorem C.2.
15

---

To proceed with (5), we first establish an error bound for RMA. Let I be a Poisson -inoomil random
variable and y 2 1. Then, we have:
+(T [ET + -) _ +
ET +
ET + +-)=
d
1
I (ETl + 1--11)- - (ET
-イ)ー
S (ET + -)- (11)
(ET + 1) +7)
For any I [ [d]. the error bound of RankDice- then given by:
1
1
|RRMA (I) - (I) -
2pj
B
+ ET +1
JEI
Σ
2pj
B(
1
1
+
r +
T T ET
T ++ ET ++
++ ET +1
JEI
1
2p
B(
+
-
T ET -j +
T ET - +1
J +1+1
JEZ
1
Dj
2pj
(T +
ET
-j ++11 1)2
(T + ET
+ +)(T + ET +1)
JEI
ΣjEr 2p, (1 + Pp)
> (T ET)
2pj
Pj (T
S
ET)
(T + ET)
T ET
JEI
Here, the third inequality follows from (11). The last inequality is because ΣJEr Pj S | = 7 and
EjErD & Ej = ET.
D Training Details
The training settings mainly follow Wang et al. [2023a,b]. For Pascal VOC, Cityscapes and ADE20K, AdamW optimizer with weight decay of 0.01 is used. The learning rate starts from le 6 and linearly warms up during the first 1% iterations to the initial learning rate 6e - 5. The learning rate is then decayed in a "poly" policy with an exponent of 1. The number of warm-up iterations is 400 for Pascal VOC and Cityscapes, and 800 for ADE20K. The total number of training iterations is 40,000 for Pascal VOC and Cityscapes, and 80,000 for ADE20K. Data augmentations including (i) random scaling in the range of [0.5, 2.0]. and (ii) random horizontal flipping with probability of 0.5.
For LiTS and KiTS, we train the models using SGD with an initial learning rate of 0.01, momentum of 0.9, and weight decay of 0.0005. The learning rate is decayed in a "poly" policy with an exponent of 0.9. The batch size is 8 and the number of epochs is 60. These two datasets are originally multi-class segmentation tasks, but convert them into binary segmentation by only treating the tumor as foreground. This because we want to compare our method with RankDice-BA, which only applicable to binary segmentation. Furthermore, since LiTS and KiTS do not include designated test sets, we employ 5-fold ross-validation to evaluate performance, following existing literature [Qin et al., 2021].
E
Additional Results
E.1 Statistical Significance Test
To validate the statistical signifi- cance of the performance improve- ment achieved by RankDice-RMA over Argmax-prob, we conduct 10 independent runs with different ran- dom seeds using UPerNet on VOC, Cityscapes, and ADE20K datasets.
Table 7: Mean performance in ' and p-values from t-tests between RankDice-RMM. RMA and Argmax-prov
Pascal VOC
Cityscapes
ADE20K
Argmax-prob
87.80 0.12
75.63 0.04
56.88 + 0.09
RankDice-RMA
88.16 0.11
76.11 + 0.07
57.67 + 0.11
p-value
1.12c-6
2.30c-13
5.96c-13
16

---

We report mean and standard devia- tion of mloU", along with the p-values from t-tests, shown in Table 7. The results indicate that the improvements are statistically significant, with p-values far below 0.01 across all datasets.
method not only achieves substantial improvement in the sense of mean performance, but also consistently outperforms Argmax-prob in every single run. For example, the results of the runs ADE20K are presented in Table 8. This consistency arises because our method is deterministic and introduces no inherent randomness. It is applied to trained models by simply replacing semmax-proh in the prediction step. Consequently, the comparison is highly stable as they share the same model.
More importantly,
Table 8: mloU! of 10 independent runs on ADE20K.
Run Argmax-prob
4
56.84
56.86
56.93
57.01
56.77
56.94
57.01
56.80
56.84
56.77
RankDice-RMA
57.56
57.66
57.71
57.86
57.59
57.73
57.84
57.59
57.68
57.52
E.2
More Qualitative Visualizations
Ground both
Argane-prch
RaNSEG-AMM
imagc
Ground truhh
Argmax
RankSEG RMA
I:
Graund Irahh
Augroe-aadd
RarASEG-RMA
Figure 3: Qualitative visualizations on Pascal VOC. From left to right: input image, ground truth. prediction by Argmax-prob, and prediction by RankSEG-RMA, The key differences are highlighted red boxes.
We provide additional qualitative visualizations in Figure to compare the proposed RankSEG-RMA with the conventional Argmax-prob method, offering further insights into how our approach enhances segmentation quality. highlighted by the red boxes in the figure, RankSEG- SEG-RE--MAA outperforms Argmax-prob primarily in capturing complete regions of challenging objects and in detecting small objects.
For instance, in the first row, where the buses are partially occluded, Argmax-prob only sparsely identifies small portion of the buses, whereas RankSHG-RMA achieves complete segmentation. In the second example, Argmax-prok fails to detect the small bottles on the table, but RankSEG-RMA successfully identifies them. Similarly, in the third example, Argmax-prob completely misses one human face, while RankSEG-RMA detects it. These examples, together with the discussions in Sections 4.3 and 4.4, demonstrate that RankSEG-RMA particularly effective for segmenting small and challenging objects.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in our abstract and introduction that proposed RankSEG- RMA significantly reduce computation cost and enable non-overlapping segmentation accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: The assumptions are clearly stated in the corresponding theorems. The proofs are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The implementation details are provided in Scction 4 and Section D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets we used are open-source datasets. The code is available in https:/ httpp:///anommouss.o 4open.sciieeeeee...............e scciencerr/RantSS--RRR -CC444
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.aa cttpssscccppccccccccpcccccccccct c/guides /
CoodeSubmis ssi aasionPoiccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The implementation details provided in Section 4 and Section D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table 3 provides the standard deviation of 10 runs. Other experiments do not suffer from randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The time consumption and used resources are provided in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /wwrrrerrrp...// hetppuli//ttisssuiddeeeooo.eeeee
Answer: [Yes]
Justification:
We have ensured that our research conforms the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose such a risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all external sources of assets and their licenses permit our use case.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs not used for method development. They are only used for writing polishing.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : http://een....eeeee. http///rpps.c/ooofeeeeeeeeeeeeee cc///oofeeeencce/22/111111 for what should or should be described.
24


Reviews:
[{"review_id": "nXeTmgAdxZ", "reviewer": "Reviewer_imyv", "summary": "The paper proposes RankSEG-RMA, an improved version of the RankSEG framework for semantic segmentation.\n\nIts core idea is to significantly reduce computational complexity via reciprocal moment approximation (RMA), while preserving theoretical consistency in optimizing Dice/IoU metrics. It also introduces a pixel-wise scoring function for non-overlapping multiclass segmentation.\nExperimental validation is conducted on natural image datasets (Cityscapes, PASCAL VOC, ADE20K) and medical image datasets (LiTS, KiTS). Compared to the traditional argmax scheme, the proposed method achieves modest improvements in average IoU/Dice and worst-case quantile performance.", "questions": "Please refer to the weaknesses and those are my main concerns. I have two minor questions and these questions would not influence my score: \n\n1. Could RankSEG-RMA be extended to instance segmentation or panoptic segmentation tasks?\n2. Can the authors quantify Dice/IoU discrepancies caused by the RMA approximation compared to original RankSEG? Could certain extreme probability distributions (for example, long-tail) negatively impact segmentation quality due to approximation?\n\n**Overall, if I misunderstand some details about this paper, please let me know.** I am looking forward to discuss with authors in rebuttal phase.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Overall, I think this paper demonstrates a high degree of originality, is supported by solid theoretical analysis, and holds strong practical value. It brings a new way for adapting RankSEG into semantic segmentation with less computation time.\n\n## Strengths\n\n1. The paper builds on the theoretical foundation of RankSEG, inheriting and strengthening theoretical consistency with Dice/IoU metrics. By employing a Bayesian segmentation rule, the authors ensure convergence to the optimal Dice/IoU solution in infinite-data scenarios, a guarantee that standard cross-entropy + argmax frameworks cannot offer. Compared to surrogate-loss-based empirical optimization, this method is more theoretically sound, highlighting high originality and quality.\n\n2. The authors cleverly propose Reciprocal Moment Approximation (RMA) to approximate IoU/Dice expectations, reducing algorithmic complexity from O(d^2) to O(d). I think this results in significant speedups (dozens of times) without considerable loss in accuracy.\n\n3. The paper addresses RankSEG’s original limitation of only supporting overlapping labels by introducing a pixel-level scoring function for non-overlapping multiclass segmentation. This strategy makes the method directly applicable to mainstream semantic segmentation benchmarks, which broadens the applicability compared to the original RankSEG.\n\n4. As RankSEG-RMA only modifies the inference step without altering model training, it can easily integrate into existing segmentation model deployments as a performance-enhancing post-processing module. This plug-and-play nature is highly appealing in industrial applications, highlighting the method’s potential significance in practice. In the experiments, the authors also demonstrate this by adopting to different methods.\n\n\n## Weaknesses\n\nDespite the above strengths, the following issues/weaknesses/concerns should be taken seriously by the authors, and I'm looking forward to the response from authors:\n\n1. Although theoretically optimal, actual empirical improvements in average metrics are relatively modest (generally within 1%).\nFor instance, the mIoU improvement on Cityscapes is less than 1%, with some metrics even slightly decreasing in certain datasets.cImprovements in worst-case quantiles are also modest (around 1%). So, I wonder that whether the complexity introduced by the method is justified by the incremental gains achieved. Such limited improvement may not always be sufficient to researchers to replace existing solutions, somewhat weakening practical significance.\n\n\n2. In my opinion, the theoretical optimality proof relies on a crucial assumption: **pixel labels being conditionally independent given the image**. However, adjacent pixels typically exhibit strong correlation (region coherence in segmentation segmentation tasks). This independence assumption simplifies theoretical derivation but is unlikely to hold in real scenarios, thus it is only approximately optimal I think. In this paper, the authors do not discuss how violating this assumption might affect method performance (Emmmm, maybe they discussed, so please point out if I missing this).\n\n3.  The introduced RMA-score for non-overlapping segmentation is essentially heuristic and does not guarantee globally optimal multiclass segmentation. Pixels are simply assigned to the highest-scoring class, a strategy close to argmax probability with adjusted scores. Thus, my concern is that In some extreme cases (e.g., very close probabilities between two classes with complex global IoU trade-offs), this greedy approach might fail to deliver globally optimal Dice/IoU. After reviewing this paper, I think it lacks rigorous analysis of the optimality or approximation ratio in multiclass scenarios, only providing empirical evidence of effectiveness.\n\n4. I also think there maybe a fairness issue. While performance on small objects improves, optimizing IoU pixel-wise inherently biases toward majority classes or large regions. RankSEG-RMA itself does not explicitly address class imbalance problem (long-tail problem)\nFor rare classes (such as some classes in LVIS), ranking optimization might be limited by scarce positive pixels.\n\n\n5. Some equtions are hard to understanding. I try my best to understand the formulas and the reasoning behind the derivations. Therefore, I suggest that the authors refine the presentation to improve clarity. Making the paper more accessible would broaden its impact and reduce the barrier to understanding.", "was_revised": true, "final_justification": "I think the response has resolved most of concerns. Considering the reviews from other reviewers, rebuttal discussions among reviewers and authors, I would keep my initial score and lean to an acceptance for this paper. Authors should update their paper according to these reviews.", "rebuttal": ">Q1.1 Although theoretically optimal, empirical improvements are relatively modest.\n\n**Reply.** Thank you for your comment. We summarize the absolute and relative gains in mIoU (with strongest backbones for argmax to avoid overstating improvements):\n\n| |VOC|Cityscapes|ADE20K|KiTS|LiTS\n-|-|-|-|-|-\nargmax|87.82|75.66|56.94|61.16|38.45\nAbs. Imp.|0.53|0.51|0.73|2.40|2.26\nRel. Imp.|0.60%|0.67%|1.27%|3.92%|5.71%\n\nWhile improvements are modest on easier datasets (VOC, Cityscapes: <1%), larger gains are obseved on more challenging datasets (ADE20K: 1.3%, KiTS: 3.9%, LiTS: 5.7%). Furthermore, improvements for difficult or small classes are substantial—for example, in VOC, *Bottle*, *Chair*, and *Potted Plant* improve by 2.56%, 2.47%, and 2.09% (see Table 4).\n\nIn summary, while the gains on easier datasets may be modest, they are noteworthy for harder datasets and challenging classes within easier datasets.\n\n>Q1.2 With some metrics slightly decreasing in certain datasets.\n\n**Reply.** Perhaps we misunderstood, yet Tables 1 and 2 show consistent gains across datasets. We would greatly appreciate it if you could point out the specific case.\n\n>Q1.3 Whether the complexity introduced is justified by the incremental gains.\n \n**Reply.** We would like to clarify that the **complexity introduced is minimal** for two reasons: a) The plug-and-play nature of RankSEG-RMA requires only minor implementation effort to deployment; b) The overall segmentation time increases by less than 10% on average, which is acceptable in practice.\n\nGiven these relatively low overheads and the improvements on challenging datasets and classes, we believe RankSEG-RMA provides a reasonable practical contribution.\n \n>Q2. The theoretical optimality proof relies on conditionally independence assumption.\n\n**Reply.** We agree that the conditional independence assumption (CIA) may be violated in some real datasets. We argue this assumption is still acceptable by following points:\n\na) **Already used in training**: CIA is **implicitly assumed when training with cross-entropy (CE) loss**, as it allows the joint likelihood to be factorized into pixel-wise terms. Without this assumption, the likelihood or CE formulation would be computationally infeasible. Specifically, the decomposed negative log-likelihood is:\n\n$$\nL_n(\\mathbf{q})=-\\log(\\prod_{i=1}^n\\mathbb{P}_{\\mathbf{q}}(\\mathbf{Y}=\\mathbf{y}_i|\\mathbf{X}=\\mathbf{x}_i))=-\\log(\\prod\\_{i=1}^n \\prod\\_{j=1}^dq_j(\\mathbf{x}_i)^{y\\_{ij}}(1-q\\_j(\\mathbf{x}_i))^{1-y\\_{ij}}) =\\sum\\_{i=1}^n\\ell\\_{\\text{CE}} (\\mathbf{y}_i,\\mathbf{q}(\\mathbf{x}_i)),\n$$\n\nwhere the second equality uses CIA, $\\mathbf{q}$ denotes the model, and $\\{(\\mathbf{x}_i, \\mathbf{y}_i)\\}\\_{i=1}^n$ is the training set. Thus, we does not introduce an additional assumption beyond standard setup. \n\nb) **Computational challenges**: Conditional dependence is hard to formulate practically, especially given the high dimensionality of segmentation. For example, even modeling pairwise dependence in a 512×512 image requires handling $512^4$ pairs, making computation infeasible.\n\nc) **Empirical precedent**: As seen in prior work, such as DeepLab v1/v2, modeling pairwise dependence with Conditional Random Fields was eventually abandoned in v3 [1] due to high computational cost and limited gains.\n\nIn summary, **CIA is a reasonable approximation once the image ($\\mathbf{X}$) delivers sufficient information for inferring $\\mathbf{Y}$, as the residual dependence in $\\mathbf{Y}$ (after removing information from $\\mathbf{X}$) becomes negligible**. Therefore, CIA remains a common and practical assumption due to its computational advantages. We agree that exploring local dependence is an interesting direction for future work.\n\n[1] Chen, et al. \"Rethinking atrous convolution for semantic image segmentation.\" ArXiv (2017).\n\n>Q3. About the heuristic and theoretical optimality of RMA-score.\n\n**Reply.** We agree RMA-score for non-overlapping is heuristic. We would like to offer following justifications for its use: a) As shown in Lemma 7 of [2], solving the Bayes rule under non-overlapping constraints is even harder than a nonlinear assignment problem (typically cubic complexity [3]), making it computationally infeasible and thus justifying the use of heuristics; b) The RMA-score has an intuitive interpretation as the expected increment in Dice/IoU according to Eq (7), providing a meaningful optimization criterion; c) It outperforms other heuristic alternatives such as probability or volume-weighted probability scores (see Sec 4.5), with gains of 0.43 and 0.39.\n\nTherefore, RMA-score offers a practical balance between theoretical motivation and computational efficiency, making it an effective choice for addressing overlaps.\n\n[2] Dai and Li. \"Rankseg: a consistent ranking-based framework for segmentation.\" JMLR (2023)\n\n[3] Kuhn. \"The Hungarian method for assignment problem.\" NRLQ (1955)\n\n>Q4. While performance on small objects improves, optimizing IoU pixel-wise biases toward large classes. RankSEG-RMA does not explicitly address class imbalance. For rare classes (in LVIS), ranking optimization might be limited by scarce positive pixels.\n\n**Reply.** Thank you for raising this important point. If we understand correctly, your concern relates to class size imbalance within an image (e.g., a small motorcycle versus a large car in a city scene).\n\nWhile RMA-score optimizes pixel-wise as in Eq (6) and Eq (7), it differs from argmax by considering the region size through the denominator terms $|\\widehat{\\mathcal{I}}\\_c|$ and $\\widehat{\\mu}\\_c$. This results in a score discount for larger regions, leading to **favor smaller regions** when appropriate. For example, even if $\\hat{p}\\_{c_1,j} > \\hat{p}\\_{c_2,j}$, our method could still assign $c_2$ to pixel $j$ if $\\hat{\\mu}\\_{c\\_1}< \\hat{\\mu}\\_{c\\_2}$. In contrast, argmax depends only on probability values and lacks this adaptivity.\n\nNumerically, our method also achieves greater gains on small classes, such as *Bottle* (2.56%) and *Chair* (2.47%), which occupy the fewest pixels in VOC. Therefore, we believe that for small-sized classes, RankSEG-RMA provides an improvement over argmax.\n \n>Q5. Some equations are hard to understand. I suggest refine the presentation to improve clarity.\n\n**Reply.** Thank you for your suggestion. We will add more explanations in the revision for clarity. Specifically, in Eq (2), the second equality expands the expectation $\\mathbb{E}(\\frac{2}{\\tau + \\Gamma + 1})$ into a sum of $d$ terms. This highlights the computational difficulty in the original RankSEG, and **motivates us to find an approximation for $\\mathbb{E}(\\frac{2}{\\tau + \\Gamma + 1})$ that is both easy to compute and as accurate as possible.**\n\nRMA (Theorem 2) addresses this by providing tight lower and upper bounds, both computable in $\\mathcal{O}(1)$. Specifically, from Eq (2) to Eq (4), where we apply RMA, the only change is replacing $\\mathbb{E}(\\frac{2}{\\tau +\\Gamma + 1})$ with $\\frac{2}{\\tau + \\mathbb{E}(\\Gamma)+1}$. This change reduces the expectation to a linear function of $\\Gamma$, rather than a non-linear one, yielding significant computational gains.\n\n>Q6. Could RankSEG-RMA be extended to instance/panoptic segmentation?\n\n**Reply.** Thank you for your question. In principle, RankSEG-RMA can be applied to any segmentation task given a (estimated) probability mask $\\mathbf{q} \\in [0,1]^{C\\times d}$. However, for instance or panoptic segmentation, where models often produce more complex outputs, additional processing would be needed.\n\nFor example, MaskFormer [4] produces a set of $(\\mathbf{p}_i, \\mathbf{m}_i)\\_{i=1}^N$ for an image, where $N$ is the number of mask candidates, $\\mathbf{p}_i \\in [0,1]^C$ denotes class probability, and $\\mathbf{m}_i \\in [0,1]^d$ denotes binary probability mask. To adapt our method, we could broadcast $\\mathbf{p}_i$ across every pixel of $\\mathbf{m}_i$ to form a probability mask $\\mathbf{q}_i \\in [0,1]^{C\\times d}$ with $q\\_{i, c, j}=p\\_{i,c}m\\_{i, j}$, and then apply RankSEG-RMA to obtain a segmentation mask. Overlaps between masks can potentially be resolved using our RMA scores or other heuristic methods in MaskFormer. We appreciate your question and will explore this potential in future work.\n\n[4] Cheng, et. al \"Per-pixel classification is not all you need for semantic segmentation.\" Neurips (2021)\n\n>Q7.1 Can you quantify discrepancies caused by the RMA compared to original RankSEG?\n\n**Reply.** We quantify the quality of the RMA both theoretically and numerically. Theoretically, Lemma 3 provides a theoretical error bound, which is tight for large mask volumes. Numerically, our method and original RankSEG yield nearly identical results (63.56 vs. 63.56 on KiTS; 40.71 vs. 40.70 on LiTS), confirming RMA’s practical accuracy. We will add more results and discussion in Table 2.\n\nFor multi-class segmentation, such quantification is not feasible since original RankSEG is not readily applicable.\n\n>Q7.2 Could extreme probability distributions (e.g., long-tail) negatively impact the quality due to approximation?\n\n**Reply.** To assess the impact of extreme distributions, we simulated a 5-class segmentation in a square grid, where pixels closer to the four corners and center are more likely to belong to classes 1-5. The logits at pixel $(h,w)$ for $c=1,\\cdots,5$ are:\n$$\\text{logits}(h,w)=\\exp(-d_c(h,w)/(H+W)),$$\n\nwhere $d_c$ denotes the distance to the four-corners (e.g., left-top) and the center. The exponential decay yields long-tailed logits. We then apply $\\text{softmax}(\\text{logits}/t)$ to obtain probabilities, where $t$ controls the long-tailness. **Larger $t$ implies a more uniform distribution, while smaller $t$ produces a more pronounced long tail by concentrating probability mass on top classes.**\n\nWith $H=W=32$ and a sample size of 3000, the results below show that our method remains effective under long-tailed distributions:\n\nt|0.7|0.8|0.9|1\n-|-|-|-|-\nargmax|13.78|13.32|12.96|12.63\nours|14.49|14.09|13.74|13.43"}, {"review_id": "yqMoS8k3Gd", "reviewer": "Reviewer_Aq2y", "summary": "This paper proposes RankSEG-RMA, an efficient semantic segmentation algorithm based on Reciprocal Moment Approximation (RMA). The algorithm effectively reduces the computational complexity of RankSEG from O(d log d) and O(d²) to O(d) while maintaining comparable performance and extending it to non-overlapping segmentation settings. Experiments demonstrate that RankSEG-RMA outperforms the conventional argmax rule across multiple datasets and models, with significant gains in small objects and challenging classes.", "questions": "please see weakness.", "limitations": "The paper compares RankSEG-RMA primarily with the argmax rule and RankDice-BA. However, it lacks a comprehensive comparison with other state-of-the-art segmentation methods that also aim to optimize IoU or Dice metrics. This makes it difficult to fully assess the novelty and superiority of the proposed method.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Adv:\n1. The introduction of Reciprocal Moment Approximation (RMA) effectively reduces the computational complexity of RankSEG, making it more feasible for practical applications.\n2. Experimental results show that RankSEG-RMA outperforms the conventional argmax rule across multiple datasets and models, particularly in small objects and challenging classes.\n3. The successful extension of the RankSEG framework to non-overlapping segmentation settings provides an effective solution for standard benchmarking.\n\nDis:\n1. The introduction of the RMA technique is somewhat complex, with detailed step-by-step explanations lacking. For example, in the proof of Theorem 2, the transition from the approximation of Poisson-binomial random variables to the final RMA formula is not intuitive for non-specialists (Section 3.1). More intermediate steps and illustrations are recommended to aid understanding.\n2. While some experiments provide standard deviations, there is no statistical significance testing for performance differences between methods. For example, in Tables 1 and 2, although mIoU and mDice values are listed, it is unclear whether these differences are statistically significant (Section 4.2). This makes it difficult for readers to assess the reliability of the performance improvements.\n3. What are the reasons for the different performance gains on different datasets of the model? I think a relevant clarification is in order.\n4. The model has very limited enhancement on some datasets and methods, for example, only 0.4% on the DeepLabV3+ model, PASCAL dataset, which leads to some doubts about the generality of the model and the validity of the proposed methods.\n5. Although the paper provides some implementation details, it lacks comprehensive information on how to reproduce the experiments, such as specific environment setups and exact commands. This could hinder the reproducibility of the results.", "was_revised": true, "final_justification": "The author has addressed most of my concerns.", "rebuttal": "> Q1. The introduction of the RMA technique is somewhat complex, with detailed step-by-step explanations lacking. For example, in the proof of Theorem 2, the transition from the approximation of Poisson-binomial random variables to the final RMA formula is not intuitive for non-specialists (Section 3.1). More intermediate steps and illustrations are recommended to aid understanding.\n\n\n**Reply.** We appreciate your feedback on this matter and will provide more detailed explanations in the revised version. \n\na) We first clarify the motivation behind using RMA. In the original RankSEG, according to Eq (2), the computational difficulty arises from expanding the expectation $\\mathbb{E}(\\frac{2}{\\tau + \\Gamma + 1})$ with respect to $\\Gamma$ into a sum of $d$ terms using the probability mass function.\n\n**Our main goal is to find a bound for $\\mathbb{E}(\\frac{2}{\\tau + \\Gamma + 1})$ that is both easy to compute and as tight as possible. To this end, RMA (Theorem 2) provides such a tight bound, and both the lower and upper bounds can be efficiently computed in $\\mathcal{O}(1)$ time.**\n\nb) Next, we provide further clarification on the proof of RMA. In the proof of Theorem 2, Lemma C.1 introduces a technique to express a reciprocal moment (RM) $\\mathbb{E}(\\frac{1}{X+a})$ in terms of the probability generating function (PGF) $\\mathbb{E}(t^X)$ for a random variable $X$. We then establish a connection between the PGF of the Poisson-binomial distribution and that of the Binomial distribution, using the arithmetic and geometric means inequality. \n\nSpecifically, denoted $\\Lambda(d, \\bar{p})$ a Binomial with $\\bar{p}=\\frac{1}{d}\\sum_{j=1}^dp_j$, we observe\n$$\n\\mathbb{E}(t^{\\Gamma}) = \\prod_{j=1}^d (1-p_j+p_jt) \\le (1-\\bar{p}+\\bar{p}t)^d = \\mathbb{E}(t^{\\Lambda}).\n$$\nThis allows us to relate the RMs of $\\Gamma$ and $\\Lambda$\n$$\n\\mathbb{E}(\\frac{1}{\\Gamma+\\tau}) = \\int_0^1 t^{\\tau-1} \\mathbb{E}(t^{\\Gamma}) dt \\le \\int_0^1 t^{\\tau-1} \\mathbb{E}(t^{\\Lambda}) dt = \\mathbb{E}(\\frac{1}{\\Lambda+\\tau}).\n$$\nFinally, we apply Lemma C.2, which provides a known upper bound for the reciprocal moment of a Binomial random variable, to conclude the proof.\n\n> Q2. There is no statistical significance testing for performance differences between methods. It is unclear whether these differences are statistically significant (Section 4.2)\n\n\n**Reply.** Thank you for your comments. We have now conducted 10 runs with different random seeds using UPerNet on ADE20k, Cityscapes, and VOC, and report the p-values from t-tests below.\n\n| | ADE20K | Cityscapes | VOC |\n|-|-|-|-|\n|Argmax-prob|$56.88\\pm 0.09$|$75.63\\pm 0.04$|$87.80 \\pm 0.12$|\n|RankSEG-RMA|$57.67\\pm 0.11$|$76.11\\pm 0.07$|$88.16 \\pm 0.11$|\n|*p-value*|5.96e-13|2.30e-13|1.12e-6|\n\nThe resulting p-values indicate that the differences are statistically significant. Moreover, **our method not only achieves a substantial improvement in the sense of mean performance, but also consistently outperforms the baseline in every single run**. For example, the results of 5 out of 10 runs on ADE20K are shown below:\n\n|run_id|1|2|3|4|5|\n|-|-|-|-|-|-|\n|Argmax-prob|56.84|56.86|56.93|57.01|56.77|\n|RankSEG-RMA|57.56|57.66|57.71|57.86|57.59|\n\nDue to time and computation resource constraints, we are currently unable to perform such statistical testing across all models, but we will make every effort to include them in the revision.\n\nWe would also like to note that the proposed RankSEG-RMA itself is deterministic and introduces no inherent randomness; it can be applied to any trained model by simply replacing the argmax-prob in the prediction step. As a result, the comparison between argmax-prob and our method is highly stable as they share the same trained model, which is further supported by the consistent improvements observed across different models and datasets.\n\n> Q3. What are the reasons for the different performance gains on different datasets of the model?\n\n**Reply.** Thank you for your insightful observation. RankSEG-RMA (7) is particularly effective for challenging classes (where class probabilities are close) and small objects (where region scale has a greater impact). For example, on datasets like PASCAL VOC, where the baseline (Argmax-prob) already achieves high performance (~85 mIoU) due to large object sizes, our method provides marginal improvements. In contrast, datasets such as ADE20K, which contain more small objects, and medical datasets (LiTS, KiTS) with inherent ambiguity and difficulty, demonstrate greater performance gains from RankSEG-RMA. Therefore, we believe that the inherent difficulty of the segmentation problem itself is a key factor influencing performance gains.\n\n> Q4. The model has very limited enhancement on some datasets and methods, for example, only 0.4% on the DeepLabV3+ model, PASCAL dataset\n\n\n**Reply.** Thank you for your comment. As noted in our response to your Q3, the limited improvement on some datasets is mainly because the baseline model already achieves high performance on these relatively easy datasets. In such cases, there is less room for further improvement. However, significant gains are observed on more challenging datasets, such as ADE20K and LiTS, as shown in the following table:\n\n|  | ADE20K | KiTS | LiTS |\n| - | - | - | - |\n| Argmax-pob | 56.94 | 61.16 | 38.45 |\n| Abs. Imp. | 0.73 | 2.40 | 2.26 |\n| Rel. Imp. | 1.27% | 3.92% | 5.71% |\n\nAdditionally, substantial improvements are also noted in difficult classes within easier datasets, with gains of 2.56% for *Bottle* and 2.09% for *Potted Plant*, as detailed in Table 4.\n\n\n> Q5. Although the paper provides some implementation details, it lacks comprehensive information on how to reproduce the experiments, such as specific environment setups and exact commands.\n\n**Reply.** Thank you for the comment. We appreciate the importance of reproducibility and will provide additional details in Appendix D to further facilitate replication of our experiments. As noted in the manuscript, we have included an anonymized GitHub link containing the codebase necessary to reproduce our results. In particular, the main algorithm can be found in `exp/rankseg_rma.py`. We will also add more comprehensive documentation and step-by-step instructions for running the experiments, and we will release trained model checkpoints to ensure that all results can be reproduced.\n\n> Q6. It lacks a comprehensive comparison with other state-of-the-art segmentation methods that also aim to optimize IoU or Dice metrics. This makes it difficult to fully assess the novelty and superiority of the proposed method.\n\n\n**Reply.** Thank you for your comment. We believe you are referring to loss functions such as soft-Dice and soft-IoU, which are designed to directly optimize Dice or IoU metrics. We provide a comprehensive comparison below to clarify the positioning of our work:\n\na) **Different roles in the segmentation pipeline**: Soft-Dice/IoU losses are used during the training phase to guide model optimization, whereas our method is a plug-and-play module that can be seamlessly integrated into a trained model at the inference stage.\n\nb) **Hyper-parameter tuning**: Perhaps due to the non-convexity of the Dice/IoU losses, they are typically combined with cross-entropy loss using a weighted sum, introducing an additional hyper-parameter. Tuning this parameter requires retraining the model, which is computationally expensive. In contrast, our method does not introduce any new hyper-parameters or require model retraining.\n\nc) **Probabilistic interpretation**: Producing reliable pixel-wise probabilities is crucial for uncertainty estimation [1], especially in high-stakes scenarios such as medical imaging. Models trained with soft-Dice/IoU loss do not output calibrated probabilities [1], locking their ability to quantify uncertainty. Our method is applied to models trained with cross-entropy loss, thus preserving this probabilistic interpretation.\n\nd) **Performance comparison**: It is acknowledged that soft-Dice/IoU losses are more effective when used independently. However, since these methods operate at different stages, we can still apply RankSEG-RMA after training with soft-IoU. Specifically, we consider $L=\\lambda L_{\\text{CE}} + (1-\\lambda) L_{\\text{soft-IoU}}$ with varying $\\lambda$ on ADE20k, and report the results as follows:\n\n|$\\lambda$|1|0.75|0.5|0.25|0|\n|-|-|-|-|-|-|\n|Argmax|56.94|59.13|60.00|60.71|54.70|\n|RankSEG-RMA|57.67|59.50|60.17|60.81|54.68|\n\nAs shown, RankSEG-RMA can still achive improvements when the model trained with the weighted loss (except for $\\lambda=0$). The performance gain decreases as $\\lambda$ decreases. This is because, as the soft-IoU loss becomes dominant (i.e., when $\\lambda$ is small), the model outputs become less calibrated and less interpretable as probabilities, which in turn reduces the effectiveness of our method.\n\n[1] Mehrtash, Alireza, et al. \"Confidence calibration and predictive uncertainty estimation for deep medical image segmentation.\" IEEE Transactions on Medical Imaging (2020): 3868-3878."}, {"review_id": "UNpDO38yYv", "reviewer": "Reviewer_xrAb", "summary": "This work introduces RankSEG-RMA, an extension of RankSEG that tackles its high computational cost and binary-class limitation by integrating Reciprocal Moment Approximation (RMA) to dramatically reduce complexity and introducing an RMA-score—an alternative to standard probability scores—for resolving overlapping regions in multi-class segmentation. Comprehensive experiments on diverse datasets and backbones demonstrate its effectiveness.", "questions": "- To further demonstrate the method’s generalizability, it would be beneficial to evaluate RankSEG-RMA's performance on more recent architectures, such as Mask2Former [r1] or CPT [r2]. Such experiments would provide a more robust assessment of its utility in the current state-of-the-art landscape.\n- There is a minor typo on line 176. The text reads \"class x,\" but it seems to be \"class c.\"\n- Line 69 points out that the original RankSEG is not suitable for panoptic segmentation. While the proposed RankSEG-RMA successfully extends its capability to multi-class semantic segmentation, its applicability to other segmentation tasks like panoptic or instance segmentation remains unclear.\n\n[r1] Cheng, Bowen, et al. \"Masked-attention mask transformer for universal image segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[r2] Tang, Quan, et al. \"Rethinking Feature Reconstruction via Category Prototype in Semantic Segmentation.\" IEEE Transactions on Image Processing (2025).", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- This work convincingly demonstrates that RankSEG-RMA not only enhances segmentation performance but also reduces computational complexity. The reported inference speed is substantially higher than that of the original RankSEG, marking a significant practical improvement.\n- The RMA-score is effective mechanism for overlapping areas.\n- The claims are well-substantiated with both theoretical derivations and comprehensive experimental analysis, lending strong credibility to the proposed method.\n\n**Weaknesses**\n- The adopted baseline, i.e., PSPNet, DeepLabV3+, SegFormer, UperNet, are actually outdated. It is strongly recommend adding one or two state-of-the-art methods to more rigorously validate the effectiveness of RankSEG-RMA.\n- The work could be further strengthened by including more qualitative visualizations. Side-by-side visual comparisons with RankSEG and argmax baseline would intuitively showcase the advantages of RankSEG-RMA.\n- The time consumed in multi-class semantic segmentation is still significantly greater than that of Argmax-prob as shown in Table 3.", "was_revised": true, "final_justification": "The rebuttal addresses most concerns. I am maintaining the current rating at 4.", "rebuttal": "> Q1. The adopted baseline models are outdated. It is recommend adding one or two state-of-the-art methods to validate the effectiveness of RankSEG-RMA, for example, Mask2Former or CPT\n\n**Reply.** Thank you for your valuable suggestion. Due to time constraints and several practical challenges, we are currently unable to include results for these models at this stage. Specifically: (a) Mask2Former outputs a set of candidate masks, which is structurally different from mainstream models. Converting its output into a standard probability mask suitable for our method would require additional effort. (b) Since CPT is a very recent model, we need more time to thoroughly review their documentation and detailed implementations to ensure a fair comparison. Yet, we appreciate your suggestion and will include the references in our discussion section. We will also try out best to incorporate them in revision.\n\nNevertheless, we believe that the models we have adopted (UNet, PSPNet, DeepLabv3+, SegFormer, UPerNet) remain among the most representative in semantic segmentation. Our primary goal is to evaluate RankSEG-RMA through a clean and fair comparison against argmax and RankSEG-BA. Furthermore, the consistent improvements demonstrated across various models and datasets suggest that our method has strong potential for generalization to other architectures.\n\nWe appreciate your feedback and will strive to incorporate these state-of-the-art models in future revision.\n\n> Q2. The work could be further strengthened by including more qualitative visualizations. Side-by-side visual comparisons with RankSEG and argmax baseline would intuitively showcase the advantages of RankSEG-RMA.\n\n\n**Reply.** Thank you for your valuable suggestion. We have included Fig. 2 to provide a visual comparison with the baseline, and we will include more qualitative visualizations and comparisons in the revision to further demonstrate our work.\n\n> Q3. The time consumed in multi-class semantic segmentation is still significantly greater than that of Argmax-prob as shown in Table 3.\n\n**Reply.** Thank you for raising this important point and providing us with the opportunity to clarify. In Table 3, we separately provide the *model-forward time* and the *prediction-algorithm time*. Yet, from a practical perspective, what we truly care about is the *total segmentation time* being the sum of these two components. \n\nThus, we have modified Table 3 for clarity as follows, presenting the values in the format of *total time (model-forward time + prediction time)*. \"NA\" indicates not applicable.\n\n|  | VOC | Cityscapes | ADE20K | LiTS | KiTS |\n| - | - | - | - | - | - |\n| Argmax-prob | 40.82 (40.77 + 0.05) | 176.30 (175.81+0.22) | 325.02 (324.59+0.43) | 14.16 (14.15+0.01) | 11.87 (11.86+0.01) |\n| RankSEG-RMA (ours) | 47.70 (40.77 + 6.93) | 185.96 (175.81+10.15) | 382.59 (324.59+58.00) | 14.49 (14.15+0.34) | 12.12 (11.86+0.26) |\n| RankSEG-BA (Dai et. al) | NA | NA | NA | 30.48 (14.15+16.33) | 21.85 (11.86+9.99) |\n\nAs indicated in the table, the total time of our method is approximately 17%, 5%, and 5% greater than that of argmax on VOC, Cityscapes, and LiTS, respectively. This demonstrates that, when considering the total time, our method maintains the entire segmentation prediction within an acceptable running time, in contrast to the original RankSEG algorithm, which required nearly twice the time.\n\nIn summary, although our method is slower than argmax in terms of prediction-algorithm time, we believe that its contribution to the total segmentation time remains acceptable because the forward pass time overwhelmingly dominates the total time. \n\n> Q4. There is a minor typo on line 176. The text reads \"class x,\" but it seems to be \"class c.\"\n\n**Reply.** Thank you for pointing this out. We will correct the typo in the revised version.\n\n> Q5. Its applicability to other segmentation tasks like panoptic or instance segmentation remains unclear.\n\n\n**Reply.** Thank you for bringing these questions to us. In principle, RankSEG-RMA can be applied to any segmentation task as long as a (estimated) probability mask $\\mathbf{q} \\in [0,1]^{C\\times d}$ is provided. However, in the context of instance segmentation or panoptic segmentation, popular deep models typically produce more complex outputs; therefore, RankSEG-RMA also requires additional processing to generate the corresponding results.\n\nFor example, MaskFormer [1] produces a set of $(\\mathbf{p}\\_i, \\mathbf{m}\\_i)\\_{i=1}^N$ for an image, where $N$ is a pre-defined number of mask candidates, $\\mathbf{p}\\_i \\in [0,1]^C$ denotes the class probability, and $\\mathbf{m}\\_i \\in [0,1]^d$ denotes the binary probability mask. To adapt our method, we first broadcast $\\mathbf{p}\\_i$ across every pixel of $\\mathbf{m}\\_i$ to form a probability mask $\\mathbf{q}\\_i \\in [0,1]^{C\\times d}$ such that $q_{i, c, j}=p\\_{i,c} m\\_{i, j}$. We then apply RankSEG-RMA to obtain a segmentation mask. The final challenge lies in resolving overlaps between different masks. We believe this can be addressed using our RMA scores or other heuristic methods proposed by MaskFormer.\n\nWe appreciate your comment and recognize the importance of exploring the potential of RankSEG-RMA in instance and panoptic segmentation tasks. We plan to investigate this further in our future work\n\n[1] Cheng, Bowen, Alex Schwing, and Alexander Kirillov. \"Per-pixel classification is not all you need for semantic segmentation.\" Advances in Neural Information Processing Systems."}, {"review_id": "cbRQ1jEYAK", "reviewer": "Reviewer_fg59", "summary": "The paper presents a novel approach to produce semantic segmentation results, where the standard argmax operation is replaced by RankSEG-RMA, their proposed approach, which improves RankSEG. In particular, the proposed reciprocal moment approximation-based approach is significantly faster than RankSEG and is not affected by RankSEG's main limitation, namely its applicability only to overlapping segments. \nThe authors introduce their method with a compelling theoretical analysis and present all proofs in the appendix. Additionally, they perform extensive experiments on several datasets to prove that their proposed approach outperforms both argmax and RankSEG, while being extremely more efficient than RankSEG.", "questions": "* The first point of discussion is the practical applicability of the proposed method compared to existing approaches. Despite being better than argmax, the proposed approach is too slow for real-world applicability. It would be interesting to visualize some more qualitative results, showing that the proposed approach has a small but significant/critical difference in performance wrt the baselines. \n\n* In the introduction (lines 45-55), the authors briefly discuss alternative methods used in the literature for attempting to directly optimize IoU, such as soft-IoU and Lovasz loss. Despite their theoretical inconsistencies, they are still among the most used methods for post-processing the output of a semantic segmentation network. Thus, to have a fair comparison to existing state-of-the-art techniques, it would be interesting to see a comparison against these methods as well.\n\n* I'd suggest a minor revision of Sec. 2 and 3, as it is not clear from the text that binary segmentation is being discussed up until Sec. 3 (line 153), when it's first explicitly mentioned. In the background section in particular, concepts such as segmentation mask and predicted mask are introduced with the underlying assumption of binary segmentation, without stating it explicitly. \n\n* In the conclusion, it is mentioned that the assumption of conditional independence in the Bayes rule is a limitation of the approach. It would be good to have an explanation of why that is, and what this assumption implies in this case.\n\n* I suggest minor revisions of the layout to make the paper look even better than it already does. Small fixes, such as avoiding text between Algorithm 1 and Figure 1, and pushing Algorithm 2 to the top of the page rather than having text on top, would help readability and avoid losing parts of the text.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "The paper is extremely well-written and explained, relatively easy to follow, and theoretically and technically sound. The authors do a good job of explaining the pertinent background knowledge necessary for a full understanding of the paper, and back up their claims and contributions with convincing proofs, extensive experiments, and careful and well-thought-out considerations.\nThe topic is also relevant and significant for the community. Despite being an extension of an existing work, the proposed approach is useful and convincing, and it's -as the authors rightfully state in the manuscript- a plug-and-play module that can be universally applied to all networks for semantic segmentation. This feature is also highlighted in the experimental evaluation, where different models are used, both CNN-based and transformer-based, showing the broad applicability of the contributed module.\n\nThe main weakness of this work is its significance in terms of practical applicability, as the performance yielded by RankSEG-RMA is slightly superior to argmax (1-1.5% mIoU in the best cases, sometimes also <1% as in Cityscapes), despite being two orders of magnitude slower on multiclass datasets. This makes the practical applicability of the method extremely limited, as it would definitely be good for benchmarking, but impossible to use for real-world deployment of any kind.", "was_revised": true, "final_justification": "I want to thank the authors for their response. They carefully addressed all reviews and responded to the reviewers' doubts. For my specific case, I am satisfied with how the authors justified their choices. I also think that they did a good job on all other reviews.\nThe paper was already in a good shape originally, and by carefully re-reading it I am still convinced that its contribution is valuable and relevant. I encourage the authors to implement all changes and extras in the final version.\nThus, I increase my rating from a 4 to a 5.", "rebuttal": "> Q1. The main weakness of this work is its significance in terms of practical applicability, by being two orders of magnitude slower than argmax on multiclass datasets.\n\n**Reply.** Thank you for raising this important point and providing us with the opportunity to clarify. In Table 3, we separately provide the *model-forward time* and the *prediction-algorithm time*. Yet, from a practical perspective, what we truly care about is the *total segmentation time*, which is the sum of these two components. \n\nTo enhance clarity, we have modified Table 3 to present the values in the format of *total time (model-forward time + prediction time)*, with \"NA\" indicating not applicable.\n\n|  | VOC | Cityscapes | ADE20K | LiTS | KiTS |\n| - | - | - | - | - | - |\n| Argmax-prob | 40.82 (40.77 + 0.05) | 176.30 (175.81+0.22) | 325.02 (324.59+0.43) | 14.16 (14.15+0.01) | 11.87 (11.86+0.01) |\n| RankSEG-RMA (ours) | 47.70 (40.77 + 6.93) | 185.96 (175.81+10.15) | 382.59 (324.59+58.00) | 14.49 (14.15+0.34) | 12.12 (11.86+0.26) |\n| RankSEG-BA (Dai et. al) | NA | NA | NA | 30.48 (14.15+16.33) | 21.85 (11.86+9.99) |\n\nAs indicated in the table, the total time of our method is approximately 17%, 5%, and 5% greater than that of argmax on VOC, Cityscapes, and LiTS, respectively. This demonstrates that our method maintains an acceptable running time for total segmentation prediction, especially in contrast to the original RankSEG algorithm, which requires nearly twice the time.\n\nIn summary, although our method is slower than argmax in terms of prediction-algorithm time, we believe that its contribution to the total segmentation time remains acceptable because it is the model-forward time overwhelmingly dominating the total time.\n\n> Q2. It would be interesting to visualize some more qualitative results, showing that the proposed approach has a small but significant/critical difference in performance wrt the baselines.\n\n**Reply.** We highly appreciate your valuable suggestion. We recognize the importance of visual comparisons in demonstrating the effectiveness of our proposed approach.\n\nWe have included Fig. 2 to provide a visual comparison with the baseline. In the revised version, we will add more qualitative visualizations to further highlight the critical differences.\n\n> Q3. Despite their theoretical inconsistencies, they are still among the most used methods semantic segmentation. It would be interesting to see a comparison against soft-IoU Loss, Lovasz Loss as well.\n\n**Reply.** Thank you for your comment. We provide a comprehensive comparison with soft-IoU loss below to clarify the positioning of our work:\n\na) **Different roles in the segmentation pipeline**: Soft-IoU losses are used during the training phase to guide model optimization, whereas our method serves a plug-and-play module that can be seamlessly integrated into a trained model at the inference stage.\n\nb) **Hyper-parameter tuning**: Perhaps due to the non-convexity of the soft-IoU losses, they are typically combined with cross-entropy loss using a weighted sum to stablize training, thus introducing an additional hyper-parameter. Tuning this parameter requires retraining the model, which is computationally expensive. In contrast, our method does not introduce any new hyper-parameters or require model retraining.\n\nc) **Probabilistic interpretation**: Producing reliable pixel-wise probabilities is crucial for uncertainty estimation [1], especially in high-stakes scenarios such as medical imaging. Models trained with soft-IoU loss do not output calibrated probabilities [1], locking their ability to quantify uncertainty. Our method, on the other hand, is typically applied to models trained with cross-entropy loss, thus preserving this probabilistic interpretation.\n\nd) **Performance comparison**: It is acknowledged that soft-IoU loss is more effective when used independently. However, since these methods operate at different stages, we can still apply RankSEG-RMA after training with soft-IoU. Specifically, we consider $L=\\lambda L_{\\text{CE}} + (1-\\lambda) L_{\\text{soft-IoU}}$ with varying $\\lambda$ on ADE20k, and report the results as follows:\n\n|$\\lambda$|1|0.75|0.5|0.25|0|\n|-|-|-|-|-|-|\n|Argmax|56.94|59.13|60.00|60.71|54.70|\n|RankSEG-RMA|57.67|59.50|60.17|60.81|54.68|\n\nAs shown, RankSEG-RMA can still achive improvements when the model trained with the weighted loss (except for $\\lambda=0$). The performance gain decreases as $\\lambda$ decreases. This trend occurs because, as the soft-IoU loss becomes dominant (i.e., when $\\lambda$ is small), the model outputs become less calibrated and less interpretable as probabilities, which in turn reduces the effectiveness of our method.\n\n[1] Mehrtash, Alireza, et al. \"Confidence calibration and predictive uncertainty estimation for deep medical image segmentation.\" IEEE Transactions on Medical Imaging 39.12 (2020): 3868-3878.\n\n> Q4. I'd suggest a minor revision of Sec. 2 and 3, as it is not clear from the text that binary segmentation is being discussed up until Sec. 3. In the background section in particular, concepts such as segmentation mask and predicted mask are introduced with the underlying assumption of binary segmentation, without stating it explicitly.\n\n**Reply.** Thank you for your suggestions to enhance our work. \n\nTo improve the readability, we will incorporate more explicit statements to clarify the scope of different sections. At the beginning of Section 2, we will explicitly state that the focus is on binary segmentation, and we will clarify that the notations and Bayes rule are tailored for this binary case. Furthermore, we will indicate that multi-class segmentation will be formally addressed in Section 3.2.\n\n> Q5. It is mentioned that the assumption of conditional independence in the Bayes rule is a limitation of the approach. It would be good to have an explanation of why that is, and what this assumption implies in this case.\n\n**Reply.** A limitation of the segmentation Bayes rule is that the ranking structure reflected in the final solution will be broken when Conditional Independence Assumption (CIA) is violated. When CIA is not satisfied, the Bayes rule is difficult to optimize and compute.\n\n\nSpecifically, the CIA states that the label distributions of two pixels are independent given the image. Intuitively, this assumption implies that the remaining information in labels at different pixel locations are independent after removing the information from the image.\n \nYet, we want to further explain why this assumption is still acceptable in segmentation applications. a) The CIA is **implicitly assumed when training with cross-entropy (CE) loss**, as it allows the joint likelihood to be factorized into pixel-wise terms. b) Conditional dependence is difficult to formulate practically, especially given the high dimensionality of segmentation data. c) Starting from v3 [2], DeepLab discards conditional random fields (CRFs) that were previously used to model pairwise label dependence, after developing more powerful neural network. This change suggests that **CIA is a reasonable approximation once the image ($\\mathbf{X}$) delivers sufficient information (and this information is captured by powerful model) for inferring the label ($\\mathbf{Y}$), the dependence among the remaining parts ($\\mathbf{Y}$ after removing information from $\\mathbf{X}$) becomes less significant**.\n\n\n[2] Chen, Liang-Chieh, et al. \"Rethinking atrous convolution for semantic image segmentation.\" arXiv preprint arXiv:1706.05587 (2017).\n\n> Q6. I suggest minor revisions of the layout to make the paper look even better than it already does. Small fixes, such as avoiding text between Algorithm 1 and Figure 1, and pushing Algorithm 2 to the top of the page rather than having text on top, would help readability and avoid losing parts of the text.\n\n\n**Reply.** We highly appreciate your helpful suggestions. Thank you for your helpful suggestions. We will revise the layout as recommended to improve readability, including avoiding text between Algorithm 1 and Figure 1 and placing Algorithm 2 at the top of the page."}]

Author rebuttal:

Meta-review: This submission proposes RankSEG-RMA, an extension of RankSEG for semantic segmentation. The key idea is to employ a Reciprocal Moment Approximation to reduce computational complexity while maintaining consistency with Dice/IoU metrics. All the reviewers rated positively for this paper, praising its theoretical soundness, computational efficiency, plug-and-play nature, and extensive experiments. However, there are also concerns regarding the magnitude of improvements, runtime overhead, and evaluation scopes. 

The authors engaged extensively with the reviewers. Their rebuttal clarified the runtime overhead (arguing that the additional inference cost is minor compared to forward pass), addressed theoretical assumptions, and provided new experiments, including statistical significance tests and results on CPT. These responses satisfied most reviewers. Reviewer's concerns about outdated baselines and practical applicability remain partly unresolved but were mitigated by the explanations.

In general, this paper makes a theoretically well-founded and practically useful contribution to segmentation tasks, and all the reviewers aggree on acceptance. There is no special reason to overturn the concensus.
Decision: Accept (poster)