SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process
Authors: Mengyu Wang
Henghui Ding
Jun Hao Liew
Jiajun Liu
Yao Zhao
Yunchao Wei
Venue: NeurIPS 2023 poster (2023)
Keywords: Refinement
Segmentation
Discrete Diffusion
TL;DR: We propose a universal segmentation refinement framework based on the discrete diffusion models.

Abstract: In this paper, we explore a principal way to enhance the quality of object masks produced by different segmentation models. We propose a model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement as a data generation process. As a result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks as inputs and refines them using a discrete diffusion process. By predicting the label and corresponding states-transition probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits a strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at [SegRefiner.git](https://github.com/MengyuWang826/SegRefiner)

SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process
Mengyu Wang 13+
Henghui Ding", Jun Hao Liew
Jiajun Liu
Yao Zhao's .,,,f Yunchao Wei
Institute of Information Science, Beijing Jiaotong University
Peng Cheng Laboratory
3 Beijing Key Laboratory of Advanced Information Science and Network
Nanyang Technological University ByteDance
xmengyn00260gaai..comm
Abstract
this paper, we explore a principal way to enhance the quality object masks produced by different segmentation models. We propose model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement a data generation process. As result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks inputs and refines them using a discrete diffusion process. By predicting the label and corresponding tratestransittonn probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at github.com/Mengya Waagggyyuuaagg22/eggeeeeee
Introduction
Although segmentation in image [35, 23, 24, 14, 11] and video [54, 37, 38, 16, 15] has been extensively studied in the past decades, obtaining accurate and detailed segmentation masks is always challenging since high-quality segmentation requires the model to capture both high-level semantic information and fine-grained texture information to make accurate predictions. This challenge is particularly pronounced for images with resolutions of 2K or higher, which requires considerable computational complexity and memory usage. As a result, existing segmentation algorithms often predict masks at a smaller size, inevitably leading to lower accuracy due to loss of fine-grained information during downsampling.
Since directly predicting high-quality masks challenging, some previous works have shifted their attention to the refinement of coarse masks obtained from a preceding segmentation model. A popular line of direction [29, 57, 27, 28] is augment the segmentation models (and features) with new
Work done during internship at ByteDance. Co-corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Image
(b) Coarse Mask
(c) CRM [44]
(d) SegRefiner (ours)
Diverse errors presented in the previous segmentation results. The top row and middle row show the significant false positives and false negatives caused by incorrect semanties, the bottom row shows the inaccurate in capturing fine-grained details. Please zoom in for better view.
Figure I:
module for masks correction. However, such approaches are usually model-specific, and hence cannot be generalized to refine coarse masks produced by other segmentation models. Another part segmentation refinement works [47, 56, 12, 44], on the other hand, resort model-agnostic approach by taking only image and the coarse segmentation as input for refinement. These methods have greater practical utility, as they are applicable to refine different segmentation models. However, the diverse types of errors (e.g., errors along object boundaries, failure in capturing fine-grained details in high-resolution images and errors due to incorrect semantics) presented in the coarse masks pose a great challenge to the refinement model thus causing underperformance (refer to Fig. 1).
In response to this challenge, draw inspiration from the working principle of denoising diffusion models [45, 22, 2]. Diffusion models perform denoising at each timestep and gradually approach the image distríbution through multiple iterations. This iterative strategy significantly reduces the difficulty of fitting the target distribution all at once, empowering diffusion models to generate high- quality images. Intuitively, by applying this strategy to the segmentation refinement task, refinement model can focus correcting only some "most obvious errors" at each step and iteratively converge to an accurate result, thus reducing the difficulty of correcting all errors in a single pass and enabling refinement model to handle more challenging instances.
Under this perspective, we present an innovative interpretation the task by representing segmenta- tion refinement as a data generation process. As a result, refinement can be implemented through sequence of denoising diffusion steps with coarse segmentation masks being the noisy version of ground truth. To work with binary masks, we further devise novel discrete diffusion process, where every pixel performs unidirectional randomly states-transition The proposed process can gradually convert ground truth into a coarse mask during training and employ the coarse mask as sampling origination during inference. generation problem, where the input image serves the condition for iteratively updating/refining the erroncous predictions in the coarse mask.
other words, we formulate the mask refinement task as conditional
To our best knowledge, we are the first to introduce diffusion-based refinement for segmentation masks. Our method, called SegRefiner, is model-agnostic, and thus applicable across different segmentation models and tasks. We extensively analyze the performance of SegRefiner across various segmentation tasks, demonstrating that our SegRefiner not only outperforms all previous model-agnostic refinement methods (with +3.42 loU, +2.21 mBA in semantic segmentation and +0.9 Mask AP, +2.2 Boundary AP in instance segmentation), but can also be effortlessly transferred to other segmentation tasks (e.g., the recently proposed dichotomous image segmentation task [40]) without any modification. Additionally, SegRefiner demonstrates a strong capability for capturing extremely fine details when applied to high-resolution images (see Fig. and Fig, 4).

---

2 Related Work
Segmentation Refinemement The aim of segmentation refinement to improve the quality of masks in pre-existing segmentation models. Some works focus on enhancing specific segmentation models. PointRend [29] employs an MLP to predict the labels of pixels with low-confidence scores output from Mask R-CNN [21]. RefineMask [57] incorporates a semantic head to Mask R-CNN as additional guidance. MaskTransfiner [27] employs an independent FCN [35] to detect incoherent regions and refines their labels with Transformer [49]. These works have demonstrated notable performance enhancements over their preceding models. However, their scope of improvement is model-specific and they lack the capacity directly refine the coarse mask derived from other models. There are also some refinement methods that adopt model-agnostic approaches, such as [47, 56, 61, 12, 44, 31]. These strategies emphasize utilizing diverse forms of inpur, including whole images, boundary patches, edge strips, etc. Even though these techniques can refine coarse masks derived from different models, their applicability remains confined to specific segmentation tasks. As a special case, SegFix [56]. which learns a mapping function between edge pixels and inner pixels and subsequently replaces inaccurate edge predictions with corresponding inner pixel predictions, is employed in both semantic segmentation and instance segmentation within the Cityscapes dataset [13]. While the performance of SegFix significantly constrained by its ability to accurately identify objects within an image, which consequently leads to decline in performance on datasets with a more extensive range of categories (e.g.. COCO [34]).
Diffusion Models for Detection and Segmentation Recently, diffusion models have received a lot of attention in research. Initial studies [45, 22, 2, 46, 8, 3] primarily sought to enhance and expand the diffusion framework. Following these, subsequent works ventured to incorporate diffusion models across a broader array of tasks [43, 9, 30] and to formulate comprehensive conditional generation frameworks [41, 58]. The application of diffusion models to detection and segmentation tasks has also been the focus of an escalating number of studies. Baranchuk et al. [4] capture the intermediate activations from diffusion models and employ an MLP execute per-pia per-pixel classification. DiffusionDet [6] and DiffusionInst [19] adapt the diffusion process to perform denoising in object boxes and mask filters. Some other works [1, 52, 53, 7, 26] consider the image segmentation task as mask generation. These studies predominantly follow the Gaussian diffusion process of DDPM [22] and leverage additional image encoder extract image features as condition to generate masks. To the best of our knowledge, our SegRefiner is the first work that applies a diffusion model to the image segmentation refinement task, and it also pioneers the abandonment of the Gaussian assumption in favor of a newly designed discrete diffusion process in diffusion- based segmentation tasks.
3 Methodology
3.1 Preliminaries: Diffusion Models
Diffusion models consist of a forward and a reverse process. The forward process ((1):T(3)) uses a Markov or None-Markov chain to gradually convert the data distribution xo q(2r)) into complete noise 2r whereas the reverse process deploys a gradual denoising procedure pa Pa(To:) that transforms random noise back into the original data distribution.
Continuous Diffusion Models The majority of existing continuous diffusion models [22, 46, 41] adheres to the Gaussian assumption and defines p(rr) N (ar (0, 1). The mean and variance of forward process are defined by hyperparameter Be and the reverse process utilizes mean and variance from model predictions, thus formulating as:
q( 1)) = (2, /1 3,0 B,1), (Mt, 1), Eo (2 ()).
(I)
po 11))) =
(2)
Discrete Diffusion Models Compared to continuous diffusion models, there less research on discrete difffusinn models. Sohl-Dickstein et al. [45] first introduce binary diffusion to reconstruct one- dimensional noisy binary sequences. Ø7 is defined to adhere to the Bernoulli distribution B(xr The forward process and reverse process are represented as:
((1//r ((1_/0t-11 (1 34) + 0.5,4);
(3)
Pa Pa(z,-110 = BZz, (l(;,,)
(4)
3

---

Fornard
Transition Sample Gunblel Somple
coarse
Faltes Cansition prolubili
Sample
Ora.a
placs
transiioone posels
Net Hol
Oo
Trataition Sample
langet
Flly
Mugz
Mefined Mark
B dennsttwise
Transttion Sumple ieput as previous makk
ispul langet mank
imput un prolubility
wise mulippiization
chamclllin concriicnation
Figure 2: An overview of the proposed SegRefiner (best viewed in color). On the right is the transition sample module we proposed, which randomly samples pixels from the current mask based on the input states-transition probabilities and change their values to match those in the target mask. During training, the transition sample module transforms the ground truth into a coarse mask, thus coarse mask is the target mask. During inference, target mask refers the predicted fine mask and this module updates the values in the coarse mask in each timestep based on the predicted fine mask and the transition probabilities.
where 3, (0, 1) hyperparameter and fo f) a model predicting Bernoulli probability, After the great success of DDPM [22], Austin et al. [2] extended the architecture of Discrete Diffusion Model more general form. They define the forward process as discrete random variable transitioning among multiple states and use states matrix Qe to characterize this process:
[Qemm.n q(xx = 2 = =m).
(5)
3.2 SegRefiner
In this work, we propose SegRefiner, with a unique discrete diffusion process, which can be applied to refine coarse masks from various segmentation models and tasks. SegRefiner performs refinement with a coarse-to-fine diffusion process. In the forward process, SegRefiner employs a discrete diffusion process which formulated unidirectional random states-transition, gradually degrading the ground truth mask into a coarse mask. In the reverse process, SegRefiner begins with provided coarse mask and gradually transforms the pixels in the coarse mask the refined state, correcting the wrongly predicted area in the coarse mask. In the following paragraphs, we will provide detailed description of the forward and reverse process.
Forward diffusion process In the forward process, gradually degrade the ground truth mask/ fine mask M fine, transiting into a coarse mask Mcoarse- In other words, we have my = MM ine and mr Mooarse: At any intermediate timestep E 21, 1), the intermediate mask m, is therefore in a transitional phase between Mine and Menarse.
We define that every pixel in has two states: fine and coarse, and the forward process is thus formulated as states-transition between these two states. Pixels in the fine state will retain their values from M fine. and vice versa. We propose new transition sample module to formulate this process. As shown in Fig. right, during forward process, the transition sample module takes the previous mask m4-1. coarse mask my and states-transition probability input and outputs a transitioned mask m,- The states-transe transition probability describes the probability of every pixel in m4-1 transitioning to the coarse state. This module first performs Gumbel-max sampling [25] according to the states-transition probability and obtains the transitioned pixels. Then, the transitioned pixels will take values from my while the om-trraniitoned pixels will keep unchanged.
Note that the transition sample module represents a unidirectional process in which only "transition to coarse state" happens. The unidirectional property ensures that the forward process will converges

---

%
%
T
Course Mask
T-T
1-T-1
1-T2
t-0
Refined Musk
Figure 3: Examples of SegRefiner's inference process. It and m, denote the state (we represent
state [0, 1] and [1,0] 0] and in this figure) and the corresponding mask, respectively. SegRefiner begins with 2T = [0, 1] and mr = Meoorser and gradually refines until we obtain fine mask mg.
to Mooarse. despite each step is completely random. This is significant difference between our SegRefiner and previous diffusion models in which forward process converges random noise.
With the reparameterization trick, we introduce a binary random variable x to formulate the above process. We represent xt as a one-hot vector to represent the state of pixel (i,j) in m, and set To iJ [1, 0] and f [0, 1] to represent the fine state and coarse state, respectively. The forward process can thus be formulated as:
31 1 31
g( (x221) 1r, where Q:
where 3e E [0, 1). and - 3 corresponds to the states- teatsss-ranittion probability used in our transition sample module. Q, is a trantssttnniition matrix. The form of Q. explicitly manifests the unidirectional property, i.e., all pixels in the coarse state will never transition back to the fine state since g( [00, 1]) [0, 1]. According Eq. (6), the marginal distribution be formulated as
Br
1- 3
a( ((a)) Q1.2...2..00 = x: 30
where B, = -,,32... 3.
Given this, we can obtain the intermediate mask my at any intermediate timestep without the need of step-by-step sampling (( allowing faster training.
Reverse diffusion process The reverse diffusion process takes coarse mask mr and gradually transforms it into a fine mask mo. However, since the fine mask mn and the reversed states-transition probability unknown, following DDPM [22], train neural network fo parameterized by 0 to predict the fine mask moje at each timestep. represented as
majt, Po (mooj) fI m,
(8)
where I is the corresponding image. Mojr and po (m)))) denote the predicted binary fine mask and its confidence score, respectively. To obtain the reversed states-transition probability, according to Eq. (6), Eq. (7), and Bayes' theorem, we first formulate the posterior at timestep / as:
q( = (( 1, (a)_(ao))daabbbbo
24Q, O 2
(9)
g(2,120)
20Q2
where the fine state Zo is set to [1, 0] during training, indicating ground truth. While during inference. To is unknown, as the predicted ma may not be entirely accurate. Since the confidence score (o())))) represents the model's level of certainty for each pixel prediction being correct, (m can also be interpreted the probability being in the fine state. Therefore, intuitively, one could
5

---

obtain the state of every pixel in majs by simply thresholding as done in [2]:
[1,0] [0, 1] otherwise,
if po((o))) 2 0.5
Ad 0
(10)
f- [L. 0J. indicating they are in the fine where pixels with higher confidence scores will have Tufe state, and vice versa. However, in such one-hot form, the values of states-transiie probabilities are determined solely by the pre-defined hyperparameters, leading to significant information loss.
Instead, we retain the soft transition and formulate 2. [pe (mojj) 1 , (Mo With this setting, the reverse diffusion process can be reformulated as
O 1-palmna Br-1
po(2221 (22 = 2t
pi.. 0,5 where pí.j 0,6
Pil (fioo e 18
(11)
Pa 1
where piu is the reversed states transition matrix. With the above reversed states-transition transition probabil- ity, my and Maje as input, the transition sample module can transit a portion of pixels to the fine state at each timestep, thereby correcting erroneous predictions.
Inference Given a coarse mask mr and its corresponding image I, we first initialize that all pixels are in the coarse state thus xه/ [0, 1]. We iterate between: (1) forward pass to obtain Moje and ( (Eq. (8)); (2) compute the reversed states-transition matrix pi- and obtain T;-1 (Eq. (11)); (3) compute the refined mask m based on 2;-1, my and Moje. The process (1)-(3) iterated until we obtain fine mask mg. Visualization examples of inference are shown in Fig. 3.
4 Experiments
4.1
Implementation Details
Model Architecture Following [39], we employ U-Net for denoising network. We modify the U-Net to take 4-channel input (concatenation image and the corresponding mask my) and output I-channel refined mask. Both input and output resolution is set to 256x256. All others remain unchanged other than the aforementioned modifications.
Objective Function Following [12], we employ combination of binary cross-entropy loss and texture loss for training our model, i.e., L = Lice + aCtezture. where texture loss is characterized as an LI loss between the segmentation gradient magnitudes of the predicted mask and the ground truth mask. a is set to 5 to balance the magnitude of both losses.
Noise Schedule Theoretically, the unidirectional property of SegRefiner ensures that any noise schedule can make the forward process converge to the coarse mask given infinite number of timesteps. However, in practice, we use much fewer timesteps (T = 6 this work) to ensure efficient inference. We designate 3r such that .T = [0, 1] for all pixels and mr Meoarse (Eq. (7)). Following DDIM [46], we directly set a linear noise schedule from 0.8 to O on 3x-
Training Strategy Our SegRefiner model has been developed into two versions for refinement of different resolution images: a low-resolution variant (hereafter referred to as LR-SegRefiner) and a high-resolution variant (hereafter referred as HRR-SegRefiner), While these two versions employ different training datasets, all other settings remain consistent. T.R-SegRefiner is trained on the LVIS dataset [20], whereas HR-Seg SeRReeeffffeer is trained on a composite datasct merged from two high-resoluticn datasets, DISSK [40] and ThinObject-5K. [32]. These datasets were chosen due to their highly accurate pixel-level annotations, thereby facilitating the training of our model to capture fine details more effectively. Following [12]. the coarse masks used for training are obtained through various morphological operations, such as randomly perturbing some edge points of the ground truth and performing dilation, erosion, etc. During training, we first train Ree-Seneefn the low-reso wew-resolution dataset until convergence. Subsequently, is fine-tuned on the high-resolution dataset yield HR- TR-SegReffner
We employed double random crop as the primary data augmentation technique. entails the random cropping of each image to generate two distinct model inputs: one encapsulating the entire foreground

---

target, and the other containing, only a partial foreground. Both crops are subsequently resized to match the model's input size. This operation ensures that the model is proficient in refining both entire objects and incomplete local patches, a capability that proved instrumental in subsequent experiments. All the following experiments were conducted on NVIDIA RTX 3090. For more details on the training process, please refer to the supplemental materials.
4.2 Semantic Segmentation
Dataset and Metries As the refinement task emphasizes the enhancement of mask quality, dataset with high annotation quality and sufficient detailed information is required to evaluate the model's performance. Hence, report the results BIG dataset [12], semantic segmentation dataset specifically designed for high-resolution images. With resolutions ranging from 2048 x 1600 to 5000 x 3600, this dataset provides challenging testbed for evaluating refinement methods. The metrics we used are the standard segmentation metric IoU and the boundary metric mBA (mean Boundary Accuracy [12]), which is commonly used in previous refinement works.
Settings Because of the high resolution of images in BIG dataset, we employ the HR-SegRefiner in this experiment. While the model's 's output size is only 256 x 256, which is insufficient for such a high-resolution dataset and results in the loss of many edge details. Consequently, during inference, we deploy the first T - timesteps to perform global refinement, which takes the resized entire image input, and the final timestep as local step that takes original-size local patches input. In order to identify the local patches that require refinement, we filter out pixels with low state-transition probabilities from the globally refined mask and use them the center points for the local patches. We allocate more timesteps for global refinement since global steps need to handle more severe errors, thus presenting a higher difficulty level. The training strategy we have employed ensures that the model can adapt to both global and local input without modification.
As shown in Tab. 1, we compare the proposed SegRefiner with three model-agnostic seman- tic segmentation refinement methods, SegFix [56], CascadePSP [12], and CRM [44]. Additionally, we include the fine-graine uineigraiedd matting method, MGMatting [55], which employs an image and mask for matting and can also be utilized for refinement purposes. The proposed SegRefiner demonstrates superior performance over previous methods when using coarse masks from four different semantic segmentation models, as evident both IoU and mBA metrics. Notably, our SegRefiner outperforms images, showcasing significant ad-
Results
CRM, which is specifically designed for ultra-hi setra-iigh-resolutiin vancements. We report the error bar (#: in gray) in this experiment due to the relatively small size of the BIG dataset (consisting of 100 testing images). The error bar represents the maximum fluctuation value among the results of five experiments, and the results Tab. I are the average of these five trials. In subsequent experiments, the stability of the results is ensured by the availability of an ample number of testing images.
4.3 Instance Segmentation
Dataset and Metrics To evaluate the effectiveness of our SegRefiner in refining instance segmenta-
tion, we select the widely-used COCO dataset [34] with LVIS [20] annotations. LVIS annotations offer superior quality and more detailed structures compared to the original COCO annotations and other commonly used instance segmentation datasets such as Cityscapes [13] and Pascal VOC [17]. This makes LVIS annotations more suitable for assessing the performance of refinement models. The evaluation metries are the Mask AP and Boundary AP. worth noting that these metries are computed using the LVIS [20] annotations on the COCO validation set. The Boundary AP metric, introduced by Cheng et al. [10], is valuable evaluation metric that measures the boundary quality of the predicted masks and is highly sensitive to the accuracy of edge prediction. It provides a detailed assessment of how well the refined masks capture the boundaries of the objects.
Table I: IoU/mBA results on the BIG dataset comparing with other mask refinement methods.
ToU/mBA
Coarse Mask
SegFix [50]
MGMarting [55]
CascadePSP [12]
CRM
SegRefiner (ours)
FCN-8s [35]
75.39
22.69/55.20
72.31 /5..32
/7.87/67.04
79.62 69.47
86.95.0m
77.81 gons
DeepLab V3+ [5]
89.42/ 60.25
89,95 / 64.34
90.49/ /7.48
92.23/74.5
91.84/74.96
94.86 100: / /77.64
RefineNet [33]
90.20 62.03
90.73 65.95
90.98 028/66 68.40
92.79 74.77
92.89/75.50
95.12
/76.9100
PSPNet [60]
90.49/59.63
91.01 /3.25
66.72/66.73
93.93/75.32
94.18/76.09
95.30
som:
777.66
Avg Improve
0.00/0.00
03,3
0.73/6.10
3.58 14.05
15.12
7.43/ 17.33
7

---

(a) Image
(b) GT
(c) U-Net [42]
(d) ISNet [44]
(e) SegRefiner (ours)
Figure 4: Qualitative comparisons with other methods on DIS5K dataset [40]. Our SegRefiner has the capability capture finer details, allowing it to discern and incorporate more subtle nuances, Please kindly zoom in for a better view. More visual results are provided in the supplemental file.
Settings In this experiment, we utilize the LR- RR-SegRefiner model. To refine each instance, we extract the bounding box region based on the coarse mask and expand by 20 pixels on each side. The extracted region is then resized to match the input size of the model. The output size is suitable for instances the COCO dataset, allowing us to perform instance-level refinement for timesteps without requiring any local patch refinement.
Results First, in Tab. 2, we compare the proposed SegRefiner with two model-agnostic instance segmentation refinement methods, BPR [47] and SegFix [56]. demonstrated in Tab. 2, our SegRe- finer achieves significantly better performance compared to these two methods. The coarse masks utilized in Tab. are obtained from Mask R-CNN to ensure consistency with the original experiments conducted in [47, 56]. This choice allows for fair and direct comparison with the previous works. Then Tab. 3, we apply our SegRefiner to other 7 instance segmentation models. Our method yields significant enhancements across models of varying performance levels. Furthermore, when compared to three model-specific instance segmentation refinement models, including PointRend [29], RefineMask [57], and Mask TransFiner [27], our SegRefiner consistently enhances their performance. These results establish SegRefiner as the leading model-agnostic refinement method for instance segmentation.
4.4 Dichotomous Image Segmentation
Dichotomous Image Segmentation (DIS) is recently introduced task by Qin et al. [40], which specifically targets the segmentation of objects with complex texture structures, such as the steel framework of bridge illustrated in Fig. 4. To facilitate research in this area, Qin al. also built the DISSK dataset, a meticulously annotated collection of 5,470 images with resolutions of 2K and above. The DIS5K dataset, characterized by its abundance of fine-grat fine-grained structural details, poses rigorous
Table 2: Comparision with other model-agnostic instance segmentation refinement methods on the COCO val set using LVIS annotations.
Mask AP
Boundary AP
Method
AP
APsa
AP
APs
APH
AP:
AP
APsp
AP1s
AP:
APa
AP2
MaskRCNN (Res50)
39.8
61.4
42.3
24.9
47.0
55.1
27.3
53.3
25.2
24.9
41.8
27.4
Segttix
61.4
48.4
-PR [47]
41.0
61.4
43.1
24,8
48.5
57.8
30.4
55.2
29.5
24.7
43.8
33.7
SegRRefieer (ours)
41.9
61.6
43.2
25.7
49.4
32.6
55.7
32.5
25.6
45.0
MaskRCNN (Rest01)
41.6
63.3
44.4
49.3
37.8
29.0
35.2
26.7
26.3
44.4
29.8
Segtix [56]
42.2
63.4
44.7
26.5
50.9
59.1
30.6
56.1
30.0
26.3
46.0
33.1
SegReflner (ours)
43.6
63.3
45.2
27.4
51,4
61.6
34,1
57.2
34,9
27.2
47.1
39,9

---

Table Tansfering our SegRefiner to other instance segmentation models.
Mask AP
Boundary AP
Method
AP
AP,
AFW
AP,
AP
APp
AP,
PointRend [29]
41.5
25.1
49.0
59.3
30.6
25.0
44.2
34,1
SegRefiner
42.8 +13
25.9.sx .65
50.4.14
61.3.2a
33.7 au
25,8 .0s
46.2-29
40.1 -Ao
RelineMask [57]
4122
48.1
59.2
30,5
23.8
34.1
SegRefiner
41.9
24.6
48.7
60.7 43
33.0 .25
24.5
44.7
39.4
Mask Transfiner [27)
42.2
25.9
49.0
60.1
31.6
25.8
44.5
35.8
Seggefiner
43.3 +1.1
26.8 .89
49.9 .69
62.0 .1.
34.4 -28
26.6 .as
45.9 -1.
41.3 +85
SOLO [S1]
37.4
19.3
45.5
56.6
24.7
19.0
39.3
27,8
SegRefiner
40.5, 401
21.7 424
49.3,33
60.9 .4.
31.3 16.
21.4 .24
44.8 .5.5
39.8.120
CondInst [48]
39.8
24.3
47.5
54.8
29.2
24.1
42.6
30.2
SegReliner
41.1
25.5
48.8
56.8
32.2
25.3
44,5
36.1 459
Query Inst [18]
424
26.7
49.5
61.4
29.9
26.4
44,5
32,3
SegRefiner
44. 118
27.5 188
51.4 .1.
64.7 13.3
34.9 (50
27.3 m99
47.2 127
42,6 110.3
Mask2Former [1]]
46.8
27.9
54.9
69.1
37.0
27.8
50.1
44.6
SegRefiner
47.4 1866
1866
55.3
+0.4
69.8
187
38.8 -18
28.4 .0.6
.0.
48.5 +43
Table 4: Transfering our SegRefiner to DIS task. SegRefiner is employed to refine the segmentation results from different models, w/o and w/ indicates the original and refined results, respectively.
Disase
Metrics
U-Net
PFNet [30]
PSPNet [00]
ICNet
[59]]
HRNee 50]
wila
w/
wio
w/
w/o
W/o
w/
u/lo
w/o
w/
DIS-VD
mBA
ToU
69.84
$4.77
58.73
75.64
-3.96
+6.6
63.69
$6.20
74,83
60.44
+4.34
62.78
56.41
71.84
60:41
+4.00
66.47
56.47
75.55
-4.91
68.94
61.02
64.43
76.62
-3.41
67.10
74.13
76.64
68.37
+1.37
DIS-TEI
IoL
44.11
12.94
49.01
12.04
47.57
49,01
11.44
40.22
49.74
-3.52
52.97
11.99
56.17
57.00
10.83
mBA
70.13
74.97
-1.44
65.13
74.67
64.45
74.67
+10.23
67.19
75.01
69:34
75.62
-4.28
73.63
75.49
DIS-TE2
ToU
mBA
69.99
$4.30
75.57
43.71
15.54
64.64
$7.04
75.05
-1.22
10:41
63.71
57.71
75.06
111:35
-2.54
56.50
66.79
35.57
60:33
-183
18.78
69.50
64.70
76.64
-3.45
17.14
74,03
67.84 .4.70
76.35 12.33
DIS-TE3
59.29
63.34
44.05
59,46
63.42
.3.66
03.44
-3.4
39.83
04:20
.4.37
67.57
.3.14
7037 .4.33
mBA
70.59
76,68
-4.09
64.00
76.06
-1..66
62.74
76.05
+13.31
66.73
76.55
-0.8
69.48
77.60
.8.12
74.65
77.30 .1.74
DIS-TE4
IoU
61.14
66.29
+8.15
39,11
66.63
47.82
57.61
66.61
.983
60.53
67.01
64.67
70.47
.530
70.12
72.31 +3.09
mRA
70,68
77.07
16.39
62.78
76.43
13365
61.88
76.43
114.5
66.42
77,06
18.64
68.00
77.86
(4777
74.35
77.66 (3.3)
demands on model's capability to perceive and capture intricate information. Thus, it serves as suitable benchmark to evaluate the performance of our refinement method. The evaluation metrics and other settings utilized in this experiment are consistent with those in Sec. 4.2.
Since the recent introduction of DIS5K, no prior refinement methods have reported results on this dataset to date. Therefore, the main aim of this experiment is to evaluate the transferability of our SegRefiner across various models. As shown in Tab. 4, our SegRefiner is applied to 6 segmentation models. The results consistently demonstrate that our SegRefiner improves the performance of each segmentation model in terms of both IoU and mBA. In Fig. 4, we provide qualitative comparisons with previous methods, which reveal the superior ability of our SegRefiner to capture fine-grained details, such as the dense and fine mesh of the chair.
Results
4.5 Ablation Study
We conduct ablation studies on the BIG dataset [44] with high-resolution images. We first investigate the effectiveness of the diffusion process in SegRefiner. The results are reported in Tab. 5a, where the term "none" refers to the utilization of a U-Net without multi-step iteration, and "w/o diffusion" denotes the direct utilization of the previous step's mask as the input for the subsequent iteration, without employing the diffusion sampling process, i.e., discarding the transition sample module. Since there is no states-transitinn probability in non-diffusion methods, we use a sliding window strategy similar to [12] in this experiment to obtain the input for local refinement. The results Tab. 5a demonstrate that the diffusion-based iterative process achieves the best performance, validating the effectiveness of the diffusion process in SegRefiner.
In Tab. 5b and Tab. 5c, we evaluate various alternatives used in prior experiments. Tab. 5b shows the analysis of global and local refinement used in high resolution images. As can be seen, global refinement significantly improves loU; however, for high-resolution images, the resulting smaller output size leads to a lower mBA. Local refinement applied to local patches of the original size greatly improves mBA, while its enhancement in IoU is less significant due to the absence of global information. The combination of local and global refinement achieves better performance both IoU and mBA. Tab. 5c presents the results corresponding to different input image sizes. Considering computational load and memory usage, 256 256 is selected as the default setting, which performs well without introducing too much computational overhead.
9

---

Table 5: Ablation studies on BIG dataset [44]. The best results are highlighted in bold and the default
settings are marked with gray.
(a) Effectiveness diffusion process.
(b) on global local refinement.
(c) Ablation on inpus size.
Iteration Type
IoU
mBA
Method
ToU
mBA
Inpar Size
ToU
mBA
nome
90,10
75.03
only global
92.43
60.58
128x
93.10
74.28
w/o diftusion
only local
63.56
256
77.64
w/ diffusion
94.85
77.64
global+local
94.85
77.64
512x512
94,97
77.56
5 Conclusion and Discussion
We propose SegRefiner, which is the first diffusion-based image segmentation refinement method with new designed discrete diffusion process. SegRefiner performs model-agnostic segmentation refinement and achieves strong empirical results in refinement of various segmentation tasks.
While SegRefiner has achieved significant improvements in accuracy. the diffusion process leads to slowdown of the inference due to the multi-step iterative strategy. As shown in Tab. 6, we conduct experiment instance segmentation about the relationship between the model's s accuracy, computational complexity, time consumption (the average time consumed per image), and the number of iteration steps. It can be observed that, while the iterative strategy has provided SegRefiner with noticeable improvement in accuracy, it has also introduced linear increase with the number of steps in both time consumption and computational complexity. As the first work applying diffusion models to the refinement task, the proposed SegRefiner primarily concentrates devising suitable diffusion process for general refinement tasks. While improving the efficiency of diffusion models will be a crucial research direction in the future, not only in the field of image generation but also in other domains where diffusion models are applied.
limitation lies in that
Table 6: The relationship between SegRefiner's accuracy, efficiency and the number of iteration steps on instance segmentation.
Steps
3
4
Mask
39.8
40.7
41.0
41.4
41.6
41.9
41.9
Boundary AP
27.3
29.9
30.6
31.5
32.1
32.5
32.6
Time
n/a
0.52
1.01
1.53
1.98
2.45
2.94
GFLOPs
n/a
249
497
746
994
1243
1492
Acknowledgement
This research was funded by the Fundamental Research Funds for the Central Universitics (202JJBZD003).
References
[1] Tomer Amit, Eliya Nachmani, Tal Shaharabany, and Lior Wolf, Segdiff: Image segmentation with diffusion prohabilistic models. CoRR, abs/21 12.00390, 2021.
[2] Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffiusion models in discrete state-spaces. In NeurIPS, volume 34, pages 17981-17993, 2021.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Goldblum. Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. CoRR, abs/2208.09392. 2022.
[4] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. ICLR, 2022.
[5] Liang- Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decodes with atrous separable convolution for semantic image segmentation. In ECCV, pages 801-818, 2018.
10

---

[6] Shoufa Chen, Peize Sun. Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. CoRR, 0ttp//2111097555 2022.
[7] Ting Chen, Lala Li, Saurahh Saxena, Geoffrey E. Hinton, and David J. Fleet. A generalist framework for panoptic segmentation of images and videos. CoRR, abs/2210.06366 2022.
[8] Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: Generating discrete data using diffusion models with colf-condiiionnn. CoRR, abs/2208. 042002, 2022.
[9] Tsai- Ssiin Chen, Chieh Hubert Lin, Hung-Yo Yu Tseng. Tsung- Yi Lin, and Ming- Hisun Yang. Motion- conditioned diffusion model for controllable video synthesis. CoRR, abs/2304. 1.0004114000 14404, 2023.
[10] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation, In CVPR, pages 15334-15342, 2021.
[11] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. CVPR, pages 1290-1299, 2022.
[12] Ho Kei Cheng, Jihoon Chung, Yu- Wing Tai, and Chi- nei-Ke Tang. Cascadepsp: Toward class- agnoosii and very high-resolution segmentation via global and local refinement. In CVPR, pages 1890-8899 2020,
[13] Marius Cordis. Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. In CVPRW, volume 2. sn, 2015.
[14] Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, and Gang Wang. Context contrasted feature and gated multi-scale aggregation for scene segmentation. In CVPR, pages 2393-2402, 2018.
[15] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. McViS: A large-scale benchmark for video segmentation with motion expressions. In ICCV, pages (694-27032 2023,
[16] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip H.S. Torr, and Song Bai. MOSE: A new dataset for video object segmentation in complex scenes. In ICCV, pages
2023.
[17] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. J. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 111(1):88-13,, January 2015.
[18] Yuxin Fang. Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. In ICCV, pages 1919,6919, 2021.
[19] Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, and Weiqiang Wang. Diffusioninst: Diffusion model for instance segmentation. CoRR, abs/2211..02733, 2022.
[20] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, pages 5356-5364, 2019.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. CVPR, pages 2961-2969, 2017.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, pages 8840-6851, 2020.
[23] Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu, Cenet: Criss-cross attention for semantic segmentation. In CVPR, pages 603-612, 2019.
[24] Zilong Huang, Yunchao Wei, Xinggang Wang, Wenyu Liu, Thomas Huang, and Humphrey Shi. Alignseg: Feature-aligned segmentation networks. IEEE TPAMI, 44(1):550-557, 2021.
[25] Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 1.01 144, 2016.
[26] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. DDP: diffusion model for dense visual prediction. CoRR, abs/2303. 17559, 2023.
[27] Lei Ke, Martin Danelljan, Xia Li, fu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. In CVPR, pages 4412 1421, 2022.
[28] Lei Ke, Henghui Ding, Martin Danelljan, Yu- fu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Video mask transfiner for high- hughityality video instance segmentation. In ECCV, pages 731-747 Springer, 2022.
11

---

[29] Alexander Kirillov, Yuxin Wu, Kaiming He. and Ross Girshick. Pointrend: rendering. In CVPR, pages 9799-9808, 2020.
Image segmentation as
[30] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Cao, Chunyuan Li, and Yong Jae Lee. GLIGEN: open-set grounded text-to-image generation. CoRR, abs/2301.0700 1070030 2023.
[31] Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, and Raquel Urtasun. Polytrans- form: Deep polygon transformer for instance segmentation. In CVPR, pages 9131-9140, 2020.
[32] Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, and Jiashi Feng. Deep interactive thin object selection. In WACV, pages 305-314, 2021.
[33] Guosheng Lin, Anton Milan, Chunhua Shen, and lan Reid. Refinenet: Multi-pal pathh refinement networks for high-resolution semantic segmentation. In CVPR, pages 1925- 1934, 2017.
[34] Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.
[35] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR. pages 3431-3440, 2015.
[36] Haiyang Mei, Ge-Peng Ji. Ziqi Wei, Xin Yang. Xiaopeng Wei, and Deng-Ping Ping Fan. Camouflaged object segmentation with distraction mining. In CVPR, pages 8772-8781, 2021.
[37] Jiaxu Miao, Xiaohan Wang. Yu Wu, Wei Li, Xu Zhang. Yunchao Wei, and Yi Yang. Large-scale vidco panoptic segmentation in the wild: A benchmark. In CVPR, pages 21033-21043, 2022.
[38] Jiaxu Miao, Yunchao Wei, Yu Wu, Chen Liang, Guangrui Li, and Yi Yang. Vspw: A large-scale dataset for vidco scene parsing in the wild. In CVPR, pages 41 4133-4143, 2021.
[39] Alexander Quinn Nichol and Prafulla Dhariwal, Improved denoising diffussio prohabilistic models. In ICML, pages 9172- PMLR, 2021.
[40] Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. In ECCV, pages 38 56. Springer, 2022.
[41] Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, pages 10684-10695, 2022.
[42] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.
[43] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE TPAMI, 2022.
[44] Tiancheng Shen, Yuechen Zhang, Lu Qi, Jason Kuen, Xingyu Xie. Jianlong Wu. Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR, pages 1310-1319, 2022.
[45] Jascha Sohl-Dickstein, Fric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In ICMI., pages 2256-2265. PMI.R, 2015.
[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[47] Chufeng Tang, Hang Chen, Xiao Li. Jianmin Li, Zhaoxiang Zhang, and Xiaolin Hu. Look closer to segment better: Boundary patch refinement for instance segmentation. In CVPR, pages 13926-13935, 2021.
[48] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV, pages 282-298. Springer, 2020.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, pages 5998-6008, 2017.
[50] Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng. Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE TPAMI, 43(10))334--336
2020.
[51] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In ECCV, pages 649 665. Springer, 2020.
12

---

[52] Junde Wu, Huihui Fang. Yu Zhang, Yehui Yang, and Yanwu Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. CoRR, abs/221 1, 2022.
[53] Junde Wu, Rao Fu, Huihui Fang. Yu Zhang, and Yanwu Xu. Medsegdiff-v2: Diffusion based medical image segmentation with transformer. CoRR, abs/2301.11798. 2023.
[54] Junfeng Wu, Yi Jiang, Song Bai, Wenging Zhang, and Xiang Bai. Seqformer: Sequential transformer for video instance segmentation. In ECCV, pages 553-569. Springer, 2022.
[55] Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang, Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In CVPR, pages 1154-1163. 2021.
[56] Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refinement for segmentation. In ECCV, pages 489-506. Springer, 2020.
[57] Gang Zhang, Xin Lu, Jingru Tan, Jianmin Li, Zhaoxiang Zhang, Quanquan Li, and Xiaolin Hu. Refinemask: Towards high-quality instance segmentation with fine-grained features. In CVPR, pages 6861-6869, 2021.
[58] Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. CoRR, ahs/2302.05543, 2023.
[59] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Ienet for real-time semantic segmentation high-resolution images. In ECCV, pages 40S-420, 2018.
[60] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, pages 2881-2890, 2017.
[61] Peng Zhou, Brian Price, Scott Cohen, Gregg Wilensky, and Larry S Davis. Deepstrip: High-resolution boundary refinement. In CVPR, pages 10558-10567, 2020.
13

---

Appendix
Here first introduce the details of SegRefiner's equation derivation in Sec. A. The implementation details are shown in Sec. B. Moreover, we provide additional qualitative results in Sec. C.
A Equation Derivation Details
In the following, we will present the derivation of the posterior distribution (Eq. (9) and the reverse transition-probablliy (Eq. (11)). Firstly, for Fq. (9), it's derived from the Bayes' theorem:
( g( [2 (o)) (ro)
(12)
According to Eq. (6) and Eq. (7), we can get all the items in Bayes' theorem as:
g (0) x1=1
(13)
g (xo)) = zo 20021-177
(14)
g( (xo) = xo Qux7
(15)
It appears to be slightly different from Eq. (6). The reason is that the results of Eq. (6) is a length-2 vector, representing the probabilities of two states respectively. While we use a scalar form here, which includes additional one-hot vector x!! and is more suitable for subsequent derivations. For the denominator of Eq. (9), we have obtained the same form. Let's consider the numerator:
zo 20) (20) = Qu0 200
= 10
(16)
(a1Q7)
(20Q1-1)07
[(x,07) (100)-))==[11
where "." refers to scalar multiplication and "o" refers to element-win multiplication. This the scalar form, by removing the -7-1, we can get the corresponding vector form just Eq. (9).
Secondly, for Eq. (LI), it's derived from Eq. (9) by substituting zo with (moo)) (
(we refer po po (mojj)) pa in the following for simplicity).
p( = (xx @ (=oQ(-1)) with vp [Po, Pol
N0Qu
xz(QZ @ [po 1 P0B3-11 11)=
[po,3,,
3
(17)
Tt
po p
3r)
BoBB
x--1
2t (Po
1
3o T
0
2t
3,)
1 1-p0
I 1t-1-
1 1-
Same as Eq. (9), we can remove of-1 and get the vector form of Eq. (11).
B Implementation Details
In this section, we give a detailed description of the model architecture and training/inference settings. The overall workflow of the training and inference process provided in Alg. and Alg. 2.
Model Architecture Following [39], we use U-Net with 4-channel input and 1-channel output. Both input and output resolution is set to 256 x 256. Considering computational load and memory usage, we set the intermediate feature channels 128 and only conduct self-attention in strides 16 and 32.
14

---

Algorithm 1 Training
Input total diffusion steps T, datasetsD> ((T, Mine; M Meoarse K) repeat
Sample (1, Menarse) ~
Sample t ~ Uni form(1....... T)
Initialize mg M fines F0 [1,0)
q(x) ( = 00 Qt
Sample xt ~ ((x) /
[ (d)) get Ta E {0, 1j2xHxx
Pixels Transition = x [0] @ Mine + xr(1] o Monarse
Take gradient descent step on VoC(fe(L, me, t), fine)
until convergence
Algorithm 2 Inference
Input total diffusion steps T, image and coarse mask (I, M Mcoarse)
Initialize Tт [0, 1), mr Mcoerse
for in {T.T ..... 1) do
mojzr Po (mojr) = fo(T,
pa 1.j 3.3 )
pi.
Sample 2.. po (x()), get 24 10.112xxxXB
Pixels Transition m = 10] O moje + 21 [1] o Mcoarse
return mg
Training Settings All experiments are conducted on 8 NVIDIA RTX3090 GPUs with Pytorch. During training. we first train the LR-SegRefiner on the LVIS dataset [20] with 120k iterations. The AdamW W optimizer is used with the initial learning rate of 4 x 10 10-4 We use multi-step learning rate schedule, which decays by 0.5 in steps 80k and 100k. Subsequently, the HR-SegRefiner is obtained from 40k-iterations fine-tuning based on the 80k checkpoint of LR-SegRefiner. Batch size is set to in each GPU.
Inference Settings In instance segmentation, we use the LR-SegRefinee to perform refinement in instance level. For each instance, we extract the bounding box region based on the coarse mask and expand it by 20 pixels on each side. The extracted region is then resized to match the input size of the model. After complete reverse diffusion process, the output is resized to the original size.
In semantic segmentation and dichotomous image segmentation, because of the high resolution of images, employ the HR-SegRefiner and conduct global-and local refinement process. In order to identify the local patches that require refinement, we filter out pixels with low state-transition probabilities from the globally refined mask and use them the center points for the local patches. We apply Non-Maximum Suppression (NMS, with 0.3 as threshold) to these patches to remove excessive overlapping.
C Qualitative Results
In this section, we provide more visual results in semantic segmentation, instance segmentation, and dichotomous image segmentation. Fig. shows the comparisons of SegRefiner and other models (including instance segmentation models and refinement models) on COCO [34] validation set. Fig. shows more comparisons between the coarse masks and refined masks on COCO validation set. These results demonstrate that the proposed SegRfiner can robustly correct inaccurate predictions in coarse masks. Fig. and Fig. show visual results on BIG dataset [12] and DIS5K dataset [40]. SegRefiner shows strong capability for capturing extremely fine details on these two high-resolution datasets.
15

---

RCD
(a) Image
RCD
(b) Mask R-CNN [21]
SOLO [51]
(d) Querylnst [18]
(e) CondInst [48]
RCD
(I) PointRend [29]
Figure 5: Visual comparisons with other instance segmentation and refinement methods on COCO dataset. Our SegRefiner can robustly correct prediction errors both outside and inside the coarse mask. (Please refer the next page for the remaining portion of this figure.)
16

---

RCD
MesineMask [57]
RCD
(h) Transfiner [27]
Mask R- R-CNN SegFix [56]
ii Mask R-CNN BPR [47]
RCD
(k) Mask R-CNN + SegRefiner (Ours)
Figure 5: Visual comparisons with other instance segmentation and refinement methods on COCO dataset. Our SegRefiner can robustly correct prediction errors both outside and inside the coarse mask.
17

---

(a) Coarse Mask
(b) Refined Mask
Figure 6: More visual results on COCO dataset. Coarse masks are obtained from Mask R-CNN [21].
Our SegRefiner corrects the errors of coarse masks (see Refined Mask).
18

---

(a) Image
(b) Coarse Mask
(c) Refined Mask
Figure 7: More visual results on BIG dataset [12]. Coarse masks obtained from Deeplab v3+ [5].
Our RegRefiner greatly enhances the mask quality (see Refined Mask). Please kindly zoom in for better view.
19

---

(a) Image
(b) Coarse Mask
(c) Refined Mask
Figure 8: More visual results DIS5K dataset [40]. Coarse masks are obtained from ISNet [40].
Our SegRefiner captures extremely fine details (see Refined Mask). Please kindly zoom in for a better view.
20


Reviews:
[{"review_id": "2nHOCb3nWH", "reviewer": "Reviewer_cPQx", "summary": "This paper proposes a novel method for improving the quality of object masks generated by different segmentation models. The method, called SegRefiner, is model-agnostic, meaning that it can be used with different segmentation models. SegRefiner works by interpreting segmentation refinement as a data generation process. This allows the work to implement the refinement process as a series of denoising diffusion steps.\n\nSpecifically, SegRefiner takes coarse masks as input and refines them using a discrete diffusion process. For each pixel, it predicts the label and corresponding state-transition probabilities. This allows the model to progressively refine the noisy masks in a conditional denoising manner.\n\nThe method is evaluated on a variety of segmentation tasks, and the results show that SegRefiner consistently outperforms previous methods. It improves both the segmentation metrics and boundary metrics across different types of coarse masks. SegRefiner also exhibits a strong capability to capture extremely fine details when refining high-resolution images.", "questions": "Please see the weakness section.", "limitations": "The authors properly addressed the limitations in the paper.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the paper is well written and clearly organized.\n\n2. The method is intuitive and straightforward. The performance improvements are clear compared to the previous baselines.\n\n3. The inference is efficient for diffusion as it only uses 6 timesteps.\n", "weaknesses": "1. Not necessarily a weakness but the model does not clearly state how the coarse masks are generated for training until later sections. While the proposed method follows previous works on this part, it would still improve the clarity if this is discussed in earlier sections and in the overview figures.\n\n2. The paper only shows one example of how each step of the diffusion process works during inference. The authors are recommended to include more examples with some highlighted boundary comparisons.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer cPQx\n### **Q1. The coarse masks used for training**\nWe very appreciate your insightful comments and will clarify how the coarse masks are generated in earlier sections and in the overview figures in our next version.\n\n### **Q2. More examples of inference**\nMore additional visualizations of the inference process are provided in Fig. 1 in the rebuttal PDF file of General Response. We will include these qualitative results in our next version to help readers gain a more intuitive understanding of this work."}, {"review_id": "aVDMDrYoSj", "reviewer": "Reviewer_ygQQ", "summary": "This work proposes diffusion-based image segmentation refinement approach, SegRefiner, using a newly designed discrete diffusion process. SegRefiner performs model-agnostic segmentation refinement that takes in coarse output masks from any segmentation model as an input and refines them through a series of diffusion steps while estimating state-transition probabilities for each pixel. Comprehensive experiments across multiple segmentation tasks demonstrates that SegRefiner consistently improves segmentation metrics and outperforms state-of-the-art model-specific and model-agnostic refinement approaches. ", "questions": "1. How is the posteior in eq(9) derived ? Similarly how do one derive eq(11) ? And hence where is eq(9) used ?\n2. Is the refinement of semantic and instance segmentation performed using zero-shot ? All I understand is LR- and HR-SegRefiner are trained using LVIS and composite dataset respectively.", "limitations": "Yes, authors have address one of the core limitation to using diffusion process. As far as my understanding, another limitation of such class-agnostic and model-agnostic approach is one should undertake each instance one at a time for refinement. This probably due to the fact there is no class information input to SegRefiner model.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality\nThis work has multiple novel contribution ranging from - being first to introduce diffusion-based image segmentation refinement approach, the use of discrete diffusion instead of Gaussian diffusion process of DDPM, being class-agnostic as well as ability to capture extremely fine-details when applied to high-resolution images.\n\nQuality\nThe paper is well-written and mostly easy to follow. Authors have put their best effort to condense lot of details into conference paper length. As the use of discrete diffusion is new in practice, some of the details can be elaborated using examples in the supplementary for the ease audience.\n\nSignificance\nThe approach taken by SegRefiner is unique. For instance, mask and states prediction task is kept separate. Similarly, there is lot of subtle details required to run this successfully as listed under the training strategy, noise schedule and other settings. All these contribution can be quite useful to other researcher pursuing different applications using generative model.", "weaknesses": "None that are significant. Have listed one under the limitation section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer ygQQ\n### **Q1. About Eq. (9) and Eq. (11)**\n**Firstly, for Eq. (9),** it's derived from the Bayes' theorem：\n\\begin{equation}\\begin{split}\nq(x_{t-1}|x_t, x_0) = \\frac{q(x_t|x_{t-1}, x_0)q(x_{t-1}|x_0)}{q(x_t|x_0)}.\n\\end{split}\\end{equation}\n\nAccording to Eq. (6) and Eq. (7), we can get all the items in Bayes' theorem as:\n\\begin{equation}\\begin{split}\nq(x_t|x_{t-1}, x_0) &= x_{t-1} Q_t x_t^T, \\\\\\\\\nq(x_{t-1}|x_0)) &= x_{0} \\bar Q_{t-1} x_{t-1}^T, \\\\\\\\\nq(x_{t}|x_0)) &= x_{0} \\bar Q_{t} x_{t}^T.\n\\end{split}\\end{equation}\nIt appears to be slightly different from Eq. (6). The reason is that the results of Eq. (6) is a length-2 vector, representing the probabilities of two states respectively. While we use a scalar form here, which includes an additional one-hot vector $x_*^T$ and is more suitable for subsequent derivations. For the denominator of Eq. (9), we have obtained the same form. Let's consider the numerator:\n\\begin{equation}\\begin{split}\nq(x_t|x_{t-1}, x_0)q(x_{t-1}|x_0) &= x_{t-1} Q_t x_t^T \\cdot x_0 \\bar Q_{t-1} x_{t-1}^T \\\\\\\\\n&= x_{t-1} (x_t Q_t^T)^T \\cdot x_0 \\bar Q_{t-1} x_{t-1}^T \\\\\\\\\n&= (x_t Q_t^T)x_{t-1}^T \\cdot (x_0 \\bar Q_{t-1}) x_{t-1}^T \\\\\\\\\n&= \\[(x_t Q_t^T) \\odot (x_0 \\bar Q_{t-1})\\] x_{t-1}^T ,\n\\end{split}\\end{equation}\nwhere \"$\\cdot$\" refers to scalar multiplication and \"$\\odot$\" refers to element-wise multiplication. This is the scalar form, by removing the $x_{t-1}^T$, we can get the corresponding vector form just as Eq. (9).   \n\n**Secondly, for Eq. (11),** it's derived from Eq. (9) by substituting $x_0$ with $[p_0, 1-p_0]$ ($p_0$ refers to $p_\\theta(\\tilde m_{0|t})^{i,j}$ in Line 162 in the paper for simplicity). Let's start from the scalar form above:\n\\begin{equation}\\begin{split}\np(x_{t-1}|x_{t}) \n&= \\frac{\\[(x_t Q_t^T) \\odot (x_0 \\bar Q_{t-1})\\] x_{t-1}^T}{x_{0} \\bar Q_{t} x_{t}^T} \\quad | \\quad x_0=\\[p_0, 1-p_0\\] \\\\\\\\\n&= \\frac{x_t (Q_t^T \\odot \\[p_0 \\bar \\beta_{t-1}, 1 - p_0 \\bar \\beta_{t-1}\\] ) x_{t-1}^T}{\\[p_0 \\bar \\beta_{t}, 1 - p_0 \\bar \\beta_{t}\\] x_t^T} \\\\\\\\\n&= \\frac{x_t \\begin{bmatrix}\n    p_0 \\bar \\beta_t & 0 \\\\\\\\\n    p_0 (\\bar \\beta_{t-1} - \\bar \\beta_t) & 1 - p_0 \\bar \\beta_{t-1}\n    \\end{bmatrix} x_{t-1}^T}{x_t \\[p_0 \\bar \\beta_{t}, 1 - p_0 \\bar \\beta_{t}\\]^T} \\\\\\\\\n&= x_t \\begin{bmatrix}\n   1 & 0 \\\\\\\\\n    \\frac {p_{0} (\\bar \\beta_{t-1} - \\bar \\beta_t)}{1 - p_0 \\bar \\beta_{t}} & \\frac{1 - p_0 \\bar \\beta_{t-1}}{1 - p_0 \\bar \\beta_{t}} \\\\\\\\\n    \\end{bmatrix} x_{t-1}^T.\n\\end{split}\\end{equation}\nSame as Eq. (9), we can remove $x_{t-1}^T$ and get the vector form of Eq.(11). Therefore, Eq. (9) is used to obtain Eq. (11). This follows the same logic of DDPM, which substitutes $x_0$ in the posterior with a predicted one and gets the reversed probability distribution.   \n\n### **Q2. Whether zero-shot or not**\nThe proposed method is class-agnostic and model-agnostic, offering the advantage of being applied in a zero-shot manner. The reported experiments, while indicative of the approach's effectiveness, do not constitute a fully rigorous zero-shot evaluation:\n* Experiment of instance segmentation is conducted on the LVIS validation set, the training set of which is used to train the LR-SegRefiner, thus it's not zero-shot.\n* For semantic segmentation, we use the HR-SegRefiner and evaluate it on the BIG dataset. It indeed involves different training and testing datasets. While many categories in BIG dataset are also presented in the composite training data. Thus it's not a completely rigorous zero-shot experiment.  \n\nTo evaluate the zero-shot capability of our approach, we report results on the FSS-1000 dataset \\[A\\], which is comprised of 1000 uncommon categories. The results on 4 unseen/novel categories (column 2 to column 5) and across the entire dataset（column 6）are presented in the table below, where the coarse masks are derived from the baseline of \\[A\\]. It can be observed that our approach achieves consistent improvements in a zero-shot manner on these uncommon categories.\n\n| Category                       | Rice cooker | Nintendo Switch | Windmill | Hot water heater | All categories |\n|:------------------------------ |:-----------:|:---------------:|:--------:|:----------------:|:-----------:|\n| Baseline              |    91.1     |      80.5       |   90.3   |       83.8       |    91.1     |\n| &nbsp;&nbsp;&nbsp;&nbsp;+ Ours |    96.3     |      94.2       |   91.7   |       87.1       |    93.4     |\n| Improvement                    |     +5.2     |      +13.7       |   +1.4    |       +3.3        |     +2.2     |\n\n\\[A\\] Li, Xiang, et al. \"FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation.\" CVPR'20.    \n\n\n### **Q3. Only one instance at a time**\nThank you for your insightful comments. It's truly a limitation that our method can only refine one mask at a time. While it's not due to its class-agnostic manner. Instead, it is constrained by two factors:\n* Firstly, refining multiple masks at once would require taking the entire image region as input, resulting in a smaller-scale output for each mask and the loss of fine-grained information. \n* Secondly, even if we tolerate the information loss, a larger challenge arises from the widespread occurrence of overlap between multiple coarse masks, which means we would need to establish a multiple-input-output parallel framework. While, without significant modifications to the model architecture, this is not significantly different from independently refining multiple masks and distinguishing them in a batch.  \n\nSince this work focuses on exploring the feasibility of diffusion in the refinement task and the superiority of our diffusion process, we have chosen the simplest approach: maintaining the U-Net architecture used in previous diffusion-based works unchanged and refining every mask independently. It's an important direction we will continue to explore in the future to design an efficient parallel model architecture to enable our method to handle multiple masks at once."}, {"review_id": "1v4QuwHAYA", "reviewer": "Reviewer_cXkh", "summary": "This manuscript presents a model-agnostic solution for segmentation refinement, which implements the refinement process of the coarse masks through a series of denoising diffusion steps. Comprehensive experiments on various segmentation tasks are conducted, including semantic segmentation, instance segmentation, and dichotomous image segmentation and the results demonstrate the superiority of the proposed method.", "questions": "See above comments.", "limitations": "As mentioned in the conclusion, the diffusion process seems to lead to a slowdown in inference due to the multi-step iterative strategy. As a post-processing step for segmentation tasks, this is indeed a major drawback. Did the authors conduct any experiments to examine the relationship between iteration step, accuracy, and speed?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- They obtain SOTA results on multiple tasks. \n- The idea sounds quite interesting. \n- This manuscript is well-formulated and -presented.", "weaknesses": "The motivation behind lacks focus. The manuscript primarily discusses the problem of the existing methods of segmentation refinement in Introduction but fails to indicate the reason why denoising diffusion models are suitable and important for the task of segmentation refinement. What are the technical challenges in applying denoising diffusion models to segmentation refinement? To address these challenges, what innovations does this paper propose regarding the application of denoising diffusion models to segmentation refinement?\n\nHow is the value of the state transition probability specifically calculated?\n\nThere are several instances where the use of others' work, such as Gumbel-max sampling, has not been properly cited.\n\nFor the DIS experiment, commonly used evaluation metrics for DIS are not employed.\n\nSome grammar and spelling errors. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer cXkh\n### **Q1. (a) Why using diffusion**\nAs mentioned in Line 40 and Fig. 1 in the paper, we found there are diverse types of errors presented in the coarse masks. This is the reason for the underperformance of existing methods because attempting to address all these diverse errors at once is extremely challenging. Therefore, a very natural idea is: **since correcting all errors simultaneously is too difficult, can we make the model correct only some \"most obvious errors\" at each step and iteratively converge to an accurate result?**  \n\nRegarding this idea, we found that this is precisely how the denoise diffusion models tackle the challenges in generative tasks: diffusion models decompose the fitting process from Gaussian noise to data samples into multiple time steps, making it easier and more accurately to fit the distribution of data samples. Therefore, transferring the diffusion model to refinement task is a very natural way to achieve the aforementioned objective.\n\n### **Q1. (b) Technical challenges and innovations in applying denoising diffusion models**\nThere are two main challenges in applying denoising diffusion models to segmentation refinement task:\n* Firstly, most existing diffusion models adopt the Gaussian assumption, representing the forward diffusion process as adding Gaussian noise, which is not suitable for binary segmentation predictions.  \n* Secondly, nearly all existing diffusion frameworks have employed random noise as the origination for sampling, whereas we aim to initiate sampling from the coarse mask, which is a binary foreground vs. background map. \n\nGiven these two challenges, The proposed method introduces the following innovations:\n* We define every pixel in the segmentation results to have two states, corresponding to fine segmentation (ground truth) and coarse segmentation. As a result, the forward and backward processes of diffusion are naturally represented as stochastic states-transition, solving the first challenge.\n* We formulate the states-transition as a unidirectional process, thus ensuring its convergence to the coarse mask during forward process, addressing the second challenge. \n* Furthermore, in the reverse diffusion process, we devised a confidence-based states-transition strategy (as shown in Eq. (11) in the paper), which ensures that pixels with the highest confidence predictions are updated first. This naturally achieves the objective of refining \"most obvious errors\" at each step, as mentioned in Q1.(a).\n\n\n### **Q2. Value of the states-transition probability**\nThe states-transition probability of forward process is defined in Eq. (6), which is determined by hyperparameter $\\beta$. In the reverse process, states-transition probability is defined in Eq. (11), which is derived from Eq. (9) by substituting $x_0$ with $p_\\theta(\\tilde m_{0|t})^{i,j}$ as mentioned in Line 162 in the paper.\n\n### **Q3. Citation, grammar and spelling problem**\nThank you for your valuable and thoughtful comments. We will carefully check the citation and English writing throughout the entire paper and make corrections in the revision.\n\n### **Q4. DIS evaluation metrics**\nIn the main paper, we chose IoU and mBA as metrics for the DIS experiment because these two metrics are the most intuitive in reflecting mask quality and accuracy. Regarding your concern, we also report the 5 metrics used in the original paper of DIS, as shown in **Tab. 1 in the rebuttal PDF file of General Response**. Experimental results demonstrate that our method consistently improves all metrics, indicating its effectiveness in improving segmentation accuracy and achieving high-quality mask refinement under different focuses.\n\n### **Q5. The relationship between iteration step, accuracy, and speed**\nWe conduct the following experiment on instance segmentation about the relationship between the model's accuracy, computational complexity, time consumption (the average time consumed per image), and the number of iteration steps. Results are shown in the table below. It can be observed that both time consumption and computational complexity increase linearly with the number of steps, while accuracy reaches saturation around 5-6 steps.    \n\n| Steps       |  0   |  1   |  2   |  3   |  4   |  5   |  6   |  7   |  8   |\n|:----------- |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| Mask AP     | 39.8 | 40.7 | 41.0 | 41.4 | 41.6 | 41.9 | 41.9 | 41.9 | 41.9 |\n| Boundary AP | 27.3 | 29.9 | 30.6 | 31.5 | 32.1 | 32.5 | 32.6 | 32.6 | 32.6 |\n| Time(s)     | n/a  | 0.52 | 1.01 | 1.53 | 1.98 | 2.45 | 2.94 | 3.44 | 3.92 |\n| FLOPs(G)    | n/a  | 249  | 497  | 746  | 994  | 1243 | 1492 | 1740 | 1989 |\n\nWe also provide a comparison between our method and existing methods in the following table. For semantic segmentation:\n\n| Method     | Time(s) | FLOPs(G) | Params(M) |\n| ---------- |:-------:|:--------:|:---------:|\n| CascadePSP |   6.2   |  26518   |   67.62   |\n| CRM        |   4.3   |   2356   |   9.27    |\n| **Ours**   |   1.6   |   1492   |  118.93   |\n\nFor instance segmentation:\n\n| Method   | Time(s) | FLOPs(G) | Params(M) |\n| -------- |:-------:|:--------:|:---------:|\n| BPR      |   1.4   |   135    |   65.85   |\n| **Ours** |   2.9   |   1492   |  118.93   |\n\nResults indicate that, in semantic segmentation, our method demonstrates faster speed and lower computational complexity. However, in instance segmentation, our method is slower than previous approaches.  \n\nThis is largely due to the high computational complexity of the U-Net model used in previous diffusion-based works (with 249 GFLOPs per step). In this work, we focuses on exploring the feasibility and superiority of diffusion process in the segmentation refinement task, thus we maintain the model structure unchanged. In the future, we will continue to explore more efficient segmentation refinement by replacing the U-Net used in our method with a more lightweight architecture.  "}, {"review_id": "EbGb5DXcUO", "reviewer": "Reviewer_oFWK", "summary": "In this paper, the authors propose a model-agnostic solution called SegRefiner to enhance the quality of object masks generated by different segmentation models. The approach introduces a novel perspective on segmentation refinement, treating it as a data generation process. By leveraging a series of denoising diffusion steps, SegRefiner achieves effective mask refinement.\nSpecifically, SegRefiner takes coarse masks as inputs and employs a discrete diffusion process to refine them. By predicting the label and states-transition probabilities for each pixel, the method progressively refines the noisy masks in a conditional denoising manner. To evaluate the performance of SegRefiner, extensive experiments are conducted on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation.\nThe experimental results demonstrate the superiority of SegRefiner in multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics for different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, SegRefiner exhibits a strong capability to capture extremely fine details when refining high-resolution images.", "questions": "I'm positive about this paper. I really like the idea to apply diffusion model to segmentation task. However, I have some concerns in the Weaknesses. I hope to see the response from the authors for those questions. Thank you.", "limitations": "The diffusion process leads to slowdown of the inference due to the multi-step iterative strategy.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper is the first one to introduce diffusion-based refinement for segmentation masks.\n(2) The proposed method, called SegRefiner, is model-agnostic, and thus applicable across different segmentation models and tasks.\n(3) The authors extensively analyze the performance of SegRefiner across various segmentation tasks, demonstrating that our SegRefiner not only outperforms all previous model agnostic refinement methods, but can also be effortlessly transferred to other segmentation tasks without any modification.", "weaknesses": "(1) In the Objective Function of sec 4.1, how do you specify the value of alpha? Do you have any ablation studies on the alpha?\n(2) In Table 3, it seems that the improvement on Boundary AP is larger than Mask AP. Why is that?\n(3) In Table 4, it seems that the improvement on ISNet is smaller than the other datasets. Could you explain the reason?\n(4) Why do you only do the ablation studies on semantic segmentation task? Have you ever tried the ablation studies on instance segmentation and Dichotomous Image Segmentation? Have you ever tried ablation studies on the number of iterations and output size? For Table 5 (c), have you ever tried a larger input size, like 1024x1024?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer oFWK\n### **Q1. Value of $\\alpha$ in the Objective Function**\nRegarding your concern, we do the following ablation study on instance segmentation with coarse masks from MaskRCNN (ResNet-50). The first row of the Table below refers to not using texture loss for training, and the remaining three rows show results with different values of $\\alpha$. \n\n| $\\mathcal{L}_{texture}$ | Mask AP | Boundary AP |\n| ----------------------- |:-------:|:-----------:|\n| None                    |  41.5   |    31.5     |\n| $\\alpha$ = 1            |  41.9   |    32.1     |\n| $\\alpha$ = 5            |  **41.9**   |    **32.6**     |\n| $\\alpha$ = 10           |  41.7   |    32.5     |\n\nIt can be observed that, on one hand, texture loss $\\mathcal{L}_{texture}$ effectively enhances the model's ability to capture boundary details. On the other hand, when the weight of texture loss becomes too large, it may compromise the model's accuracy.  \n\nThis is in line with intuition. Since texture loss only incurs loss at the edges of the segmentation results, it can enhance the model's sensitivity to capture edge information. However, as these edge details often stem from low-level features, an excessively large weight may cause the model to overly focus on low-level features and overlook semantic consistency, resulting in erroneous predictions.  \n\nTherefore, in our experiments, we chose $\\alpha=5$, which well balances the two losses. \n\n### **Q2. Why Boundary AP improves more**\nWe greatly appreciate your observation! The more enhancements on Boundary AP effectively showcase the SegRefiner's capacity to capture more edge details and better refine coarse predictions. As illustrated in Fig. 1 and 4 of the main paper, existing segmentation models can provide segmentation results that roughly resemble the real shape of objects, but exhibit significant coarseness in edge details. While Mask IoU/AP is not sensitive to these edge errors, as the pixels near edges constitute only a small portion of the foreground. Instead, Boundary IoU/AP is solely related to the prediction accuracy of the edge region, which better reflects the precision of segmentation results in terms of fine details. For a clear illustration of this, please refer to **Fig. 2 in the rebuttal PDF file of General Response**. \n\n### **Q3. Why results on ISNet improve less**\nSince ISNet is specifically designed for the DIS task and utilizes a specialized method of intermediate supervision strategy, it is better suited for the DIS task compared to other baselines in Tab. 4. As a result, predictions of ISNet has fewer potential errors that need to be corrected, leading to smaller improvement with our method.\n\n### **Q4. (a) Ablation studies on other tasks**\nThere are two main reasons: \n* Compared to instance segmentation, semantic segmentation and Dichotomous Image Segmentation involve both global and local refinements, allowing for analyzing more variables in ablation studies. \n* Refinements on semantic segmentation and Dichotomous Image Segmentation share exactly the same experimental settings. While the dataset for semantic segmentation is smaller. Considering the time and computational resources required for experiments, conducting ablation studies on semantic segmentation can be more efficient.  \n\nWe have also conducted the ablation study on “effectiveness of the diffusion process” on instance segmentation to address your concern. As shown in following table, the results are consistent with those in Tab. 5(a) in the paper.\n\n| Iteration Type | Mask AP | Boundary AP |\n|:--------------:|:-------:|:-----------:|\n|      none      |  40.9   |    30.1     |\n| w/o diffusion  |  41.4   |    31.0     |\n|  w/ diffusion  |  41.9   |    32.6     |\n\n\n### **Q4. (b) Ablation studies on iterations**\nWe conduct the following experiment in the table below about the number of iteration steps. It can be observed that both time consumption and computational complexity increase linearly with the number of steps, while accuracy reaches saturation around 5-6 steps. Considering the time and computational resource consumption, we aim to keep the number of iterations as low as possible. Therefore, we set it to 6.\n\n| Steps       |  0   |  1   |  2   |  3   |  4   |  5   |  6   |  7   |  8   |\n|:----------- |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| Mask AP     | 39.8 | 40.7 | 41.0 | 41.4 | 41.6 | 41.9 | 41.9 | 41.9 | 41.9 |\n| Boundary AP | 27.3 | 29.9 | 30.6 | 31.5 | 32.1 | 32.5 | 32.6 | 32.6 | 32.6 |\n| Time(s)     | n/a  | 0.52 | 1.01 | 1.53 | 1.98 | 2.45 | 2.94 | 3.44 | 3.92 |\n| FLOPs(G)    | n/a  | 249  | 497  | 746  | 994  | 1243 | 1492 | 1740 | 1989 |\n\n\n### **Q4. \\(c\\) Ablation studies on output size**\nIn Tab. 5\\(c\\), we conducted ablation studies only on three output sizes: 128, 256, and 512. The reason is that smaller sizes (e.g., 64) would result in less detailed output even on low-resolution datasets like COCO and LVIS, leading to the loss of many fine-grained details. On the other hand, larger sizes (e.g., 1024) are wasteful since they already exceed the original size of images in low-resolution datasets. Additionally, they would introduce unacceptable computational complexity and memory usage.  "}, {"review_id": "qQxBV7o0CS", "reviewer": "Reviewer_Xhma", "summary": "This paper develops a method based discrete diffusion process to refine coarse segmentation masks with input images in the context of semantic image segmentation. In this work, the refinement problem has been formulated as an iterative data generation process. To work with binary masks, as adapted for most segmentation scenarios, the authors proposed discrete state translation with mathematical analysis. Experimental results seem solid about the superiority of the proposed methods against several parallel methods on three segmentation tasks.", "questions": "- throughout the paper, the authors utilized examples with binary masks to illustrate the idea, including Fig. 2, Equ. 6-11, etc. Could the authors explain how exactly to apply this method to a multi-class segmentation problem?\n\n- Since the authors utilize Gumbel-softmax sampling in the forward, it might exhibit some kind of diversities for the transitioned masks in the intermediate steps (even the final output–coarse mask remains the same). However, for the backward process, if my understanding is not wrong, how can this algorithm ensure a certain degree of diversity as no stochastic operations are involved?\n\n- The forward process has ground truth for the source and target masks, while for the backward process, the paper applied simple thresholding. This can cause instability of convergence. Did the authors meet any optimization difficulties?\n\n- In the training strategy, why exactly two versions of SegRefiner were proposed?", "limitations": "The authors have adequately addressed the limitation of this paper while I am expecting more details, as stated in the previous comments.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: this paper is new for its adaptation of discrete prior to tackling binary segmentation masks. although the discrete diffusion model is not new, its application in segmentation can be an interesting attempt.  In order to handle a smooth translation from a fine mask to a coarse one, the authors proposed a transition sample module in the forward process. This is different from adding traditional Gaussian noise as in continuous diffusion models. \n\nQuality: the quality of this paper is relatively good, through multiple reading passes. The paper is kind of self-contained, including strong motivation, mathematical analysis, and experimental results. \n\nClarity: This paper is not easy to read for my first pass. After reading it several times, one can have a better understanding of certain concepts and illustrations\n\nSignificance: I give high significance to this paper as it solves a common problem in a novel way as well as the obtained results seem to be greatly improved in a wide range of segmentation tasks and datasets.", "weaknesses": "Personally, the biggest concern of this paper is the clarification. Basically, the authors devised a discrete process to bridge the fine mask with the coarse one, by using a transitioned mask. This mask is unidirectional, which means only the transition to the target happens.  This forward process consists of a continuous Q_{t} and a discrete sampling based on probability defined on Q_{t}. These processes are not well illustrated and explained in the paper. In addition, the paper refers to source and target masks differently for forward and backward processes, which can confuse readers and a better nomination would be suggested.\n\nAlthough this paper presents extensive experimental results, it lacks a comparison of computational complexity and time and resources. Despite the fact that the authors had a small discussion on this aspect in the conclusion, it is much better to provide quantitative results either in the main paper or in the supplementary material.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Xhma\n### **Q1. Clarification of Method Description**\nThank you for your valuable and thoughtful comments and we will rewrite the Method section to enhance clarification. Due to the confusion caused by \"source and target masks\", we will replace them with \"fine and coarse masks\" in both forward and reverse process. The description of the overall method will be modified as follows: \n* In the forward process, fine mask corresponds to the ground truth. The forward process involves binary sampling based on the probability $Q_t$ (Eq. 6), gradually transforming the fine mask into a coarse one, yielding masks of varying coarseness levels used for training. \n* In the reverse process, coarse mask is the previous segmentation result which needed to be refined, while the fine mask comes from our model's prediction. The reverse process involves sampling according to the probability $P_t$ (Eq. 11) and progressively correcting coarse predictions.\n### **Q2. Quantitative results of accuracy, computational complexity and time**\nTable below shows the relationship between the model's accuracy, efficiency and the number of iteration steps on instance segmentation. It can be observed that both time consumption and computational complexity increase linearly with the number of steps, while accuracy reaches saturation around 5-6 steps.    \n\n| Steps       |  0   |  1   |  2   |  3   |  4   |  5   |  6   |  7   |  8   |\n|:----------- |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| Mask AP     | 39.8 | 40.7 | 41.0 | 41.4 | 41.6 | 41.9 | 41.9 | 41.9 | 41.9 |\n| Boundary AP | 27.3 | 29.9 | 30.6 | 31.5 | 32.1 | 32.5 | 32.6 | 32.6 | 32.6 |\n| Time(s)     | n/a  | 0.52 | 1.01 | 1.53 | 1.98 | 2.45 | 2.94 | 3.44 | 3.92 |\n| FLOPs(G)    | n/a  | 249  | 497  | 746  | 994  | 1243 | 1492 | 1740 | 1989 |\n\nWe also provide a comparison between our method and previous methods in the following table. For semantic segmentation:\n\n| Method     | Time(s) | FLOPs(G) | Params(M) |\n|:---------- |:-------:|:--------:|:---------:|\n| CascadePSP |   6.2   |  26518   |   67.62   |\n| CRM        |   4.3   |   2356   |   9.27    |\n| **Ours**   |   1.6   |   1492   |  118.93   |\n\nFor instance segmentation:\n\n| Method   | Time(s) | FLOPs(G) | Params(M) |\n|:-------- |:-------:|:--------:|:---------:|\n| BPR      |   1.4   |   135    |   65.85   |\n| **Ours** |   2.9   |   1492   |  118.93   |\n\nResults indicate that, in semantic segmentation, our method demonstrates faster speed and lower computational complexity. However, in instance segmentation, our method is slower than previous approaches.  \nThis is largely due to the high computational complexity of the U-Net model used in previous diffusion-based works (with 249 GFLOPs per step). While this work focuses on exploring the feasibility and superiority of diffusion process in refinement task, thus we simply maintain the model structure unchanged. In the future, we will continue to explore more efficient segmentation refinement by replacing the U-Net with a more lightweight architecture.\n### **Q3. Applying to multi-class segmentation problem**\nAs mentioned in the paper, our method refines coarse segmentation results in a class-agnostic manner. Its mechanism is first obtaining prompts from the coarse mask to determine the objects needed to be segmented and then providing accurate masks. Thus our method only needs to perform binary foreground-background prediction without the need to predict categories.  As for coarse segmentation results with multiple classes, the proposed method can refine the coarse mask of every class sequentially.  \n### **Q4. Gumbel-softmax. Stochastic operations in backward process**\nIn the backward process (referred as reverse process in the paper), there are also stochastic operations. During the backward process, our method first get the predicted fine mask and the reversed states-transition probability as shown in Eq.(8)-(11). Then, it performs Gumbel-softmax sampling on this probability and replace the sampled pixels with the values of the predicted fine mask. This process is also stochastic, ensuring a certain degree of diversity.\n### **Q5. Thresholding in backward process**\nWe are sorry that we may not have fully understood your meaning. If the \"backward process\" you referred to is the reverse diffusion process, it's the inference process of our method and does not affect the stability of convergence and optimization of training.  \nWhile there is another similar concern. As you mentioned, during the inference process, we replace the ground truth with a predicted mask, which could potentially introduce a gap between training and inference. To address this issue, we propose the states-transition strategy shown in Eq. (11). The states-transition probability is related to the confidence of the prediction, ensuring that pixels with high confidence are prioritized to be replaced by the predicted values. This helps to bridge the gap between training and inference.\n### **Q6. Why two versions**\nThis is mainly because low-resolution and high-resolution segmentation tasks have different characteristics, resulting in significant distribution gap between corresponding datasets. \n* Low-resolution segmentation focus on accurately identifying foreground objects without introducing semantic errors, and the datasets exhibit complex scenes with multiple objects. \n* High-resolution segmentation emphasizes capturing very fine details (such as hair, railings, etc.), and the datasets are object-centric.  \n\nIf mixing these two types of datasets together for training, the distribution gap can lead to optimization difficulties and affecting the model's accuracy on both low and high resolutions. Therefore, in this work, we have chosen the simplest approach: separating it into two versions. While it's possible to combine different-resolution datasets for training with a more carefully crafted strategy, and we will continue to explore this issue in our future work."}]

Author rebuttal:
## General Response
We have uploaded a PDF file here, which includes:
* **Table 1:** (to Reviewer cXkh Q4) An additional experiment to evaluate SegRefiner on DIS-5K dataset with the commonly used metrics for Dichotomous Image Segmentation. 
* **Figure 1:** (to Reviewer cPQx Q2) Visualizations of how each step of SegRefiner works during inference.
* **Figure 2:** (to Reviewer oFWK Q2) An example to illustrate why Boundary AP improves more in refinement of instance segmentation.

Meta-review: The submission was reviewed by 5 reviewers. All reviewers agree that the ideas presented in the paper are interesting and find the approach to sound. Moreover, the reviewers complement the presentation of the paper and find the validation convincing. AC agrees with the reviewers and recommend to accept the submission.
Decision: Accept (poster)