Title: SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process
Venue: NeurIPS 2023
Accepted: yes
Keywords: Refinement

Segmentation

Discrete Diffusion
Abstract: In this paper, we explore a principal way to enhance the quality of object masks produced by different segmentation models. We propose a model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement as a data generation process. As a result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks as inputs and refines them using a discrete diffusion process. By predicting the label and corresponding states-transition probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits a strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at [SegRefiner.git](https://github.com/MengyuWang826/SegRefiner)

SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process

Abstract

In this paper, we explore a principal way to enhance the quality of object masks produced by different segmentation models. We propose a model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement as a data generation process. As a result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks as inputs and refines them using a discrete diffusion process. By predicting the label and corresponding states-transition probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits a strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at github.com/MengyuWang826/SegRefiner.

1 Introduction

Although segmentation in image [35, 23, 24, 14, 11] and video [54, 37, 38, 16, 15] has been extensively studied in the past decades, obtaining accurate and detailed segmentation masks is always challenging since high-quality segmentation requires the model to capture both high-level semantic information and fine-grained texture information to make accurate predictions. This challenge is particularly pronounced for images with resolutions of 2K or higher, which requires considerable computational complexity and memory usage. As a result, existing segmentation algorithms often predict masks at a smaller size, inevitably leading to lower accuracy due to the loss of fine-grained information during downsampling.

Since directly predicting high-quality masks is challenging, some previous works have shifted their attention to the refinement of coarse masks obtained from a preceding segmentation model. A popular line of direction [29, 57, 27, 28] is to augment the segmentation models (and features) with a new

(a) Image

(b) Coarse Mask

(c) CRM [44]

(d)

SegRefiner

(ours)

module for masks correction. However, such approaches are usually model-specific, and hence cannot be generalized to refine coarse masks produced by other segmentation models. Another part segmentation refinement works [47, 56, 12, 44], on the other hand, resort to model-agnostic approach by taking only an image and the coarse segmentation as input for refinement. These methods have greater practical utility, as they are applicable to refine different segmentation models. However, the diverse types of errors ( e.g. , errors along object boundaries, failure in capturing fine-grained details in high-resolution images and errors due to incorrect semantics) presented in the coarse masks pose a great challenge to the refinement model thus causing underperformance (refer to Fig. 1).

In response to this challenge, we draw inspiration from the working principle of denoising diffusion models [45, 22, 2]. Diffusion models perform denoising at each timestep and gradually approach the image distribution through multiple iterations. This iterative strategy significantly reduces the difficulty of fitting the target distribution all at once, empowering diffusion models to generate highquality images. Intuitively, by applying this strategy to the segmentation refinement task, refinement model can focus on correcting only some 'most obvious errors' at each step and iteratively converge to an accurate result, thus reducing the difficulty of correcting all errors in a single pass and enabling refinement model to handle more challenging instances.

Under this perspective, we present an innovative interpretation of the task by representing segmentation refinement as a data generation process. As a result, refinement can be implemented through a sequence of denoising diffusion steps with coarse segmentation masks being the noisy version of ground truth. To work with binary masks, we further devise a novel discrete diffusion process, where every pixel performs unidirectional randomly states-transition . The proposed process can gradually convert ground truth into a coarse mask during training and employ the coarse mask as sampling origination during inference. In other words, we formulate the mask refinement task as a conditional generation problem, where the input image serves as the condition for iteratively updating/refining the erroneous predictions in the coarse mask.

To our best knowledge, we are the first to introduce diffusion-based refinement for segmentation masks. Our method, called SegRefiner, is model-agnostic, and thus applicable across different segmentation models and tasks. We extensively analyze the performance of SegRefiner across various segmentation tasks, demonstrating that our SegRefiner not only outperforms all previous model-agnostic refinement methods (with +3.42 IoU, +2.21 mBA in semantic segmentation and +0.9 Mask AP, +2.2 Boundary AP in instance segmentation), but can also be effortlessly transferred to other segmentation tasks ( e.g. , the recently proposed dichotomous image segmentation task [40]) without any modification. Additionally, SegRefiner demonstrates a strong capability for capturing extremely fine details when applied to high-resolution images (see Fig. 1 and Fig. 4).

2 Related Work

Segmentation Refinemement The aim of segmentation refinement is to improve the quality of masks in pre-existing segmentation models. Some works focus on enhancing specific segmentation models. PointRend [29] employs an MLP to predict the labels of pixels with low-confidence scores output from Mask R-CNN [21]. RefineMask [57] incorporates a semantic head to Mask R-CNN as additional guidance. MaskTransfiner [27] employs an independent FCN [35] to detect incoherent regions and refines their labels with a Transformer [49]. These works have demonstrated notable performance enhancements over their preceding models. However, their scope of improvement is model-specific and they lack the capacity to directly refine the coarse mask derived from other models. There are also some refinement methods that adopt model-agnostic approaches, such as [47, 56, 61, 12, 44, 31]. These strategies emphasize utilizing diverse forms of input, including whole images, boundary patches, edge strips, etc. Even though these techniques can refine coarse masks derived from different models, their applicability remains confined to specific segmentation tasks. As a special case, SegFix [56], which learns a mapping function between edge pixels and inner pixels and subsequently replaces inaccurate edge predictions with corresponding inner pixel predictions, is employed in both semantic segmentation and instance segmentation within the Cityscapes dataset [13]. While the performance of SegFix is significantly constrained by its ability to accurately identify objects within an image, which consequently leads to a decline in performance on datasets with a more extensive range of categories ( e.g. , COCO [34]).

Diffusion Models for Detection and Segmentation Recently, diffusion models have received a lot of attention in research. Initial studies [45, 22, 2, 46, 8, 3] primarily sought to enhance and expand the diffusion framework. Following these, subsequent works ventured to incorporate diffusion models across a broader array of tasks [43, 9, 30] and to formulate comprehensive conditional generation frameworks [41, 58]. The application of diffusion models to detection and segmentation tasks has also been the focus of an escalating number of studies. Baranchuk et al. [4] capture the intermediate activations from diffusion models and employ an MLP to execute per-pixel classification. DiffusionDet [6] and DiffusionInst [19] adapt the diffusion process to perform denoising in object boxes and mask filters. Some other works [1, 52, 53, 7, 26] consider the image segmentation task as mask generation. These studies predominantly follow the Gaussian diffusion process of DDPM [22] and leverage an additional image encoder to extract image features as condition to generate masks. To the best of our knowledge, our SegRefiner is the first work that applies a diffusion model to the image segmentation refinement task, and it also pioneers the abandonment of the Gaussian assumption in favor of a newly designed discrete diffusion process in diffusion-based segmentation tasks.

3 Methodology

3.1 Preliminaries: Diffusion Models

Diffusion models consist of a forward and a reverse process. The forward process q ( x 1: T | x 0 ) uses a Markov or None-Markov chain to gradually convert the data distribution x 0 ∼ q ( x 0 ) into complete noise x T whereas the reverse process deploys a gradual denoising procedure p θ ( x 0: T ) that transforms a random noise back into the original data distribution.

Continuous Diffusion Models The majority of existing continuous diffusion models [22, 46, 41] adheres to the Gaussian assumption and defines p ( x T ) = N ( x T | 0 , 1) . The mean and variance of forward process are defined by a hyperparameter β t and the reverse process utilizes mean and variance from model predictions, thus formulating as:

Discrete Diffusion Models Compared to continuous diffusion models, there is less research on discrete diffusion models. Sohl-Dickstein et al. [45] first introduce binary diffusion to reconstruct onedimensional noisy binary sequences. x T is defined to adhere to the Bernoulli distribution B ( x T | 0 . 5) . The forward process and reverse process are represented as:

Forward

Reverse

𝑚 "

!|#

𝑝

$

𝑚 "

!|#

𝑚 "

!

𝑚 "

!

U-Net

𝑛𝑛

$

gumbel

noise

Gumbel-Max Sample

states-transition

probability

non-transitioned

pixels

transitioned pixels

target mask

previous mask

Transition Sample

𝐶

⋯

channel-dim concatenation

𝐶

element-wise addition

element-wise multiplication

𝑞 𝑥

#

𝑥

#%&

)

Ground Truth

Refined Mask

Transition

Sample

𝑚

!

𝑚

'

Transition

Sample

T

𝑚

&

⋯

×(𝑇 - 1)

𝑝

$

𝑥

#%&

𝑥

#

)

T

Transition Sample

input as previous mask

input as target mask

input as probability

transitioned mask

1 - 𝛽

#

fine

𝛽

#

coarse

c

f

c

f

Coarse Mask

O

口

where β t ∈ (0 , 1) is a hyperparameter and f b ( x t , t ) is a model predicting Bernoulli probability. After the great success of DDPM [22], Austin et al. [2] extended the architecture of Discrete Diffusion Model to a more general form. They define the forward process as a discrete random variable transitioning among multiple states and use states-transition matrix Q t to characterize this process:

3.2 SegRefiner

In this work, we propose SegRefiner, with a unique discrete diffusion process, which can be applied to refine coarse masks from various segmentation models and tasks. SegRefiner performs refinement with a coarse-to-fine diffusion process. In the forward process, SegRefiner employs a discrete diffusion process which is formulated as unidirectional random states-transition, gradually degrading the ground truth mask into a coarse mask. In the reverse process, SegRefiner begins with a provided coarse mask and gradually transforms the pixels in the coarse mask to the refined state, correcting the wrongly predicted area in the coarse mask. In the following paragraphs, we will provide a detailed description of the forward and reverse process.

Forward diffusion process In the forward process, we gradually degrade the ground truth mask/ fine mask M fine , transiting it into a coarse mask M coarse . In other words, we have m 0 = M fine and m T = M coarse . At any intermediate timestep t ∈ { 1 , 2 , ..., T -1 } , the intermediate mask m t is therefore in a transitional phase between M fine and M coarse .

We define that every pixel in m t has two states: fine and coarse, and the forward process is thus formulated as states-transition between these two states. Pixels in the fine state will retain their values from M fine , and vice versa . We propose a new transition sample module to formulate this process. As shown in Fig. 2 right, during forward process, the transition sample module takes the previous mask m t -1 , coarse mask m T and a states-transition probability as input and outputs a transitioned mask m t . The states-transition probability describes the probability of every pixel in m t -1 transitioning to the coarse state. This module first performs Gumbel-max sampling [25] according to the states-transition probability and obtains the transitioned pixels. Then, the transitioned pixels will take values from m T while the non-transitioned pixels will keep unchanged.

Note that the transition sample module represents a unidirectional process in which only 'transition to coarse state' happens. The unidirectional property ensures that the forward process will converges

t = T

t = T-1

t = T-2

t = 1

t = 0

Coarse Mask

Refined Mask

𝑥

!

𝑚

!

𝑥

!

𝑚

!

to M coarse , despite each step is completely random. This is a significant difference between our SegRefiner and previous diffusion models in which forward process converges to a random noise.

With the reparameterization trick, we introduce a binary random variable x to formulate the above process. We represent x i,j t as a one-hot vector to represent the state of pixel ( i, j ) in m t and set x i,j 0 = [1 , 0] and x i,j T = [0 , 1] to represent the fine state and coarse state, respectively. The forward process can thus be formulated as:

where β t ∈ [0 , 1] , and 1 -β t corresponds to the states-transition probability used in our transition sample module. Q t is a states-transition matrix. The form of Q t explicitly manifests the unidirectional property, i.e. , all pixels in the coarse state will never transition back to the fine state since q ( x t | [0 , 1]) = [0 , 1] . According to Eq. (6), the marginal distribution can be formulated as

where ¯ β t = β 1 β 2 . . . β t . Given this, we can obtain the intermediate mask m t at any intermediate timestep t without the need of step-by-step sampling q ( x t | x t -1 ) , allowing faster training.

Reverse diffusion process The reverse diffusion process takes a coarse mask m T and gradually transforms it into a fine mask m 0 . However, since the fine mask m 0 and the reversed states-transition probability is unknown, following DDPM [22], we train a neural network f θ parameterized by θ to predict the fine mask ˜ m 0 | t at each timestep, represented as

where I is the corresponding image. ˜ m 0 | t and p θ ( ˜ m 0 | t ) denote the predicted binary fine mask and its confidence score, respectively. To obtain the reversed states-transition probability, according to Eq. (6), Eq. (7), and Bayes' theorem, we first formulate the posterior at timestep t -1 as:

where the fine state x 0 is set to [1 , 0] during training, indicating ground truth. While during inference, x 0 is unknown, as the predicted ˜ m 0 | t may not be entirely accurate. Since the confidence score p θ ( ˜ m 0 | t ) represents the model's level of certainty for each pixel prediction being correct, p θ ( ˜ m 0 | t ) can also be interpreted as the probability being in the fine state. Therefore, intuitively, one could

obtain the state of every pixel in ˜ m 0 | t by simply thresholding as done in [2]:

where pixels with higher confidence scores will have x i,j 0 | t = [1 , 0] , indicating they are in the fine state, and vice versa . However, in such one-hot form, the values of states-transition probabilities are determined solely by the pre-defined hyperparameters, leading to significant information loss. Instead, we retain the soft transition and formulate x i,j 0 | t = [ p θ ( ˜ m 0 | t ) i,j , 1 -p θ ( ˜ m 0 | t ) i,j ] . With this setting, the reverse diffusion process can be reformulated as

where P i,j θ,t is the reversed states-transition matrix. With the above reversed states-transition probability, m t and ˜ m 0 | t as input, the transition sample module can transit a portion of pixels to the fine state at each timestep, thereby correcting erroneous predictions.

Inference Given a coarse mask m T and its corresponding image I , we first initialize that all pixels are in the coarse state thus x i,j T = [0 , 1] . We iterate between: (1) forward pass to obtain ˜ m 0 | t and p θ ( ˜ m 0 | t ) (Eq. (8)); (2) compute the reversed states-transition matrix P i,j θ,t and obtain x t -1 (Eq. (11)); (3) compute the refined mask m t -1 based on x t -1 , m t and ˜ m 0 | t . The process (1)-(3) is iterated until we obtain fine mask m 0 . Visualization examples of inference are shown in Fig. 3.

4 Experiments

4.1 Implementation Details

Model Architecture Following [39], we employ U-Net for our denoising network. We modify the U-Net to take in 4-channel input (concatenation of image and the corresponding mask m t ) and output a 1-channel refined mask. Both input and output resolution is set to 256 × 256. All others remain unchanged other than the aforementioned modifications.

Objective Function Following [12], we employ a combination of binary cross-entropy loss and texture loss for training our model, i.e. , L = L bce + α L texture , where texture loss is characterized as an L1 loss between the segmentation gradient magnitudes of the predicted mask and the ground truth mask. α is set to 5 to balance the magnitude of both losses.

Noise Schedule Theoretically, the unidirectional property of SegRefiner ensures that any noise schedule can make the forward process converge to the coarse mask given an infinite number of timesteps. However, in practice, we use much fewer timesteps ( T = 6 this work) to ensure efficient inference. We designate ¯ β T = 0 such that x T = [0 , 1] for all pixels and m T = M coarse (Eq. (7)). Following DDIM [46], we directly set a linear noise schedule from 0.8 to 0 on ¯ β t .

Training Strategy Our SegRefiner model has been developed into two versions for refinement of different resolution images: a low-resolution variant (hereafter referred to as LR-SegRefiner ) and a high-resolution variant (hereafter referred to as HR-SegRefiner ). While these two versions employ different training datasets, all other settings remain consistent. LR-SegRefiner is trained on the LVIS dataset [20], whereas HR-SegRefiner is trained on a composite dataset merged from two high-resolution datasets, DIS5K [40] and ThinObject-5K [32]. These datasets were chosen due to their highly accurate pixel-level annotations, thereby facilitating the training of our model to capture fine details more effectively. Following [12], the coarse masks used for training are obtained through various morphological operations, such as randomly perturbing some edge points of the ground truth and performing dilation, erosion, etc. During training, we first train LR-SegRefiner on the low-resolution dataset until convergence. Subsequently, it is fine-tuned on the high-resolution dataset to yield HR-SegRefiner .

We employed double random crop as the primary data augmentation technique. It entails the random cropping of each image to generate two distinct model inputs: one encapsulating the entire foreground

target, and the other containing only a partial foreground. Both crops are subsequently resized to match the model's input size. This operation ensures that the model is proficient in refining both entire objects and incomplete local patches, a capability that proved instrumental in subsequent experiments. All the following experiments were conducted on 8 NVIDIA RTX 3090. For more details on the training process, please refer to the supplemental materials.

4.2 Semantic Segmentation

Dataset and Metrics As the refinement task emphasizes the enhancement of mask quality, a dataset with high annotation quality and sufficient detailed information is required to evaluate the model's performance. Hence, we report the results on BIG dataset [12], a semantic segmentation dataset specifically designed for high-resolution images. With resolutions ranging from 2048 × 1600 to 5000 × 3600 , this dataset provides a challenging testbed for evaluating refinement methods. The metrics we used are the standard segmentation metric IoU and the boundary metric mBA (mean Boundary Accuracy [12]), which is commonly used in previous refinement works.

Settings Because of the high resolution of images in BIG dataset, we employ the HR-SegRefiner in this experiment. While the model's output size is only 256 × 256 , which is insufficient for such a high-resolution dataset and results in the loss of many edge details. Consequently, during inference, we deploy the first T -1 timesteps to perform global refinement, which takes the resized entire image as input, and the final timestep as a local step that takes original-size local patches as input. In order to identify the local patches that require refinement, we filter out pixels with low state-transition probabilities from the globally refined mask and use them as the center points for the local patches. We allocate more timesteps for global refinement since global steps need to handle more severe errors, thus presenting a higher difficulty level. The training strategy we have employed ensures that the model can adapt to both global and local input without modification.

Results As shown in Tab. 1, we compare the proposed SegRefiner with three model-agnostic semantic segmentation refinement methods, SegFix [56], CascadePSP [12], and CRM [44]. Additionally, we include the fine-grained matting method, MGMatting [55], which employs an image and mask for matting and can also be utilized for refinement purposes. The proposed SegRefiner demonstrates superior performance over previous methods when using coarse masks from four different semantic segmentation models, as evident in both IoU and mBA metrics. Notably, our SegRefiner outperforms CRM, which is specifically designed for ultra-high-resolution images, showcasing significant advancements. We report the error bar ( ± in gray) in this experiment due to the relatively small size of the BIG dataset (consisting of 100 testing images). The error bar represents the maximum fluctuation value among the results of five experiments, and the results in Tab. 1 are the average of these five trials. In subsequent experiments, the stability of the results is ensured by the availability of an ample number of testing images.

4.3 Instance Segmentation

Dataset and Metrics To evaluate the effectiveness of our SegRefiner in refining instance segmentation, we select the widely-used COCO dataset [34] with LVIS [20] annotations. LVIS annotations offer superior quality and more detailed structures compared to the original COCO annotations and other commonly used instance segmentation datasets such as Cityscapes [13] and Pascal VOC [17]. This makes LVIS annotations more suitable for assessing the performance of refinement models. The evaluation metrics are the Mask AP and Boundary AP. It worth noting that these metrics are computed using the LVIS [20] annotations on the COCO validation set. The Boundary AP metric, introduced by Cheng et al. [10], is a valuable evaluation metric that measures the boundary quality of the predicted masks and is highly sensitive to the accuracy of edge prediction. It provides a detailed assessment of how well the refined masks capture the boundaries of the objects.

(a) Image

(b) GT

(c) U-Net [42]

(d) ISNet [44]

(e)

SegRefiner

(ours)

Settings In this experiment, we utilize the LR-SegRefiner model. To refine each instance, we extract the bounding box region based on the coarse mask and expand it by 20 pixels on each side. The extracted region is then resized to match the input size of the model. The output size is suitable for instances in the COCO dataset, allowing us to perform instance-level refinement for all timesteps without requiring any local patch refinement.

Results First, in Tab. 2, we compare the proposed SegRefiner with two model-agnostic instance segmentation refinement methods, BPR [47] and SegFix [56]. As demonstrated in Tab. 2, our SegRefiner achieves significantly better performance compared to these two methods. The coarse masks utilized in Tab. 2 are obtained from Mask R-CNN to ensure consistency with the original experiments conducted in [47, 56]. This choice allows for a fair and direct comparison with the previous works. Then in Tab. 3, we apply our SegRefiner to other 7 instance segmentation models. Our method yields significant enhancements across models of varying performance levels. Furthermore, when compared to three model-specific instance segmentation refinement models, including PointRend [29], RefineMask [57], and Mask TransFiner [27], our SegRefiner consistently enhances their performance. These results establish SegRefiner as the leading model-agnostic refinement method for instance segmentation.

4.4 Dichotomous Image Segmentation

Dichotomous Image Segmentation (DIS) is a recently introduced task by Qin et al. [40], which specifically targets the segmentation of objects with complex texture structures, such as the steel framework of bridge illustrated in Fig. 4. To facilitate research in this area, Qin et al. also built the DIS5K dataset, a meticulously annotated collection of 5,470 images with resolutions of 2K and above. The DIS5K dataset, characterized by its abundance of fine-grained structural details, poses rigorous

demands on a model's capability to perceive and capture intricate information. Thus, it serves as a suitable benchmark to evaluate the performance of our refinement method. The evaluation metrics and other settings utilized in this experiment are consistent with those in Sec. 4.2.

Results Since the recent introduction of DIS5K, no prior refinement methods have reported results on this dataset to date. Therefore, the main aim of this experiment is to evaluate the transferability of our SegRefiner across various models. As shown in Tab. 4, our SegRefiner is applied to 6 segmentation models. The results consistently demonstrate that our SegRefiner improves the performance of each segmentation model in terms of both IoU and mBA. In Fig. 4, we provide qualitative comparisons with previous methods, which reveal the superior ability of our SegRefiner to capture fine-grained details, such as the dense and fine mesh of the chair.

4.5 Ablation Study

We conduct ablation studies on the BIG dataset [44] with high-resolution images. We first investigate the effectiveness of the diffusion process in SegRefiner. The results are reported in Tab. 5a, where the term 'none' refers to the utilization of a U-Net without multi-step iteration, and 'w/o diffusion' denotes the direct utilization of the previous step's mask as the input for the subsequent iteration, without employing the diffusion sampling process, i.e. , discarding the transition sample module. Since there is no states-transition probability in non-diffusion methods, we use a sliding window strategy similar to [12] in this experiment to obtain the input for local refinement. The results in Tab. 5a demonstrate that the diffusion-based iterative process achieves the best performance, validating the effectiveness of the diffusion process in SegRefiner.

In Tab. 5b and Tab. 5c, we evaluate various alternatives used in prior experiments. Tab. 5b shows the analysis of global and local refinement used in high resolution images. As can be seen, global refinement significantly improves IoU; however, for high-resolution images, the resulting smaller output size leads to a lower mBA. Local refinement applied to local patches of the original size greatly improves mBA, while its enhancement in IoU is less significant due to the absence of global information. The combination of local and global refinement achieves better performance in both IoU and mBA. Tab. 5c presents the results corresponding to different input image sizes. Considering computational load and memory usage, 256 × 256 is selected as the default setting, which performs well without introducing too much computational overhead.

5 Conclusion and Discussion

We propose SegRefiner, which is the first diffusion-based image segmentation refinement method with a new designed discrete diffusion process. SegRefiner performs model-agnostic segmentation refinement and achieves strong empirical results in refinement of various segmentation tasks.

While SegRefiner has achieved significant improvements in accuracy, one limitation lies in that the diffusion process leads to slowdown of the inference due to the multi-step iterative strategy. As shown in Tab. 6, we conduct an experiment on instance segmentation about the relationship between the model's accuracy, computational complexity, time consumption (the average time consumed per image), and the number of iteration steps. It can be observed that, while the iterative strategy has provided SegRefiner with a noticeable improvement in accuracy, it has also introduced a linear increase with the number of steps in both time consumption and computational complexity. As the first work applying diffusion models to the refinement task, the proposed SegRefiner primarily concentrates on devising a suitable diffusion process for general refinement tasks. While improving the efficiency of diffusion models will be a crucial research direction in the future, not only in the field of image generation but also in other domains where diffusion models are applied.

References

Tomer Amit, Eliya Nachmani, Tal Shaharabany, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. CoRR , abs/2112.00390, 2021.

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurIPS , volume 34, pages 17981-17993, 2021.

Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. CoRR , abs/2208.09392, 2022.

Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Labelefficient semantic segmentation with diffusion models. In ICLR , 2022.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV , pages 801-818, 2018.

Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. CoRR , abs/2211.09788, 2022.

Ting Chen, Lala Li, Saurabh Saxena, Geoffrey E. Hinton, and David J. Fleet. A generalist framework for panoptic segmentation of images and videos. CoRR , abs/2210.06366, 2022.

Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. CoRR , abs/2208.04202, 2022.

Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. Motionconditioned diffusion model for controllable video synthesis. CoRR , abs/2304.14404, 2023.

Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In CVPR , pages 15334-15342, 2021.

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskedattention mask transformer for universal image segmentation. In CVPR , pages 1290-1299, 2022.

Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In CVPR , pages 8890-8899, 2020.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. In CVPRW , volume 2. sn, 2015.

Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, and Gang Wang. Context contrasted feature and gated multi-scale aggregation for scene segmentation. In CVPR , pages 2393-2402, 2018.

[15]

Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy.

MeViS: A large-scale benchmark for video segmentation with motion expressions. In

ICCV

, pages 2694-2703, 2023.

Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip H.S. Torr, and Song Bai. MOSE: A new dataset for video object segmentation in complex scenes. In ICCV , pages 20224-20234, 2023.

M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJCV , 111(1):98-136, January 2015.

Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. In ICCV , pages 6910-6919, 2021.

Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, and Weiqiang Wang. Diffusioninst: Diffusion model for instance segmentation. CoRR , abs/2212.02773, 2022.

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR , pages 5356-5364, 2019.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In CVPR , pages 2961-2969, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , pages 6840-6851, 2020.

Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. In CVPR , pages 603-612, 2019.

Zilong Huang, Yunchao Wei, Xinggang Wang, Wenyu Liu, Thomas S Huang, and Humphrey Shi. Alignseg: Feature-aligned segmentation networks. IEEE TPAMI , 44(1):550-557, 2021.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 , 2016.

Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. DDP: diffusion model for dense visual prediction. CoRR , abs/2303.17559, 2023.

Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. In CVPR , pages 4412-4421, 2022.

Lei Ke, Henghui Ding, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Video mask transfiner for high-quality video instance segmentation. In ECCV , pages 731-747. Springer, 2022.

Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR , pages 9799-9808, 2020.

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: open-set grounded text-to-image generation. CoRR , abs/2301.07093, 2023.

Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, and Raquel Urtasun. Polytransform: Deep polygon transformer for instance segmentation. In CVPR , pages 9131-9140, 2020.

Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, and Jiashi Feng. Deep interactive thin object selection. In WACV , pages 305-314, 2021.

Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR , pages 1925-1934, 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755. Springer, 2014.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , pages 3431-3440, 2015.

Haiyang Mei, Ge-Peng Ji, Ziqi Wei, Xin Yang, Xiaopeng Wei, and Deng-Ping Fan. Camouflaged object segmentation with distraction mining. In CVPR , pages 8772-8781, 2021.

Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In CVPR , pages 21033-21043, 2022.

Jiaxu Miao, Yunchao Wei, Yu Wu, Chen Liang, Guangrui Li, and Yi Yang. Vspw: A large-scale dataset for video scene parsing in the wild. In CVPR , pages 4133-4143, 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , pages 8162-8171. PMLR, 2021.

Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. In ECCV , pages 38-56. Springer, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , pages 234-241. Springer, 2015.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE TPAMI , 2022.

Tiancheng Shen, Yuechen Zhang, Lu Qi, Jason Kuen, Xingyu Xie, Jianlong Wu, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR , pages 1310-1319, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Chufeng Tang, Hang Chen, Xiao Li, Jianmin Li, Zhaoxiang Zhang, and Xiaolin Hu. Look closer to segment better: Boundary patch refinement for instance segmentation. In CVPR , pages 13926-13935, 2021.

Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV , pages 282-298. Springer, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , pages 5998-6008, 2017.

Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE TPAMI , 43(10):3349-3364, 2020.

Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In ECCV , pages 649-665. Springer, 2020.

Junde Wu, Huihui Fang, Yu Zhang, Yehui Yang, and Yanwu Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. CoRR , abs/2211.00611, 2022.

Junde Wu, Rao Fu, Huihui Fang, Yu Zhang, and Yanwu Xu. Medsegdiff-v2: Diffusion based medical image segmentation with transformer. CoRR , abs/2301.11798, 2023.

Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai. Seqformer: Sequential transformer for video instance segmentation. In ECCV , pages 553-569. Springer, 2022.

Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang, Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In CVPR , pages 1154-1163, 2021.

Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refinement for segmentation. In ECCV , pages 489-506. Springer, 2020.

Gang Zhang, Xin Lu, Jingru Tan, Jianmin Li, Zhaoxiang Zhang, Quanquan Li, and Xiaolin Hu. Refinemask: Towards high-quality instance segmentation with fine-grained features. In CVPR , pages 6861-6869, 2021.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. CoRR , abs/2302.05543, 2023.

Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In ECCV , pages 405-420, 2018.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR , pages 2881-2890, 2017.

Peng Zhou, Brian Price, Scott Cohen, Gregg Wilensky, and Larry S Davis. Deepstrip: High-resolution boundary refinement. In CVPR , pages 10558-10567, 2020.

Appendix

Here we first introduce the details of SegRefiner's equation derivation in Sec. A. The implementation details are shown in Sec. B. Moreover, we provide additional qualitative results in Sec. C.

A Equation Derivation Details

In the following, we will present the derivation of the posterior distribution (Eq. (9)) and the reverse transition-probability (Eq. (11)). Firstly, for Eq. (9), it's derived from the Bayes' theorem:

According to Eq. (6) and Eq. (7), we can get all the items in Bayes' theorem as:

It appears to be slightly different from Eq. (6). The reason is that the results of Eq. (6) is a length-2 vector, representing the probabilities of two states respectively. While we use a scalar form here, which includes an additional one-hot vector x T ∗ and is more suitable for subsequent derivations. For the denominator of Eq. (9), we have obtained the same form. Let's consider the numerator:

where " · " refers to scalar multiplication and " ⊙ " refers to element-wise multiplication. This is the scalar form, by removing the x T t -1 , we can get the corresponding vector form just as Eq. (9).

Secondly, for Eq. (11), it's derived from Eq. (9) by substituting x 0 with [ p θ ( ˜ m 0 | t ) i,j , 1 -p θ ( ˜ m 0 | t ) i,j ] (we refer p θ ( ˜ m 0 | t ) i,j to p 0 in the following for simplicity).

Same as Eq. (9), we can remove x T t -1 and get the vector form of Eq. (11).

B Implementation Details

In this section, we give a detailed description of the model architecture and training/inference settings. The overall workflow of the training and inference process are provided in Alg. 1 and Alg. 2.

Model Architecture Following [39], we use a U-Net with 4-channel input and 1-channel output. Both input and output resolution is set to 256 × 256 . Considering computational load and memory usage, we set the intermediate feature channels to 128 and only conduct self-attention in strides 16 and 32.

Training Settings All experiments are conducted on 8 NVIDIA RTX3090 GPUs with Pytorch. During training, we first train the LR-SegRefiner on the LVIS dataset [20] with 120k iterations. The AdamW optimizer is used with the initial learning rate of 4 × 10 -4 . We use a multi-step learning rate schedule, which decays by 0.5 in steps 80k and 100k. Subsequently, the HR-SegRefiner is obtained from 40k-iterations fine-tuning based on the 80k checkpoint of LR-SegRefiner . Batch size is set to 8 in each GPU.

Inference Settings In instance segmentation, we use the LR-SegRefiner to perform refinement in instance level. For each instance, we extract the bounding box region based on the coarse mask and expand it by 20 pixels on each side. The extracted region is then resized to match the input size of the model. After a complete reverse diffusion process, the output is resized to the original size.

In semantic segmentation and dichotomous image segmentation, because of the high resolution of images, we employ the HR-SegRefiner and conduct a global-and-local refinement process. In order to identify the local patches that require refinement, we filter out pixels with low state-transition probabilities from the globally refined mask and use them as the center points for the local patches. We apply Non-Maximum Suppression (NMS, with 0.3 as threshold) to these patches to remove excessive overlapping.

C Qualitative Results

In this section, we provide more visual results in semantic segmentation, instance segmentation, and dichotomous image segmentation. Fig. 5 shows the comparisons of SegRefiner and other models (including instance segmentation models and refinement models) on COCO [34] validation set. Fig. 6 shows more comparisons between the coarse masks and refined masks on COCO validation set. These results demonstrate that the proposed SegRfiner can robustly correct inaccurate predictions in coarse masks. Fig. 7 and Fig. 8 show visual results on BIG dataset [12] and DIS5K dataset [40]. SegRefiner shows a strong capability for capturing extremely fine details on these two high-resolution datasets.

(a) Image

(b) Mask R-CNN [21]

(c) SOLO [51]

(d) QueryInst [18]

(e) CondInst [48]

(f) PointRend [29]

RCDols

RCDols

RCDol

RCDolrs

RCDol

HEE

RCDols

PE

(h) Transfiner [27]

(j) Mask R-CNN + BPR [47]

RCDoles

11222

RCDol

RCDol

RCDol

HTE

RCDol

(k) Mask R-CNN + SegRefiner (Ours)

(a) Coarse Mask

(b) Refined Mask

Ewitte

SP-AWF

SP-AWF

(a) Image

(b) Coarse Mask

(c) Refined Mask

(a) Image

(b) Coarse Mask

(c) Refined Mask

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a method based discrete diffusion process to refine coarse segmentation masks with input images in the context of semantic image segmentation. In this work, the refinement problem has been formulated as an iterative data generation process. To work with binary masks, as adapted for most segmentation scenarios, the authors proposed discrete state translation with mathematical analysis. Experimental results seem solid about the superiority of the proposed methods against several parallel methods on three segmentation tasks.\n\nSTRENGTHS:\nOriginality: this paper is new for its adaptation of discrete prior to tackling binary segmentation masks. although the discrete diffusion model is not new, its application in segmentation can be an interesting attempt.  In order to handle a smooth translation from a fine mask to a coarse one, the authors proposed a transition sample module in the forward process. This is different from adding traditional Gaussian noise as in continuous diffusion models. \n\nQuality: the quality of this paper is relatively good, through multiple reading passes. The paper is kind of self-contained, including strong motivation, mathematical analysis, and experimental results. \n\nClarity: This paper is not easy to read for my first pass. After reading it several times, one can have a better understanding of certain concepts and illustrations\n\nSignificance: I give high significance to this paper as it solves a common problem in a novel way as well as the obtained results seem to be greatly improved in a wide range of segmentation tasks and datasets.\n\nWEAKNESSES:\nPersonally, the biggest concern of this paper is the clarification. Basically, the authors devised a discrete process to bridge the fine mask with the coarse one, by using a transitioned mask. This mask is unidirectional, which means only the transition to the target happens.  This forward process consists of a continuous Q_{t} and a discrete sampling based on probability defined on Q_{t}. These processes are not well illustrated and explained in the paper. In addition, the paper refers to source and target masks differently for forward and backward processes, which can confuse readers and a better nomination would be suggested.\n\nAlthough this paper presents extensive experimental results, it lacks a comparison of computational complexity and time and resources. Despite the fact that the authors had a small discussion on this aspect in the conclusion, it is much better to provide quantitative results either in the main paper or in the supplementary material.\n\nQUESTIONS:\n- throughout the paper, the authors utilized examples with binary masks to illustrate the idea, including Fig. 2, Equ. 6-11, etc. Could the authors explain how exactly to apply this method to a multi-class segmentation problem?\n\n- Since the authors utilize Gumbel-softmax sampling in the forward, it might exhibit some kind of diversities for the transitioned masks in the intermediate steps (even the final output–coarse mask remains the same). However, for the backward process, if my understanding is not wrong, how can this algorithm ensure a certain degree of diversity as no stochastic operations are involved?\n\n- The forward process has ground truth for the source and target masks, while for the backward process, the paper applied simple thresholding. This can cause instability of convergence. Did the authors meet any optimization difficulties?\n\n- In the training strategy, why exactly two versions of SegRefiner were proposed?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitation of this paper while I am expecting more details, as stated in the previous comments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a model-agnostic solution called SegRefiner to enhance the quality of object masks generated by different segmentation models. The approach introduces a novel perspective on segmentation refinement, treating it as a data generation process. By leveraging a series of denoising diffusion steps, SegRefiner achieves effective mask refinement.\nSpecifically, SegRefiner takes coarse masks as inputs and employs a discrete diffusion process to refine them. By predicting the label and states-transition probabilities for each pixel, the method progressively refines the noisy masks in a conditional denoising manner. To evaluate the performance of SegRefiner, extensive experiments are conducted on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation.\nThe experimental results demonstrate the superiority of SegRefiner in multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics for different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, SegRefiner exhibits a strong capability to capture extremely fine details when refining high-resolution images.\n\nSTRENGTHS:\n(1) This paper is the first one to introduce diffusion-based refinement for segmentation masks.\n(2) The proposed method, called SegRefiner, is model-agnostic, and thus applicable across different segmentation models and tasks.\n(3) The authors extensively analyze the performance of SegRefiner across various segmentation tasks, demonstrating that our SegRefiner not only outperforms all previous model agnostic refinement methods, but can also be effortlessly transferred to other segmentation tasks without any modification.\n\nWEAKNESSES:\n(1) In the Objective Function of sec 4.1, how do you specify the value of alpha? Do you have any ablation studies on the alpha?\n(2) In Table 3, it seems that the improvement on Boundary AP is larger than Mask AP. Why is that?\n(3) In Table 4, it seems that the improvement on ISNet is smaller than the other datasets. Could you explain the reason?\n(4) Why do you only do the ablation studies on semantic segmentation task? Have you ever tried the ablation studies on instance segmentation and Dichotomous Image Segmentation? Have you ever tried ablation studies on the number of iterations and output size? For Table 5 (c), have you ever tried a larger input size, like 1024x1024?\n\nQUESTIONS:\nI'm positive about this paper. I really like the idea to apply diffusion model to segmentation task. However, I have some concerns in the Weaknesses. I hope to see the response from the authors for those questions. Thank you.\n\nLIMITATIONS:\nThe diffusion process leads to slowdown of the inference due to the multi-step iterative strategy.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents a model-agnostic solution for segmentation refinement, which implements the refinement process of the coarse masks through a series of denoising diffusion steps. Comprehensive experiments on various segmentation tasks are conducted, including semantic segmentation, instance segmentation, and dichotomous image segmentation and the results demonstrate the superiority of the proposed method.\n\nSTRENGTHS:\n- They obtain SOTA results on multiple tasks. \n- The idea sounds quite interesting. \n- This manuscript is well-formulated and -presented.\n\nWEAKNESSES:\nThe motivation behind lacks focus. The manuscript primarily discusses the problem of the existing methods of segmentation refinement in Introduction but fails to indicate the reason why denoising diffusion models are suitable and important for the task of segmentation refinement. What are the technical challenges in applying denoising diffusion models to segmentation refinement? To address these challenges, what innovations does this paper propose regarding the application of denoising diffusion models to segmentation refinement?\n\nHow is the value of the state transition probability specifically calculated?\n\nThere are several instances where the use of others' work, such as Gumbel-max sampling, has not been properly cited.\n\nFor the DIS experiment, commonly used evaluation metrics for DIS are not employed.\n\nSome grammar and spelling errors.\n\nQUESTIONS:\nSee above comments.\n\nLIMITATIONS:\nAs mentioned in the conclusion, the diffusion process seems to lead to a slowdown in inference due to the multi-step iterative strategy. As a post-processing step for segmentation tasks, this is indeed a major drawback. Did the authors conduct any experiments to examine the relationship between iteration step, accuracy, and speed?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes diffusion-based image segmentation refinement approach, SegRefiner, using a newly designed discrete diffusion process. SegRefiner performs model-agnostic segmentation refinement that takes in coarse output masks from any segmentation model as an input and refines them through a series of diffusion steps while estimating state-transition probabilities for each pixel. Comprehensive experiments across multiple segmentation tasks demonstrates that SegRefiner consistently improves segmentation metrics and outperforms state-of-the-art model-specific and model-agnostic refinement approaches.\n\nSTRENGTHS:\nOriginality\nThis work has multiple novel contribution ranging from - being first to introduce diffusion-based image segmentation refinement approach, the use of discrete diffusion instead of Gaussian diffusion process of DDPM, being class-agnostic as well as ability to capture extremely fine-details when applied to high-resolution images.\n\nQuality\nThe paper is well-written and mostly easy to follow. Authors have put their best effort to condense lot of details into conference paper length. As the use of discrete diffusion is new in practice, some of the details can be elaborated using examples in the supplementary for the ease audience.\n\nSignificance\nThe approach taken by SegRefiner is unique. For instance, mask and states prediction task is kept separate. Similarly, there is lot of subtle details required to run this successfully as listed under the training strategy, noise schedule and other settings. All these contribution can be quite useful to other researcher pursuing different applications using generative model.\n\nWEAKNESSES:\nNone that are significant. Have listed one under the limitation section.\n\nQUESTIONS:\n1. How is the posteior in eq(9) derived ? Similarly how do one derive eq(11) ? And hence where is eq(9) used ?\n2. Is the refinement of semantic and instance segmentation performed using zero-shot ? All I understand is LR- and HR-SegRefiner are trained using LVIS and composite dataset respectively.\n\nLIMITATIONS:\nYes, authors have address one of the core limitation to using diffusion process. As far as my understanding, another limitation of such class-agnostic and model-agnostic approach is one should undertake each instance one at a time for refinement. This probably due to the fact there is no class information input to SegRefiner model.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for improving the quality of object masks generated by different segmentation models. The method, called SegRefiner, is model-agnostic, meaning that it can be used with different segmentation models. SegRefiner works by interpreting segmentation refinement as a data generation process. This allows the work to implement the refinement process as a series of denoising diffusion steps.\n\nSpecifically, SegRefiner takes coarse masks as input and refines them using a discrete diffusion process. For each pixel, it predicts the label and corresponding state-transition probabilities. This allows the model to progressively refine the noisy masks in a conditional denoising manner.\n\nThe method is evaluated on a variety of segmentation tasks, and the results show that SegRefiner consistently outperforms previous methods. It improves both the segmentation metrics and boundary metrics across different types of coarse masks. SegRefiner also exhibits a strong capability to capture extremely fine details when refining high-resolution images.\n\nSTRENGTHS:\n1. Overall, the paper is well written and clearly organized.\n\n2. The method is intuitive and straightforward. The performance improvements are clear compared to the previous baselines.\n\n3. The inference is efficient for diffusion as it only uses 6 timesteps.\n\nWEAKNESSES:\n1. Not necessarily a weakness but the model does not clearly state how the coarse masks are generated for training until later sections. While the proposed method follows previous works on this part, it would still improve the clarity if this is discussed in earlier sections and in the overview figures.\n\n2. The paper only shows one example of how each step of the diffusion process works during inference. The authors are recommended to include more examples with some highlighted boundary comparisons.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nThe authors properly addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}