SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset and Metrics As the refinement task emphasizes the enhancement of mask quality, a dataset with high annotation quality and sufficient detailed information is required to evaluate the model's performance. Hence, we report the results on BIG dataset [12], a semantic segmentation dataset specifically designed for high-resolution images. With resolutions ranging from  $2048 \times 1600$  to  $5000 \times 3600$ , this dataset provides a challenging testbed for evaluating refinement methods. The metrics we used are the standard segmentation metric IoU and the boundary metric mBA (mean Boundary Accuracy [12]), which is commonly used in previous refinement works.
Settings Because of the high resolution of images in BIG dataset, we employ the  $HR-SegRefiner$  in this experiment. While the model's output size is only  $256 \times 256$ , which is insufficient for such a high-resolution dataset and results in the loss of many edge details. Consequently, during inference, we deploy the first  $T - 1$  timesteps to perform global refinement, which takes the resized entire image as input, and the final timestep as a local step that takes original-size local patches as input. In order to identify the local patches that require refinement, we filter out pixels with low state-transition probabilities from the globally refined mask and use them as the center points for the local patches. We allocate more timesteps for global refinement since global steps need to handle more severe errors, thus presenting a higher difficulty level. The training strategy we have employed ensures that the model can adapt to both global and local input without modification.
Results As shown in Tab. 1, we compare the proposed SegRefiner with three model-agnostic semantic segmentation refinement methods, SegFix [56], CascadePSP [12], and CRM [44]. Additionally, we include the fine-grained matting method, MGMatting [55], which employs an image and mask for matting and can also be utilized for refinement purposes. The proposed SegRefiner demonstrates superior performance over previous methods when using coarse masks from four different semantic segmentation models, as evident in both IoU and mBA metrics. Notably, our SegRefiner outperforms CRM, which is specifically designed for ultra-high-resolution images, showcasing significant advancements. We report the error bar  $(\pm$  in gray) in this experiment due to the relatively small size of the BIG dataset (consisting of 100 testing images). The error bar represents the maximum fluctuation value among the results of five experiments, and the results in Tab. 1 are the average of these five trials. In subsequent experiments, the stability of the results is ensured by the availability of an ample number of testing images.
4.3 Instance Segmentation
Dataset and Metrics To evaluate the effectiveness of our SegRefiner in refining instance segmentation, we select the widely-used COCO dataset [34] with LVIS [20] annotations. LVIS annotations offer superior quality and more detailed structures compared to the original COCO annotations and other commonly used instance segmentation datasets such as Cityscapes [13] and Pascal VOC [17]. This makes LVIS annotations more suitable for assessing the performance of refinement models. The evaluation metrics are the Mask AP and Boundary AP. It worth noting that these metrics are computed using the LVIS [20] annotations on the COCO validation set. The Boundary AP metric, introduced by Cheng et al. [10], is a valuable evaluation metric that measures the boundary quality of the predicted masks and is highly sensitive to the accuracy of edge prediction. It provides a detailed assessment of how well the refined masks capture the boundaries of the objects.
Table 1: IoU/mBA results on the BIG dataset comparing with other mask refinement methods.
IoU/mBA	Coarse Mask	SegFix [56]	MGMatting [55]	CascadePSP [12]	CRM [44]	SegRefiner (ours)
FCN-8s [35]	72.39 / 53.63	72.69 / 55.21	72.31 / 57.32	77.87 / 67.04	79.62 / 69.47	86.95 ±0.06 / 72.81 ±0.05
DeepLab V3+ [5]	89.42 / 60.25	89.95 / 64.34	90.49 / 67.48	92.23 / 74.59	91.84 / 74.96	94.86 ±0.04 / 77.64 ±0.05
RefineNet [33]	90.20 / 62.03	90.73 / 65.95	90.98 / 68.40	92.79 / 74.77	92.89 / 75.50	95.12 ±0.05 / 76.93 ±0.05
PSPNet [60]	90.49 / 59.63	91.01 / 63.25	91.62 / 66.73	93.93 / 75.32	94.18 / 76.09	95.30 ±0.03 / 77.46 ±0.04
Avg Improve	0.00 / 0.00	0.47 / 3.30	0.73 / 6.10	3.58 / 14.05	4.01 / 15.12	7.43 / 17.33
(c) U-Net [42]
Figure 4: Qualitative comparisons with other methods on DIS5K dataset [40]. Our SegRefiner has the capability to capture finer details, allowing it to discern and incorporate more subtle nuances. Please kindly zoom in for a better view. More visual results are provided in the supplemental file.
(d) ISNet [44]
(e) SegRefiner (ours)
Settings In this experiment, we utilize the  $LR$ -SegRefiner model. To refine each instance, we extract the bounding box region based on the coarse mask and expand it by 20 pixels on each side. The extracted region is then resized to match the input size of the model. The output size is suitable for instances in the COCO dataset, allowing us to perform instance-level refinement for all timesteps without requiring any local patch refinement.
Results First, in Tab. 2, we compare the proposed SegRefiner with two model-agnostic instance segmentation refinement methods, BPR [47] and SegFix [56]. As demonstrated in Tab. 2, our SegRefiner achieves significantly better performance compared to these two methods. The coarse masks utilized in Tab. 2 are obtained from Mask R-CNN to ensure consistency with the original experiments conducted in [47, 56]. This choice allows for a fair and direct comparison with the previous works. Then in Tab. 3, we apply our SegRefiner to other 7 instance segmentation models. Our method yields significant enhancements across models of varying performance levels. Furthermore, when compared to three model-specific instance segmentation refinement models, including PointRend [29], RefineMask [57], and Mask TransFiner [27], our SegRefiner consistently enhances their performance. These results establish SegRefiner as the leading model-agnostic refinement method for instance segmentation.
4.4 Dichotomous Image Segmentation
Dichotomous Image Segmentation (DIS) is a recently introduced task by Qin et al. [40], which specifically targets the segmentation of objects with complex texture structures, such as the steel framework of bridge illustrated in Fig. 4. To facilitate research in this area, Qin et al. also built the DIS5K dataset, a meticulously annotated collection of 5,470 images with resolutions of 2K and above. The DIS5K dataset, characterized by its abundance of fine-grained structural details, poses rigorous
Table 2: Comparison with other model-agnostic instance segmentation refinement methods on the COCO val set using LVIS annotations.
Method	Mask AP	Boundary AP
AP	\(AP_{50}\)	\(AP_{75}\)	\(AP_{S}\)	\(AP_{M}\)	\(AP_{L}\)	AP	\(AP_{50}\)	\(AP_{75}\)	\(AP_{S}\)	\(AP_{M}\)	\(AP_{L}\)
MaskRCNN (Res50)	39.8	61.4	42.3	24.9	47.0	55.1	27.3	53.3	25.2	24.9	41.8	27.4
+SegFix [56]	40.6	61.4	42.8	25.0	48.4	56.6	29.1	53.7	28.0	24.9	43.6	30.8
+BPR [47]	41.0	61.4	43.1	24.8	48.5	57.8	30.4	55.2	29.5	24.7	43.8	33.7
+SegRefiner (ours)	41.9	61.6	43.2	25.7	49.4	58.8	32.6	55.7	32.5	25.6	45.0	37.3
MaskRCNN (Res101)	41.6	63.3	44.4	26.5	49.5	57.8	29.0	55.2	26.7	26.3	44.4	29.8
+SegFix [56]	42.2	63.4	44.7	26.5	50.9	59.1	30.6	56.1	30.0	26.3	46.0	33.1
+BPR [47]	42.8	63.3	45.3	26.1	51.0	60.6	32.0	57.3	31.6	25.9	46.3	36.3
+SegRefiner (ours)	43.6	63.3	45.2	27.4	51.4	61.6	34.1	57.2	34.9	27.2	47.1	39.9
Table 3: Tansfering our SegRefiner to other instance segmentation models.
Method	Mask AP	Boundary AP
AP	\(AP_S\)	\(AP_M\)	\(AP_L\)	AP	\(AP_S\)	\(AP_M\)	\(AP_L\)
PointRend [29]	41.5	25.1	49.0	59.3	30.6	25.0	44.2	34.1
+SegRefiner	42.8 +1.3	25.9 +0.8	50.4 +1.4	61.3 +2.0	33.7 +3.1	25.8 +0.8	46.2 +2.0	40.1 +6.0
RefineMask [57]	41.2	24.0	48.1	59.2	30.5	23.8	43.5	34.1
+SegRefiner	41.9 +0.7	24.6 +0.6	48.7 +0.6	60.7 +1.5	33.0 +2.5	24.5 +0.7	44.7 +1.2	39.4 +5.3
Mask Transfiner [27]	42.2	25.9	49.0	60.1	31.6	25.8	44.5	35.8
+SegRefiner	43.3 +1.1	26.8 +0.9	49.9 +0.9	62.0 +1.9	34.4 +2.8	26.6 +0.8	45.9 +1.4	41.3 +5.5
SOLO [51]	37.4	19.3	45.5	56.6	24.7	19.0	39.3	27.8
+SegRefiner	40.5 +3.1	21.7 +2.4	49.3 +3.8	60.9 +4.3	31.3 +6.6	21.4 +2.4	44.8 +5.5	39.8 +12.0
CondInst [48]	39.8	24.3	47.5	54.8	29.2	24.1	42.6	30.2
+SegRefiner	41.1 +1.3	25.5 +1.2	48.8 +1.3	56.8 +2.0	32.2 +2.6	25.3 +1.2	44.5 +1.9	36.1 +5.9
QueryInst [18]	42.4	26.7	49.5	61.4	29.9	26.4	44.5	32.3
+SegRefiner	44.2 +1.8	27.5 +0.8	51.4 +1.9	64.7 +3.3	34.9 +5.0	27.3 +0.9	47.2 +2.7	42.6 +10.3
Mask2Former [11]	46.8	27.9	54.9	69.1	37.0	27.8	50.1	44.6
+SegRefiner	47.4 +0.6	28.5 +0.6	55.3 +0.4	69.8 +0.7	38.8 +1.8	28.4 +0.6	51.0 +0.9	48.5 +4.2
Table 4: Transfering our SegRefiner to DIS task. SegRefiner is employed to refine the segmentation results from 6 different models. w/o and w/ indicates the original and refined results, respectively.
Dataset	Metrics	U-Net [42]	PFNet [36]	PSPNet [60]	ICNet [59]	HRNet[50]	ISNet[44]
w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/
DIS-VD	IoU	54.77	58.73+3.96	56.20	60.44+4.24	56.41	60.41+4.00	56.47	61.38+4.91	61.02	64.43+3.41	67.10	68.27+1.17
mBA	69.84	75.44+5.6	63.69	74.83+11.14	62.78	74.84+12.06	66.47	75.55+9.08	68.94	76.62+7.68	74.13	76.64+2.51
DIS-TE1	IoU	44.11	47.05+2.94	46.97	49.01+2.04	47.57	49.01+1.44	46.22	49.74+3.52	50.98	52.97+1.99	56.17	57.00+0.83
mBA	70.13	74.97+4.84	65.13	74.67+9.54	64.45	74.67+10.22	67.19	75.01+7.82	69.34	75.62+6.28	73.63	75.49+1.86
DIS-TE2	IoU	54.39	58.10+3.71	57.04	60.26+3.22	57.71	60.25+2.54	56.50	60.33+3.83	61.22	64.70+3.48	67.05	67.84+0.79
mBA	69.99	75.57+5.58	64.64	75.05+10.41	63.71	75.06+11.35	66.79	75.57+8.78	69.50	76.64+7.14	74.03	76.35+2.32
DIS-TE3	IoU	59.29	63.34+4.05	59.46	63.42+3.96	59.90	63.44+3.54	59.83	64.20+4.37	64.43	67.57+3.14	69.94	70.87+0.93
mBA	70.59	76.68+6.09	64.00	76.06+12.06	62.74	76.05+13.31	66.73	76.55+9.82	69.48	77.60+8.12	74.65	77.39+2.74
DIS-TE4	IoU	61.14	66.29+5.15	59.11	66.63+7.52	57.61	66.64+9.03	60.53	67.01+6.48	64.67	70.47+5.80	70.12	72.21+2.09
mBA	70.68	77.07+6.39	62.78	76.43+13.65	61.88	76.43+14.55	66.42	77.06+10.64	68.09	77.86+9.77	74.35	77.66+3.31
demands on a model's capability to perceive and capture intricate information. Thus, it serves as a suitable benchmark to evaluate the performance of our refinement method. The evaluation metrics and other settings utilized in this experiment are consistent with those in Sec. 4.2.
Results Since the recent introduction of DIS5K, no prior refinement methods have reported results on this dataset to date. Therefore, the main aim of this experiment is to evaluate the transferability of our SegRefiner across various models. As shown in Tab. 4, our SegRefiner is applied to 6 segmentation models. The results consistently demonstrate that our SegRefiner improves the performance of each segmentation model in terms of both IoU and mBA. In Fig. 4, we provide qualitative comparisons with previous methods, which reveal the superior ability of our SegRefiner to capture fine-grained details, such as the dense and fine mesh of the chair.
We conduct ablation studies on the BIG dataset [44] with high-resolution images. We first investigate the effectiveness of the diffusion process in SegRefiner. The results are reported in Tab. 5a, where the term "none" refers to the utilization of a U-Net without multi-step iteration, and "w/o diffusion" denotes the direct utilization of the previous step's mask as the input for the subsequent iteration, without employing the diffusion sampling process, i.e., discarding the transition sample module. Since there is no states-transition probability in non-diffusion methods, we use a sliding window strategy similar to [12] in this experiment to obtain the input for local refinement. The results in Tab. 5a demonstrate that the diffusion-based iterative process achieves the best performance, validating the effectiveness of the diffusion process in SegRefiner.
In Tab. 5b and Tab. 5c, we evaluate various alternatives used in prior experiments. Tab. 5b shows the analysis of global and local refinement used in high resolution images. As can be seen, global refinement significantly improves IoU; however, for high-resolution images, the resulting smaller output size leads to a lower mBA. Local refinement applied to local patches of the original size greatly improves mBA, while its enhancement in IoU is less significant due to the absence of global information. The combination of local and global refinement achieves better performance in both IoU and mBA. Tab. 5c presents the results corresponding to different input image sizes. Considering computational load and memory usage,  $256 \times 256$  is selected as the default setting, which performs well without introducing too much computational overhead.
Table 5: Ablation studies on BIG dataset [44]. The best results are highlighted in bold and the default settings are marked with gray.
(a) Effectiveness of the diffusion process.
Iteration Type	IoU	mBA
none	94.10	75.03
w/o diffusion	94.11	75.10
w/ diffusion	94.85	77.64
Method	IoU	mBA
only global	92.43	60.58
only local	89.73	63.56
global+local	94.85	77.64
Input Size	IoU	mBA
128×128	93.10	74.28
256×256	94.85	77.64
512×512	94.97	77.56
We propose SegRefiner, which is the first diffusion-based image segmentation refinement method with a new designed discrete diffusion process. SegRefiner performs model-agnostic segmentation refinement and achieves strong empirical results in refinement of various segmentation tasks.
While SegRefiner has achieved significant improvements in accuracy, one limitation lies in that the diffusion process leads to slowdown of the inference due to the multi-step iterative strategy. As shown in Tab. 6, we conduct an experiment on instance segmentation about the relationship between the model's accuracy, computational complexity, time consumption (the average time consumed per image), and the number of iteration steps. It can be observed that, while the iterative strategy has provided SegRefiner with a noticeable improvement in accuracy, it has also introduced a linear increase with the number of steps in both time consumption and computational complexity. As the first work applying diffusion models to the refinement task, the proposed SegRefiner primarily concentrates on devising a suitable diffusion process for general refinement tasks. While improving the efficiency of diffusion models will be a crucial research direction in the future, not only in the field of image generation but also in other domains where diffusion models are applied.
Table 6: The relationship between SegRefiner's accuracy, efficiency and the number of iteration steps on instance segmentation.
Steps	0	1	2	3	4	5	6
Mask AP	39.8	40.7	41.0	41.4	41.6	41.9	41.9
Boundary AP	27.3	29.9	30.6	31.5	32.1	32.5	32.6
Time (s)	n/a	0.52	1.01	1.53	1.98	2.45	2.94
GFLOPs	n/a	249	497	746	994	1243	1492
This research was funded by the Fundamental Research Funds for the Central Universities (2023JBZD003).
[1] Tomer Amit, Eliya Nachmani, Tal Shaharabany, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. CoRR, abs/2112.00390, 2021.
[2] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurIPS, volume 34, pages 17981-17993, 2021.
[3] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. CoRR, abs/2208.09392, 2022.
[4] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In ICLR, 2022.
[5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 801-818, 2018.
[52] Junde Wu, Huihui Fang, Yu Zhang, Yehui Yang, and Yanwu Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. CoRR, abs/2211.00611, 2022.
[53] Junde Wu, Rao Fu, Huihui Fang, Yu Zhang, and Yanwu Xu. Medsegdiff-v2: Diffusion based medical image segmentation with transformer. CoRR, abs/2301.11798, 2023.
[54] Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai. Seqformer: Sequential transformer for video instance segmentation. In ECCV, pages 553-569. Springer, 2022.
[55] Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang, Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In CVPR, pages 1154-1163, 2021.
[56] Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refinement for segmentation. In ECCV, pages 489–506. Springer, 2020.
[57] Gang Zhang, Xin Lu, Jingru Tan, Jianmin Li, Zhaoxiang Zhang, Quanquan Li, and Xiaolin Hu. Refinemask: Towards high-quality instance segmentation with fine-grained features. In CVPR, pages 6861-6869, 2021.
[58] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. CoRR, abs/2302.05543, 2023.
[59] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In ECCV, pages 405-420, 2018.
[60] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, pages 2881-2890, 2017.
[61] Peng Zhou, Brian Price, Scott Cohen, Gregg Wilensky, and Larry S Davis. Deepstrip: High-resolution boundary refinement. In CVPR, pages 10558-10567, 2020.
Here we first introduce the details of SegRefiner's equation derivation in Sec. A. The implementation details are shown in Sec. B. Moreover, we provide additional qualitative results in Sec. C.
A Equation Derivation Details
In the following, we will present the derivation of the posterior distribution (Eq. (9)) and the reverse transition-probability (Eq. (11)). Firstly, for Eq. (9), it's derived from the Bayes' theorem:
$$
q \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) = \frac {q \left(x _ {t} \mid x _ {t - 1} , x _ {0}\right) q \left(x _ {t - 1} \mid x _ {0}\right)}{q \left(x _ {t} \mid x _ {0}\right)}. \tag {12}
$$
According to Eq. (6) and Eq. (7), we can get all the items in Bayes' theorem as:
$$
q \left(x _ {t} \mid x _ {t - 1}, x _ {0}\right) = x _ {t - 1} Q _ {t} x _ {t} ^ {T}, \tag {13}
$$
$$
q \left(x _ {t - 1} \mid x _ {0}\right) = x _ {0} \bar {Q} _ {t - 1} x _ {t - 1} ^ {T}, \tag {14}
$$
$$
q \left(x _ {t} \mid x _ {0}\right) = x _ {0} \bar {Q} _ {t} x _ {t} ^ {T}. \tag {15}
$$
It appears to be slightly different from Eq. (6). The reason is that the results of Eq. (6) is a length-2 vector, representing the probabilities of two states respectively. While we use a scalar form here, which includes an additional one-hot vector  $x_{*}^{T}$  and is more suitable for subsequent derivations. For the denominator of Eq. (9), we have obtained the same form. Let's consider the numerator:
$$
\begin{array}{l} q (x _ {t} | x _ {t - 1}, x _ {0}) q (x _ {t - 1} | x _ {0}) = x _ {t - 1} Q _ {t} x _ {t} ^ {T} \cdot x _ {0} \bar {Q} _ {t - 1} x _ {t - 1} ^ {T} \\ = x _ {t - 1} \left(x _ {t} Q _ {t} ^ {T}\right) ^ {T} \cdot x _ {0} \bar {Q} _ {t - 1} x _ {t - 1} ^ {T} (16) \\ = \left(x _ {t} Q _ {t} ^ {T}\right) x _ {t - 1} ^ {T} \cdot \left(x _ {0} \bar {Q} _ {t - 1}\right) x _ {t - 1} ^ {T} (16) \\ = \left[ \left(x _ {t} Q _ {t} ^ {T}\right) \odot \left(x _ {0} \bar {Q} _ {t - 1}\right) \right] x _ {t - 1} ^ {T}, \\ \end{array}
$$
where " $\cdot$ " refers to scalar multiplication and " $\odot$ " refers to element-wise multiplication. This is the scalar form, by removing the  $x_{t-1}^T$ , we can get the corresponding vector form just as Eq. (9).
Secondly, for Eq. (11), it's derived from Eq. (9) by substituting  $x_0$  with  $[p_{\theta}(\tilde{m}_{0|t})^{i,j}, 1 - p_{\theta}(\tilde{m}_{0|t})^{i,j}]$  (we refer  $p_{\theta}(\tilde{m}_{0|t})^{i,j}$  to  $p_0$  in the following for simplicity).
$$
\begin{array}{l} p (x _ {t - 1} | x _ {t}) = \frac {[ (x _ {t} Q _ {t} ^ {T}) \odot (x _ {0} \bar {Q} _ {t - 1}) ] x _ {t - 1} ^ {T}}{x _ {0} \bar {Q} _ {t} x _ {t} ^ {T}}, w i t h x _ {0} = [ p _ {0}, 1 - p _ {0} ] \\ = \frac {x _ {t} \left(Q _ {t} ^ {T} \odot \left[ p _ {0} \bar {\beta} _ {t - 1} , 1 - p _ {0} \bar {\beta} _ {t - 1} \right]\right) x _ {t - 1} ^ {T}}{\left[ p _ {0} \bar {\beta} _ {t} , 1 - p _ {0} \bar {\beta} _ {t} \right] x _ {t} ^ {T}} \\ = \frac {x _ {t} \left[ \begin{array}{c c} p _ {0} \bar {\beta} _ {t} & 0 \\ p _ {0} (\bar {\beta} _ {t - 1} - \bar {\beta} _ {t}) & 1 - p _ {0} \bar {\beta} _ {t - 1} \end{array} \right] x _ {t - 1} ^ {T}}{x _ {t} [ p _ {0} \bar {\beta} _ {t} , 1 - p _ {0} \bar {\beta} _ {t} ] ^ {T}} \tag {17} \\ = x _ {t} \left[ \begin{array}{c c} 1 & 0 \\ \frac {p _ {0} (\bar {\beta} _ {t - 1} - \bar {\beta} _ {t})}{1 - p _ {0} \bar {\beta} _ {t}} & \frac {1 - p _ {0} \bar {\beta} _ {t - 1}}{1 - p _ {0} \bar {\beta} _ {t}} \end{array} \right] x _ {t - 1} ^ {T}. \\ \end{array}
$$
Same as Eq. (9), we can remove  $x_{t - 1}^{T}$  and get the vector form of Eq. (11).
In this section, we give a detailed description of the model architecture and training/inference settings. The overall workflow of the training and inference process are provided in Alg. 1 and Alg. 2.
Model Architecture Following [39], we use a U-Net with 4-channel input and 1-channel output. Both input and output resolution is set to  $256 \times 256$ . Considering computational load and memory usage, we set the intermediate feature channels to 128 and only conduct self-attention in strides 16 and 32.
Algorithm 1 Training  
Input total diffusion steps  $T$ , datasets  $D = \{(I, M_{fine}, M_{coarse})^K\}$   
repeat  
Sample  $(I, M_{fine}, M_{coarse}) \sim D$   
Sample  $t \sim \text{Uniform}(1, \ldots, T)$   
Initialize  $m_0 = M_{fine}$ ,  $x_0^{i,j} = [1,0]$ $q(x_t^{i,j} | x_0^{i,j}) = x_0^{i,j} \bar{Q}_t$   
Sample  $x_t^{i,j} \sim q(x_t^{i,j} | x_0^{i,j})$ , get  $x_t \in \{0,1\}^{2 \times H \times W}$   
Pixels Transition  $m_t = x_t[0] \odot M_{fine} + x_t[1] \odot M_{coarse}$   
Take gradient descent step on  $\nabla_\theta \mathcal{L}(f_\theta(I, m_t, t), M_{fine})$   
until convergence
Algorithm 2 Inference  
Input total diffusion steps  $T$ , image and coarse mask  $(I, M_{coarse})$   
Initialize  $x_{T} = [0,1]$ ,  $m_{T} = M_{coarse}$   
for  $t$  in  $\{T, T-1,\dots,1\}$  do  
 $\tilde{m}_{0|t}, p_{\theta}(\tilde{m}_{0|t}) = f_{\theta}(I, m_t, t)$ $p_{\theta}(x_{t-1}^{i,j}|x_t^{i,j}) = x_t^{i,j}P_{\theta,t}^{i,j}$   
Sample  $x_t^{i,j} \sim p_{\theta}(x_{t-1}^{i,j}|x_t^{i,j})$ , get  $x_t \in \{0,1\}^{2 \times H \times W}$   
Pixels Transition  $m_{t-1} = x_{t-1}[0] \odot \tilde{m}_{0|t} + x_{t-1}[1] \odot M_{coarse}$   
return  $m_0$
Training Settings All experiments are conducted on 8 NVIDIA RTX3090 GPUs with Pytorch. During training, we first train the  $LR-SegRefiner$  on the LVIS dataset [20] with 120k iterations. The AdamW optimizer is used with the initial learning rate of  $4 \times 10^{-4}$ . We use a multi-step learning rate schedule, which decays by 0.5 in steps 80k and 100k. Subsequently, the  $HR-SegRefiner$  is obtained from 40k-iterations fine-tuning based on the 80k checkpoint of  $LR-SegRefiner$ . Batch size is set to 8 in each GPU.
Inference Settings In instance segmentation, we use the LR-SegRefiner to perform refinement in instance level. For each instance, we extract the bounding box region based on the coarse mask and expand it by 20 pixels on each side. The extracted region is then resized to match the input size of the model. After a complete reverse diffusion process, the output is resized to the original size.
In semantic segmentation and dichotomous image segmentation, because of the high resolution of images, we employ the HR-SegRefiner and conduct a global-and-local refinement process. In order to identify the local patches that require refinement, we filter out pixels with low state-transition probabilities from the globally refined mask and use them as the center points for the local patches. We apply Non-Maximum Suppression (NMS, with 0.3 as threshold) to these patches to remove excessive overlapping.