High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity
Authors: Qian Yu
Peng-Tao Jiang
Hao Zhang
Jinwei Chen
Bo Li
Lihe Zhang
Huchuan Lu
Venue: ICLR 2025 Poster (2025)
Keywords: dichotomous image segmentation
diffusion models
high-resolution image segmentation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In the realm of high-resolution (HR), fine-grained image segmentation, the primary challenge is balancing broad contextual awareness with the precision required for detailed object delineation, capturing intricate details and the finest edges of objects. Diffusion models, trained on vast datasets comprising billions of image-text pairs, such as SD V2.1, have revolutionized text-to-image synthesis by delivering exceptional quality, fine detail resolution, and strong contextual awareness, making them an attractive solution for high-resolution image segmentation. To this end, we propose DiffDIS, a diffusion-driven segmentation model that taps into the potential of the pre-trained U-Net within diffusion models, specifically designed for high-resolution, fine-grained object segmentation. By leveraging the robust generalization capabilities and rich, versatile image representation prior of the SD models, coupled with a task-specific stable one-step denoising approach, we significantly reduce the inference time while preserving high-fidelity, detailed generation. Additionally, we introduce an auxiliary edge generation task to not only enhance the preservation of fine details of the object boundaries, but reconcile the probabilistic nature of diffusion with the deterministic demands of segmentation. With these refined strategies in place, DiffDIS serves as a rapid object mask generation model, specifically optimized for generating detailed binary maps at high resolutions, while demonstrating impressive accuracy and swift processing. Experiments on the DIS5K dataset demonstrate the superiority of DiffDIS, achieving state-of-the-art results through a streamlined inference process. The source code will be publicly available at \href{https://github.com/qianyu-dlut/DiffDIS}{DiffDIS}.

Published as a conference paper at ICLR 2025
HIGH-PRECISION DICHOTOMOUS IMAGE SEGMENTA- TION VIA PROBING DIFFUSION CAPACITY
Qian Yu 12- Peng-Tao Jiang*+ Hao Zhang² Jinwei Chen? Bo Li Lihe Zhang' Huchuan Lu'
Dalian University of Technology vvivo Mobile Communication Co., Ltd
{ms. yuqian) aeggail dlut edu. en, {pt. jiang, Jibrajevivo.com zhangl the, Juhuchuan e @dlut www.l..e...........ooeo. edu.cn
Figure 1: Dichotomous image segmentation results using our DifIDIS.
ABSTRACT
In the realm of high-resolution (HR), fine-grained image segmentation, the pri- mary challenge balaneing broad contextual awareness with the precision re- quired for detailed object delineation, capturing intricate details and the finest edges objects. Diffusion models, trained on vast datasets comprising billions of image-text pairs, such as SD V2.1, have revolutionized text-to-image synthesis by delivering exceptional quality, fine detail resolution, and strong contextual aware- ness, making them attractive solution for high-resolution image segmentation. To this end, we propose DiffDIS, a diffusion-driven segmentation model that taps into the potential of the pre-trained U-Net within diffusion models, specifically designed for high-rese regouutooouuion, fine- grained object segmentation. By leveraging the robust generalization capabilities and rich, versatile image representation prior of the SD models, coupled with task-specific stable one-step denoising approach, we significantly reduce the inference time while preserving nigh-fidelity, detailed generation. Additionally, we introduce an auxiliary edge generation task to not only enhance the preservation of fine details of the object boundaries, but reconcile the probabilistic nature of diffusion with the deterministic demands of segmenta- tion. With these refined strategies place, DifIDIS serves as a rapid object mask generation model, specifically optimized for generating detailed binary maps at high resolutions, while demonstrating impressive accuracy and swift processing.
Work was done during interning at vivo. The first two authors share equal contributions. Corresponning authors.

---

Published as a conference paper at ICLR 2025
Experiments the DIS5K dataset demonstrate the superiority of DifIDIS, achiev- ing state-of-the-aae art results through a streamlined inference process. The source code will be publicly available at DifiDIS
INTRODUCTION
Hlih-aacurauyy dichotomous image segmentation (DIS) (Qin et al.
2022) aims to accurately identify
sallegory-anoosie foreground objects within natural scenes, which is fundamental for a wide range of
scene understanding applications, including AR/VR applications (Tian et al. 2022: Qin et al.) 2
image editing (Goferman et al., (01T), and 3D shape reconstruction (Liu et al., 2021). Different from existing segmentation tasks, DIS focuses on challenging high-resolution (HR) fine-grained
object segmentation, regardless of their characteristics. These objects often encompass a greater volume of information and exhibit richer detail, thereby demanding more refined feature selection and more sophisticated algorithms for segmentation. Current CNN-(Pe et al. 2023/Qin et (2022) and Transformer- (Xie al. 2022 Kim et al. 2022 Yu et al. 2024) methods, despite their robust feature extraction capabilities, often face challenges in balancing receptive field expansion
with detail preservation in high-resolution images (Xie et al. 2022 Yu et al. 2024).
Diffusion probabilistic models (DPMs), by predicting noise variables across the entire image, have been demonstrated by numerous studies (Ji al., [023] Wang et al.] 2023) to show promise in maintaining a global receptive field while more accurately learning the target distribution. More- over, Stable Diffusion (SD) (Rombach et al., 2022) has emerged as a significant leap forward in the domain of DPMs. Trained on vast datasets comprising billions of images, it exhibits robust gen- eralization capabilities and offers a rich, versatile image representation, positioning as an ideal candidate for tasks demanding both macroscopic context and microscopic precision. Its power is further underscored by its significant contributions to fine-grained feature extraction, as evidenced by recent studies (Ke et al.) 2024 Zhang et al. 2024 Zavadski et al. 2024 Hu et al.) [2023] Xu et al.) 2024) that have harnessed the SD's capabilities to capture the subtleties detail. The advancements in these methods inspired that diffusion could be powerful tool for improving the accuracy and robustness of high-resolution image segmentation.
However, there are several challenges when leveraging the diffusion model to DIS: (1) The in- herent high-dimensional, continuous feature space of DPM conflicts with the discrete nature of binary segmentation, potentially leading to a discrepancy in the predictive process. (2) More- over, the diffusion models often suffer from lengthy inference time. Due to the recurrent na- ture of diffusion models, it usually takes more than 100 steps for DPM to generate satisfying re- sults (Ke et al., 2024, which further exacerbates the already slow inference speeds of HR images due to their substantial data volume. (3) There is a fundamental conflict between the stochas- tic nature of diffusion and the deterministic outcomes required for image perception tasks.
Against this backdrop, we propose DiffDIS, ad- dressing both aforementioned challenges and task- specific complexities of DIS, focusing en- hanced processing speed, stronger detail percep- tion, and higher determinism, achieved through the
Table 1: The restorative capability of VAE.
Metric VAE
iggar t
E f
Sm 0.985
M1
0.993
0.999
0.002
following strategies: First, we found that the Variational Autoencoder (VAE) (Kingmal 2013) has demonstrated the capability nearly achieve perfect reconstruction of binary masks (See Tab. [] Therefore, following SD (Rombach et al.) 2022), by mapping the masks through the VAE into the latent space, we can not only effectively leverage the strengths of diffusion models for denoising and refinement within it, but also significantly reduces the computational cost associated with processing HR image segmentation. Then, by employing a direct one-step denoising paradigm (See Eqn. 3 and integrating the pretrained parameters of SD-Turbo (Sauer et al. 2023 Parmar et al.] 2024). which feature a smoother probability curve compared to the standard SD V2.1 parameters, we streamline the network into end-to-end model. This approach facilitates one-step denoising while leveraging its robust generalization capabilities and preserving the intricate details of high-resolution objects. Moreover, we introduce joint predicting strategy for mask and edge, which enhances mask gen- eration through finer constraints while improving controllability of diffusion models for perception tasks. Finally, we condition the diffusion model on RGB latent representations and propose Scale-
2

---

Published as a conference paper at ICLR 2025
Wise Conditional Injection to enable multi- grantig,,,,,ar,, longerange feature interactions, ensuring more refined feature preservation and selection.
Generally, our main contributions can be summarized as follows: (1) We propose DifiDIS, lever- aging the powerful prior of diffusion models for the DIS task, elegantly navigating the traditional struggle to effectively balance the trade-off between receptive field expansion and detail preserva- tion in traditional discriminative learning-basse based methods.
(2) We transform the recurrent nature of diffusion models into an end-to-end framework by implementing straightforward one-step denois- ing, significantly accelerating the inference speed. (3) We introduce an auxiliary edge generation task, complemented by an effective interactive module, to achieve a nuanced balance in detail rep- resentation while also enhancing the determinism of the generated masks. (4) We advance the field forward by outperforming almost all metrics on the DIS benchmark dataset, and thus establish a new SoTA in this space. Additionally, it boasts an inference speed that is orders of magnitude faster than traditional multi-step diffusion approaches without compromising accuracy.
RELATED WORKS
2.1
CONVENTIONAL APPROACH TO DEAL WITH DIS WORKS
Dichotomous Image Segmentation (DIS) is formulated as a category-agnostie task that focuses on accurately segmenting objects with varying structural complexities, independent of their specific characteristics. It includes high-resolution images containing salient (Pang et al.] 2220;; Zhao et al., 2020 2024ch, camouflaged (Pang et al.] 2024b 2022 Fan al., 2020 Pang et al.) 2024a), and meticulous (Lie et al. 2021 Yang et al. (2020) instances in various backgrounds, integrating sev- eral context-dependent (hhao et al.) 2024b a) segmentation tasks into unified benchmark. When dealing with DIS, it is essential to consider the demand for highly precise object delineation, captur- ing the finest internal details of objects. Upon introducing the DIS dataset, Qin et al. also presented IS-Net (Qin et al. 2022). model specifically crafted to address the DIS challenge, utilizing the U and intermediate supervision to mitigate overfitting risks. PF-DIS (Zhou et al. 2023) uti- lized frequency prior generator and featured harmonization module to identify fnneggrained object boundaries in DiS. UDÜN (Pei et al., 2023) proposed a unite-divide-ume
scheme to disentangle the trunk and structure segmentation for DIS. InSPyReNet (Kim al., (022) was constructed to gener- ated HR outputs with a multi-resolution pyramid blending at the testing stage. Recent approaches have incorporated multi muult-granularity, cues that hamness both global and local information to enhance localization accuracy and detail fidelity. BiRefNet (Zheng et al., (024) employed a bilateral refer- ence strategy, leveraging patches of the original images at their native scales as internal references and harnessing gradient priors as external references. Recently, MVANet (Yu et al., 2024) modeled the DIS task as a multi-view object perception problem, leveraging the complementary localiza- tion and refinement among views to process HR, fine-detail images. Despite their commendable performance, existing methods have not effectively balanced the semantic dispersion of HR targets within a limited receptive field with the loss of high-precisin details in a larger receptive field when tackling DIS. Compared to conventional approaches, our diffusion based framework has excelled in achieving high-quality background removal, with fast processing times, and offers straightforward integration.
2.2 DIFFUSION MODELS FOR DENSE PREDICTION AND EFFICIENT INFERENCE IN DIFFUSION
With the recent success of diffusion models in generation tasks, there has been a noticeable rise in interest in incorporating them into dense visual prediction tasks. Several pioneering works attempted to apply the diffusion model to visual perception tasks, e.g. g.image segmentation (Amit et al.) 202T
Ji et al. 2023 Wang et al. 2023). matting (Hu et al. 2023 2024), depth estimation (Ke al. 2024 Zhang et al. 2024 Zavadski et 2024). edge detection Ye et al. 2024) et al.. Since the pioneering work (Amit et al. 2021) introduced diffusion methods to solve image segmentation, several follow-
ups use diffusion to attempt their respective tasks. (Ji et al. 2023) formulated the dense visual prediction tasks as a general conditional denoising process. (Hu et al., 2023) pioneered the use of diffusion in matting, decoupling encoder and decoder to stabilize performance with uniform time intervals. (Hu et al.
2024) ingeniously trained the model to paint on a fixed pure green screen backdrop. (Ye al."
2024 utilized decoupled architecture for faster denoising and an adaptive
3

---

Published as a conference paper at ICLR 2025
Fourier filter to adjust latent features specific frequencies. In the depth estimation field, several recent works have leveraged diffusion for high-fidelity finegrainedd generation. (Ke et al. (024) introduced latent diffusion model based SD (Rombach et al.] )022). with fine-tuning for depth estimation, achieving strong performance on natural images. (Zavadski et al. (024) extracts a rich, frozen image representation from SD, termed preimage, which is then refined for downstream tasks. Inspired by their work, we observe that duffusionn-gnerall complex data distributions and generating realistic texture details.
methods naturally excel at modeling
However, traditional multi-step generation models encounter a challenge: the recurrent structure of diffusion models often necessitates over 100 steps produce satisfactory outputs. In response, various proposals have emerged to expedite the inference process. Distillation-base methods have demonstrated remarkable speedups by optimizing the original diffusion model's weights with en- hanced schedulers or architectures. (Luo et al. 2023) achieves a few-step inference in conditional image generation through self-consistency enforcement. SD-Turbo (Sauer et al.) 2023) employs Adversarial Diffusion Distillation (ADD), utilizing pre-trained SD model to denoise images and calculate adversarial and distillation losses, facilitating rapid, high-quality generation. Recently, GenPercept (Xu et al. 2024) presents novel perspective the diffusion process tion between RGB images and perceptual targets, effectively harnessing the pre-trained U-Net for various downstream image understanding tasks. Given that SD-Turbo refines both high-frequency and low-frequency information through distillation (Sauer et al. 2023), ) by harnessing its efficient prior for image generation, we can sustain competitive performance in single-step denoising sce- nario while achieving high-fidelity, fine-grainn treeggaaied mask generation (See Fig.
interpola-
3 PRELIMINARIES
Diffusion probabilistic models (Ho et al. 2020) have emerged as highly successful approaches for generating images by modeling the inverse process of diffusion process from Gaussian noise. It defines a Markovian chain of diffusion forward process g(<e (z)) by gradually adding noise to input data Z0
It = VājJo + VI- <~ 10,
(I)
where is pure Gaussian noise map. As increases, ã₂ gradually decreases, leading I; approx- imate the Gaussian noise. By predicting the noise, the loss can be written as:
T
L (co) = Σ Ezg E20- [||so (Vā,zu + vi - -c]
During the reverse process in DDPM (Ho et al. 2020), given a random sampled Gaussian noise IT ~ (O, I), we utilize a Gaussian distribution Po x,) to approximate the true posterior distribution, whose mean is estimated by the neural network, and the variance, denoted as of. is derived from the noise schedule. To this end, we can repeat the following denoising process for te (T,T-1,.... 1,..., 1} to predict final denoised result Fo.
1 Vat
1- ai
x--1
It
V1 ā Eo (Ze,t) + oee,
(3)
In the context of our method, during the training phase, instead of randomly selecting t, we initialize t to its maximum value T, i.e., 999. This initialization enables the network to directly learn the probability distribution for a single-step denoising process. From Eqn. 3 the one-step denoising formula can be derived as follows. Since or is relatively small when T 999, its impact on the zo can be considered negligible. The formula is given by:
Zo Denoise (Ea(zT, T' cond, diab), ET v - arroooT.. T)
(4)
Var
where cond represents the conditional input, specifically the image latent, while diab denotes dis- criminative labels used to generate dishimiicrrimieative embeddings (See Sec. 4.2) Accordingly,
the training objective of our method can be reformulated as minimizing the expected squared error between the denoised output and the ground truth latent xo. as expressed by:
min Et.cc ||00 -xall 1 8
(5)
4

---

Published as a conference paper at ICLR 2025
Latent Space
DRIA
VAE
UNet
Batch-wise Concat
Injector
Batch- Batch-DDicriminntiiee
C
Channel-wise Concat
T: Timestep
Embedding
Supervision
N: High-step Noise
min Baccond Denoise (tu(ity, T. c.ndd, chan).
Figure 2: Overall framework of DINIDS. To start with, the inputs are encoded into the latent space. We concatenate the noisy mask and its corresponding edge latent along the batch dimension, utilizing batch- discriminative embedding for diffcrentiation. Then, we employ the RGB latent as conditioning factor through channel comcatenation, along with multi-scalc conditional injection into the U-Net encoder layers. The noise obtained is processed through a direct single-step denoising approach to yield a clean latent prediction.
4
METHOD
4.1
OVERALL ARCHITECTURE
The workflow of DifIDIS is outlined in Alg. Algorithm 1 First, the imputs are encoded into the VAE's latent space, capturing the essential features tinuous, nigh-dimensional representation. Then, denoising model applied to mitigate noise, aiming for the best denoising effects to refine the representations and enhance clarity of the fea- tures. We utilize the RGB latent conditioning factor to assist in generating more authentic struc- tural details for the mask. Meanwhile, we improve the information flow by multi-tasking to further boost the accuracy. We concatenate the mask and edge on batch basis and apply Batch Discrimi- native Embedding (Fu al. (024) obtain dis- tinctive embeddings for each. These embeddings are then fed into the denoising network together to help distinguish between mask and edge.
&Trainng Process
con-
Input: cond:
conditional image latent, m: mask latent, e: edge latent, dyas: discriminative labels while converged do
t t= T
c N2 I)
demb BDE( diab)
m, Varm + / āqe
Vare + v1 C+t
pprod... "pred, = 66 =60(m,, ct; cond, L, demb)
pred,... (m, - vI ai X (peod
Vār
pred (e+ VI- - āq X Epres, aa
Perform Gradient descent steps on V Looaa
end while return 0
Within the U-Net architecture, specifically in the mid-block where semantic information is most con- centrated, we strategically integrate an additional attention module called Detail-Ban maaal-BBlancingg Interac- tive Attention. This module not only enhances the determinism of the generative model's predictions (See Fig. 4 but also facilitates a concise yet potent interaction between the nigh-dimensional noisy features of masks and edges, detailed in Section 4.2 Upon acquiring the noise estimated from the U-Net, we proceed with a direct one-step denoising transition from the high-step noisy latents, in accordance with the methodology outlined in Eqn. 4 We subsequently utilize the initial mask and edge latent codes obtained from the VAE for direct supervision of the denoised mask and edge latent codes. For an illustrative training diagram, refer to Fig. 2
4.2
TRAINING STRATEGY
In high-resolution tasks, the complexity and detail richness make it tough to capture fine features. Furthermore, for diffusion- architectures, accurately depicting detailed features during few- step generation poses a challenge, especially as details may be overlooked during denoising in the latent space, which is often reduced to a the size of the input. Some studies (Pei et al. 2023 Chen al.) 20233 Zhao et al. 2022) have indicated that incorporating additional edge constraitt improves the boundary segmentation performance of masks. Given these considerations, we propose an inte- grated and streamlined network architecture that concurrently predicts noise for both the mask and edge, leveraging auxiliary edge information to constrain the network. These dual prediction streams operate within the same network structure and share parameters, with batch discriminative embed-
5

---

Published as a conference paper at ICLR 2025
cond
Injector
Injector
Self-attention
T
Vanilla Cross-attention
Empty embedding
ReeSlokk
Heralock
AtBlock
DBIR
Attalock
ReeSlock
AttBlock
Split Batch
OBB
Qn
Kiw
K.&.F
UNet
Add
Injector
cond,
Conv
Dot product
Mutual Scaled Dot Dot-product Attention
ZeroConv
ZeroCorv
Projection
feat,
Batch wise Concat
ZeroConv
Zero Convolution
Mutual Cross Domain Attention
Figure 3: The structure of tention (DBIA).
Embedding (BDE) and the Detail Baail-BBlaacing Interactive At-
Base
EEdge
+Edge+DBIA
Img
Figure 4: Visualizing the reduction of diffusion process stochasticity through edge integration. With the
addition of the edge auxiliary prediction task, the controllability and alignment of the generated mask have been enhanced, particularly in the areas highlighted by red boxes. The introduction of DBIA improves the sharpness and quality of the generated details.
ding applied to distinguish between them effectively. Within the U-Net architecture, specifically in the mid-block where semantic information is most concentrated, incorporate task-specific en- hancement by upgrading the original attention module to a Detail-Balancing Interactive Attention. This mechanism aligns the attention regions of both the mask and edge streams, facilitating more efficient interaction and complementarity between the two.
aatch-Discriminative Embedding Inspired by (Fu al. 2024 Long et al. 2024). we incorporate additional discriminative labels i.e., diab within batches to enable a single stable diffusion model generate multiple types of outputs simultaneously, ensuring seamless domain processing without mutual interference. Specifically, the dias is first represented in binary form and then encoded using positional encoding (Mildenhall al.) (021). After passing through learnable projection head, the resulting mutch-discriminativv embeddings are combined with the time embeddings through an element-wise addition. The combined embeddings are then fed into the ResBlocks, enhancing the model's 's capacity to produce batch-specific outputs. The process is illustrated in Fig.
Detail-Balancine Interactive Attention To ensure the continuity and alignment of mask and edge, as well as to harmonize their semantic cues and attentioned areas, we introduce Interactive Attention (DBIA). which includes, similar to traditional architectures, self-attention module and vanilla cross-attentin module, along with newly designed Mutual Attention mechanism. DBIA designed to facilitate simple yet effective exchange of information between the two domains, with the ultimate goal of generating outputs that well-aligned in terms of both edge details and semantic content.
Specifically, initialize the Mutual self-attention module, and project the mask and edge features to produce the query (Qm). ) key (K,)), and value (V,, m) matrices for the mask, and correspondingly for the edge (Q.. K., Va), then,
Attention module with the parameters of the

---

Published as a conference paper at ICLR 2025
Algorithm Sampling Process
VAE
Input:
cond:
conditional image latent, d.lab:
U
discriminative labels
Pure k(0.1)
sample
Batch- Discriinaaiin Embedding
Mr, er ~ NN,
demb = BDE(dlab)
Epeed Epeed_ (MT, ET, cond, T, demb)
pped (MT VI - är X Epredon Vār
Wlpred VAE(Scale(Hmod)
return Wppred
Batch-wise Concat
Channel-wise Concat
Injector
Figure Inference procedure of our DifIDIS.
perform mutual scaled dot-product attention between these two sets, utilizing the query features to effectively query the corresponding key and value features from the alternative domain. The formulation is as follows:
Qmmk' dd
QuKkn Kr
Attention, softmax
V
Attention, softmax
V
(6)
vdle
m
This method ensures direct and efficient exchange of information between two domains, fostering robust interaction that enhances the overall feature representation.
Our method differs from the approach of (Fu et al., 2024) in how we handle feature interaction. In their work, keys (K) and values (V) are concatenated into a unified representation, and cross- attention is computed using domain-specific queries (Q) on this combined feature. Our method performs targeted attention between mask and edge features, enhancing discrimination and align- ment while preserving information integrity (Tab. 5}. This approach has been observed not only enhance the fine detail rendering but also, to our delight, significantly reduce the stochastic tenden- cies inherent in diffusion processes (See in Fig.
4.3 SCALE- WISE CONDITIONAL INJECTION
In traditional diffusion models for dense prediction tasks, conditioning is typically done the in- put stage by concatenating along the channel dimension with the variable to be denoised (Ke et al.
2024 Fu et al. 2024). This approach can lead to information loss in later stages. To establish long- range and profound conditional guidance, we introduce Scale- Wise Conditional Injection (SWCI) enhance multi-granulae perception and facilitate deep visual interactions. Specifically, we incor- porate multi-scale conditions into the corresponding layers of the U-Net encoder (See Fig. 33 and employ three injector heads, each composed of a simple convolution layer and two zero convolution layers (Zhang et al. 2023) (See Fig. 3 Each injector head receives conditional latent code that is resized to corresponding scales as input. The resulting outputs are then integrated at the junction of the last three layers in the U-Net encoder, facilitating the generation of more authentic structural de- tails. In this way, we introduce multi- cullti-grauularityy information for semantic and structural perception, without causing excessive interference with RGB texture.
4.4 ONE-STEP MASK SAMPLING
A depiction of the sampling pipeline can be seen in Fig. 5 During inference, we first encode the RGB image into the latent space using a VAE encoder. Next, we sample the starting variable from standard Gaussian noise, which serves as the initialization for both the mask and edge in the latent space. Similar to the training phase, these two components are concatenated a batch, with batch-discriminative embedding applied to effectively distinguish between them. The concatenated components, conditioned on the RGB latent representation, are then fed into the U-Net to predict the noise. Building upon the established DDPM approach (Ho et al. 22020), one-step sampling process, as detailed in Fig. S Finally, the mask and edge map are decoded from the latent code using the VAE decoder and are post-processed by averaging the channels.
implement streamlined

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets and Metrics Similar to previous works (Yu et al. 2024 Kim et al. 2022 we conducted
training on the DIS5K training dataset, which consists of 3,000 images spanning 225 categories. Validation and testing were performed the DIS5K validation and test datasets, referred to as DIS-VD and DIS-TE, respectively. The DIS-TE dataset is further divided into four subsets (DIS- TEI, DIS-TE-TE DIS-TE3, DIS-TE4), each containing 500 images with progressively more complex morphological structures. Following MVANet (Yu et al. 2024), we employ a total five evaluation metrics, concentrating measuring the precision of the foreground areas as well as the intricacy of the structural details across the compared models, including max F-measure (Fgnar) (Perazzi et al.
2012), weighted F-measure (FF) (Margolin et al., 2014), structural similarity measure (Sm) (Fan al. 2017). E-measure (E") (Fan et al. 2018) and mean absolute error (MAE, M) (Perazzi al.
2012
Implementation Details Experiments were implemented in PyTorch and conducted on a single NVIDIA H800 GPU. During training. the original images were resized 1024 x 1024 for training. We use SD V (Rombach et al. (202)) as our backbone, and initialize the model with the parame- ters from SD-Turbo (Sauer et al., 2023). To mitigate the risk of overfitting in diffusion models when trained on a relatively small dataset, we apply several data augmentation techniques, including ran- dom horizontal flipping, cropping, rotation, and CutMix Yunn et al. 2019). For optimization, we use the Adam optimizer, setting the initial learning rate to 3 x 10 The batch size is configured as 4. The maximum number of training epochs was set to 90. During evaluation, we binarize the predicted maps to filter out minor noise for accuracy calculation.
5.2 COMPARISON
Quantitative Evaluation In this study, we benchmark our proposed DifIDIS against six DIS- only models, including IS-Net (Qin et al. 2022). FP-DIS (Zhou al. 2023), UDUN (Pei et al. 2023), InSPyReNet (Kim al., (022), BiRefNet (Zheng et al.) 2024, and MVANet (Yu et al. 2024 J. Additionally, we incorporate GenPercept (Xu et al. 2024), a diffusion-
model that has been experimentally applied to DIS. We also include four widely recognized segmentation mod- els: BSANet (Zhu et al.) 2022), ISDNet (Guo et al.) 2022). IFA (Hu et al., 2022). and PGNet (Xiel standardize the input size of the comparison models to 1024 x 1024. The results, as shown in Tab. 2 indicate that our method outperforms others and achieves state-of-the-att performance.
al., 2022). For fair comparison,
Qualitative Evaluation Fig. 6 presents qualitative comparison of our approach with previous ofate-of-the-art methods, highlighting our method's enhanced capability to perceive fine regions with greater clarity. As depicted in Fig. 6 our model adeptly captures precise object localization and edge details across a variety of complex scenes, showcasing its robust performance in high- accuracy segmentation tasks.
5.3 ABLATION
In this section, we analyze the effects of each component and evaluate the impact of various pre- trained parameters and denoising paradigms on experimental accuracy. All results are tested on the DIS-VD dataset.
Effectiveness of each component In Tab. 3. the first row represents the SD model using a single-step denoising paradigm as described in Eqn. 4 initialized with SD- Turbo pre-trained parameters. Subsequent rows represent the incremental addition auxiliary edge prediction, the DBIA mech- anism, the SWCI module, and CutMix data augmentation, respectively. The pro- gressive trend illustrates the utility of each module, demonstrating their collective contribution to the model's performance.
Table Ablation experiments of components.
Edge
DBIA
SWCI
CutMix
Froux
Em
S.,
.M1
0.882
0.930
0.039
0.891
0.934
0.888
0.036
0.896
0.943
0.895
0.032
0.897
0.945
0.899
0.030
0.908
0.948
0.904
0.029
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparison of DISSK with representative methods. represents the lower value better, while t represents the higher value is better. The best score is highlighted in bold, and the second is underlined.
Danasess
Metric
BSANee
ISDNet
IEA
PGNet
IS Net
JP DIS
UDUN
InSPyReNet
BiReeNet
MVANNe
GenPercept
Ours
0.738
0.763
0.749
0.798
0.791
0,823
0.825
0.859
0.397
0.904
0.844
0.908
DISVDD
P71 1
0,615
0.691
0.653
0.733
0.717
0.763
0.763
0.834
0.863
0.863
0.824
0.858
0,307
0.852
0.829
0.879
0.556
0.891
0.892
0.914
0.937
0.941
0.924
0.948
Sw.7
0.786
0.303
0.785
0.824
0.813
0.843
0.838
0.900
0.905
0.005
0.863
.M1
0.100
0,080
0.088
0.067
0.074
0,062
0.069
0.042
0.006
0033
0.044
0.029
0.683
0.717
0.673
0.754
0.740
0.784
0.784
0.845
0,365
0.322
0.807
0.883
DI3-727
0.545
0.643
0.573
0.690
0.662
0.713
0.720
0.788
0.829
0.823
0.781
0.862
0.773
0.824
0.785
0.848
0.520
0.860
0.854
0.874
0.917
0.911
0.889
0,933
S. 1
0.756
0.782
0.745
0.800
0.787
0.821
0.817
0.873
0.389
0.879
0.852
0,891
MM
0.008
0.067
0.074
0.060
0.059
0.043
0.016
0.043
0.030
0.752
0.783
0.755
0.807
0.799
0.827
0.829
0.894
0,906
0.910
0.849
0.917
DI3-72
0.608
0.714
0.666
0.743
0.728
0.067
0.768
0.846
0.826
0.874
0.827
0.895
0.815
0.365
0.895
0.690
0.858
0.893
0.686
0.916
0.943
0.944
0.922
0.951
S..7 7
0.796
0.817
0.793
0.833
0.823
0.845
0.843
0.905
0,913
0.915
0.869
0.913
M
0.008
0.072
0.065
0.070
0.059
0.058
0.036
0.043
0.026
0.817
0.797
0.849
0.830
0.868
0.865
0.919
0.930
0.029
0.862
0.934
DIS-73
0.660
0.747
0.705
0.785
0.758
0.811
0.809
0.871
0.388
0.890
0.839
0.916
0.540
0.393
0.361
0.911
0.883
0.922
0.917
0.940
0.951
0.954
0.935
0.964
5.
0.844
0.836
0.865
0.918
0.918
0.920
0.869
0.91
M
0.000
0.077
0.056
0.064
0.049
0.050
0.034
0.029
0.031
0.043
0.025
0.794
0.790
0.831
0.827
0.846
0.846
0.905
0.906
0.912
0.841
0.909
DMS-784
0.640
0.725
0.700
0.774
0.753
0.788
0.792
0.848
0.366
0.857
0.821
0,893
0.899
0.570
0.906
0.901
0.936
0.944
0.934
S.. T
0.811
0.830
0.852
0.849
0,965
0.902
0,903
0.849
0.896
M
0.107
0.079
0.065
0.072
0.059
0.042
0.088
0.041
0.049
0.032
Fg
0.744
0.778
0.755
0.809
0.799
0.831
0.831
0.891
0,900
0.908
0.840
0.911
Oheroil
DSS- 185-181141
Py-1
0.618
0,707
0.661
0.746
0.725
0.770
0.772
0.838
0.365
0.861
0.817
0.892
0,811
0,864
0.832
0.885
0.855
0
0.892
0.917
0:938
0.336
0.920
0.950
0,789
0,812
0.791
0.830
0.819
0.847
0.844
0.900
0,906
0.904
0.860
0.905
.
0.098
0.073
0.064
0.063
0,070
0.087
0.057
0.039
0.004
0.035
0.044
0,028
Table 4: Ablation experiments of the pre-trained parameters and denoising steps. The asterisk (*)
indicates that the timestep is not fixed during training.
Pre-trained Params
Train step
Infer step
Fymus f
ET f
Sm f
M 4
Inf. Time J
Train from seratch
0.682
0.758
0.687
0.103
0.33
0.883
0,930
0.880
0.039
0.33
SD-Turbo
1+
0.876
0.918
0.878
0.041
0.52
0.879
0.933
0.881
0.038
0.52
0.853
0.910
0,862
0.048
0.33
SDV2.1
0.837
0.888
0.831
0.052
1.40
Diverse Pre-trained Parameters and Denoising Steps As shown in Tab. 4 to further investigate the impact of SD's powerful pretrained prior on high-resolution DIS tasks and to explore the effects of various denoising paradigms, we experimented with the following variants based on the vanilla SD backbone. All experiments shared the same network architecture:
By comparing the results of the 1s, 2nd and 5th rows, which correspond to different pretrained pa- rameters loaded into the U-Net, it's evident that SD-Turbo outperforms others in one-step denoising. This superior performance can be attributed to its efficient, few-step image generation capabilities acquired through distillation. The 2nd 3rd and 4th rows of Table illustrate the varying impacts of different few-step denoising paradigms. Our approach, as mentioned in the paper, is reflected in the 2nd row. The 4th row represents a fixed two-step denoising paradigm with timesteps [999, 499] used for both training and testing, which, compared to the second row, shows slightly better accuracy but at the expense of doubled inference and training time. The 3rd row modifies the 4th by introduc- ing random timestep selection from (999, 499) during training, followed by one-step denoising process. During testing, a fixed two-step denoising procedure is employed, using the timesteps [999, 499]. While this approach reduces training time, it results in lower accuracy than the 2nd row. The performance decrease is probably because the model wasn't 't consistently trained for the specific two-step denoising needed at test time. The last two rows of Table 4 demonstrate the compara-
9

---

Published as a conference paper at ICLR 2025
121
Image
GT
Ours
GenPercept
MVANet
BiReiNet
InSPyReNet
ISNet
Figure 6: Visual comparison of different DIS methods.
tive efficacy between one-step denoising paradigms and traditional multi-step approaches (Ke al, 2024 Rombach al. (022) in the context of high-precision HR object segmentation. The final row, which employs a random timestep between 0 and 1000 to introduce noise into the mask latent during training, followed by ten-step denoising process during testing, similar to (Ke et al.) 20244, yields significantly lower results than one-step paradigms. This diminished performance is attributed to the randomness of multi-step denoising, where stochastic variations are progressively amplified each step. This leads to increased unpredictability and generative characteristies, conflicting with the deterministic nature of binary segmentation tasks that require unambiguous outputs.
Comparison of Different Interac- tion Methods in DBIA To compare the differences in cross-domain in-
Table 5: Ablation experiments of the interaction DBIA.
teraction methods between our ap- proach and (Fu et al. 2024), we re-
Edge
DBIA
DBIA
P
E
Sy f
M1
placed the interaction method in the additional cross-attention mechanism of DBIA with a fusion-orie fusion-oriented ap- proach similar to that used in (Fu et al. 2024). As shown in the 2nd line, targeted interaction is more likely to boost generation accuracy and enhance information exchange.
0.896
0.943
0.895
0.032
0.891
0.940
0.894
0.035
CONCLUSION
In this paper, we made the attempt to harness the exceptional performance and prior knowledge of diffusion architectures to transcend the limitations of traditional discriminative learning-bass based frameworks in HR, fine-grained object segmentation, aiming at generating detailed binary maps at high resolutions, while demonstrating impressive accuracy swift processing. Our approach used one-step denoising paradigm to generate detailed binary maps quickly and accurately. To handle the complexity and detail richness of DIS segmentation, we introduced additional edge constraints and upgraded the attention module to Detail-Balancin Interactive Attention, enhancing both detail clarity and the generative certainty of the diffusion model. We also incorporated multi-scale con- ditional injection into the U-Net, introducing multi- cratiiigraularttyy information for enhanced semantic and structural perception. Extensive experiments demonstrate DiffDIS's excellent performance on the DIS dataset.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China under Grant 62431004 and 62276046, and by Dalian Science and Technology Innovation Foundation under Grant 2023JJ12GX01..
REFERENCES
Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXi::2112.00390 2021.
Tianyou Chen, Jin Xiao, Xiaoguang Hu, Guofeng Zhang, and Shaojie Wang. Adaptive fusion net- work for rgb-d salient object detection. Neurocomputing, 522:152-1644 2023.
Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-measure: to evaluate foreground maps. In Proceedings of the IEEE international conference on computer vision, pp. 4548 4557, 2017.
measure: A new way
Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. Enhanced- alignment measure for binary foreground map evaluation. arXiv preprint arXiv: 10421, 2018.
Deng-Ping Fan, Ge-Peng Ji, Guolei Sun, Ming-Ming Cheng, Jianbing Shen, and Ling Shao. Cam ouflaged object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2777-2787, 2020.
Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. arXiv preprint arXiv:2403, /2013, 2024.
Stas Goferman, Lihi Zelnik- and Ayellet Tal. Context-aware saliency detection. IEEE trans- actions on pattern analysis and machine intelligence, 34(10):1915-1926, 2011.
Shaohua Guo, Liang Liu, Zhenye Gan, Yabiao Wang, Wuhao Zhang, Chengjie Wang, Guannan Jiang. Wei Zhang, Ran Yi, Lizhuang Ma, et al. Isdnet: Integrating shallow and deep networks for efficient ultra-high resolution segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4361-4370, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Hanzhe Hu, Yinbo Chen, Jiarui Xu, Shubhankar Borse, Hong Cai, Fatih Porikli, and Xiaolong Wang- Learning implicit feature alignment function for semantic segmentation. European Conference Computer Vision, pp. 487-505. Springer, 2022.
Xiaobin Hu, Xu Peng, Donghao Luo, Xiaozhong Ji, Jinlong Peng, Zhengkai Jiang, Jiangning Zhang, Taisong Jin, Chengjie Wang, and Rongrong Ji. Diffumatting: Synthesizing arbitrary objects with matting-level annotation. arXiv preprint arXiv:2403.066 2024.
Yihan Hu, Yiheng Lin, Wei Wang. Yao Zhao, Yunchao Wei, and Humphrey Shi. Diffusion for natural image matting, arXiv preprint 2rXiv:2312.05915,22 2023.
Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21741-2172 2023.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-basee image generators for monocular depth estimation. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp- 9492-9502, 2024.
11

---

Published as a conference paper at ICLR 2025
Taehun Kim, Kunhee Kim, Joonyeong Lee, Dongmin Cha, Jiho Lee, and Daijin Kim. Revisiting image pyramid structure for high resolution salient object detection. In Proceedings of the Asian Conference Computer Vision, pp. 108-124, 2022.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint urXiv: 2013.
Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, and Jiashi Feng. Deep interactive thin object selection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 305-314, 2021.
Feng Liu, Luan Tran, and Xiaoming Liu. Fully understanding generic objects: Modeling, segmen- tation, and reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 7423-7433, 2021.
Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 9970-9980, 2024.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprini urXiv:2310.04378. 2023.
Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How evaluate foreground maps? Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 248-255, 2014.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-1))) 106, 2021.
Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Multi-scale interactive network for salient object detection. In CVPR, pp. 9413-9422, 2020.
Youwei Pang, Xiaoqi Zhao, Tian-Zhu Xiang, Lihe Zhang, and Huchuan Lu. Zoom in and out: A mixed-scale triplet network for camouflaged object detection. In CVPR, pp. 2160-2170, 2022.
Youwei Pang. Xiaoqi Zhao, Tian-Zhu Xiang, Lihe Zhang, and Huchuan Lu. Zoomnext: A unified collaborative pyramid network for camouflaged object detection. IEEE TPAMI, 2024a.
Youwei Pang, Xiaoqi Zhao, Jiaming Zuo, Lihe Zhang, and Huchuan Lu. Open-vocabulary camou- flaged object segmentation. ECCV, 2024b.
Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, and Jun-Yan Yan Zhu. One-step image translation with text-to-image models. arXiv preprint arXiv:2403.1203: 2024.
Jialun Pei, Zhangjun Zhou, Yueming Jin, He Tang, and Pheng-An Ann Heng. boosting trunk and structure for aeghraccuracy dichotomous image segmentation. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 2139-2147, 2023.
divide-unite Joint
Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and Alexander Hornung- Saliency filters: Con- trast based filtering for salient region detection. 2012 IEEE conference on computer vision and pattern recognition, pp. 733-740. IEEE, 2012.
Xuebin Qin, Deng-Ping Fan, Chenyang Huang. Cyril Diagne, Zichen Zhang, Adrià Cabeza Sant' Anna, Albert Suarez, Martin Jagersand, and Ling Shao. Boundary- segmentation net- work for mobile and web applications. arXiv preprint 10Ii:2201100704,, 2021.
Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accu- rate dichotomous image segmentation. In European Conference on Computer Vision, pp. 38-56. Springer, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. arXiv preprint arXiv:2311.17442, 2023.
Yang Tian, Hualong Bai, Shengdong Zhao, Chi- Wing Fu, Chun Yu, Haozhao Qin, Qiong Wang. and Pheng-Ann Heng. Kine-appendage: Enhancing freehand vr interaction through transformations of virtual appendages. IEEE Transactions on Visualization and Computer Graphics, 2022.
Hefeng Wang, Jiale Cao, Rao Muhammad Anwer, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Dformer: Diffusion-guided transformer for universal image segmentation. arXiv preprint arXiv:2306.034377 2023.
Chenxi Xie, Changqun Xia, Mingcan Ma, Zhirui Zhao, Xiaowu Chen, and Jia Li. Pyramid graft- ing network for one-stage high resolution saliency detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 11717-11726, 2022.
Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. Diffusion models trained with large data are transferable visual models. arXiv preprint arXiv:2403. 060900 2024.
Chenglin Yang. Yilin Wang, Jianming Zhang, He Zhang, Zhe Lin, and Alan Yuille. Meticulous object segmentation. arXiv preprint arXiv:20/2.07181, 2020.
Yunfan Ye, Kai Xu, Yuhang Huang, Renjiao Yi, and Zhiping Cai. Diffusionedge: Diffusion prob- abilistic model for crisp edge detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6675-6683, 2024.
Qian Yu, Xiaoqi Zhao, Youwei Pang. Lihe Zhang, and Huchuan Lu. Multi-view aggregation network for dichotomous image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3921399133 -3390,,,,, 2024.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Denis Zavadski, Damjan Kalšan, and Carsten Rother. Primedepth: Efficient monocular depth esti- mation with stable diffusion preimage. arXiv preprint 2rXiv:2409.09144 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Xiang Zhang, Bingxin Ke, Hayko Riemenschneider, Nando Metzger, Anton Obukhov, Markus Gross, Konrad Schindler, and Christopher Schroers. Betterdepth: Plug-and-play diffusion refiner for zero-shot monocular depth estimation. arXiv preprint arXiv:2407. 109222777777..... 2024.
Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu, and Lei Zhang. Suppress and balance: A simple gated network for salient object detection. ECCV, pp. 35-51, 2020.
Xiaoqi Zhao, Youwei Pang, Lihe Zhang, and Huchuan Lu. Joint learning of salient object detection, depth estimation and contour extraction. IEEE 77P, 31:7350-7366 7362, 2022.
Xiaoqi Zhao, Youwei Pang, Shijie Chang, Yuan Zhao, Lihe Zhang, Huchuan Lu, Jinsong Ouyang, Georges El Fakhri, and Xiaofeng Liu. Inspiring the next generation of segment anything mod- els: Comprehensively evaluate sam and sam 2 with diverse prompts towards context-dependent concepts under different scenes. arXiv preprint arXiv:2412.0/241 2024a.
Xiaoqi Zhao, Youwei Pang, Wei Ji, Baicheng Sheng, Jiaming Zuo. Lihe Zhang, and Huchuan Lu. Spider: A unified framework for context-dependent concept understanding. In ICML, 2024b.
Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu, and Lei Zhang. Towards diverse binary segmentation via simple yet general gated network. IJCV, pp. 1-78, 2024c.
Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolution dichotomous image segmentation. arXiv preprint M.Xi::4011.03007, 2024.
13

---

Published as a conference paper at ICLR 2025
Yan Zhou, Bo Dong, Yuanfeng Wu, Wentao Zhu, Geng Chen, and Yanning Zhang. Dichotomous image segmentation with frequency priors. In ICCAI, volume 1, pp. 3, 2023.
Hongwei Zhu, Peng Li, Haoran Xie, Xuefeng Yan, Dong Liang, Dapeng Chen, Mingqiang Wei, and Jing Qin. can find you! boundary-guided separated attention network for camouflaged object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3608-3616, 2022.
14


Reviews:
[{"review_id": "0QbRDNdbDR", "reviewer": "Reviewer_N1et", "summary": "This paper proposes a model named DiffDIS for high-resolution binary image segmentation, incorporating strategies like single-step denoising, edge-assisted generation, and multi-scale conditional injection to enhance segmentation accuracy and inference speed. The authors validate DiffDIS’s performance on the DIS5K dataset, showing promising results. While the design is sound and experimental results are clearly presented, the paper’s novelty and certain implementation details could benefit from further clarification. In the rebuttal, the author addressed these concerns.", "questions": "please see the weakness part", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tMulti-level Design Innovations: The paper combines single-step denoising, edge-assisted generation, and multi-scale conditional injection to address the challenges of high-resolution segmentation, balancing speed and detail retention effectively.\n2.\tComprehensive Experiments: The experimental setup on the DIS5K dataset is thorough, with comparisons to multiple specialized and general segmentation models. Ablation studies illustrate each component’s contribution, supporting the rationale behind the model design.\n3.\tClarity: The paper is well-organized, with clear descriptions of the model and results, making it accessible to readers.", "weaknesses": "1.\tClarify Novelty of the Single-Step Denoising: While the single-step denoising strategy indeed boosts inference efficiency, a similar concept has been explored in models like GenPercept. I suggest that the authors clarify if DiffDIS’s single-step denoising incorporates task-specific optimizations for DIS tasks, to better highlight its originality.\n\t2.\tElaborate on the Edge-Assisted Generation’s Distinctiveness and Adaptation for High-Resolution Segmentation: The edge-assisted generation approach in DiffDIS appears similar to the edge-guided inpainting technique used in EdgeConnect, albeit applied in segmentation rather than inpainting. To avoid the impression that this is a simple adaptation from inpainting, I suggest the authors discuss any specific adjustments or optimizations made for high-resolution segmentation in DiffDIS.\n\t3.\tAdvantages of Joint Edge and Mask Prediction with Experimental Validation: DiffDIS performs joint edge and mask prediction, unlike stage-wise processing. Further discussion on the specific advantages of joint prediction, especially in handling fine details and complex boundaries, would strengthen this choice. Additionally, including experimental comparisons between joint prediction and stage-wise prediction would provide valuable insights into its effectiveness.\n\t4.\tComparative Analysis of Training Time: While DiffDIS’s inference time is shown to be efficient, comparative analysis of training times is absent. Including training time comparisons would provide a more holistic view of DiffDIS’s computational efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DnO1jOVSuC", "reviewer": "Reviewer_V4Fe", "summary": "The authors proposed a method for dichotomous segmentation using a Stable Diffusion prior, finding that introducing edges into the segmentation task can enhance performance. They introduced the BDE and DBIA modules, which can distinguish between different tasks and achieve better detail generation capabilities. The method efficiently utilizes one-step sampling and shows significant improvement over previous methods across multiple test projects.", "questions": "1. In emu-edit, they introduced a method called task embedding, it is a learnable embedding added to the time step embedding, to distinguish the different task in multi-task training. As your Batch-Discriminative Embedding contains more detailed design, do you have any performance comparison between your Batch-Discriminative Embedding and the task embedding?\n2. For the one step mask sampling, in 4.4 and the fig 5 only state that it is building upon the established DDPM, but no more detailed description and implemented the one step sampling, could you please provide more details on it?\n3. From the paper, it shows that the author used SD2.1 and SD turbo as initialization weights, and there is a channel-wise concat operation before feeding into the UNet, which changes the number of channels in the input layer. I would like to know how the pretrained weights are handled for the input layer when they are used (eg. duplicate/zero).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The author discovered that the introduction of edges can enhance the detail and performance of segmentation. They used batch discriminative embedding to distinguish between edges and segmentation. This is a novel method.\n2. The author provided detailed experiments that demonstrate the method's strong performance across multiple aspects, and also included an ablation study to prove the effectiveness of each module.", "weaknesses": "1. The description of the one step inference is not comprehensive enough, please see Q2\n2. For dichotomous segmentation, using an RGB 3-channel VAE to encode a single-channel segmentation mask might be a bit overkill. As an advancement in dichotomous segmentation, some earlier works have used diffusion models for matting, which also achieved very good results. However, considering that it can produce decent results in just one step, it is acceptable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64sNpkLFt4", "reviewer": "Reviewer_U4V6", "summary": "This paper address the problem of Dichotomous Image Segmentation (DIS) with a generative foundation model, StableDiffusion, by modifying with several key modules.", "questions": "1. Latent diffusion models such as SD would map the original image into low-dimensional latents e.g. 32x32x4 for 512x512x3 input. I do not understand well how the low dimensional latent is decoded to high-resolution mask.\n2. I would like to see if the proposed methods can be transferrable to other architectures, e.g. pixel-space diffusion UNets and latent DiT.\n\nThe followings seem typos or grammar issues, which do not affect my ratings:\n1. line 64, \"in balance receptive field expansion\" should be \"in balancing ...\"\n2. line 72, \"It’s power is\" should be \"Its power is\"\n3. line 363, \"conloution\" should be \"convolution\"\n4. line 530, \"attmpt\" should be \"attempt\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method reaches SOTA performance and beat other concurrent diffusion-based approach on DIS datasets.\n2. This is an early attempt that uses a pre-trained generative model for challenging DIS task.\n3. The method is efficient in comparison to the line of work that follows SegDiff, which runs diffusion process for more time-steps.\n4. The ablation studies include both quantitative numbers and qualitative visualizations, which are helpful for understanding how the whole framework is designed.", "weaknesses": "1. The application scenario seems limited. The task setting is only limited to Dichotomous Image Segmentation. It could be more convincing if the authors can also address the applicability of this approach in more settings, e.g. image matting, foreground object segmentation, edge detection.\n2. Running diffusion for one-step for segmentation is not a great contribution. This paper might miss some related work that is in the line of DDPM-Seg [1]. A lot of recent work that uses StableDiffusion for unsupervised semantic segmentation and open-vocabulary segmentation is indeed one-step in inference time.\n3. The methods are not novel. From the ablation studies, the most prominent modules are Batch-Discriminative Embedding and Detail-Balancing Interactive Attention (DBIA). However, Batch-Discriminative Embedding is proposed by previous work and this work is more of applying that module for DIS task. DBIA is a modified attention module but specifically designed for DIS.\n\n[1] Label-Efficient Semantic Segmentation with Diffusion Models. ICLR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d5aCABI4Nn", "reviewer": "Reviewer_8bJM", "summary": "This paper proposes a DiffDIS framework based on SD V2.1 to tackle the fine-grained dichotomous image segmentation task. The proposed DiffDIS finetunes the diffusion U-Net in the VAE-encoded latent space and introduces several modifications to enhance the edge perception, including:  1) The edge-assisted training strategy introduces batch-discriminative embedding to enable the mask and edge prediction in a single SD model and conducts interactive attention between the mask and edge branches, 2) Add to zero convolution to enhance the condition injection at different scales. The paper is easy to follow. The proposed method only uses single-step denoising to enhance efficiency and achieves SoTA performance on all the benchmarks.", "questions": "1. In Algorithm 1, the authors finetune diffusion U-Net by only considering t=T and directly optimizing with $x_0$  instead of noise, where the diffusion UNet seems to degenerate into a common UNet and is inconsistent with SD. Why did the authors not train the diffusion U-Net using the consistent objective with SD and deploy an efficient ODE solver to enable efficient inference?\n2. Considering the randomness of the noise sampling process, is the model sensitive to the sampled noise in the inference stage? It is suggested that an analysis of the performance be made using varied noise.\n3. Can this performance be improved by sending some text embedding given by the caption model instead of using the empty text embedding?\n4. What is the difference between the proposed Detail-Balancing Interactive Attention (Eqn.6) and the common cross-attention deployed between the mask and edge feature?\n5.  The multi-scale injector introduces condition signals into blocks. There lacks a comparison with the common condition/signal injection methods, such as the cross-attention in SD.\n6. Since VAE contains a large number of parameters, the authors should give a comparison in terms of the inference speed and parameters with state-of-the-art methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method is technically sound and streamlined, unleashing the dimension-reduction ability of VAE and the representation capacity of diffusion U-Net in the perceptional task.\n+ The paper is well-organized and easy to follow.\n+ The proposed method achieves superior performance gains on the benchmark datasets.\n+ Some experimental observations, such as the restorative capacity of VAE (Tab. 1) and the influence of time-step (Tab. 4) might be valuable to the community.", "weaknesses": "+ Some concerns about the technical contribution should be clarified: see Q4 and Q5.\n+ Some concerns about the robustness and model efficiency should be addressed: see Q2 and Q6\n+ Some other concerns about the methodology should be tackled: see Q1 and Q3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)