High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity

## Abstract

Abstract In the realm of high-resolution (HR), fine-grained image segmentation, the primary challenge is balancing broad contextual awareness with the precision required for detailed object delineation, capturing intricate details and the finest edges of objects.
Diffusion models, trained on vast datasets comprising billions of image-text pairs, such as SD V2.1, have revolutionized text-to-image synthesis by delivering exceptional quality, fine detail resolution, and strong contextual awareness, making them an attractive solution for high-resolution image segmentation.
To this end, we propose DiffDIS, a diffusion-driven segmentation model that taps into the potential of the pre-trained U-Net within diffusion models, specifically designed for high-resolution, fine-grained object segmentation.
By leveraging the robust generalization capabilities and rich, versatile image representation prior of the SD models, coupled with a task-specific stable one-step denoising approach, we significantly reduce the inference time while preserving high-fidelity, detailed generation.
Additionally, we introduce an auxiliary edge generation task to not only enhance the preservation of fine details of the object boundaries, but reconcile the probabilistic nature of diffusion with the deterministic demands of segmentation.
With these refined strategies in place, DiffDIS serves as a rapid object mask generation model, specifically optimized for generating detailed binary maps at high resolutions, while demonstrating impressive accuracy and swift processing.
Experiments on the DIS5K dataset demonstrate the superiority of DiffDIS, achieving state-of-the-art results through a streamlined inference process.
Our code will be made publicly available.

## 1 Introduction

High-accuracy dichotomous image segmentation (DIS) (Qin et al., ) aims to accurately identify category-agnostic foreground objects within natural scenes, which is fundamental for a wide range of scene understanding applications, including AR/VR applications (Tian et al., ; Qin et al., ), image editing (Goferman et al., ), and 3D shape reconstruction (Liu et al., ).
Different from existing segmentation tasks, DIS focuses on challenging high-resolution (HR) fine-grained object segmentation, regardless of their characteristics.
These objects often encompass a greater volume of information and exhibit richer detail, thereby demanding more refined feature selection and more sophisticated algorithms for segmentation.
Current CNN-(Pei et al., ; Qin et al., ) and Transformer-based (Xie et al., ; Kim et al., ; Yu et al., ) methods, despite their robust feature extraction capabilities, often face challenges in balance receptive field expansion with detail preservation in high-resolution images (Xie et al., ; Yu et al., ).

Diffusion probabilistic models (DPMs), by predicting noise variables across the entire image, have been demonstrated by numerous studies (Ji et al., ; Wang et al., ) to show promise in maintaining a global receptive field while more accurately learning the target distribution.
Moreover, Stable Diffusion (SD) (Rombach et al., ) has emerged as a significant leap forward in the domain of DPMs. Trained on vast datasets comprising billions of images, it exhibits robust generalization capabilities and offers a rich, versatile image representation, positioning it as an ideal candidate for tasks demanding both macroscopic context and microscopic precision.
It’s power is further underscored by its significant contributions to fine-grained feature extraction, as evidenced by recent studies (Ke et al., ; Zhang et al., ; Zavadski et al., ; Hu et al., ; Xu et al., ) that have harnessed the SD’s capabilities to capture the subtleties of detail.
The advancements in these methods inspired us that diffusion could be a powerful tool for improving the accuracy and robustness of high-resolution image segmentation.

However, there are several challenges when leveraging the diffusion model to DIS:
On the one hand, for diffusion models: (1) The inherent high-dimensional, continuous feature space of DPM conflicts with the discrete nature of binary segmentation, potentially leading to a discrepancy in the predictive process. (2) Moreover, the diffusion models often suffer from lengthy inference time. Due to the recurrent nature of diffusion models, it usually takes more than 100 steps for DPM to generate satisfying results (Ke et al., ), which further exacerbates the already slow inference speeds of HR images due to their substantial data volume. (3) There is a fundamental conflict between the stochastic nature of diffusion and the deterministic outcomes required for image perception tasks. On the other hand, for the DIS task: As image resolution increases, so does the intricacy of the visual information and the corresponding computational demands, necessitating more refined feature selection.

**Table 1: The restorative capability of VAE.**
| Metric | $F_{\beta}^{max}\uparrow$ | $E^{m}_{\phi}\uparrow$ | $S_{m}\uparrow$ | $\mathcal{M}$$\downarrow$ |
| --- | --- | --- | --- | --- |
| VAE | 0.993 | 0.999 | 0.985 | 0.002 |

Against this backdrop, we propose DiffDIS leveraging the following strategies:
First, we found that the Variational Autoencoder (VAE) (Kingma, ) has demonstrated the capability to nearly achieve perfect reconstruction of binary masks (See Tab. 1) .
Therefore, following SD (Rombach et al., ), by mapping the masks through the VAE into the latent space, we can not only effectively leverage the strengths of diffusion models for denoising and refinement within it, but also significantly reduces the computational cost associated with processing high-accuracy HR image segmentation.
Then, by employing a direct one-step denoising paradigm (See Eqn. 3) and integrating the pretrained parameters of SD-Turbo (Sauer et al., ; Parmar et al., ), which feature a smoother probability curve compared to the standard SD V2.1 parameters, we aim to streamline the network into an end-to-end model.
This approach facilitates one-step denoising while leveraging its robust generalization capabilities and preserving the intricate details of high-resolution objects.
Moreover,
we introduce a joint predicting strategy for mask and edge, where the edge task assists in the training phase.
This not only enhances the controllability of fine details but mitigates the misalignment between the generative nature of diffusion models and the determinism required in perception tasks.
Finally, we utilize RGB latent representations as conditions for the diffusion model and introduce Scale-Wise Conditional Injection to establish multi-granular, long-range, and profound visual interactions.

In this paper, we successfully trained a one-step SD model that generates high-resolution, high-quality results in DIS, preserving fine details while accurately segmenting complex structures (See Fig. 1) .
Generally, our main contributions can be summarized as follows:

- •
We propose DiffDIS, leveraging the powerful prior of diffusion models for the DIS task, elegantly navigating the traditional struggle to effectively balance the trade-off between receptive field expansion and detail preservation in traditional discriminative learning-based methods.
- •
We transform the recurrent nature of diffusion models into an end-to-end framework by implementing straightforward one-step denoising, significantly accelerating the inference speed.
- •
We introduce an auxiliary edge generation task, complemented by an effective interactive module, to achieve a nuanced balance in detail representation while also enhancing the determinism of the generated masks.
- •
We significantly advance the field forward by outperforming almost all metrics on the DIS benchmark dataset, and thus establish a new SoTA in this space. Additionally, it boasts an inference speed that is orders of magnitude faster than traditional multi-step diffusion approaches, without compromising accuracy, demonstrating the superiority of DiffDIS.

## 2 Related works

### 2.1 Conventional Approach to Deal With DIS Works

Dichotomous Image Segmentation (DIS) is formulated as a category-agnostic task that focuses on accurately segmenting objects with varying structural complexities, independent of their specific characteristics.
It includes high-resolution images of camouflaged (Fan et al., ; Le et al., ),
salient (Pang et al., ; Zhao et al., ),
and meticulous (Liew et al., ; Yang et al., ) objects in various backgrounds.
What distinguishes DIS from conventional segmentation tasks is the demand for highly precise object delineation, including the finest internal details of objects.
Upon introducing the DIS dataset, Qin et al. also presented IS-Net (Qin et al., ), a model specifically crafted to address the DIS challenge, utilizing the U^2Net and intermediate supervision to mitigate overfitting risks.
PF-DIS (Zhou et al., ) utilized a frequency prior generator and feature harmonization module to identify fine-grained object boundaries in DIS.
UDUN (Pei et al., ) proposed a unite-divide-unite scheme to disentangle the trunk and structure segmentation for high-accuracy DIS.
InSPyReNet (Kim et al., ) was constructed to generate HR outputs with a multi-resolution pyramid blending at the testing stage.
Recent approaches have incorporated multi-granularity cues that harness both global and local information to enhance detail fidelity and object localization accuracy.
BiRefNet (Zheng et al., ) employed a bilateral reference strategy, leveraging patches of the original images at their native scales as internal references and harnessing gradient priors as external references.
Recently, MVANet (Yu et al., ) modeled the DIS task as a multi-view object perception problem, leveraging the complementary localization and refinement among different views to process HR, fine-detail images.
Despite their commendable performance, existing methods have not effectively balanced the semantic dispersion of HR targets within a limited receptive field with the loss of high-precision details associated with a larger receptive field when tackling DIS.
Compared to conventional approaches, our utilization of a diffusion architecture has excelled in achieving high-quality background removal, with fast processing times (0.3-0.4s), and offers a straightforward integration.

### 2.2 Diffusion Models for Dense Prediction and Efficient Inference in Diffusion

With the recent success of diffusion models in generation tasks, there has been a noticeable rise in interest in incorporating them into dense visual prediction tasks. Several pioneering works attempted to apply the diffusion model to visual perception tasks, e.g.image segmentation (Amit et al., ; Ji et al., ; Wang et al., ), matting (Hu et al., ; ),
depth estimation (Ke et al., ; Zhang et al., ; Zavadski et al., ),
edge detection (Ye et al., ) et al..
Since the pioneering work (Amit et al., ) introduced diffusion methods to solve image segmentation, several follow-ups use diffusion to attempt their respective tasks.
(Ji et al., ) formulated the dense visual prediction tasks as a general conditional denoising process.
(Hu et al., ) pioneered the use of diffusion in matting, decoupling encoder and decoder to stabilize performance with uniform time intervals.
(Hu et al., ) ingeniously trained the model to paint on a fixed pure green screen backdrop.
(Ye et al., ) utilized a decoupled architecture for faster denoising and an adaptive Fourier filter to adjust latent features at specific frequencies.
In the depth estimation field, several recent works have leveraged diffusion for high-fidelity, fine-grained generation.
(Ke et al., ) introduced a latent diffusion model based on SD (Rombach et al., ), with fine-tuning for depth estimation, achieving strong performance on natural images.
(Zavadski et al., ) extracts a rich, frozen image representation from SD, termed preimage, which is then refined for downstream tasks.
Inspired by their work, we observe that diffusion-generative methods naturally excel at modeling complex data distributions and generating realistic texture details.

However, traditional multi-step generation models encounter a challenge: the recurrent structure of diffusion models often necessitates over 100 steps to produce satisfactory outputs. In response, various proposals have emerged to expedite the inference process.
Distillation-based methods have demonstrated remarkable speedups by optimizing the original diffusion model’s weights with enhanced schedulers or architectures.
(Luo et al., ) achieves a few-step inference in conditional image generation through self-consistency enforcement.
SD-Turbo (Sauer et al., ) employs Adversarial Diffusion Distillation (ADD), utilizing a pre-trained SD model to denoise images and calculate adversarial and distillation losses, facilitating rapid, high-quality generation.
Recently, GenPercept (Xu et al., ) presents a novel perspective on the diffusion process as an interpolation between RGB images and perceptual targets, effectively harnessing the pre-trained U-Net for various downstream image understanding tasks.
Given that SD-Turbo refines both high-frequency and low-frequency information through distillation (Sauer et al., ), by harnessing its efficient prior for image generation, we can sustain competitive performance in a single-step denoising scenario while achieving high-fidelity, fine-grained mask generation.

## 3 Preliminaries

Diffusion probabilistic models (Ho et al., ) have emerged as highly successful approaches for generating images by modeling the inverse process of a diffusion process from Gaussian noise.
It defines a Markovian chain of diffusion forward process $q(x_{t}|x_{0})$ by gradually adding noise to input data $x_{0}$ :

$$ $\displaystyle x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}} \epsilon,\quad\epsilon\sim\mathcal{N}(0,I),$ (1) $$

where $\epsilon$ is a pure Gaussian noise map. As $t$ increases, $\bar{\alpha}_{t}$ gradually decreases, leading $x_{t}$ to approximate the Gaussian noise.
By predicting the noise, the loss can be written as:

$$ $\displaystyle\mathcal{L}\left(\epsilon_{\theta}\right)=\sum_{t=1}^{T}\mathbb{E }_{x_{0}\sim q\left(x_{0}\right),\epsilon\sim\mathcal{N}(0,I)}\left[\left\| \epsilon_{\theta}\left(\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}} \epsilon\right)-\epsilon\right\|_{2}^{2}\right],$ (2) $$

During the reverse process in DDPM (Ho et al., ), given a random sampled Gaussian noise $x_{T}\sim\mathcal{N}(0,I)$, we utilize a Gaussian distribution $p_{\theta}(\mathbf{x}_{t+1}\mid\mathbf{x}_{t})$ to approximate the true posterior distribution, whose mean is estimated by the neural network, and the variance, denoted as $\sigma_{t}^{2}$, is derived from the noise schedule. To this end, we can repeat the following denoising process for $t\in\{T,T-1,\ldots,1\}$ to predict final denoised result $x_{0}$.

$$ $\displaystyle x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t }}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}\left(x_{t},t\right)\right)+ \sigma_{t}\epsilon,$ (3) $$

In the context of our method, during the training phase, instead of randomly selecting $t$, we initialize $t$ to its maximum value $T$, i.e., $999$. This initialization enables the network to directly learn the probability distribution for a single-step denoising process.
From Eqn. 3, the one-step denoising formula can be derived as follows. Since $\sigma_{T}$ is relatively small when $T=999$, its impact on the $x_{0}$ can be considered negligible. The formula is given by:

$$ $\displaystyle\hat{x}_{0}=\mathtt{Denoise}\left(\epsilon_{\theta}(x_{T},T,cond, d_{lab}),\epsilon\right)=\frac{x_{T}-\sqrt{1-\bar{\alpha}_{T}}\epsilon_{\theta }(x_{T},T)}{\sqrt{\bar{\alpha}_{T}}},$ (4) $$

where $cond$ represents the conditional input, specifically the image latent, while $d_{lab}$ denotes discriminative labels used to generate batch-discriminative embeddings (See Sec. 4.2 ) . Accordingly, the training objective of our method can be reformulated as minimizing the expected squared error between the denoised output and the ground truth latent $x_{0}$, as expressed by:

$$ $\displaystyle\min_{\theta}\mathbb{E}_{t,\epsilon,cond}\left\|\hat{x}_{0}-x_{0} \right\|_{2}^{2},$ (5) $$

Figure: Figure 2: Overall framework of DiffDIS. To start with, the inputs are encoded into the latent space. We concatenate the noisy mask and its corresponding edge latent along the batch dimension, utilizing batch-discriminative embedding for differentiation. Then, we employ the RGB latent as a conditioning factor through channel-wise concatenation, along with multi-scale conditional injection into the U-Net encoder layers. The noise obtained is processed through a direct single-step denoising approach to yield a clean latent prediction.
Refer to caption: x2.png

## 4 Method

### 4.1 Overall Architecture

The workflow of DiffDIS is outlined in Alg. 1:

First, the inputs are encoded into the VAE’s latent space, capturing the essential features in a continuous, high-dimensional representation.
Then, a denoising model is applied to mitigate noise, aiming for the best denoising effects to refine the representations and enhance the clarity of the features.
We utilize the RGB latent as a conditioning factor to assist in generating more authentic structural details for the mask.
Meanwhile, we improve the information flow by multi-tasking to further boost the accuracy.
We concatenate the mask and edge on a batch basis and apply Batch Discriminative Embedding (Fu et al., ) to obtain distinctive embeddings for each. These embeddings are then fed into the denoising network together to help distinguish between mask and edge.

Figure: Algorithm 1 Training Process

Within the U-Net architecture, specifically in the mid-block where semantic information is most concentrated, we strategically integrate an additional attention module called Detail-Balancing Interactive Attention.
This module not only enhances the determinism of the generative model’s predictions (See Fig. 4) but also facilitates a concise yet potent interaction between the high-dimensional noisy features of masks and edges, as detailed in Section 4.2.
Upon acquiring the noise estimated from the U-Net, we proceed with a direct one-step denoising transition from the high-step noisy latents, in accordance with the methodology outlined in Eqn. 4.
We subsequently utilize the initial mask and edge latent codes obtained from the VAE for direct supervision of the denoised mask and edge latent codes.
For an illustrative training diagram, refer to Fig. 2.

Figure: Figure 3: The structure of Batch-Discriminative Embedding (BDE) and the Detail-Balancing Interactive Attention (DBIA) .
Refer to caption: x3.png

Figure: Figure 4: Visualizing the reduction of diffusion process stochasticity through edge integration. With the addition of the edge auxiliary prediction task, the controllability and alignment of the generated mask have been enhanced, particularly in the areas highlighted by red boxes. The introduction of DBIA improves the sharpness and quality of the generated details.
Refer to caption: x4.png

### 4.2 Edge-Assisted Training Strategy

In high-resolution tasks, the complexity and detail richness make it tough to capture fine features. Furthermore, for diffusion-based architectures, accurately depicting detailed features during few-step generation poses a challenge, especially as details may be overlooked during denoising in the latent space, which is often reduced to $\frac{1}{8}$ the size of the input.
Some studies (Pei et al., ; Chen et al., ; Cheng et al., ) have indicated that incorporating additional edge constraints improves the boundary segmentation performance of masks.
Given these considerations, we propose an integrated and streamlined network architecture that concurrently predicts noise for both the mask and edge, leveraging auxiliary edge information to constrain the network.
These dual prediction streams operate within the same network structure and share parameters, with batch discriminative embedding applied to distinguish between them effectively.
Within the U-Net architecture, specifically in the mid-block where semantic information is most concentrated, we incorporate a task-specific enhancement by upgrading the original attention module to a Detail-Balancing Interactive Attention.
This mechanism aligns the attention regions of both the mask and edge streams, facilitating more efficient interaction and complementarity between the two.

Batch-Discriminative Embedding
Inspired by (Fu et al., ; Long et al., ), we incorporate additional discriminative labels , i.e., $d_{lab}$, within batches to enable a single stable diffusion model to generate multiple types of outputs simultaneously, ensuring seamless domain processing without mutual interference.
Specifically, the $d_{lab}$ is first represented in binary form and then encoded using positional encoding (Mildenhall et al., ).
After passing through a learnable projection head, the resulting batch-discriminative embeddings are combined with the time embeddings through an element-wise addition.
The combined embeddings are then fed into the ResBlocks, enhancing the model’s capacity to produce batch-specific outputs.
The process is illustrated in Fig. 3

Detail-Balancing Interactive Attention
To ensure the continuity and alignment of mask and edge, as well as to harmonize their semantic cues and attentioned areas, we introduce Detail-Balancing Interactive Attention (DBIA) , which includes, similar to traditional architectures, a self-attention module and a vanilla cross-attention module, along with our newly designed Mutual Cross-Domain Attention mechanism.
DBIA is designed to facilitate a simple yet effective exchange of information between the two domains, with the ultimate goal of generating outputs that are well-aligned in terms of both edge details and semantic content.

Specifically, we initialize the Mutual Cross-Domain Attention module with the parameters of the self-attention module, and project the mask and edge features to produce the query ($\mathbf{Q}_{m}$), key ($\mathbf{K}_{m}$), and value ($\mathbf{V}_{m}$) matrices for the mask, and correspondingly for the edge ($\mathbf{Q}_{e}$, $\mathbf{K}_{e}$, $\mathbf{V}_{e}$), then, we perform mutual scaled dot-product attention between these two sets, utilizing the query features to effectively query the corresponding key and value features from the alternative domain.
The formulation is as follows:

$$ $\displaystyle\text{Attention}_{m}$ $\displaystyle=\text{softmax}\left(\frac{\mathbf{Q}_{m}\mathbf{K}_{e}^{T}}{ \sqrt{d_{k}}}\right)\mathbf{V}_{e},$ $\displaystyle\text{Attention}_{e}$ $\displaystyle=\text{softmax}\left(\frac{\mathbf{Q}_{e}\mathbf{K}_{m}^{T}}{ \sqrt{d_{k}}}\right)\mathbf{V}_{m}$ (6) $$

This method ensures a direct and efficient exchange of information between two domains, fostering a robust interaction that enhances the overall feature representation.

Our method differs from the approach of (Fu et al., ) in how we handle feature interaction.
In their work, the keys ($\mathbf{K}$) and values ($\mathbf{V}$) from different domains are concatenated into a unified representation, and cross-attention is performed by applying the queries ($\mathbf{Q}$) from each domain to the concatenated $\mathbf{K}$ and $\mathbf{V}$.
Our method surpasses theirs by enabling direct, targeted attention operations that facilitate a more nuanced interaction between mask and edge features, leading to better discrimination and alignment without the dilution of information (See in Tab. 5) .

This approach has been observed to not only enhance the fine detail rendering but also, to our delight, significantly reduce the stochastic tendencies inherent in diffusion processes ( See in Fig. 4) .
Consequently, through in-depth cross-domain interaction and feature complementation, it not only effectively bridges the gap between the generative potential and the determinism essential for precise segmentation tasks but also yields more refined details in the predicted results.

### 4.3 Scale-Wise Conditional Injection

In traditional diffusion models for dense prediction tasks, conditioning is typically done at the input stage by concatenating along the channel dimension with the variable to be denoised (Ke et al., ; Fu et al., ). This approach can lead to information loss in later stages. To establish long-range and profound conditional guidance, we introduce Scale-Wise Conditional Injection to create multi-granular, long-range, and deep visual interactions.
Specifically, we incorporate multi-scale conditions into the corresponding layers of the U-Net encoder (See Fig. 3) . Specifically, we employ three injector heads, each composed of a simple convolutional layer and two zero conloution layers (Zhang et al., ) (See Fig. 3). Each injector head receives conditional latent code that is resized to corresponding scales as input.
The resulting outputs are then integrated at the junction of the last three layers in the U-Net encoder, facilitating the generation of more authentic structural details.
In this way, we introduce multi-granularity information for semantic and structural perception, without causing excessive interference with RGB texture.

Figure: Algorithm 2 Sampling Process
Refer to caption: x5.png

### 4.4 One-Step Mask Sampling

A depiction of the sampling pipeline can be seen in Fig. 5.
During inference, we first encode the RGB image into the latent space using a VAE encoder.
Next, we sample the starting variable from standard Gaussian noise, which serves as the initialization for both the mask and edge in the latent space.
Similar to the training phase, these two components are concatenated in a batch, with batch-discriminative embedding applied to effectively distinguish between them.
The concatenated components, conditioned on the RGB latent representation, are then fed into the U-Net to predict the noise.
Building upon the established DDPM approach (Ho et al., ), we implement a streamlined one-step sampling process, as detailed in Fig. 5.
Finally, the mask and edge map are decoded from the latent code using the VAE decoder and are post-processed by averaging the channels.

## 5 Experiments

### 5.1 Experimental Settings

Datasets and Metrics
Similar to previous works (Yu et al., ; Kim et al., ), we conducted training on the DIS5K training dataset, which consists of 3,000 images spanning 225 categories.
Validation and testing were performed on the DIS5K validation and test datasets, referred to as DIS-VD and DIS-TE, respectively.
The DIS-TE dataset is further divided into four subsets (DIS-TE1, DIS-TE2, DIS-TE3, DIS-TE4), each containing 500 images with progressively more complex morphological structures.

Following MVANet (Yu et al., ), we employ a total of five evaluation metrics, concentrating on measuring the precision of the foreground areas as well as the intricacy of the structural details across the compared models, including max F-measure ($F_{\beta}^{max}$) (Perazzi et al., ),
weighted F-measure ($F_{\beta}^{\omega}$) (Margolin et al., ),
structural similarity measure ($S_{m}$) (Fan et al., ),
E-measure ($E^{m}_{\phi}$) (Fan et al., )
and mean absolute error (MAE, $\mathcal{M}$) (Perazzi et al., ).

Implementation Details Experiments were implemented in PyTorch and conducted on a single NVIDIA H800 GPU.
During training, the original images were resized to $1024\times 1024$ for training. We use SD V2.1 (Rombach et al., ) as the backbone, and initialize the model with the parameters from SD-Turbo (Sauer et al., ).
To mitigate the risk of overfitting in diffusion models when trained on a relatively small dataset, we apply several data augmentation techniques, including random horizontal flipping, cropping, rotation, and CutMix (Yun et al., ).
For optimization, we use the Adam optimizer, setting the initial learning rate to $3\times 10^{-5}$. The batch size is configured as $2$.
The maximum number of training epochs was set to $90$.
During evaluation, we binarize the predicted maps for accuracy calculation.

### 5.2 Comparison

Quantitative Evaluation
In this study, we benchmark our proposed DiffDIS against six DIS-only models, including IS-Net (Qin et al., ), FP-DIS (Zhou et al., ), UDUN (Pei et al., ), InSPyReNet (Kim et al., ), BiRefNet (Zheng et al., ), and MVANet (Yu et al., ).
Additionally, we incorporate GenPercept (Xu et al., ), a diffusion-based model that has been experimentally applied to DIS.
We also include four widely recognized segmentation models: BSANet (Zhu et al., ), ISDNet (Guo et al., ), IFA (Hu et al., ), and PGNet (Xie et al., ).
For a fair comparison, we standardize the input size of the comparison
models to $1024\times 1024$.
The results, as shown in Tab. 2, indicate that our method outperforms others and achieves state-of-the-art performance.

**Table 2: Quantitative comparison of DIS5K with 11 representative methods. $\downarrow$ represents the lower value is better, while $\uparrow$ represents the higher value is better. The best score is highlighted in bold, and the second is underlined.**
| *Datasets* | *Metric* | BSANet | ISDNet | IFA | PGNet | IS-Net | FP-DIS | UDUN | InSPyReNet | BiRefNet | MVANet | GenPercept | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| *<br>DIS-VD<br>* | $F_{\beta}^{max}$ $\uparrow$ | 0.738 | 0.763 | 0.749 | 0.798 | 0.791 | 0.823 | 0.823 | 0.889 | 0.897 | 0.913 | 0.844 | 0.918 |
| $F^{\omega}_{\beta}$ $\uparrow$ | 0.615 | 0.691 | 0.653 | 0.733 | 0.717 | 0.763 | 0.763 | 0.834 | 0.863 | 0.856 | 0.824 | 0.888 |  |
| $E^{m}_{\phi}$ $\uparrow$ | 0.807 | 0.852 | 0.829 | 0.879 | 0.856 | 0.891 | 0.892 | 0.914 | 0.937 | 0.938 | 0.924 | 0.948 |  |
| $S_{m}$ $\uparrow$ | 0.786 | 0.803 | 0.785 | 0.824 | 0.813 | 0.843 | 0.838 | 0.900 | 0.905 | 0.905 | 0.863 | 0.904 |  |
| $\mathcal{M}$ $\downarrow$ | 0.100 | 0.080 | 0.088 | 0.067 | 0.074 | 0.062 | 0.059 | 0.042 | 0.036 | 0.036 | 0.044 | 0.029 |  |
| *<br>DIS-TE1<br>* | $F_{\beta}^{max}$ $\uparrow$ | 0.683 | 0.717 | 0.673 | 0.754 | 0.740 | 0.784 | 0.784 | 0.845 | 0.866 | 0.893 | 0.807 | 0.903 |
| $F^{\omega}_{\beta}$ $\uparrow$ | 0.545 | 0.643 | 0.573 | 0.680 | 0.662 | 0.713 | 0.720 | 0.788 | 0.829 | 0.823 | 0.781 | 0.862 |  |
| $E^{m}_{\phi}$ $\uparrow$ | 0.773 | 0.824 | 0.785 | 0.848 | 0.820 | 0.860 | 0.864 | 0.874 | 0.917 | 0.911 | 0.889 | 0.933 |  |
| $S_{m}$ $\uparrow$ | 0.754 | 0.782 | 0.746 | 0.800 | 0.787 | 0.821 | 0.817 | 0.873 | 0.889 | 0.879 | 0.852 | 0.891 |  |
| $\mathcal{M}$ $\downarrow$ | 0.098 | 0.077 | 0.088 | 0.067 | 0.074 | 0.060 | 0.059 | 0.043 | 0.036 | 0.037 | 0.043 | 0.030 |  |
| *<br>DIS-TE2<br>* | $F_{\beta}^{max}$ $\uparrow$ | 0.752 | 0.783 | 0.758 | 0.807 | 0.799 | 0.827 | 0.829 | 0.894 | 0.906 | 0.925 | 0.849 | 0.927 |
| $F^{\omega}_{\beta}$ $\uparrow$ | 0.628 | 0.714 | 0.666 | 0.743 | 0.728 | 0.767 | 0.768 | 0.846 | 0.876 | 0.874 | 0.827 | 0.895 |  |
| $E^{m}_{\phi}$ $\uparrow$ | 0.815 | 0.865 | 0.835 | 0.880 | 0.858 | 0.893 | 0.886 | 0.916 | 0.943 | 0.944 | 0.922 | 0.951 |  |
| $S_{m}$ $\uparrow$ | 0.794 | 0.817 | 0.793 | 0.833 | 0.823 | 0.845 | 0.843 | 0.905 | 0.913 | 0.915 | 0.869 | 0.913 |  |
| $\mathcal{M}$ $\downarrow$ | 0.098 | 0.072 | 0.085 | 0.065 | 0.070 | 0.059 | 0.058 | 0.036 | 0.031 | 0.030 | 0.042 | 0.026 |  |
| *<br>DIS-TE3<br>* | $F_{\beta}^{max}$ $\uparrow$ | 0.783 | 0.817 | 0.797 | 0.843 | 0.830 | 0.868 | 0.865 | 0.919 | 0.920 | 0.936 | 0.862 | 0.938 |
| $F^{\omega}_{\beta}$ $\uparrow$ | 0.660 | 0.747 | 0.705 | 0.785 | 0.758 | 0.811 | 0.809 | 0.871 | 0.888 | 0.890 | 0.839 | 0.916 |  |
| $E^{m}_{\phi}$ $\uparrow$ | 0.840 | 0.893 | 0.861 | 0.911 | 0.883 | 0.922 | 0.917 | 0.940 | 0.951 | 0.954 | 0.935 | 0.964 |  |
| $S_{m}$ $\uparrow$ | 0.814 | 0.834 | 0.815 | 0.844 | 0.836 | 0.871 | 0.865 | 0.918 | 0.918 | 0.920 | 0.869 | 0.919 |  |
| $\mathcal{M}$ $\downarrow$ | 0.090 | 0.065 | 0.077 | 0.056 | 0.064 | 0.049 | 0.050 | 0.034 | 0.029 | 0.031 | 0.042 | 0.025 |  |
| *<br>DIS-TE4<br>* | $F_{\beta}^{max}$ $\uparrow$ | 0.757 | 0.794 | 0.790 | 0.831 | 0.827 | 0.846 | 0.846 | 0.905 | 0.906 | 0.911 | 0.841 | 0.916 |
| $F^{\omega}_{\beta}$ $\uparrow$ | 0.640 | 0.725 | 0.700 | 0.774 | 0.753 | 0.788 | 0.792 | 0.848 | 0.866 | 0.857 | 0.823 | 0.893 |  |
| $E^{m}_{\phi}$ $\uparrow$ | 0.815 | 0.873 | 0.847 | 0.899 | 0.870 | 0.906 | 0.901 | 0.936 | 0.940 | 0.944 | 0.934 | 0.955 |  |
| $S_{m}$ $\uparrow$ | 0.794 | 0.815 | 0.841 | 0.811 | 0.830 | 0.852 | 0.849 | 0.905 | 0.902 | 0.903 | 0.849 | 0.896 |  |
| $\mathcal{M}$ $\downarrow$ | 0.107 | 0.079 | 0.085 | 0.065 | 0.072 | 0.061 | 0.059 | 0.042 | 0.038 | 0.041 | 0.049 | 0.032 |  |
| *<br>Overall<br>* | $F_{\beta}^{max}$ $\uparrow$ | 0.744 | 0.778 | 0.755 | 0.809 | 0.799 | 0.831 | 0.831 | 0.891 | 0.900 | 0.916 | 0.840 | 0.921 |
| $F^{\omega}_{\beta}$ $\uparrow$ | 0.618 | 0.707 | 0.661 | 0.746 | 0.726 | 0.770 | 0.772 | 0.838 | 0.865 | 0.855 | 0.817 | 0.892 |  |
| $E^{m}_{\phi}$ $\uparrow$ | 0.811 | 0.864 | 0.832 | 0.885 | 0.858 | 0.895 | 0.892 | 0.917 | 0.938 | 0.938 | 0.920 | 0.947 |  |
| $S_{m}$ $\uparrow$ | 0.789 | 0.812 | 0.791 | 0.830 | 0.819 | 0.847 | 0.844 | 0.900 | 0.906 | 0.905 | 0.860 | 0.905 |  |
| $\mathcal{M}$ $\downarrow$ | 0.098 | 0.073 | 0.084 | 0.063 | 0.070 | 0.057 | 0.057 | 0.039 | 0.034 | 0.035 | 0.044 | 0.028 |  |

Qualitative Evaluation
Fig. 6 presents a qualitative comparison of our approach with previous state-of-the-art methods, highlighting our method’s enhanced capability to perceive fine regions with greater clarity.
As depicted in Fig. 6, our model adeptly captures precise object localization and edge details across a variety of complex scenes, showcasing its robust performance in high-accuracy segmentation tasks.

### 5.3 Ablation

In this section, we analyze the effects of each component and evaluate the impact of various pre-trained parameters and denoising paradigms on experimental accuracy. All results are tested on the DIS-VD dataset.

**Table 3: Ablation experiments of components.**
| Edge | DBIA | SWCI | CutMix | $F_{\beta}^{max}\uparrow$ | $E^{m}_{\phi}\uparrow$ | $S_{m}\uparrow$ | $\mathcal{M}$$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | 0.895 | 0.916 | 0.880 | 0.046 |
| $\checkmark$ |  |  |  | 0.902 | 0.921 | 0.888 | 0.036 |
| $\checkmark$ | $\checkmark$ |  |  | 0.917 | 0.943 | 0.895 | 0.032 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 0.918 | 0.945 | 0.899 | 0.030 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.918 | 0.948 | 0.904 | 0.029 |

Effectiveness of each component
In Tab. 3, the first row represents the SD model using a single-step denoising paradigm as described in Eqn. 4, initialized with SD-Turbo pre-trained parameters. Subsequent rows represent the incremental addition of auxiliary edge prediction, the DBIA mechanism, the SWCI module, and CutMix data augmentation, respectively. The progressive trend illustrates the utility of each module, demonstrating their collective contribution to the model’s performance.

**Table 4: Ablation experiments of the pre-trained parameters and denoising steps. The asterisk (*) indicates that the timestep is not fixed during training.**
| Pre-trained Params | Train step | Infer step | $F_{\beta}^{max}\uparrow$ | $E^{m}_{\phi}\uparrow$ | $S_{m}\uparrow$ | $\mathcal{M}\downarrow$ | Inf. Time $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Train from scratch | 1 | 1 | 0.747 | 0.758 | 0.687 | 0.103 | 0.33 |
| SD-Turbo | 1 | 1 | 0.895 | 0.916 | 0.880 | 0.046 | 0.33 |
| 1* | 2 | 0.889 | 0.913 | 0.878 | 0.047 | 0.52 |  |
| 2 | 2 | 0.894 | 0.923 | 0.881 | 0.038 | 0.52 |  |
| SDV2.1 | 1 | 1 | 0.890 | 0.910 | 0.869 | 0.048 | 0.33 |
| 1* | 10 | 0.823 | 0.878 | 0.854 | 0.052 | 1.40 |  |

Diverse Pre-trained Parameters and Denoising Steps
As shown in Tab. 4, to further investigate the impact of SD’s powerful pretrained prior on high-resolution DIS tasks and to explore the effects of various denoising paradigms, we experimented with the following variants based on the vanilla SD backbone. All experiments shared the same network architecture:

By comparing the results of the $1^{\text{st}}$, $2^{\text{nd}}$, and $5\textsuperscript{th}$ rows, which correspond to different pretrained parameters loaded into the U-Net, it’s evident that SD-Turbo outperforms others in one-step denoising. This superior performance can be attributed to its efficient, few-step image generation capabilities acquired through distillation.

The $2^{\text{nd}}$, $3^{\text{rd}}$, and $4\textsuperscript{th}$ rows of Table 4 illustrate the varying impacts of different few-step denoising paradigms.
Our approach, as mentioned in the paper, is reflected in the $2^{\text{nd}}$ row.
The $4\textsuperscript{th}$ row represents a fixed two-step denoising paradigm with timesteps [999, 499] used for both training and testing, which, compared to the second row, shows a slightly better accuracy but at the expense of doubled inference and training time.
The $3^{\text{rd}}$ row modifies the $4\textsuperscript{th}$ by introducing a random timestep selection from (999, 499) during training, followed by a one-step denoising process.
During testing, a fixed two-step denoising procedure is employed, using the timesteps [999, 499]. While this approach reduces training time, it results in lower accuracy than the $2^{\text{nd}}$ row.
The performance decrease is probably because the model wasn’t consistently trained for the specific two-step denoising needed at test time.

The last two rows of Table 4 demonstrate the comparative efficacy between one-step denoising paradigms and traditional multi-step approaches (Ke et al., ; Rombach et al., ) in the context of high-precision, HR object segmentation.
The final row, which employs a random timestep between 0 and 1000 to introduce noise into the mask latent during training, followed by a ten-step denoising process during testing, similar to (Ke et al., ), yields significantly lower results than one-step paradigms.
This diminished performance is attributed to the increased prediction variance stemming from the model’s exposure to inconsistent data distributions during training.

**Table 5: Ablation experiments of the interaction in DBIA.**
| Edge | DBIA | $\text{DBIA}_{geo}$ | $F_{\beta}^{max}\uparrow$ | $E^{m}_{\phi}\uparrow$ | $S_{m}\uparrow$ | $\mathcal{M}$$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| $\checkmark$ | $\checkmark$ |  | 0.917 | 0.943 | 0.895 | 0.032 |
| $\checkmark$ |  | $\checkmark$ | 0.912 | 0.935 | 0.894 | 0.035 |

Figure: Figure 6: Visual comparison of different DIS methods.
Refer to caption: x6.png

Comparison of Different Interaction Methods in DBIA
To compare the differences in cross-domain interaction methods between our approach and (Fu et al., ), we replaced the interaction method in the additional cross-attention mechanism of DBIA with a fusion-oriented approach similar to that used in (Fu et al., ). As shown in the $2^{\text{nd}}$ line, our targeted interaction is more likely to boost generation accuracy and enhance information exchange.

## 6 Conclusion

In this paper, we made the attmpt to harness the exceptional performance and prior knowledge of diffusion architectures to transcend the limitations of traditional discriminative learning-based frameworks in HR, fine-grained object segmentation, aiming at generating detailed binary maps at high resolutions, while demonstrating impressive accuracy and swift processing. Our approach used a one-step denoising paradigm to generate detailed binary maps quickly and accurately. To handle the complexity and detail richness of DIS segmentation, we introduced additional edge constraints and upgraded the attention module to Detail-Balancing Interactive Attention, enhancing both detail clarity and the generative certainty of the diffusion model. We also incorporated multi-scale conditional injection into the U-Net, introducing multi-granularity information for enhanced semantic and structural perception. Extensive experiments demonstrate DiffDIS’s excellent performance on the DIS dataset.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: please see the weakness part
soundness: 3
strengths: 1.	Multi-level Design Innovations: The paper combines single-step denoising, edge-assisted generation, and multi-scale conditional injection to address the challenges of high-resolution segmentation, balancing speed and detail retention effectively.
2.	Comprehensive Experiments: The experimental setup on the DIS5K dataset is thorough, with comparisons to multiple specialized and general segmentation models. Ablation studies illustrate each component’s contribution, supporting the rationale behind the model design.
3.	Clarity: The paper is well-organized, with clear descriptions of the model and results, making it accessible to readers.
summary: This paper proposes a model named DiffDIS for high-resolution binary image segmentation, incorporating strategies like single-step denoising, edge-assisted generation, and multi-scale conditional injection to enhance segmentation accuracy and inference speed. The authors validate DiffDIS’s performance on the DIS5K dataset, showing promising results. While the design is sound and experimental results are clearly presented, the paper’s novelty and certain implementation details could benefit from further clarification. In the rebuttal, the author addressed these concerns.
weaknesses: 1.	Clarify Novelty of the Single-Step Denoising: While the single-step denoising strategy indeed boosts inference efficiency, a similar concept has been explored in models like GenPercept. I suggest that the authors clarify if DiffDIS’s single-step denoising incorporates task-specific optimizations for DIS tasks, to better highlight its originality.
	2.	Elaborate on the Edge-Assisted Generation’s Distinctiveness and Adaptation for High-Resolution Segmentation: The edge-assisted generation approach in DiffDIS appears similar to the edge-guided inpainting technique used in EdgeConnect, albeit applied in segmentation rather than inpainting. To avoid the impression that this is a simple adaptation from inpainting, I suggest the authors discuss any specific adjustments or optimizations made for high-resolution segmentation in DiffDIS.
	3.	Advantages of Joint Edge and Mask Prediction with Experimental Validation: DiffDIS performs joint edge and mask prediction, unlike stage-wise processing. Further discussion on the specific advantages of joint prediction, especially in handling fine details and complex boundaries, would strengthen this choice. Additionally, including experimental comparisons between joint prediction and stage-wise prediction would provide valuable insights into its effectiveness.
	4.	Comparative Analysis of Training Time: While DiffDIS’s inference time is shown to be efficient, comparative analysis of training times is absent. Including training time comparisons would provide a more holistic view of DiffDIS’s computational efficiency.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. In emu-edit, they introduced a method called task embedding, it is a learnable embedding added to the time step embedding, to distinguish the different task in multi-task training. As your Batch-Discriminative Embedding contains more detailed design, do you have any performance comparison between your Batch-Discriminative Embedding and the task embedding?
2. For the one step mask sampling, in 4.4 and the fig 5 only state that it is building upon the established DDPM, but no more detailed description and implemented the one step sampling, could you please provide more details on it?
3. From the paper, it shows that the author used SD2.1 and SD turbo as initialization weights, and there is a channel-wise concat operation before feeding into the UNet, which changes the number of channels in the input layer. I would like to know how the pretrained weights are handled for the input layer when they are used (eg. duplicate/zero).
soundness: 4
strengths: 1. The author discovered that the introduction of edges can enhance the detail and performance of segmentation. They used batch discriminative embedding to distinguish between edges and segmentation. This is a novel method.
2. The author provided detailed experiments that demonstrate the method's strong performance across multiple aspects, and also included an ablation study to prove the effectiveness of each module.
summary: The authors proposed a method for dichotomous segmentation using a Stable Diffusion prior, finding that introducing edges into the segmentation task can enhance performance. They introduced the BDE and DBIA modules, which can distinguish between different tasks and achieve better detail generation capabilities. The method efficiently utilizes one-step sampling and shows significant improvement over previous methods across multiple test projects.
weaknesses: 1. The description of the one step inference is not comprehensive enough, please see Q2
2. For dichotomous segmentation, using an RGB 3-channel VAE to encode a single-channel segmentation mask might be a bit overkill. As an advancement in dichotomous segmentation, some earlier works have used diffusion models for matting, which also achieved very good results. However, considering that it can produce decent results in just one step, it is acceptable.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Latent diffusion models such as SD would map the original image into low-dimensional latents e.g. 32x32x4 for 512x512x3 input. I do not understand well how the low dimensional latent is decoded to high-resolution mask.
2. I would like to see if the proposed methods can be transferrable to other architectures, e.g. pixel-space diffusion UNets and latent DiT.

The followings seem typos or grammar issues, which do not affect my ratings:
1. line 64, "in balance receptive field expansion" should be "in balancing ..."
2. line 72, "It’s power is" should be "Its power is"
3. line 363, "conloution" should be "convolution"
4. line 530, "attmpt" should be "attempt"
soundness: 3
strengths: 1. The proposed method reaches SOTA performance and beat other concurrent diffusion-based approach on DIS datasets.
2. This is an early attempt that uses a pre-trained generative model for challenging DIS task.
3. The method is efficient in comparison to the line of work that follows SegDiff, which runs diffusion process for more time-steps.
4. The ablation studies include both quantitative numbers and qualitative visualizations, which are helpful for understanding how the whole framework is designed.
summary: This paper address the problem of Dichotomous Image Segmentation (DIS) with a generative foundation model, StableDiffusion, by modifying with several key modules.
weaknesses: 1. The application scenario seems limited. The task setting is only limited to Dichotomous Image Segmentation. It could be more convincing if the authors can also address the applicability of this approach in more settings, e.g. image matting, foreground object segmentation, edge detection.
2. Running diffusion for one-step for segmentation is not a great contribution. This paper might miss some related work that is in the line of DDPM-Seg [1]. A lot of recent work that uses StableDiffusion for unsupervised semantic segmentation and open-vocabulary segmentation is indeed one-step in inference time.
3. The methods are not novel. From the ablation studies, the most prominent modules are Batch-Discriminative Embedding and Detail-Balancing Interactive Attention (DBIA). However, Batch-Discriminative Embedding is proposed by previous work and this work is more of applying that module for DIS task. DBIA is a modified attention module but specifically designed for DIS.

[1] Label-Efficient Semantic Segmentation with Diffusion Models. ICLR 2022.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Algorithm 1, the authors finetune diffusion U-Net by only considering t=T and directly optimizing with $x_0$  instead of noise, where the diffusion UNet seems to degenerate into a common UNet and is inconsistent with SD. Why did the authors not train the diffusion U-Net using the consistent objective with SD and deploy an efficient ODE solver to enable efficient inference?
2. Considering the randomness of the noise sampling process, is the model sensitive to the sampled noise in the inference stage? It is suggested that an analysis of the performance be made using varied noise.
3. Can this performance be improved by sending some text embedding given by the caption model instead of using the empty text embedding?
4. What is the difference between the proposed Detail-Balancing Interactive Attention (Eqn.6) and the common cross-attention deployed between the mask and edge feature?
5.  The multi-scale injector introduces condition signals into blocks. There lacks a comparison with the common condition/signal injection methods, such as the cross-attention in SD.
6. Since VAE contains a large number of parameters, the authors should give a comparison in terms of the inference speed and parameters with state-of-the-art methods.
soundness: 3
strengths: + The proposed method is technically sound and streamlined, unleashing the dimension-reduction ability of VAE and the representation capacity of diffusion U-Net in the perceptional task.
+ The paper is well-organized and easy to follow.
+ The proposed method achieves superior performance gains on the benchmark datasets.
+ Some experimental observations, such as the restorative capacity of VAE (Tab. 1) and the influence of time-step (Tab. 4) might be valuable to the community.
summary: This paper proposes a DiffDIS framework based on SD V2.1 to tackle the fine-grained dichotomous image segmentation task. The proposed DiffDIS finetunes the diffusion U-Net in the VAE-encoded latent space and introduces several modifications to enhance the edge perception, including:  1) The edge-assisted training strategy introduces batch-discriminative embedding to enable the mask and edge prediction in a single SD model and conducts interactive attention between the mask and edge branches, 2) Add to zero convolution to enhance the condition injection at different scales. The paper is easy to follow. The proposed method only uses single-step denoising to enhance efficiency and achieves SoTA performance on all the benchmarks.
weaknesses: + Some concerns about the technical contribution should be clarified: see Q4 and Q5.
+ Some concerns about the robustness and model efficiency should be addressed: see Q2 and Q6
+ Some other concerns about the methodology should be tackled: see Q1 and Q3

Meta-review: While utilizing diffusion model for image segmentation is not new, this paper proposes a variant of diffusion model for high-precision dichotomous image segmentation. During inference, the method is efficient with single-step denoising from random Gaussian noise. Edge information is integrated with mask via Batch Discriminative Embedding (Fu. et, al. 2024).
Many recent or concurrent works on diffusion models for image segmentation rely on single-step denoising. The method here is slightly different as it denoises random Gaussian noise for edge and mask at the last step conditioned on an image. It is not clear though why single step suffices for denoising of mask and edge map, though this can be due to the simplicity of such modalities compared to information-rich images. The Batch Discriminative Embedding block is also not new.
The technical novelty of this paper is not significant, but the method is technically sound, and the results are impressive and convincing. Hence, the AC recommends acceptance of this paper as a poster.

Decision: Accept (Poster)