Title: MaskFactory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dichotomous Image Segmentation; Diffusion Model; Synthetic Data
Abstract: Dichotomous Image Segmentation (DIS) tasks require highly precise annotations, and traditional dataset creation methods are labor intensive, costly, and require extensive domain expertise. Although using synthetic data for DIS is a promising solution to these challenges, current generative models and techniques struggle with the issues of scene deviations, noise-induced errors, and limited training sample variability. To address these issues, we introduce a novel approach, Mask Factory, which provides a scalable solution for generating diverse and precise datasets, markedly reducing preparation time and costs. We first introduce a general mask editing method that combines rigid and non-rigid editing techniques to generate high-quality synthetic masks. Specially, rigid editing leverages geometric priors from diffusion models to achieve precise viewpoint transformations under zero-shot conditions, while non-rigid editing employs adversarial training and self-attention mechanisms for complex, topologically consistent modifications. Then, we generate pairs of high-resolution image and accurate segmentation mask using a multi-conditional control generation method. Finally, our experiments on the widely-used DIS5K dataset benchmark demonstrate superior performance in quality and efficiency compared to existing methods. The code is available at https://qian-hao-tian.github.io/MaskFactory/.

MaskFactory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation

Abstract

Dichotomous Image Segmentation (DIS) tasks require highly precise annotations, and traditional dataset creation methods are labor intensive, costly, and require extensive domain expertise. Although using synthetic data for DIS is a promising solution to these challenges, current generative models and techniques struggle with the issues of scene deviations, noise-induced errors, and limited training sample variability. To address these issues, we introduce a novel approach, MaskFactory , which provides a scalable solution for generating diverse and precise datasets, markedly reducing preparation time and costs. We first introduce a general mask editing method that combines rigid and non-rigid editing techniques to generate high-quality synthetic masks. Specially, rigid editing leverages geometric priors from diffusion models to achieve precise viewpoint transformations under zeroshot conditions, while non-rigid editing employs adversarial training and selfattention mechanisms for complex, topologically consistent modifications. Then, we generate pairs of high-resolution image and accurate segmentation mask using a multi-conditional control generation method. Finally, our experiments on the widely-used DIS5K dataset benchmark demonstrate superior performance in quality and efficiency compared to existing methods. The code is available at https: //qian-hao-tian.github.io/MaskFactory/ .

1 Introduction

Dichotomous image segmentation (DIS) aims to accurately segment objects from natural images [1, 2], a critical task in various computer vision applications, including medical imaging [3, 4, 5], autonomous driving [6] and connectomics research [7, 8]. However, traditional methods for collecting datasets for DIS tasks are labor-intensive, costly, and require extensive domain expertise. Recently, synthetic data has emerged as a promising solution for generating diverse and precise datasets at scale, offering a scalable and cost-effective means for model training. Given the importance of high-quality synthetic data for training models, developing methods to generate such datasets for DIS tasks is crucial. Although generative models have been employed to assist in synthetic dataset production, they face significant limitations in terms of controllability, precision, and diversity.

Recent synthetic methods [9, 10, 11] utilize diffusion models [12] to synthesize image-mask pairs from textual cues and attention maps. Despite progress, they still encounter three major challenges: (1) Controllability: Text-guided generation scheme may deviate from real-world scenes, especially with fine-grained labels, leading to uncontrollable generated images. (2) Precision: The use of attention maps can introduce noise, which degrades mask fidelity and poses challenges for applications

Synthetic mask

Rigid Mask Editing

Original Source

Non-rigid Mask Editing

Synthetic mask

requiring precise pixel-level segmentation, such as DIS [2]. 3. Diversity: Previous works [13, 14, 15] are relatively uniform, with limited ability to generate diverse training samples, which makes them unsuitable for providing sufficient variability in training data.

Generating diverse human images often poses challenge due to the complexity of prompts, which limits their applicability in tasks requiring extensive variability. It's worthy noting that the expansion of datasets with fine granularity, such as DIS datasets, currently lacks an effective solution. In fact, we observe that existing DIS datasets predominantly consist of single-source orthophotos featuring limited shape diversity among similar objects. The generation of more diverse DIS images that accurately reflect real-world distributions remains an unresolved challenge.

To address these challenges, we propose MaskFactory, a novel two-stage method that efficiently generates high-quality synthetic datasets for DIS tasks. Based on the geometric characteristics of objects, our approach simultaneously considers both rigid and non-rigid transformations of target objects. This means that when generating synthetic datasets, we take into account not only changes in viewpoints (rigid transformations) but also deformations (non-rigid transformations), as illustrated in Figure 1. In the first stage, rigid transformations are driven by geometric priors learned from large-scale diffusion models, enabling precise viewpoint changes and simulating diverse variations in observation angles and scales. Non-rigid transformations leverage prompts provided by large language models to accurately alter the shape of target objects via attention-based editing, ensuring topological consistency before and after editing through topology-preserving adversarial training. Consequently, even for masks with complex geometries, high-quality and diverse synthetic masks can be generated for DIS tasks. In the second stage, we utilize multiple control inputs, including masks, canny edges, and prompts representing class information, to guide the generation process. These inputs are fed into diffusion models to produce high-resolution images that match the prepared segmentation masks. This approach ensures high consistency between images and masks while enhancing the realism and diversity of the dataset.

Our method significantly enhances the realism and diversity of generated datasets. We validate the effectiveness of MaskFactory on the DIS5K dataset [2], specifically designed to evaluate DIS performance. Experimental results show that our approach outperforms existing dataset synthesis methods in terms of structural preservation and error metrics, achieving an average performance gain of 8.8%. These findings highlight the potential of MaskFactory to provide the data diversity and annotation precision required for DIS tasks while significantly reducing the time and costs associated with dataset preparation.

In summary, the contributions of our work are listed as follows:

We introduce MaskFactory, a novel approach that generate high-quality datasets for DIS task in terms of quality, precision, and efficiency.

(1.a) Rigid Mask Editing

GLYPH(cmap:d835)(∙)

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

(1.b) Non-rigid Mask Editing

GLYPH(cmap:d835)(∙)

Give a prompt to

change the shape

A round table

Decoder

Decoder

Encoder

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Canny

Topology   Preserving   Loss

Decoder

Encoder

Mask Pool

{GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

}

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Q

K

V

Prompt Pool

{GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

}

noise

(2.a) Edge Condition

(2.b) Prompt Condition

Synthetic  Data Pool

{GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

, GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

}

Stage1: Mask Editing

Stage2:  Image Generation

We propose a two-step method that synthesizes high-quality and diverse object masks via masking editing and generates corresponding high-resolution images using a multi-conditional control generation method.

Experimental results on the DIS5K dataset demonstrate the superior performance of MaskFactory, compared to existing dataset synthesizing methods.

2 Related work

Synthetic data. Synthetic data has garnered significant attention in various machine learning and computer vision tasks, such as natural language processing [16], object detection [17], and image segmentation [10]. For instance, Lin et al. [17] demonstrated that synthetic data can enhance performance in object detection, especially in scenarios with limited access to real-world data. Generative adversarial networks (GANs) and variational autoencoders (VAEs) are two popular deep learning-based methods for generating synthetic data [18]. Recent studies have focused on improving the quality and diversity of synthetic data [19] and exploring its use in few-shot learning [20]. Additionally, synthetic data has been shown to enhance the interpretability and explainability of machine learning models [21]. Benefiting from recent advancements in diffusion models [12, 22], methods such as DatasetDM [9] and Dataset Diffusion [10] can generate high-quality synthetic data for various computer vision tasks. However, these methods may introduce additional errors due to the inclusion of pseudo-labels. These errors arise because pseudo-labels can be noisy and inaccurate, leading to suboptimal training data. Although some research [23] employs controlnet-like control schemes [24] to mitigate these issues, the generated synthetic data may still suffer from noise and errors, making it unsuitable for the DIS task. Therefore, in this paper, we focus on generating high-quality synthetic data for the DIS task, where the generated image-mask pairs need to be highly precise and accurate.

Dichotomous image segmentation. DIS has seen substantial progress with the advent of highresolution imaging technologies. Representative approaches include the intermediate supervision strategy in IS-Net [2], the frequency prior method [1], and the unite-divide-unite strategy by [25]. However, these methods have enhanced segmentation accuracy but often struggle to capture extremely fine details. Thus, recent progressive refinement strategies, e.g., BASNet [26], emphasize the importance of auxiliary information such as gradient maps and multi-scale inputs. These methods propose to use gradient features and ground truth supervision to enhance the learning of weak features in complex regions. Then, BiRefNet [1] maintains high-resolution inputs and employs a bilateral reference framework to better capture intricate details. The latest works, such as the multi-view aggregation network by Yu et al. [27] and the interactive segmentation approach by Liu et al. [28], further advance the field by integrating diverse prompts and enhancing feature representation for high-quality segmentation.

3 Method

3.1 Overview of MaskFactory

Current image segmentation methods are significantly constrained by their dependence on limited manually annotated data, which hampers both performance and generalization. To mitigate this, pseudo-label generation is often employed to augment training datasets. However, in the context of DIS, these methods frequently introduce artifacts that degrade segmentation quality. Additionally, existing image editing techniques often fail to preserve the topological structure of binary masks, resulting in discontinuities and overlaps within target regions.

To address these challenges, we introduce the MaskFactory framework, designed to generate a large number of high-quality synthetic image-mask pairs G = { ( g r i , g i m ) } i M =1 from an original dataset D = { ( I r i , I i m ) } N i =1 . This approach aims to enhance the performance of DIS models. As illustrated in Figure 2, our framework comprises two main stages: mask editing and image generation.

In the mask editing stage, source masks from the original dataset are transformed using both rigid and non-rigid editing methods, resulting in a set of high-precision synthetic masks G m = { g i m } i M =1 . Rigid mask editing generates synthetic masks from various perspectives, while non-rigid mask editing employs a topology-preserving adversarial training mechanism to edit masks according to semantic prompts while retaining the structural integrity of the source masks.

In the image generation stage, a multi-conditional control generation method is utilized to produce realistic RGB images G r = { g r i } i M =1 that correspond precisely to the synthetic masks, using the latter as conditioning constraints.

3.2 Mask Editing Stage

3.2.1 Rigid Mask Editing

Rigid mask editing aims to preserve detailed information from the source masks through rigid transformations. We leverage the Zero123 [29] method, which employs a viewpoint-conditioned diffusion model ψ θ to manipulate masks' perspectives. Given the relative camera rotation and translation T i for the desired viewpoint, ψ θ synthesizes a new mask g i m based on the source mask I i m , such that g i m = ψ θ ( I m ′ i , T i ) , where I m ′ i is the inverted image of the source mask I i m to ensure the main component is non-zero.

3.2.2 Non-Rigid Mask Editing

Non-rigid mask editing, inspired by MasaCtrl [30], is a critical component of MaskFactory. Unlike MasaCtrl, which directly manipulates the source mask I i m using a textual prompt P i to generate a synthetic mask g i m , we introduce a topology-preserving adversarial training mechanism to mitigate artifacts and structural degradation in binary mask editing. The textual prompt P i is derived from a pool of prompts { p i m } that are generated using GPT-4 based on the original images. These prompts provide detailed descriptions that guide the mask editing process. This module consists of a generator G θ and a discriminator D ϕ . The generator transforms noise z into a synthetic mask g i m under the guidance of a textual prompt P i and the source mask I i m . A mutual attention mechanism aligns the

query features Q t of g i m with the key and value features K s , V s of I i m , ensuring consistency during editing. To avoid foreground-background confusion, a mask M is extracted from the cross-attention maps to guide the model's focus.

Topology-Preserving Adversarial Training. To maintain the structural information of the source mask, we first extract an edge map E s = E ( I i m ) using an edge detection operator E , obtaining key points V = { v j } N v j =1 . We then construct a graph T = ( V , E s ) based on these key points. The discriminator D ϕ performs adversarial training on the structural graphs T g and T s of the synthetic mask g i m and the source mask, respectively, ensuring topological consistency.

The training objective of the discriminator D ϕ is to maximize:

where p data ( T s ) and p gen ( T g ) represent the distributions of the structural graphs of the source masks and the synthetic masks, respectively. Conversely, the training objective of the generator G θ is to minimize the discriminative power of the discriminator:

Through topology-preserving adversarial training, the non-rigid editing module effectively retains the structural information from the source masks during the editing process, generating high-quality, artifact-free synthetic masks.

The overall loss function L total for non-rigid mask editing encompasses the adversarial loss L GAN, the content loss L content, and the structure preservation loss L structure :

where λ 1 and λ 2 are balancing factors.

The adversarial loss L GAN is defined as:

The content loss L content measures the semantic consistency between the synthetic mask and the textual prompt:

where g i m is the synthetic mask and I i m is the source mask.

The structure preservation loss L structure evaluates the difference between the structural graphs of the synthetic mask and the source mask:

These components ensure that the editing process maintains the structural and semantic consistency of the source masks.

3.3 Image Generation Stage

Following the mask editing stage, we obtain a synthetic mask pool { g i m } i M =0 comprising finely detailed synthetic masks generated through the aforementioned transformations. In the subsequent image generation stage, we accurately generate corresponding RGB images { g r i } i M =0 for the masks in the synthetic mask pool using a multi-condition control generation method. The primary segmentation of the RGB images aligns with the corresponding synthetic masks. Inspired by ControlNet [24], we introduce a multi-condition control generation method to achieve precise RGB image generation. This method simultaneously injects the segmentation condition c s i and the canny condition c y i to steer the denoising process of the random Gaussian noise, ensuring the generated RGB images { g r i } i M =0 correspond accurately to the synthetic masks { g i m } i M =0 .

Since the synthetic mask itself serves as the segmentation condition c s i , when a synthetic mask g i m is provided, we only need to extract the canny condition c y i using the canny operator:

bed

A bed frame made out of wooden pallets.

fan

A fan is sitting on a window sill.

bicycle

A bicycle with a basket on the front of it.

gate

An iron gate is in front of a house.

bench

A wooden bench is sitting on a sidewalk.

(a) Raw Image

(b) MaskFactory

(c) DatasetDM [9]

(d) DatasetDiffusion[10]

After obtaining the canny and segmentation conditions, we input them into block B θ , which consists of a set of neural layers. Finally, these conditions are injected into the pre-trained diffusion model M θ , controlling the noise z denoising process to generate the corresponding RGB image. We refer readers to [24] for more details.

4 Experiment and Results

4.1 Dataset & Metrics

We conduct our experiments on the DIS5K dataset, which comprises 5,479 high-resolution images featuring camouflaged, salient, or meticulous objects in various backgrounds. The DIS5K dataset is divided into three subsets: DIS-TR (3,000 images) for training, DIS-VD (470 images) for validation, and DIS-TE (2,000 images) for testing. For data augmentation, we utilize the mask portion of the training subset (DIS-TR).

To evaluate our models, we employ a diverse set of metrics to ensure comprehensive performance assessment. These metrics include max F 1 [31], which balances precision and recall, providing a harmonic mean that is indicative of overall accuracy; F ω β [32], a weighted F-measure that compensates for class imbalances, with values ranging from 0 to 1, where higher values denote superior performance; M (Mean Absolute Error)[33], which calculates the average absolute difference between the predicted and ground truth masks, with lower values signifying better accuracy; S α [34], a structural similarity measure that evaluates the preservation of significant structures within the image, with values closer to 1 indicating better performance; and E ϕ M [35], an enhanced measure that considers both pixel-level and image-level information for a more holistic evaluation, where higher values represent better performance. Collectively, these metrics provide a robust framework for assessing the effectiveness and reliability of our segmentation models.

4.2 Impletement Details

Our image editing framework is implemented using PyTorch and trained on 8 NVIDIA GeForce RTX 3090 GPUs. We employ both non-rigid and rigid editing approaches to manipulate images in the DIS-TR dataset, utilizing spatial transformers and affine transformations to model non-rigid and rigid deformations, respectively. The hyperparameters used in our model are as follows: a batch size of 16, an image size of 512x512, 5 editing iterations, a learning rate of 0.001, a weight decay of 0.0001, 1000 diffusion steps, and a diffusion step size of 0.1. The hyperparameters in Equ 3 are set to λ 1 = 0 . 8 and λ 2 = 0 . 5 .

We train the segmentation model using the DIS-TR subset of the DIS5K dataset, utilizing 2 NVIDIA GeForce RTX 3090 GPUs. The input size to the network is 512x512, with a learning rate of 0.0001 and a batch size of 48. The model is optimized using the Adam optimizer over a total of 800 epochs. The edited images generated by our model are combined with the original training set to form a new training set, enabling the model to learn from both original and edited images.

We evaluate our model on the DIS5K datasets, using the test sets from each dataset. For a comprehensive comparison, we also reproduce two state-of-the-art diffusion-based image generation methods, DatasetDM and Dataset Diffusion, using their default parameters. The generated images are compared with those from our approach, with the visualization of the results shown in Figure 3.

4.3 Results

Results by Generated Image Count. We evaluate the performance of our proposed method, MaskFactory, against two state-of-the-art baselines, DatasetDM [9] and Dataset Diffusion [10], on the DIS5k dataset. The DIS5k dataset comprises five sub-datasets: DIS-VD, DIS-TE1, DIS-TE2, DIS-TE3, and DIS-TE4. We use the robust segmentation method IS-Net [2] as a baseline for the DIS task. For the two baselines, DatasetDM and Dataset Diffusion, we use the default parameters provided in their open-source implementations.

First, we identify the best-performing model on the DIS-VD validation set and then evaluate its performance on the other sub-datasets. The models are trained on the DIS-TR dataset, which is augmented with generated datasets of varying sizes: 2500, 5000, 7500, and 10000 images.

The experimental results are presented in Table 1. Our proposed method, MaskFactory, consistently outperforms the baselines across all sub-datasets and evaluation metrics. As the number of generated images increases, the performance of MaskFactory improves, achieving the best results with 10000 generated images. Notably, MaskFactory attains the highest maxF 1 scores across all sub-datasets, with improvements ranging from 0.044 to 0.052 compared to the IS-Net baseline.

In contrast, while DatasetDM and Dataset Diffusion also show some performance gains, they encounter the issue of collapse when the generated data exceeds 5000 images. In these cases, the segmentation network's performance stagnates or even degrades. For the DIS segmentation task, the use of pseudo-labels could introduce additional errors, leading to performance declines.

Furthermore, MaskFactory demonstrates superior performance in terms of the weighted F-measure ( F ω β ), S-measure ( S α ), and Enhanced-alignment Measure ( E ϕ M ). The Mean Absolute Error ( M ) is also consistently lower for MaskFactory compared to the baselines, indicating more accurate segmentation results.

Results by Segmentation Network. After achieving stable improvements in IS-Net's performance, we examined the generalizability of our approach by applying the same configuration to several state-of-the-art segmentation networks on the DIS5K dataset. Each network was trained using the DIS-TR dataset augmented with 10,000 generated image pairs. The networks considered in this study include FP-DIS [36], UDUN [25], BiRefNet [1], and SAMHQ [37], all implemented with their default parameters.

The experimental results, shown in Table 2, demonstrate that our proposed method consistently enhances the performance of all evaluated networks across the five sub-datasets of DIS5K. Notably, significant improvements were observed in the maxF 1 and E ϕ M metrics. For instance, on the DISTE1 sub-dataset, our method increased the maxF 1 score of IS-Net from 0.740 to 0.784, FP-DIS from 0.784 to 0.805, UDUN from 0.784 to 0.799, BiRefNet from 0.866 to 0.882, and SAMHQ from 0.897 to 0.905.























































































































































































































































































Additionally, the Mean Absolute Error ( M ) decreased for all networks when applying our method, indicating more accurate segmentation results. The S-measure ( S α ) also consistently improved across all sub-datasets and networks, highlighting the effectiveness of our approach in capturing structural similarity.

On the DIS-VD sub-dataset, used for validation, our method boosted the maxF 1 score of IS-Net from 0.791 to 0.835, FP-DIS from 0.823 to 0.851, UDUN from 0.823 to 0.847, BiRefNet from 0.897 to 0.905, and SAMHQ from 0.842 to 0.847. These findings underscore the generalizability of our approach, as it enhances the performance of diverse segmentation networks without requiring any network-specific modifications.

Visual Results. As shown in Figure 5 (Appendix), our approach achieves precise results, comprising both rigid and non-rigid transformations. The non-rigid transformations, illustrated in columns (b) and (c), enable shape editing, such as removing a table corner or merging two backpacks into one. In contrast, the rigid transformations, demonstrated in columns (d), (e), and (f), primarily involve viewpoint changes, showcasing the original mask

rotated in 3D space. Notably, our method effectively preserves the topological structure information of the original image, including the holes on the chair back. This allows for the low-cost generation of high-precision, diverse data pairs. To investigate differences between generated and real images, we analyzed image and mask distributions. Specifically, we used the CLIP model[38] to extract features from 300 real images, as well as images and masks generated by DatasetDM, DatasetDiffusion, and MaskFactory. We then applied UMAP[39] for dimensionality reduction on these 300 features. The feature distributions of masks and images are visualized in Figures 4(a) and 4(b), respectively.

In the mask editing domain, MaskFactory demonstrates superior mask fidelity compared to other generation methods, primarily due to the incorporation of topological consistency constraints. This results in a feature distribution that closely aligns with that of real images. Conversely, for RGB images, the prior from diffusion VAE introduces a larger disparity between the generated and real

image distributions. However, the distribution generated by MaskFactory shows a greater overlap with the real image distribution compared to the other two methods.

Furthermore, we quantified the differences using cosine similarity, as presented in Table 3. The results indicate that our method achieves the closest distribution to real images, further validating the effectiveness of MaskFactory in generating realistic masks and images.

5 Discussion

5.1 Ablation Study

Mask Generation Type Ablation. In our study, we implemented mask rigid editing, non-rigid editing, and mixed editing-each leveraging our novel mask control technology tailored for specific application scenarios. Rigid editing is designed for scenarios requiring precise geometric adjustments, primarily focusing on viewpoint and scale transformations. Non-rigid editing caters to applications

needing high adaptability, handling topologically consistent deformations and complex, dynamic image edits. Mixed editing combines the advantages of both approaches, offering a comprehensive solution. We further evaluated the performance gains of each editing strategy. Our experimental results, as shown in Table 4.

From the table, we observe that mixed editing achieves the highest performance across most metrics. Specifically, it achieves the highest maxF 1 score and S α , indicating superior structural fidelity and segmentation quality. The slight improvement in M and E ϕ M further underscores the versatility and effectiveness of mixed editing in creating diverse and realistic image-mask combinations.

Loss Function Ablation. We introduce content and structure losses into our model. The Discriminative Loss, implemented via a discriminator, evaluates the differences between generated and real images, aiming to enhance the realism and quality of the outputs. The Edge Constraint Loss focuses on maintaining edge coherence during image editing, which is critical for preserving detailed structural information. We conducted ablation experiments to evaluate the impact of each loss function. The experimental results, shown in Table 5.

From the table, we observe that the combination of all three loss functions ( L GAN, L content, and L structure) achieves the highest maxF 1 score, indicating the best performance in terms of structural fidelity and realism.

5.2 Limitation

Despite the favorable outcomes achieved by our method, it still encounters significant issues. Although we experimented with different conditions, the results are shown in

Table 6. ControlNet sometimes produces unnatural images with stark foreground-background distinctions, necessitating additional harmonization. Complex scenarios can yield unrealistic elements, such as improperly positioned objects. Additionally, our method relies on pre-annotated image-mask pairs, limiting its ability to generate data autonomously and requiring high-quality initial annotations.

5.3 Conclusion

This paper introduces MaskFactory, a novel two-stage approach for generating high-quality synthetic datasets for DIS tasks. By combining rigid and non-rigid mask editing techniques and using multi-conditional control for image generation, MaskFactory produces diverse and precise synthetic image-mask pairs, significantly reducing dataset preparation time and costs. Experiments on the DIS5K benchmark demonstrate the superior performance

of MaskFactory compared to existing methods in terms of quality and efficiency.

References

Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolution dichotomous image segmentation. CAAI AIR , 3:9150038, 2024.

Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. In ECCV , 2022.

Lian Liu, Han Zhou, Jiongquan Chen, Sijing Liu, Wenlong Shi, Dong Ni, Deng-Ping Fan, and Xin Yang. Instructive feature enhancement for dichotomous medical image segmentation. In MICCAI , 2023.

Yinda Chen, Che Liu, Xiaoyu Liu, Rossella Arcucci, and Zhiwei Xiong. Bimcv-r: A landmark dataset for 3d ct text-image retrieval. In MICCAI , 2024.

Yinda Chen, Wei Huang, Xiaoyu Liu, Shiyu Deng, Qi Chen, and Zhiwei Xiong. Learning multiscale consistency for self-supervised electron microscopy instance segmentation. In ICASSP , 2024.

Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In ECCV , 2020.

Xiaoyu Liu, Miaomiao Cai, Yinda Chen, Yueyi Zhang, Te Shi, Ruobing Zhang, Xuejin Chen, and Zhiwei Xiong. Cross-dimension affinity distillation for 3d em neuron segmentation. In CVPR , 2024.

Yinda Chen, Wei Huang, Shenglong Zhou, Qi Chen, and Zhiwei Xiong. Self-supervised neuron segmentation with multi-agent reinforcement learning. In IJCAI , 2023.

Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. NeurIPS , 2023.

Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation. NeurIPS , 2023.

Yinda Chen, Che Liu, Wei Huang, Sibo Cheng, Rossella Arcucci, and Zhiwei Xiong. Generative text-guided 3d vision-language pretraining for unified medical image segmentation. arXiv preprint arXiv:2306.04811 , 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Aysim Toker, Marvin Eisenberger, Daniel Cremers, and Laura Leal-Taixé. Satsynth: Augmenting imagemask pairs through diffusion models for aerial semantic segmentation. In CVPR , 2024.

Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models. NeurIPS , 2023.

Yinda Chen, Haoyuan Shi, Xiaoyu Liu, Te Shi, Ruobing Zhang, Dong Liu, Zhiwei Xiong, and Feng Wu. Tokenunify: Scalable autoregressive visual pre-training with mixture token prediction. arXiv preprint arXiv:2405.16847 , 2024.

Dingquan Wang and Jason Eisner. Synthetic data made to order: The case of parsing. In EMNLP , 2018.

Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao. Explore the power of synthetic data on few-shot object detection. In CVPR , 2023.

Alvaro Figueira and Bruno Vaz. Survey on synthetic data generation, evaluation methods and gans. Math. , 10(15):2733, 2022.

Aman Kishore, Tae Eun Choe, Junghyun Kwon, Minwoo Park, Pengfei Hao, and Akshita Mittel. Synthetic data generation using imitation training. In CVPR , 2021.

Tung Nguyen, Sudhanshu Agrawal, and Aditya Grover. Expt: Synthetic pretraining for few-shot experimental design. NeurIPS , 2023.

Yang Liu, Sujay Khandagale, Colin White, and Willie Neiswanger. Synthetic benchmarks for scientific research in explainable machine learning. In NeurIPS Dataset Track , 2021.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Qi Chen, Xiaoxi Chen, Haorui Song, Zhiwei Xiong, Alan Yuille, Chen Wei, and Zongwei Zhou. Towards generalizable tumor synthesis. CVPR , 2024.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , 2023.

Jialun Pei, Zhangjun Zhou, Yueming Jin, He Tang, and Pheng-Ann Heng. Unite-divide-unite: Joint boosting trunk and structure for high-accuracy dichotomous image segmentation. In ACM MM , 2023.

Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand. Basnet: Boundary-aware salient object detection. In CVPR , 2019.

Qian Yu, Xiaoqi Zhao, Youwei Pang, Lihe Zhang, and Huchuan Lu. Multi-view aggregation network for dichotomous image segmentation. In CVPR , 2024.

Qin Liu, Jaemin Cho, Mohit Bansal, and Marc Niethammer. Rethinking interactive image segmentation with low latency high quality and diverse prompts. In CVPR , 2024.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In CVPR , 2023.

Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In CVPR , 2023.

Radhakrishna Achanta, Sheila Hemami, Francisco Estrada, and Sabine Susstrunk. Frequency-tuned salient region detection. In CVPR , 2009.

Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How to evaluate foreground maps? In CVPR , 2014.

Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and Alexander Hornung. Saliency filters: Contrast based filtering for salient region detection. In CVPR , 2012.

Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-measure: A new way to evaluate foreground maps. In ICCV , 2017.

Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. Enhanced-alignment measure for binary foreground map evaluation. In IJCAI , 2018.

Yan Zhou, Bo Dong, Yuanfeng Wu, Wentao Zhu, Geng Chen, and Yanning Zhang. Dichotomous image segmentation with frequency priors. In IJCAI , 2023.

Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality. NeurIPS , 36, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Appendix

A Pseudocode for the MaskFactory Algorithm

MaskFactory is a two-stage approach for generating high-quality synthetic datasets for DIS tasks. In the first stage, existing ground truth masks undergo rigid and non-rigid editing to generate diverse synthetic masks. Rigid editing uses geometric priors from diffusion models for precise viewpoint transformations, while non-rigid editing employs adversarial training and self-attention mechanisms for complex shape modifications while preserving topology.

In the second stage, the generated masks and their corresponding Canny edges serve as conditions, along with category prompts, to guide the generation of high-resolution RGB images using a multiconditional control generation method. This process ensures consistency between the generated images and masks while enhancing dataset realism and diversity. Our pseudocode visible Algorithm 1

B Dataset Details

We conduct our experiments on the DIS5K dataset, which comprises 5,479 high-resolution images featuring camouflaged, salient, or meticulous objects in various backgrounds. The DIS5K dataset is divided into three subsets: DIS-TR (3,000 images) for training, DIS-VD (470 images) for validation, and DIS-TE (2,000 images) for testing. For data augmentation, we utilize the mask portion of the training subset (DIS-TR).

To evaluate our models, we employ a diverse set of metrics to ensure comprehensive performance assessment. These metrics include max F 1 , which balances precision and recall, providing a harmonic mean that is indicative of overall accuracy; F ω β , a weighted F-measure that compensates for class

imbalances, with values ranging from 0 to 1, where higher values denote superior performance; M (Mean Absolute Error), which calculates the average absolute difference between the predicted and ground truth masks, with lower values signifying better accuracy; S α , a structural similarity measure that evaluates the preservation of significant structures within the image, with values closer to 1 indicating better performance; and E ϕ M , an enhanced measure that considers both pixel-level and image-level information for a more holistic evaluation, where higher values represent better performance. Collectively, these metrics provide a robust framework for assessing the effectiveness and reliability of our segmentation models.

C Mathematical Details

C.1 Diffusion-Based Image Generation

The diffusion model employed in MaskFactory for image generation follows the formulation introduced by Ho et al. [40]. Given a data distribution x 0 ∼ q ( x 0 ) , the forward diffusion process is defined as a Markov chain that gradually adds Gaussian noise to the data:

where β t ∈ (0 , 1) is a variance schedule. The reverse process is learned by a neural network ϵ θ that predicts the noise added at each step:

The objective is to maximize the variational lower bound :

C.2 Topology-Preserving Adversarial Training

The topology-preserving adversarial training in MaskFactory involves a generator G θ and a discriminator D ϕ . The generator aims to minimize the adversarial loss:

while the discriminator tries to maximize the adversarial loss:

The generator and discriminator are updated alternately to reach an equilibrium.

C.3 Structural Graph Construction

To preserve the topological structure of the source masks during editing, MaskFactory constructs structural graphs T s and T g for the source and synthetic masks, respectively. The structural graph T = ( V, E ) consists of a set of vertices V = { v j } N v j =1 representing key points and a set of edges E representing their connectivity.

The structure preservation loss is defined as the L1 distance between the structural graphs:

where T g ( i, j ) and T s ( i, j ) denote the edge weights between vertices i and j ' in the synthetic and source mask structural graphs, respectively.

By minimizing the structure preservation loss along with the adversarial and content losses, MaskFactory ensures that the edited masks maintain the topological structure of the source masks.

D Visualization of the results using two editing methods from MaskFactory.

In this section, we demonstrate MaskFactory's ability to edit masks for common and fine-grained objects.

D.1 Visualization of Common Object Mask Editing

We selected common household items such as tables, chairs, bags, and musical instruments for editing using MaskFactory. Non-rigid edits were performed with different prompts, and rigid edits were performed from different viewpoints. The editing results and corresponding RGB image generations are shown in Figure 5. MaskFactory exhibits strong topological structure preservation and produces diverse editing outcomes, as demonstrated by the variety of modifications made to both rigid and non-rigid objects.

D.2 Visualization of Fine-Grained Object Mask Editing

We selected fine-grained objects from MaskFactory's generated masks, such as ornate European chandeliers, iron gates, birdcages, and seahorses, to showcase MaskFactory's detailed editing capabilities. Even with complex geometries, MaskFactory can perform topology-preserving edits without losing the original mask's semantic information. These intricate structures play a crucial role in segmentation metrics. As illustrated in Figure 6, the visual results demonstrate the model's ability to handle fine-grained details while maintaining the integrity of the original mask's structure.

D.3 Visualization Results with Canny Constraints

After incorporating Canny edge detection as a constraint, the visual results of MaskFactory show a significant improvement in boundary precision. The Canny edges effectively guide the generation process, resulting in images with clearer and more accurate boundary details, avoiding vague or ambiguous transition areas. The visualizations demonstrate that the Canny edges not only better constrain the contours of the generated images but also enhance the overall fidelity and visual quality of the output. Compared to models without edge constraints, our approach produces more detailed and structurally coherent images, as shown in Figure 7.

D.4 Topological Structure Visualization

We performed topological structure visualizations on selected samples to assess the model's ability to preserve topology during the editing process. These visualizations clearly show how our method retains the topological structure of the original data while allowing for effective manipulation when necessary. Whether dealing with complex geometric shapes or subtle structural modifications, the topological visualizations illustrate that our approach reliably preserves geometric consistency and topological features throughout the editing process, as demonstrated in Figure 8.

(a) raw image

(b) rigid 1

(c) rigid 2

(d) non-rigid 1

(e) non-rigid 2

(f) non-rigid 3

Real Image

Source Mask

Synthetic Image

Synthetic mask

Rigid Mask Editing

Original Source

Non-rigid Mask Editing

Synthetic Image

Synthetic mask

Sythetic

Generated

Generated

mask(depth)

image(w/o canny)

Raw

Topology

MaskFactory

(rigid)

Topology

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This paper presents a method for generating and editing images with highprecision masks, and announces the open-sourcing of a DIS generation dataset.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section 5.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: This paper's experiments and methods do not introduce new theories; the theoretical foundations used are detailed in Section C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: This paper includes a link to the anonymized code in the abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: please check in our project page.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: please check in our project page.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: please check in our project page.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: please check in our project page.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: please check in our project page.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: please check in our project page.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: please check in our project page.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generative method for dichotomous image segmentation. In detail, non-rigid and rigid editing techniques are used to generate high-quality synthetic masks. Those masks are leveraged for segmentation model training, which typically requires expensive dichotomous image labeling.\n\nSTRENGTHS:\nThis paper introduces an approach for generating high-quality datasets for DIS. In my opinion, the methodology is carefully designed and experiments show the effectiveness. Overall I think this is an interesting and insightful paper.\n\nWEAKNESSES:\n1. There seems to have some typos/errors in the paper.\n  \n   a. In line 140, the (V,E) should be (V, E_s). The E_s does not show in section 3.2.2 except line 139.\n  \n   b. The font should be consistent, e.g., line 139,140 and Appendix. A Algorithm 1 line 7-11.\n  \n   c. The Figure 2 does not show what is discussed in Section 3.3. In the Figure 2, the prompt pool is used for generation. But in Section 3.3, the authors say that the segmentation masks and the edge conditions are used. If the text prompt is not used, it is meaningless to introduce the text in the last row of Figure 2 Stage 1 and Section 3.2.2. Also, here the detailed description of {p^m_i} is missing in the paper, though it is quite clear. Only have P_i shows in Section 3.2.2 and at least {p^m_i} should show in the following sections for consistency.\n  \n   d. \"Topylogy\" should be \"Topology\" in Figure 2.\n  \n   e. Remove full stops in the title of Section 4, 4.1, 4.2.\n  \n2. The authors argue that inspired by ControlNet, canny condition is also used. However, since there is no pretrained ControlNet used, some layers are trained according to line 173 and 174, it is unclear whether the canny condition is useful. Though more conditions present more information intuitively, the canny is already actually included in the informative mask condition. So some ablation studies should be presented to show that the additional condition do helps. But according to Section 5.2, is the pretrained Canny ControlNet used? Need to clarify in the main paper.\n  \n3. The method has limited increase compared with SAM-HQ, which finetunes SAM with additional branch on some prepared data.\n  \n4. It seems the impact of this method itself is limited, especially DIS is not so important. Maybe the authors can think about extending the method to a broader scope. Though I think this is a minor issue.\n\nQUESTIONS:\nSome suggestions\n\n1. The authors use Zero123 for rigid mask editing. I am curious whether ImageDream [1] can be used here. Both of them can generate novel view images. [1] can takes the additional text as the input and I am interested in whether it can be used to directly change the shape of the mask like this paper shows that \"A round table\" can change the shape in Figure 2.\n  \n2. For the topology mentioned in Section 3.2.2, is it easy to show some topology structures of some samples? It would be better to include topology results for better illustration.\n  \n3. For Table 1, the numbers (2500, 5000, ...) are too close to the line.\n  \n4. The authors can include more discussion on related works [2,3]. It is interesting to discuss whether the data filtering strategy in [2] can be combined in this work. And [3], which also takes a dataset as the input and outputs a new one, also includes edited masks for the generation of specially tuned generative models, sharing the similar core idea of this paper.\n\nAs I discussed in Strength, overall this is an interesting paper. And I also love the area of synthetic date generation. However, it seems this paper is finished too rush. There are some improper points in the paper, discussed in the Weakness. I believe the authors should revise the paper thoroughly for precise presentation. And it is very important to decribe the method as how it is implemented in fact. It seems the text, masks, and the canny are all used in the generation according to line 388, then the Figure 2, Section 3.3 including Formula 8, and the algorithm in the appendix should be modified. Considering too much issues that need to be resolved, I prefer rating this paper as \"Borderline reject\" and I am sure to consider raising my rating to \"Borderline accept\" after the rebuttal for my possible misunderstanding and the authors' proper clarification.\n\n[1] Wang, Peng, and Yichun Shi. \"Imagedream: Image-prompt multi-view diffusion for 3d generation.\" *arXiv preprint arXiv:2312.02201* (2023).\n\n[2] Yang, Lihe, et al. \"Freemask: Synthetic images with dense annotations make stronger segmentation models.\" *Advances in Neural Information Processing Systems* 36 (2024).\n\n[3] Zhu, Lingting, et al. \"Generative Enhancement for 3D Medical Images.\" *arXiv preprint arXiv:2403.12852* (2024).\n\nLIMITATIONS:\nThe limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

comments: SUMMARY:
This paper introduces a new approach for generating diverse and precise datasets. The authors first introduce a mask editing method that combines rigid and non-rigid editing techniques to generate high-quality synthetic masks leveraging geometric priors from diffusion models and adversarial training and self-attention mechanisms for complex, topologically consistent modifications. Then, the authors generate pairs of images and segmentation mask control generation methods. Finally, the experiments on the DIS5K dataset benchmark demonstrate superior performance compared to existing methods.

STRENGTHS:
1. The paper provides a comprehensive and innovative solution to the challenges of DIS dataset creation. The methodology is well-detailed, and the experimental results robustly support the claims of superior performance.

2. The paper is well-written and easy to read.

WEAKNESSES:
1. My major concern is the novelty of the method. Although the paper works on a new task to generate simulated data, the method seems to be a combination of some unrelated approaches including Zero-123, GPT, ControlNet et al. 

2. Whether the generated data can boost the Dichotomous Image Segmentation task is not validated. For example, the ControlNet generated images can be unrealistic, have a large domain gap between real-world images and training images, and the generated images seem not very diversified. Can generated images really help this task?

3. The approach is evaluated on only one dataset, which is limited.

QUESTIONS:
1. The proposed method is only evaluated on one task, which is only limited to only one object. How about applying the approach on general semantic segmentation datasets, such as VOC, ADE20K and CityScapes?

LIMITATIONS:
See Weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-08

comments: SUMMARY:
⁤The paper introduces MaskFactory, a method for producing high-quality synthetic datasets for Dichotomous Image Segmentation (DIS) tasks. ⁤⁤The method includes a two-stage process: mask editing (combining rigid and non-rigid transformations) and image generation using multi-conditional control methods. ⁤⁤The suggested method improves model performance in DIS tasks by providing variation in synthetic data. ⁤

STRENGTHS:
- The manuscript is well-written and constructed.
- The authors conduct comprehensive ablation studies to highlight the contribution of each component of their method, further validating their approach.
- The proposed MaskFactory framework shows improvements in the quality, and diversity of synthetic datasets, which is critical for advancing DIS applications.

WEAKNESSES:
There is one limitation that authors in the limitations section acknowledge issues with unnatural images in complex scenarios, but it could delve deeper into how these might impact practical applications.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
Please see the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MaskFactory, a novel approach aimed at addressing the challenges of generating high-quality synthetic datasets for Dichotomous Image Segmentation (DIS) tasks. \nThe authors tackle the challenges by leveraging a blend of rigid and non-rigid editing techniques to generate accurate synthetic masks. Rigid editing utilizes geometric priors from diffusion models for precise viewpoint transformations, while non-rigid editing incorporates adversarial training and mutual self-attention mechanisms for complex modifications. With the accurate generated segmentation masks, a multi-conditional control generation method is then employed to create high-resolution images. The efficacy of MaskFactory is demonstrated through experiments on the widely-used DIS5K dataset, showing superior quality and efficiency in dataset generation compared to existing methods, thus significantly reducing preparation time and costs.\n\nSTRENGTHS:\n1. The scientific problem discussed here is worth researching, and there is still much room for exploration in this area. If the synthetic technology can be leveraged to substantially expand the simulated image-mask pairs based on existing data, it would benefit model training, especially with the current large multimodal models.\n2. This paper takes into consideration both rigid mask editing and non-rigid mask editing, combining them to jointly contribute to mask generation. Additionally, during the non-rigid mask editing process, a topology preserving loss is introduced to maintain the topological structure of the original mask.\n3. The experimental results are very encouraging. Compared to previous methods, the data distribution generated in this paper is closer to the real data. As the amount of synthetic data involved in training increases, the model's segmentation capability also increases proportionally.\n\nWEAKNESSES:\n1. The writing of this paper is too brief. It does not clearly show how the authors specifically carry out the mask editing part, or what unique designs they have implemented in the editing process.\n2. The paper lacks novelty in certain aspects. For the rigid mask editing part, using viewpoint changes to generate masks is a very common practice. The network architecture and synthesis method in the non-rigid mask editing part are completely identical to those used in MaskFactory. Although an additional topology preserving loss has been added, this loss is hard to regard as an independent innovation. Furthermore, the ablation study does not present results using only L_GAN and L_content, making it difficult to intuitively understand the significant impact of L_structure.\n3. Some of the table headers are too narrow.\n\nQUESTIONS:\n1. Can rigid mask editing and non-rigid mask editing be integrated into one framework, rather than just being simply combined?\n2. Can the improvement brought by the  L_structure loss be intuitively reflected in the ablation study?\n\nLIMITATIONS:\nThe method faces challenges including the occasional production of unnatural images with stark foreground-background distinctions and inaccuracies in complex scenarios. Additionally, it relies on pre-annotated image-mask pairs, limiting autonomous data generation and requiring high-quality initial annotations. However, these issues are acceptable as there is extensive research aimed at resolving the inherent problems with ControlNet, which does not detract from the effectiveness of the method presented in this paper. Furthermore, the authors could explore using pseudo-labels for mask correction and generation to reduce dependency on pre-annotated image-mask pairs.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}