MaskFactory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation
Authors: Haotian Qian
Yinda Chen
Shengtao Lou
Fahad Khan
Xiaogang Jin
Deng-Ping Fan
Venue: NeurIPS 2024 poster (2024)
Keywords: Dichotomous Image Segmentation; Diffusion Model; Synthetic Data
Primary area: machine_vision
TL;DR: Our method uses diffusion models to generate fine-grained data for dichotomous image segmentation, addressing annotation challenges and boosting model performance.

Abstract: Dichotomous Image Segmentation (DIS) tasks require highly precise annotations, and traditional dataset creation methods are labor intensive, costly, and require extensive domain expertise. Although using synthetic data for DIS is a promising solution to these challenges, current generative models and techniques struggle with the issues of scene deviations, noise-induced errors, and limited training sample variability. To address these issues, we introduce a novel approach, Mask Factory, which provides a scalable solution for generating diverse and precise datasets, markedly reducing preparation time and costs. We first introduce a general mask editing method that combines rigid and non-rigid editing techniques to generate high-quality synthetic masks. Specially, rigid editing leverages geometric priors from diffusion models to achieve precise viewpoint transformations under zero-shot conditions, while non-rigid editing employs adversarial training and self-attention mechanisms for complex, topologically consistent modifications. Then, we generate pairs of high-resolution image and accurate segmentation mask using a multi-conditional control generation method. Finally, our experiments on the widely-used DIS5K dataset benchmark demonstrate superior performance in quality and efficiency compared to existing methods. The code is available at https://qian-hao-tian.github.io/MaskFactory/.

MaskFactory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation
Haotian Qian YD Chen 1- Shengtao Lou't
Fahad Shahbaz Khan 3.4
Xiaogang Jin't
Deng-Ping Fan
State Key Lab of CAD&CG, Zhejiang University 2VCIP&CS, Nankai University MBBZAI
"Linköping University
Abstract
Dichotomous Image Segmentation (DIS) tasks require highly precise annotations, and traditional dataset creation methods are labor intensive, costly, and require extensive domain expertise. Although using synthetic data for DIS a promising solution to these challenges, current generative models and techniques struggle with the issues of scene deviations, noise-induced errors, and limited training sample variability. To address these issues, introduce novel approach, MaskFactory, which provides a scalable solution for generating diverse and precise datasets, markedly reducing preparation time and costs. We first introduce a general mask editing method that combines rigid and non-rigid editing techniques to generate high-quality synthetic masks. Specially, rigid editing leverages geometric priors from diffusion models to achieve precise viewpoint transformations under zero- shot conditions, while non-rigid editing employs adversarial training and self- attention mechanisms for complex, topologically consistent modifications. Then, we generate pairs of bigh-resolution image and accurate segmentation mask using a multi-condiiional control generation method. Finally, our experiments on the widely-used DIS5K dataset benchmark demonstrate superior performance in quality and efficiency compared to existing methods. The code is available at https: //qian- hanntian github. io/MaskFactoryy
Introduction
Dichotomous image segmentation (DIS) aims to accurately segment objects from natural images 2122 a critical task in various computer vision applications, including medical imaging 3 4 [ autonomous driving [6] and connectomics research [7] However, traditional methods for collecting datasets for DIS tasks are labor-intensive, costly, and require extensive domain expertise. Recently, synthetic data has emerged as a promising solution for generating diverse and precise datasets at scale, offering scalable and cost-effective means for model training. Given the importance of high-quality synthetic data for training models, developing methods to generate such datasets for DIS tasks is crucial. Although generative models have been employed to assist in synthetic dataset production, they face significant limitations in terms of controllability, precision, and diversity.
Recent synthetic methods 9 10 utilize diffusion models [2] synthesize image-mask pairs from textual cues and attention maps. Despite progress, they still encounter three major challenges: (1) Controllability: Text-guided generation scheme may deviate from real-world scenes, especially with fine-grained labels, leading to uncontrollable generated images. (2) Precision: The use of attention maps can introduce noise, which degrades mask fidelity and poses challenges for applications
"qqual contribution. Deng-Ping Fan served as project leader for this work.
Corresponning author. Contact the author at jin@ cad.zju.edu.cn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original Source
Rigid Mask Editing
Non-rigid Mask Editing
Real Image
Source Mask
Synthetic Image Synthetic mask
Synthetic Image Synthetic mask
Figure 1: Shows the edited masks from the first stage and the corresponding images generated in the second stage. In the examples, we transformed the viewpoint of park benches and tables from frontal view to a top-down view and edited their shapes, changing park benches from curved to square edges and tables from square to circular shapes.
requiring precise pixel-level segmentation, such as DIS 22. 3. Diversity: Previous works [3 14 15
are relatively uniform, with limited ability to generate diverse training samples, which makes them unsuitable for providing sufficient variability in training data.
Generating diverse human images often poses challenge due to the complexity of prompts, which limits their applicability in tasks requiring extensive variability. It's worthy noting that the expansion of datasets with fine granularity, such as DIS datasets, currently lacks an effective solution. In fact, we observe that existing DIS datasets predominantly consist of single-source orthophotos featuring limited shape diversity among similar objects. The generation of more diverse DIS images that accurately reflect real-world distributions remains an unresolved challenge.
To address these challenges, we propose MaskFactory, novel two-stage method that efficiently generates high high-quality synthetic datasets for DIS tasks. Based on the geometric characteristies of objects, our approach simultaneously considers both rigid and non-rigid transformations of target objects. This means that when generating synthetic datasets, we take into account not only changes viewpoints (rigid transformations) but also deformations (non-rigid transformations), illustrated in Figure [ In the first stage, rigid transformations are driven by geometric priors learned from large-scale diffusion models, enabling precise viewpoint changes and simulating diverse variations in observation angles and scales. Non-rigid transformations leverage prompts provided by large language models to accurately alter the shape of target objects via attention-base editing, ensuring topological consistency before and after editing through appoloyy-presevvigg adversarial training. Consequently, even for masks with complex geometries, high-quality and diverse synthetic masks be generated for DIS tasks. In the second stage, utilize multiple control inputs, including masks, canny edges, and prompts representing class information, to guide the generation process. These inputs are fed into diffusion models to produce nigh-resolution images that match the prepared segmentation masks. This approach ensures high consistency between images and masks while enhancing the realism and diversity of the dataset.
Our method significantly enhances the realism and diversity of generated datasets. We validate the effectiveness of MaskFactory the DIS5K dataset [2), specifically designed to evaluate DIS performance. Experimental results show that our approach outperforms existing dataset synthesis methods in terms of structural preservation and error metrics, achieving average performance gain of 8.8%. These findings highlight the potential of MaskFactory to provide the data diversity and annotation precision required for DIS tasks while significantly reducing the time and costs associated with dataset preparation.
In summary, the contributions of our work are listed as follows:
We introduce MaskFactory, a novel approach that generate high-quality datasets for DIS task in terms of quality, precision, and efficiency.

---

(1.a) Rigid Mask Editing 2()
Incoder
Topology Preserving Loss
Dos
Stagel: Mask Editing
(1.b) Non-rigid Mask Editing N(-)
Encader
Canny
Give prompt change the shape
Decader
round table
(2.a) Edge Condition
Stage2: Image Generation
Synthetie Data Pool
Mask Pool (
noise
Prompt Pool (p
(2.b) Prompt Condition
Figure 2: Workflow of MaskFactory. In the first stage, we generate new masks by applying rigid and non-rigid editing to the existing ground truth masks. In the second stage, we use the generated masks and their corresponding extracted Canny edges as conditions, along with a prompt representing the category, to generate RGB images. This process forms paired data for our generative model.
We propose a two-step method that synthesizes high-quality and diverse object masks via masking editing and generates corresponding high-resolution images using multi-conditional control generation method.
Experimental results on the DIS5K dataset demonstrate the superior performance of MaskFactory, compared to existing dataset synthesizing methods.
2
Related work
Synthetic data. Synthetic data has garnered significant attention in various machine learning and computer vision tasks, such natural language processing [16), object detection [17), and image segmentation [10] For instance, Lin et al. [17] demonstrated that synthetic data can enhance performance in object detection, especially in scenarios with limited access real-world data. Generative adversarial networks (GANs) and variational autoencoders (VAEs) are two popular deep learning- bbsed methods for generating synthetic data [8] Recent studies have focused improving the quality and diversity of synthetic data [19 and exploring its use in few-shot learning 20]. Additionally, synthetic data has been shown to enhance the interpretability and explainability of (21). Benefiting from recent ad advancmeents in diffusion models [12] (22).
machine learning models
methods such as DatasetDM 9 and Dataset Diffusion [10] can generate high-quality synthetic data for various computer vision tasks. However, these methods may introduce additional errors due the inclusion of pseudo-labels. These errors arise because pseudo- labels can be noisy and inaccurate, leading to suboptimal training data. Although some research (23 employs controlnet
control schemes [24] to mitigate these issues, the generated synthetic data may still suffer from noise and errors, making it unsuitable for the DIS task. Therefore, in this paper, we focus on generating high-quality synthetic data for the DIS task, where the generated image-mask pairs need be highly precise and accurate.
3

---

Dichotomous image segmentation. DIS has seen substantial progress with the advent of high- resolution imaging technologies. Representative approaches include the intermediate supervision strategy in IS-Net [2], the frequency prior method [ and the unite-divide-unie strategy by [25] However, these methods have enhanced segmentation accuracy but often struggle to capture extremely fine details. Thus, recent progressive refinement strategies, e.g., BASNet (26), emphasize the importance auxiliary information such as gradient maps and multi-scale inputs. These methods propose to gradient features and ground truth supervision to enhance the learning of weak features in complex regions. Then, BiRefNet ( maintains high-resolution inputs and employs a bilateral reference framework to better capture intricate details. The latest works, such as the multi-view aggregation network by Yu et al. [27] and the interactive segmentation approach by Liu et al. (28), further advance the field by integrating diverse prompts and enhancing feature representation for high-quality segmentation.
3 Method
3.1 Overview of MaskFactory
Current image segmentation methods are significantly constrained by their dependence on limited manually annotated data, which hampers both performance and generalization. To mitigate this, pseudo-label generation is often employed to augment training datasets. However, the context of DIS, these methods frequently introduce artifacts that degrade segmentation quality, Additionally, existing image editing techniques often fail to preserve the topological structure of binary masks, resulting in discontinuities and overlaps within target regions.
To address these challenges, we introduce the MaskFactory framework, designed to generate large number of high-quality synthetic image-mask pairs G {(o ..."" ) from an original dataset D = {1E 1 } This approach aims to enhance the performance of DIS models. As illustrated in Figure 2 our framework comprises two main stages: mask editing and image generation.
In the mask editing stage, source masks from the original dataset are transformed using both rigid and non-rigid editing methods, resulting in set of high precision synthetic masks GmD {9 fi=1- Rigid mask editing generates synthetic masks from various perspectives, while non-rigid mask editing employs
adversarial training mechanism edit masks according to semantic prompts while retaining the structural integrity of the source masks.
In the image generation stage, a multi-conditional control generation method is utilized to produce realistic RGB images G" = [of j that correspond precisely to the synthetic masks, using the latter as conditioning constraints.
3.2 Mask Editing Stage
3.2.1 Rigid Mask Editing
Rigid mask editing aims to preserve detailed information from the source masks through rigid transformations. We leverage the Zero123 [29] method, which employs a viewpoint-condition diffusion model to manipulate masks' perspectives. Given the relative camera rotation and translation T; for the desired viewpoint, the synthesizes a new mask g.T based on the source mask I," such that 9," = (I Tr), where I.tl is the inverted image of the source mask 1," to ensure the main component non-zero.
3.2.2 Non-Rigid Mask Editing
Non-rigid mask editing, inspired by MasaCtrl [30], a critical component of MaskFactory. Unlike MasaCtrl, which directly manipulates the source mask ITTh using a textual prompt P; to generate synthetic mask 9."h we introduce topology- topology-preserving adversarial training mechanism to mitigate artifacts and structural degradation in binary mask editing. The textual prompt P, is derived from a pool of prompts {p(T) that are generated using GPT-4 based on the original images. These prompts provide detailed descriptions that guide the mask editing process. This module consists of a generator Go and discriminator Dz. The generator transforms noise into a synthetic mask g," under the guidance of textual prompt P. and the source mask I," A mutual attention mechanism aligns the

---

query features Q; of 9,"h with the key and value features K, V. of 1,", ensuring consistency during editing. To avoid foreground-bec preground-backgroum background confusion, a mask M extracted from the cross-attention maps to guide the model's focus.
Hopology-Preserving Adversarial Training. To maintain the structural information of the source mask, we first extract an edge map E, = E(TT)) using an edge detection operator &, obtaining key points V {wg 1 We then construct a graph T = (V,&s) based these key points. The discriminator D& performs adversarial training on the structural graphs To and T, of the synthetic mask 9," and the source mask, respectively, ensuring topologiaal consistency.
The training objective of the discriminator DΦ is to maximize:
max ET, PTs) D& (T.T)) (Ta (T,) (og) - Dz(Tg)));
where Paata (T,) and Pgen (Tg) represent the distributions of the structural graphs of the source masks and the synthetic masks, respectively. Conversely, the training objective of the generator Go is to minimize the discriminative power of the discriminator:
min TT [log(1 (T,))
(2)
Through nopology-preserving adversarial training, the non-rigid editing module effectively retains the structural information from the source masks during the editing process, generating high-quality, artifact-free synthetic masks.
The overall loss function Lioal for non-rigid mask editing encompasses the adversarial loss LGAN, the content loss Leontent, and the structure preservation loss Laructrr:
Livill LGAN + Leuntent + >2
where A and 22 are balancing factors.
The adversarial loss LGAN is defined as:
LGAN (T.) flog (T.T))) + (T, (T))))
The content loss Leonteut measures the semantic consistency between the synthetic mask and the textual prompt:
Leumtent 11 17"1l1, where 9.th is the synthetic mask and 1.th is the source mask.
The structure preservation loss Lamuture evaluates the difference between the structural graphs of the synthetic mask and the source mask:
Lstructure
ITg -Tlli.
(6)
These components ensure that the editing process maintains the structural and semantic consistency of the source masks.
3.3 Image Generation Stage
Following the mask editing stage, we obtain synthetic mask pool {. detailed synthetic masks generated through the aforementioned transformations. In the subsequent image generation stage, we accurately generate corresponding RGB images {o] ].M Jizo for the masks in the synthetic mask pool using a multi-condition control generation method. The primary segmentation of the RGB images aligns with the corresponding synthetic masks. Inspired by ControlNet 24,, we introduce multi-condition control generation method to achieve precise RGB image generation. This method simultaneously injects the segmentation condition oj and the canny condition c/' to steer the denoising process of the random Gaussian noise, ensuring the generated RGB images {o] ) Ji correspond accurately to the synthetic masks (9." M
Jizo comprising finely
T-
Since the synthetic mask itself serves as the segmentation condition ef, when a synthetic mask 9f" is provided, we only need to extract the canny condition of using the canny operator:
9 c²' Cannylg.")
5

---

bed
A bed frame made out of wooden pallets.
fan
fan is sitting a window sill.
bicvcle
bicycle with basket on the front of it.
gate
An iron gate is in front of house.
bench
A wooden bench is sitting on a sidewalk.
(a) Raw Image
(b) MaskFactory
(c) DatasetDM 9
(d) DatasetDiffusion [0]]
Figure 3: compared with baseline methods
After obtaining the canny and segmentation conditions, input them into block Ba, which consists of a set of neural layers. Finally, these conditions are injected into the pre-trained diffusion model Mg, controlling the noise denoising process to generate the corresponding RGB image. We refer readers to [24] for more details.
gf = (o(c))) Bo(c))
(8)
Experiment and Results
4.1 Dataset & Metrics
We conduct our experiments on the DIS5K dataset, which comprises 5,479 high-resolution images featuring camouflaged, salient, or meticulous objects in various backgrounds. The DIS5K dataset is divided into three subsets: DIS-TR (3,000 images) for training, DIS- VD (470 images) for validation, and DIS-TE (2,000 images) for testing. For data augmentation, we utilize the mask portion of the training subset (DIS-TR).
To evaluate our models, we employ diverse set of metrics to ensure comprehensive performance assessment. These metrics include max F, (1], which balances precision and recall, providing harmonic mean that is indicative of overall accuracy; Fy (32), a weighted F-measure that compensates for class imbalances, with values ranging from O to 1, where higher values denote superior perfor- mance; M (Mean Absolute Brr) which calculates the average absolute difference between the predicted and ground truth masks, with lower values signifying better accuracy; S., (34), structural similarity measure that evaluates the preservation of significant structures within the image, with values closer to indicating better performance; and E (35) an enhanced measure that considers both pixel-level and image-level information for more holistic evaluation, where higher values represent better performance. Collectively, these metries provide robust framework for assessing the effectiveness and reliability of our segmentation models.

---

Table I:
Comparison of experimental results with different generation methods, illustrating the impact of using various numbers of generated images on segmentation task performance.
DatasetDM
Dataset Diffusion
[0]
MaskFactory
Dataset IS-Net
2500
5000
7500
10000
2500
5000
7500
10000
2500
5000
7500
10000
DIS VD
marFi
0.791
0.792
0.791
0.788
0.785
0.793
0.780
0.771
0.767
10.831
0.833
0.834
0.835
Py
0.717
0.720
0.719
0.712
0.710
0.726
0.726
0.716
0.710
0.725
0.754
0.757
0.759
M
0.074
0.076
0.076
0.075
0.077
0.074
0.077
0.076
0.077
0.073
0.073
0.071
0.072
S 1
0.813
0.814
0,810
0.807
0,805
0.826
0.821
0.804
0.790
0.832
0.855
0.860
0.866
0.856
0.869
0.864
0.864
0.860
0.868
0.859
0.852
0.838
0.880
0.911
0.914
0.923
DIS-TEI TEI
marFF t
0.740
0.744
0.744
0.740
0.736
00.741
0.727
0.721
0.719
00.776
0.777
0.779
0.784
P
0.662
0,670
0.668
0.663
0.659
0.675
0.669
0.667
0,654
0.677
0.700
0.703
0.705
M
0.074
0.075
0.077
0.075
0.076
0.076
0.074
0.077
0.078
0.073
0.071
0.071
0.073
Sa 0.787
0.791
0.786
0.781
0.778
0.790
0.776
0.760
0.754
0.803
0.822
0.826
0.829
0.820
0.825
0.819
0.813
0.810
0.827
0.816
0.802
0.798
0.853
0.866
0.868
0.875
DIS DIS-TE2 TE2
maLE: F1 t 0.799
0.810
0.807
0.801
0.793
00.01
0.799
0.782
0.776
00,008
0.814
0.819
0.822
P 1 0.728
0.741
0.736
0.736
0.734
0.739
0.737
0.721
0.712
0.739
0.764
0.765
0.772
M L. 0.070
0.072
0.070
0.071
0.071
0.072
0.071
0.073
0.071
0.068
0.067
0.068
0.067
S. f 0.826
0.833
0.828
0.823
0.821
0.833
0.832
0.811
0.795
0.849
0.855
0.861
0.865
0.858 MM
0.870
0.868
0.866
0.865
0.861
0,860
0.850
0.834
0.868
0.895
0.901
0.903
miaFi
0.830
0.846
0.845
0.839
0.833
0.838
0.834
0.831
0.812
0.841
0.850
0.867
0.870
DIS
0.758
0.770
0.766
0.763
0.757
0.769
0.752
0.733
0.728
0.761
0.782
0.783
0.785
M
0.064
0.066
0.066
0.065
0.066
0.066
0.065
0.066
0.065
0.062
0.062
0.062
0.063
Sa f
0.836
0.848
0.844
0.839
0.832
0.844
0.837
0.823
0.819
0.842
0.859
0.866
0.878
B f 0.883
0.894
0.887
0.883
0.878
0.900
0.896
0.878
0.858
0.911
0.915
0.918
0.926
maaFi f 0.827
0.833
0.826
0.820
0.816
0.830
0,818
0.804
0.801
0.856
0.876
0.876
0.879
DIS-TEL
P3 0.753
0.759
0.754
0.750
0.746
0.760
0.759
0.758
0.747
0.789
0.824
0.824
0.830
M
0.072
0.074
0.073
0.073
0.075
0.074
0.072
0.076
0.074
0.071
0.069
0.071
0.072
Sa
0.830
0.846
0.839
0.838
0,833
0,841
0.830
0.822
0.815
0,842
0.852
0.860
0.862
0.870
0.885
0.881
0.873
0.868
0.873
0.866
0.845
0.836
0.891
0.917
0.917
0.923
4.2 Impletement Details
Our image editing framework is implemented using PyTorch and trained on 8 NVIDIA GeForce RTX 3090 GPUs. We employ both non-rigid and rigid editing approaches to manipulate images in the DIS-TR dataset, utilizing spatial transformers and affine transformations to model non-rigid and rigid deformations, respectively. The hyperparameters used in our model are as follows: a batch size of 16, an image size of 512x512, 5 editing iterations, a learning rate of 0.001, weight decay of 0,0001, 1000 diffusion steps, and a diffusion step size of 0.1. The hyperparameters in Equ Jare set to A = 0.8 and 22= = 0.5.
We train the segmentation model using the DIS-TR subset of the DIS5K dataset, utilizing 2 NVIDIA GeForce RTX 3090 GPUs. The input size the network is 512x512, with learning rate of 0.0001 and a batch size of 48. The model is optimized using the Adam optimizer over a total of 800 epochs. The edited images generated by our model are combined with the original training set to form a new training set, enabling the model to learn from both original and edited images.
We evaluate model the DISSK datasets, using the test sets from each dataset. For com- prehensive comparison, we also reproduce two oftte-of-thearnt diffusion-based image generation methods, DatasetDM and Dataset Diffusion, using their default parameters. The generated images are compared with those from our approach, with the visualization of the results shown in Figure 3
4.3 Results
Results by Generated Image Count. We evaluate the performance of our proposed method, MaskFactory, against two state-of-the-art baselines, DatasetDM (9] and Dataset Diffusion [10]. on the DIS5k dataset. The DISSk dataset comprises five sub-datasets: DIS-VD, DIS-TEI, DIS-TE2, DIS-TE3, and DIS-TE4. We use the robust segmentation method IS-Net (2) as baseline for the DIS task. For the two baselines, DatasetDM and Dataset Diffusion, we use the default parameters provided in their ooen-source implementations.
7

---

Table Comparison of experimental results using different segmentation schemes, demonstrating
performance gains with fixed number of generated images.
IS-Net [2]
FP-DIS (36)
UDUN
[5]
BiRefNet
SAM-HQ (37)
Method
w/o Ours
w Ours
w/o Ours
w Ours
w/o Ours
w Ours
w/o Ours
w Ours
w/o Ours
w Ours
DIS-TEI
marFi f
0.740
0.784
0.784
0.805
0.784
0.799
0.866
0.882
0.897
0.905
M 1
0.074
0.073
0.060)
0.063
0.059
0.057
0.036
0.033
0.019
0.018
Sa f
0.787
0.829
0.821
0.859
0.817
0.830
0.889
0.900
0.907
0.911
0.820
0.875
0.855
0.885
0.846
0.849
0.915
0.916
0.943
0.949
DIS-TE2
mapFi
0.799
0.822
0.827
0.849
0.829
0.849
0,906
0,910
0.889
0.894
M +
0,070
0,067
0.059
0.061
0.058
0.055
0.031
0.029
0.029
0.030
Sa 0.823
0.865
0.845
0.862
0.843
0.866
0.913
0.921
0.883
0.889
0.858
0.903
0.889
0.893
0.886
0.894
0,947
0,957
0,928
0.937
DIS-TE3
mar f
0.830
0.870
0.868
0.911
0.865
0.888
0.920
0.937
0.851
0.853
M 1
0.064
0.063
0.049
0.051
0.050
0.047
0.029
0.027
0.045
0.043
Sa f
0.836
0.878
0.871
0.892
0.865
0.885
0.918
0.918
0.851
0.854
E
0.883
0.926
0.903
0.924
0.913
0.931
0.951
0.957
0.897
0.906
mazFi f
0.827
0.879
0.846
0.882
0,846
0.851
0,906
0,916
0.763
0.764
DIS-TER
M.
0,072
0,072
0,061
0.063
0.059
0.053
0.038
0.035
0.088
0.084
Sa
0.830
0.862
0.852
0.856
0.849
0.873
0.901
0.911
0.799
0.806
FM t
0.870
0.923
0.891
0.935
0.891
0.895
0.933
0.941
0.843
0.850
DIS-VD
man: t
0.791
0.835
0.823
0.851
0.823
0.847
0.897
0.905
0.842
0.847
M 4
0.074
0.072
0.062
0.065
0.059
0.058
0.036
0.033
0.045
0.044
Sa f
0,813
0.866
0.843
0.873
0.838
0.857
0.905
0.909
0.848
0.850
E
0.856
0.923
0.873
0.880
0.876
0.901
0.931
0.941
0.896
0.903
First, we identify the best-performing model on the DIS-VD validation set and then evaluate its performance on the other sub-datasets. The models are trained on the DIS-TR- dataset, which is augmented with generated datasets of varying sizes: 2500, 5000, 7500, and 1 10000 images.
The experimental results are presented in T Our proposed method, MaskFactory, consistently outperforms the baselines across all sub-datasets and evaluation metrics. As the number of generated images increases, the performance of MaskFactory improves, achieving the best results with 10000 generated images. Notably, MaskFactory attains the highest max Fi scores across all sub-datasets, with improvements ranging from 0.044 to 0.052 compared to the IS-Net baseline.
In contrast, while DatasetDM and Dataset Diffusion also show some performance gains, they encounter the issue of collapse when the generated data exceeds 5000 images. In these cases, the segmentation network's 's performance stagnates even degrades. For the DIS segmentation task, the use of pseudo-labels could introduce additional errors, leading to performance declines.
Furthermore, MaskFactory demonstrates superior performance in terms of the weighted F-measure (FF)), S-measure (Sa), and Enhanced- alignmmen Measure (E ). The Mean Absolute Error (M) is also consistently lower for MaskFactory compared to the baselines, indicating more accurate segmentation results.
Results by Segmentation Network. After achieving stable improvements in IS-Net's performance, we examined the generalizability of our approach by applying the same configuration to several state-of-the-arr segmentation networks on the DIS5K dataset. Each network was trained using the DIS-TR dataset augmented with 10,000 generated image pairs. The networks considered in this
study include FP-DIS (36]. UDUN 25 BiRefNet 1. and SAMHQ 37,, all implemented with their
default parameters.
The experimental results, shown in Table 2 demonstrate that our proposed method consistently enhances the performance of all evaluated networks across the five sub-datasets of DIS5K. Notably, significant improvements were observed in the maxF, and Ey metrics. For instance, on the DIS- TEI sub-dataset, our method increased the max/, score of IS-Net from 0.740 to 0.784, FP-DIS from 0.784 to 0.805, UDUN from 0.784 to 0.799, BiRefNet from 0.866 to 0,882, and SAMHQ from 0.897 to 0.905.
8

---

Mask Faciury (Rigid)
Mask Factury (Non Rigid)
Mask Factory (Mix)
100
UMAPS
Murk tuskry (hom-- Dalaett
GT
Pactayy (lagh
Mark Faakey dito DesucDMN
DasseDM
-25
-25
2.0
2.5
500
7.5
10.0
125
-25
0.0
2.5
5.0
7.5
33,0
12.5
15.0
18
12
14
UMAP1
UMAP
UMXP
(a) Differences in UMAP distribution of generated mask images
Mask Factory (Rigid)
Mask Faclery (Non-Rigid)
Mask Factory (Mix)
150
12
Mark taney (Maa
12
10
1000
10
UMAP2
25
Misk Puatrry Stipko
Mark Faicery (honelipp))
-2.
-2.5
0.
2.5
<5
15
100
12.5
-25
00
2.1
5.0
7.5
1000
12.5
10
12
UMAP
UMAP
1/MAP 1
(b) Differences in UMAP distribution of generated corresponding RGB images
Figure 4: UMAP Distribution Differences
Additionally, the Mean Absolute Error (M) decreased for all networks when applying our method, indicating more accurate segmentation results. The S-measure (S.) also consistently improved across all sub-datasets and networks, highlighting the effectiveness of our approach in capturing structural similarity.
On the DIS-VD sub-dataset, used for validation, method boosted the mazFi score of IS-Net from 0.791 to 0.835, FP-DIS from 0.823 to 0.851, UDUN from 0.823 to 0.847, BiRefNet from 0.897 to 0.905, and SAMHQ from 0.842 to 0.847. These findings underscore the generalizability of our approach, as it enhances the performance of diverse segmentation networks without requiring any network-specific modifications.
Visual Results. As shown in Figure 5 (Appendix), our approach achieves precise results, comprising both rigid and non-rigid transformations. The non-rigid transfor- mations, illustrated in columns (b) and (c), enable shape editing, such as removing a table corner or merging two backpacks into one. In contrast, the rigid transformations, demonstrated in columns (d), (e), and (f), primarily in- volve viewpoint changes, showcasing the original mask
Table 3: Similarity with original dataset.
Generate Type CLIP
UMAP
DatasetDM 9 0.6683
0.7017
Dataset Diffusion [0]
0.6217
0.6349
FaskFactoryrriiiddl
0.8791
0.8961
MaskFactory(monn-ige
0.9147
0.9346
MaskFactoryyAll)
0.8967 0.9103
rotated in 3D space. Notably, our method effectively preserves the topological structure information of the original image, including the holes on the chair back. This allows for the low-cost generation of nigh-precision diverse data pairs. To investigate differences between generated and real images, analyzed image and mask distributions. Specifically, we used the CLIP model to extract features from 300 real images, as well images and masks generated by DatasetDM, DatasetDiffusion, and MaskFactory. We then applied UMAP 39 for dimensionality reduction on these 300 features. The feature distributions of masks and images visualized in Figures 4 a) and (b), respectively.
In the mask editing domain, MaskFactory demonstrates superior mask fidelity compared to other generation methods, primarily due to the incorporation of topological consistency constraints. This results in a feature distribution that closely aligns with that of real images. Conversely, for RGB images, the prior from diffusion VAE introduces larger disparity between the generated and real

---

image distributions. However, the distribution generated by MaskFactory shows a greater overlap with the real image distribution compared the other two methods.
Furthermore, we quantified the differences using cosine similarity, as presented in Table 3 The results indicate that our method achieves the closest distribution real images, further validating the effectiveness of MaskFactory in generating realistic masks and images.
5
Discussion
5.1
Ablation Study
Mask Generation Type Ablation. In our study, im- plemented mask rigid editing, non-rigid editing, and mixed editing- Seating-eacch leveraging our novel mask control technol- ogy tailored for specific application scenarios. Rigid edit- ing designed for scenarios requiring precise geometric adjustments, primarily focusing on viewpoint and scale transformations. Non-rigid editing caters to applications
Table 4: Ablations on generation types.
Type maz f
S. f F t
Rigid 0.768
0.074
0.807 0.867
Non-Rigid
0.771
0.074
0.796 0.858
Mix 0.784 0.073 0.829 0.875
needing high adaptability, handling topologically consistent deformations and complex, dynamic image edits. Mixed editing combines the advantages of both approaches, offering comprehensive solution. We further evaluated the performance gains of each editing strategy. Our experimental results, as shown in Table 4
From the table, we observe that mixed editing achieves the highest performance across most metrics. Specifically, it achieves the highest marFi score and S,,, indicating superior structural fidelity and segmentation quality. The slight improvement in A and Ex further underscores the versatility and effectiveness of mixed editing in creating diverse and realistic image-mask combinations.
Loss Function Ablation. We introduce content and structure losses into model. The Discriminative Loss, implemented via discriminator, evaluates the differences between gen- erated and real images, aiming to enhance the realism and quality of the outputs. The Edge Constraint Loss focuses on maintaining edge coherence during image editing, which critical for preserving detailed structural information. We conducted ablation experiments to evaluate the impact of each loss function. The experimental results, shown in Table 5
From the table, we observe that the combination of all three loss functions (LGAN, Loontent, and ) achieves highest maz score, indicating the best performance in terms of structural fidelity and realism.
Table 5: Ablations on loss functions.
Loss Function
maaFi f
4
LEAN Lecatens Lesructure
0.778
0.073
5.2 Limitation
0.745
0.075
0.751
0.074
Despite the favorable outcomes achieved by our method, it still encounters significant issues. Although we exper- imented with different conditions, the results are shown in T ControlNet sometimes produces unnatural images with stark foreground-backgrounnd distinc- tions, necessitating additional harmonization. Complex scenarios can yield unrealistic elements, such as improperly positioned objects. Additionally, our method relies on pre-annotated image-mask pairs, limiting its ability to generate data autonomously and requiring high-quality initial annotations.
0.784
0.073
5.3 Conclusion
This paper introduces MaskFactory, a novel two-stage approach for generating high-quality synthetic datasets for DIS tasks. By combining rigid and non-rigid mask editing techniques and using multi-conditional control for image generation, MaskFactory produces diverse and precise synthetic image-mask pairs, significantly reducing dataset preparation time and costs. Experiments the DISSK benchmark demonstrate the superior performance of MaskFactory compared to existing methods in terms of quality and efficiency.
Table Ablations on conditions.
Mask Prompt Canny mazFi t
M 4
0.778
0.075
0.782
0.073
0.764
0.080
0.784
0.073
10

---

Acknowledgements
We thank Yuyan Huang. Hong Liu, and Wenjun Ji for fruitful discussions during the course of this project. Haoqian Qian and Xiaogang Jin were supported by Key R&D Program of Zhejiang (No.2024C01069) Deng-Ping Fan was supported by NSFC (No.62476143).
References
[1] Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolution dichotomous image segmentation. CAAI AIR, 3:9150038, 2024.
[2] Xucbin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. In ECCV, 2022.
[3] Lian Liu, Han Zhou, Jiongquan Chen, Sijing Liu, Wenlong Shi, Dong Ni, Deng-Ping Fan, and Xin Yang. Instructive feature enhancement for dichotomous medical image segmentation. MICCA, 2023.
[4] Yinda Chen, Che Liu, Xiaoyu Liu, Rossella Arcucci, and Zhiwei Xiong. Bimev-r: A landmark dataset for text-image retrieval. In MICCAI, 2024.
[5] Yinda Chen, Wei Huang, Xiaoyu Liu, Shiyu Deng, Qi Chen, and Zhiwei Xiong. Learning multiscale consistency for self-supervised electron microscopy instance segmentation. In ICASSP, 2024.
[6] Chenfeng Xu, Bichen Wu, Zining Wang. Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In ECCV, 2020.
[7] Xiaoyu Liu, Miaomiao Cai, Yinda Chen, Yueyi Zhang, Te Shi, Ruobing Zhang, Xuejin Chen, and Zhiwei Xiong. Cross-dimension affinity distillation for 3d em neuron segmentation. In CVPR, 2024.
[8] Yinda Chen, Wei Huang, Shenglong Zhou, Qi Chen, and Zhiwei Xiong. Self-supervised neuron segmenta- tion with multi-agent reinforcement learning. In IJCAI, 2023.
[9] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei Hc, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. NewrIPS, 2023.
[10] Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-hased synthetic data generation for pixel-level semantic segmentation. NeurIPS, 2023.
[11] Yinda Chen, Che Liu, Wei Huang. Sibo Cheng, Rossella Arcucci, and Zhiwei Xiong. Generative guxded 3d vision- language pretraining for unified medical image segmentation. arXiv preprint 14X1::2306.0481, 2023.
[12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffuusio models. In CVPR, 2022.
[13] Aysim Toker, Marvin Eisenberger, Daniel Cremers, and Laura Tai-T Satsynth: Augmenting image- mask pairs through diffusion models for acrial semantic segmentation. In CVPR, 2024.
[14] Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models. NeurIPS, 2023.
[15] Yinda Chen, Haoyuan Shi, Xiaoyu Liu, Te Shi, Ruobing Zhang, Dong Liu, Zhiwei Xiong, and Feng Wu. Tokenunify: Scalable autoregressive visual pre- pre-training with mixture token prediction. arXiv preprint arXiv:2405.168.4 16847, 2024.
[16] Dingquan Wang and Jason Eisner. Synthetic data made to order: The case of parsing. In EMNLP, 2018.
[17] Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao. Explore the power of synthetic data on few-shot object detection. CVPR, 2023.
[18] Alvaro Figucira and Bruno Vaz. Survey synthetic data generation, evaluation methods and gans, Math., 10(15):2733, 2022.
[19] Aman Kishore, Tae Eun Choe, Junghyun Kwon, Minwoo Park, Pengfei Hao, and Akshita Mitell Synthetic data generation using imitation training. In CVPR, 2021.
[20] Tung Nguyen, Sudhanshu Agrawal, and Aditya Grover. Expt: Synthetic pretraining for few-shot experi- mental design. NewrIPS, 2023.

---

[21] Yang Liu, Sujay Khandagale, Colin White, and Willie Neiswanger. Synthetic benchmarks for scientific research in explainable machine learning. In NeurlPS Dalaxel Track, 2021.
[22] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint xxXiv:2307.01952.222 2023.
[23] Qi Chen, Xiaoxi Chen, Haorui Song, Zhiwei Xiong, Alan Yuille, Chen Wei, and Zongwei Zhou. Towards generalizable tumor synthesis. CVPR, 2024.
[24] Lvmin Zhang, Anyi Rao. and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In ICCV, 2023.
[25] Jialun Pei, Zhangjun Zhou, Yueming Jin, He Tang, and Pheng-Ann Heng. Unite-divide-unite: Joint boosting trunk and structure for high-aceuracy dichotomous image segmentation. In ACM MM, 2023.
[26] Xuebin Qin, Zichen Zhang, Chenyang Huang. Chao Gao, Masood Dehghan, and Martin Jagersand. Basnet: Boundary-aware salient object detection. CVPR, 2019.
[27] Qian Yu, Xiaoqi Zhao, Youwei Pang, Lihe Zhang, and Huchuan Lu. Multi-view aggregation network for dichotomous image segmentation. CVPR, 2024.
[28] Qin Liu, Jaemin Cho, Mohit Bansal, and Mare Niethammer. Rethinking interactive image segmentation with low latency high quality and diverse prompts. CVPR, 2024.
[29] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. CVPR, 2023.
[30] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yingiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. CVPR, 2023.
[31] Radhakrishna Achanta, Sheila Hemami, Francisco Estrada, and Sabine Susstrunk. Frequency-tune salient region detection. CVPR, 2009.
[32] Ran Margolin, Lihi Zelnik-Manor, and Ayellct Tal. How evaluate foreground maps? In CVPR, 2014.
[33] Federico Perazzi, Philipp Krähenbühl, Yacl Pritch, and Alexander Hornung. Saliency filters: Contrast based filtering for salient region detection. In CVPR, 2012.
[34] Deng-Pine Fan. Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure- A new way evaluate foreground maps. In ICCV, 2017.
[35] Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. aahanced-alignmeee measure for binary foreground map evaluation. IJCAI, 2018
[36] Yan Zhou, Bo Dong, Yuanfeng Wu, Wentao Zhu, Geng Chen, and Yanning Zhang. Dichotomous image segmentation with frequency priors. In IJCAI, 2023.
[37] Lei Ke, Mingqiao Ye, Martin Danelljan, Yu- Yu-Wing Tai, Chi Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality. NeurlPS. 36, 2023.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[39] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
2018
[40] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
12

---

Appendix
A Pseudocode for the MaskFactory Algorithm
MaskFactory is two-stage approach for generating high-quality synthetic datasets for DIS tasks. In the first stage, existing ground truth masks undergo rigid and non-rigid editing to generate diverse synthetic masks. Rigid editing uses geometric priors from diffusion models for precise viewpoint transformations, while non-rigid editing employs adversarial training and self-attention mechanisms for complex shape modifications while preserving topology.
In the second stage, the generated masks and their corresponding Canny edges serve as conditions, along with category prompts, to guide the generation of high-resolution RGB images using multi- conditional control generation method. This process ensures consistency between the generated images and masks while enhancing dataset realism and diversity. Our pseudocode visible Algorithm
Algorithm 1: MaskFactory Algorithm
Input :D 1
Original dataset
Output: 9f ) Synthetic dataset
Stage 1: Mask Editing
for i <- 1 to N do
Step 1.1: Rigid Mask Editing
r." +- Invert ( // Invert source mask
oeth < -o TT))
Apply viewpoint transformation
Step 1.2: Non-Rigid Mask Editing
E, + E(I,") // Extract edge map from source mask
{ // Obtain key points from edge map
Tg - (V,E,) Construct source mask structural graph
10
gth + I.. // Generate synthetic mask
T. = (V,E,)/ Construct synthetic mask structural graph
12
LGAN ET,xpsan (T, flog D& (T.)) ET, ET, (T, [log (T,)))
Lcontent + |
1T
14
Estructure + T T.ll
15
Liotal & LGAN + X Ccontent + AyCCstrcture
16
Update Go and D& to minimize Ltotal
Stage 2: Image Generation
18
for i +-1 to M do
c 9 // Segmentation condition
c + Canny // Canny condition
21
2 ~ N(O0 1) // Sample Gaussian noise
22
of - Mo(z, (ct), (e())) // Generate RGB image
return G ((g] 9 M
B Dataset Details
We conduct our experiments on the DIS5K dataset, which comprises 5,479 high-resolutinn images featuring camouflaged, salient, meticulous objects in various backgrounds. The DIS5K dataset is divided into three subsets: DIS-TR (3,000 images) for training, DIS- VD (470 images) for validation, and DIS-TE (2,000 images) for testing. For data augmentation, utilize the mask portion of the training subset (DIS-TR).
To evaluate our models, we employ diverse set of metrics to ensure comprehensive performance assessment. These metrics include max F,, which balances precision and recall, providing a harmonic mean that is indicative of overall accuracy; Fg, weighted F-measure that compensates for class
13

---

imbalances, with values ranging from O to 1, where higher values denote superior performance; M (Mean Absolute Error), which calculates the average absolute difference between the predicted and ground truth masks, with lower values signifying better accuracy: Sa, a structural similarity measure that evaluates the preservation of significant structures within the image, with values closer to indicating better performance; and Ex/- an enhanced measure that considers both pixel-level and image-level information for more holistic evaluation, where higher values represent better performance. Collectively, these metrics provide a robust framework for assessing the effectiveness and reliability of our segmentation models.
C Mathematical Details
C.1 Diffusion-Based Image Generation
The diffusion model employed MaskFactory for image generation follows the formulation in- troduced by Ho et al. [40]. Given a data distribution zo (xo)) the forward diffusion process is defined as a Markov chain that gradually adds Gaussian noise to the data:
(( = Na(ri / 3,
where E (0, is variance schedule. The reverse process is learned by neural network eg that predicts the noise added at each step:
pe(x 1x)) = =N):=N((xx-1 (Z0ss1;HH(a oi)
(10)
where no (z(,,)t)
24 -- Po co( and a, = 1 - Br, ā. = IT.= 0g..
voa The objective is maximize the variational lower bound:
po po (xte)
C Ey(so) Be or
Σ
log
(11)
g( No)
C.2 Topeeogg Adversarial Training
The adversarial training in MaskFactory involves a generator Go and discriminator Dø. The generator aims minimize the adversarial loss:
LGAN (G) = (T, log(1 - D (Ty))),
(12)
while the discriminator tries to maximize the adversarial loss:
LGAN D) = ET, (T..) (log DT(T.)) BT, Prm (o (log( (T(T)))
(13)
The generator and discriminator are updated alternately to reach an equilibrium.
C.3 Structural Graph Construction
To preserve the topological structure of the source masks during editing, MaskFactory constructs structural graphs T, and T, for the source and synthetic masks, respectively. The structural graph
T = (V, E) consists of set of vertices V = {1y} 1 representing key points and a set of edges E representing their connectivity.
The structure preservation loss is defined as the LI distance between the structural graphs:
S = 1|T, -Tall=
Σ
[T,() - (11
(14)
(UEE
14

---

where To (i. and (i,) denote the edge weights between vertices and j in the synthetic and source mask structural graphs, respectively.
By minimizing the structure preservation loss along with the adversarial and content losses, MaskFactory ensures that the edited masks maintain the topological structure of the source masks.
D Visualization of the results using two editing methods from MaskFactory.
In this section, we demonstrate MaskFactory's ability to edit masks for common and fine-grained objects.
D.1 Visualization of Common Object Mask Editing
We selected common household items such as tables, chairs, bags, and musical instruments for editing using MaskFactory. Non-rigid edits were performed with different prompts, and rigid edits were performed from different viewpoints. The editing results and corresponding RGB image generations are shown Figure MaskFactory exhibits strong topological structure preservation and produces diverse editing outcomes, as demonstrated by the variety of modifications made to both rigid and non-rigid objects.
D.2 Visualization of Fine-Grained Object Mask Editing
We selected fine-grained objects from MaskFactory's generated masks, such as ornate European chandeliers, iron gates, birdcages, and seahorses, showcase MaskFactory's detailed editing capabil- ities. Even with complex geometries, MaskFactory can perform prroogyypreserving edits without losing the original mask's s semantic information. These intricate structures play a crucial role in segmentation metrics. As illustrated in Figure O the visual results demonstrate the model's ability to handle fine fine-graine grained details while maintaining the integrity of the original mask's structure.
D.3 Visualization Results with Canny Constraints
After incorporating Canny edge detection as constraint, the visual results of MaskFactory show a significant improvement in boundary precision. The Canny edges effectively guide the generation process, resulting in images with clearer and more accurate boundary details, avoiding vague or ambiguous transition areas. The visualizations demonstrate that the Canny edges not only better constrain the contours of the generated images but also enhance the overall fidelity and visual quality of the output. Compared models without edge constraints, our approach produces more detailed and structurally coherent images, as shown in Figure 7
D.4 Topological Structure Visualization
We performed topological structure visualizations selected samples assess the model's ability to preserve topology during the editing process. These visualizations clearly show how our method retains the topological structure of the original data while allowing for effective manipulation when necessary. Whether dealing with complex geometric shapes or subtle structural modifications, the topological visualizations illustrate that our approach reliably preserves geometric consistency and topological features throughout the editing process, as demonstrated in Figure g
15

---

(a) image
(b) rigid
(c) rigid 2 (d) non-rigid
(e) non-rigid (f) non-rigid
Figure 5: Visual results common object mask editing. The model demonstrates strong topological
structure preservation and diverse editing outcomes with both rigid and non-rigid edits.
16

---

Original Source
Rigid Mask Editing
Non-rigid Mask Editing
Real Image
Source Mask
Synthetic Image
Synthetic mask
Synthetic Image
Synthetic mask
Figure 6: Visual results of fine-gra fine-grained object mask editing. The model successfully edits complex structures without compromising the original mask's semantic information.
17

---

Sythetic mask(depll
Generated image(w/o canny)
Generated image(w w canny)
Figure 7: Canny condition visual results. The generated images show improved boundary precision and better structural coherence.
Raw
Topology
MaskFactory (rigid)
Topology
Figure 8: Topological Structure Visualization. The visualizations demonstrate the model's ability to maintain and manipulate topology during the editing process.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper presents method for generating and editing images with high- precision masks, and announces the open-sourcing DIS generation dataset.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 5.2
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

This paper's ss experiments and methods do not introduce new theories; the theoretical foundations used are detailed in Section
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer Section
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
This paper includes a link to the anonymized code in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips wwwp//bllcggguiieeeeeeeeeeeeeeee /wweddee/ooooebbmicccce
wwiiiisssopoiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Please refer to Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: h/tp:::////uis/.eeeeeeeee https///cppp...ttt.ccccc........
Answer: [Yes]
Justification: please check in our project page.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: please check in our project page.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: please check in our project page.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: please check in our project page.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer:
[NA]
Justification:
please check in our project page.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: please check in our project page.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: please check in our project page.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "EMloKkfzuw", "reviewer": "Reviewer_nBLG", "summary": "This paper introduces MaskFactory, a novel approach aimed at addressing the challenges of generating high-quality synthetic datasets for Dichotomous Image Segmentation (DIS) tasks. \nThe authors tackle the challenges by leveraging a blend of rigid and non-rigid editing techniques to generate accurate synthetic masks. Rigid editing utilizes geometric priors from diffusion models for precise viewpoint transformations, while non-rigid editing incorporates adversarial training and mutual self-attention mechanisms for complex modifications. With the accurate generated segmentation masks, a multi-conditional control generation method is then employed to create high-resolution images. The efficacy of MaskFactory is demonstrated through experiments on the widely-used DIS5K dataset, showing superior quality and efficiency in dataset generation compared to existing methods, thus significantly reducing preparation time and costs.", "questions": "1. Can rigid mask editing and non-rigid mask editing be integrated into one framework, rather than just being simply combined?\n2. Can the improvement brought by the  L_structure loss be intuitively reflected in the ablation study?", "limitations": "The method faces challenges including the occasional production of unnatural images with stark foreground-background distinctions and inaccuracies in complex scenarios. Additionally, it relies on pre-annotated image-mask pairs, limiting autonomous data generation and requiring high-quality initial annotations. However, these issues are acceptable as there is extensive research aimed at resolving the inherent problems with ControlNet, which does not detract from the effectiveness of the method presented in this paper. Furthermore, the authors could explore using pseudo-labels for mask correction and generation to reduce dependency on pre-annotated image-mask pairs.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The scientific problem discussed here is worth researching, and there is still much room for exploration in this area. If the synthetic technology can be leveraged to substantially expand the simulated image-mask pairs based on existing data, it would benefit model training, especially with the current large multimodal models.\n2. This paper takes into consideration both rigid mask editing and non-rigid mask editing, combining them to jointly contribute to mask generation. Additionally, during the non-rigid mask editing process, a topology preserving loss is introduced to maintain the topological structure of the original mask.\n3. The experimental results are very encouraging. Compared to previous methods, the data distribution generated in this paper is closer to the real data. As the amount of synthetic data involved in training increases, the model's segmentation capability also increases proportionally.", "weaknesses": "1. The writing of this paper is too brief. It does not clearly show how the authors specifically carry out the mask editing part, or what unique designs they have implemented in the editing process.\n2. The paper lacks novelty in certain aspects. For the rigid mask editing part, using viewpoint changes to generate masks is a very common practice. The network architecture and synthesis method in the non-rigid mask editing part are completely identical to those used in MaskFactory. Although an additional topology preserving loss has been added, this loss is hard to regard as an independent innovation. Furthermore, the ablation study does not present results using only L_GAN and L_content, making it difficult to intuitively understand the significant impact of L_structure.\n3. Some of the table headers are too narrow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' thorough evaluation and constructive feedback. We are grateful for the opportunity to clarify and expand on our work.\n\n### Q1: Can you elaborate on the methodology, especially regarding mask editing and unique designs?\n\nWe will expand the methodology section in the revised manuscript to include:\n\n1. **Rigid Editing**: We will provide a comprehensive explanation of our rigid editing process using Zero123. As stated in Section 3.2.1: \"We leverage the Zero123 method, which employs a viewpoint-conditioned diffusion model ψθ to manipulate masks' perspectives.\" We will elaborate on how this allows for precise geometric adjustments, focusing on viewpoint and scale transformations.\n\n2. **Non-rigid Editing**: We will offer a more detailed description of our non-rigid editing process, including:\n   - Topology-preserving adversarial training\n   - Mutual attention mechanism\n   - Foreground-background focus guidance\n\n   As mentioned in Section 3.2.2: \"We introduce a topology-preserving adversarial training mechanism to mitigate artifacts and structural degradation in binary mask editing.\" We will expand on how this mechanism works and its importance in maintaining structural integrity.\n\n3. **Multi-conditional Control Generation**: We will provide a more in-depth explanation of our image generation stage, as described in Section 3.3: \"We introduce a multi-condition control generation method to achieve precise RGB image generation.\"\n\nWe will also include architectural diagrams and pseudocode to enhance clarity and understanding of our methodology.\n\n### Q2: Can you clarify your contributions beyond existing methods?\n\nOur work presents several key innovations in the field of mask editing and dataset generation for DIS tasks:\n\n1. **Specific Improvements for DIS Tasks**: Our method is designed for Dichotomous Image Segmentation (DIS) tasks, which require extremely high fidelity and precision in mask editing. Our approach generates diverse samples while maintaining high accuracy.\n\n2. **Methodological Innovations**: \n   - Topology-preserving adversarial training for non-rigid editing, ensuring structural integrity of edited masks\n   - Combination of rigid and non-rigid editing techniques, providing comprehensive mask editing capabilities\n   - Multi-conditional control generation method for precise image-mask pair creation\n\n3. **Performance Improvements**: As demonstrated in Table 2 of our paper, our method consistently enhances the performance of various state-of-the-art segmentation networks across multiple datasets, proving its effectiveness and generality.\n\n4. **Addressing Unique Challenges in DIS Tasks**: Our method addresses the need for high-precision annotations and diverse training samples in DIS tasks, which has rarely been specifically targeted in previous methods.\n\n### Q3: Can you provide more comprehensive ablation study results?\n\nWe have conducted additional experiments to provide a complete picture of the impact of each loss component:\n\n| L_GAN | L_content | L_structure | maxF1 ↑ | M ↓   |\n|-------|-----------|-------------|---------|-------|\n| ✔     |           |             | 0.778   | 0.073 |\n|       | ✔         |             | 0.745   | 0.075 |\n|       |           | ✔           | 0.751   | 0.074 |\n| ✔     | ✔         |             | 0.780   | 0.074 |\n| ✔     |           | ✔           | 0.782   | 0.073 |\n| ✔     | ✔         | ✔           | 0.784   | 0.073 |\n\nThese results demonstrate that while L_GAN alone provides a strong baseline, the combination of all three loss components yields the best performance. This underscores the importance of our multi-faceted approach to mask editing and image generation.\n\n### Q4: Can rigid and non-rigid mask editing be integrated into one framework?\n\nCurrently, we have not integrated these two frameworks, primarily because the rigid editing part cannot use topology preservation constraints. After viewpoint transformation, the detected key points undergo significant shifts, making topology preservation challenging.\n\nIn future work, we plan to explore the following directions to attempt integration:\n\n- Develop adaptive loss functions that can handle both rigid and non-rigid transformations\n- Design multi-stage editing processes that handle rigid and non-rigid edits separately while maintaining overall consistency\n- Research novel architectures that can automatically select appropriate editing techniques based on the input and desired transformation\n\n### Q5: How do you plan to address the limitations mentioned in the paper?\n\nWe are actively addressing these issues, particularly focusing on the problem of unnatural images:\n\n1. **Unnatural Images**: We recognize that this issue primarily stems from VAE training. We attempted to train a specialized VAE using the DIS dataset, but due to limited data, the results were not as effective as using a larger general dataset. Here are our preliminary results:\n\n   | VAE Training Data | maxF1 ↑ | M ↓   | E_φ^M ↑ |\n   |-------------------|---------|-------|---------|\n   | General dataset   | 0.784   | 0.073 | 0.875   |\n   | Small DIS dataset | 0.762   | 0.079 | 0.853   |\n\n   In the future, we plan to explore:\n   - Data augmentation techniques to effectively increase the size of the DIS dataset\n   - Transfer learning methods, using VAEs pre-trained on large-scale datasets and then fine-tuned on DIS data\n\n2. **Inaccuracies in Complex Scenarios**: We are developing hierarchical generation approaches and improved attention mechanisms to better capture long-range dependencies in complex scenes.\n\n3. **Reliance on Pre-annotated Pairs**: We are researching self-supervised learning methods and weak supervision strategies to reduce the need for extensive pre-annotated data.\n\nWe will include preliminary results and discussions of these ongoing efforts in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YWlO8FuG2", "reviewer": "Reviewer_AsEU", "summary": "⁤The paper introduces MaskFactory, a method for producing high-quality synthetic datasets for Dichotomous Image Segmentation (DIS) tasks. ⁤⁤The method includes a two-stage process: mask editing (combining rigid and non-rigid transformations) and image generation using multi-conditional control methods. ⁤⁤The suggested method improves model performance in DIS tasks by providing variation in synthetic data. ⁤", "questions": "Please see the weaknesses.", "limitations": "Please see the weaknesses.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The manuscript is well-written and constructed.\n- The authors conduct comprehensive ablation studies to highlight the contribution of each component of their method, further validating their approach.\n- The proposed MaskFactory framework shows improvements in the quality, and diversity of synthetic datasets, which is critical for advancing DIS applications.", "weaknesses": "There is one limitation that authors in the limitations section acknowledge issues with unnatural images in complex scenarios, but it could delve deeper into how these might impact practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's perceptive observation. We provide a more in-depth analysis below:\n\n**Q: The paper acknowledges issues with unnatural images in complex scenarios, but could delve deeper into how these might impact practical applications.**\n\nA: We address this concern by focusing on two key points:\n\n1. DIS data scarcity for VAE training:\n   We opted for a more general VAE to ensure better generalization. Additional experiments show:\n\n   | VAE Training Data | maxF1 ↑ | M ↓ | E_φ^M ↑ |\n   |-------------------|---------|------|----------|\n   | General dataset   | 0.784   | 0.073| 0.875    |\n   | Small DIS dataset | 0.762   | 0.079| 0.853    |\n\n   A DIS-specific VAE produces more visually coherent images but leads to performance decrease due to overfitting.\n\n2. Impact on performance:\n   Despite occasional unnatural images, our method significantly improves downstream segmentation performance. We achieve a 5.6% improvement in maxF1 score compared to the baseline, suggesting increased data diversity outweighs drawbacks of some unnatural samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ojleJct3kS", "reviewer": "Reviewer_h8fH", "summary": "This paper introduces a new approach for generating diverse and precise datasets. The authors first introduce a mask editing method that combines rigid and non-rigid editing techniques to generate high-quality synthetic masks leveraging geometric priors from diffusion models and adversarial training and self-attention mechanisms for complex, topologically consistent modifications. Then, the authors generate pairs of images and segmentation mask control generation methods. Finally, the experiments on the DIS5K dataset benchmark demonstrate superior performance compared to existing methods.", "questions": "1. The proposed method is only evaluated on one task, which is only limited to only one object. How about applying the approach on general semantic segmentation datasets, such as VOC, ADE20K and CityScapes?", "limitations": "See Weakness", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The paper provides a comprehensive and innovative solution to the challenges of DIS dataset creation. The methodology is well-detailed, and the experimental results robustly support the claims of superior performance.\n\n2. The paper is well-written and easy to read.", "weaknesses": "1. My major concern is the novelty of the method. Although the paper works on a new task to generate simulated data, the method seems to be a combination of some unrelated approaches including Zero-123, GPT, ControlNet et al. \n\n2. Whether the generated data can boost the Dichotomous Image Segmentation task is not validated. For example, the ControlNet generated images can be unrealistic, have a large domain gap between real-world images and training images, and the generated images seem not very diversified. Can generated images really help this task?\n\n3. The approach is evaluated on only one dataset, which is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's comprehensive examination of our work. Our responses to each point are as follows:\n\n### Q1: Novelty of the method and combination of existing approaches\n\nWhile our approach does integrate existing methods, its novelty lies in:\n\n1. **Task-specific adaptations for DIS tasks**: Our method is specifically tailored for Dichotomous Image Segmentation, addressing unique challenges such as high-precision requirements and the need for diverse training samples.\n\n2. **Synergistic integration of rigid and non-rigid editing**: We combine these techniques in a novel way to provide comprehensive mask editing capabilities, enhancing both geometric precision and topological consistency.\n\n3. **Topology-preserving adversarial training**: This innovative approach ensures structural integrity during non-rigid editing, which is crucial for maintaining the quality of binary masks in DIS tasks.\n\nThese innovations collectively contribute to a method that is greater than the sum of its parts, specifically designed to meet the demands of DIS tasks.\n\n### Q2: Validation of generated data boosting DIS performance\n\nWe have rigorously validated the effectiveness of our generated data in improving DIS performance:\n\n1. Our method achieves a 5.6% improvement in maxF1 score over the baseline, demonstrating significant performance gains.\n\n2. We employ multi-conditional control generation to enhance consistency between generated images and masks, addressing the concern of unrealistic images.\n\n3. The combination of rigid and non-rigid editing increases data diversity, mitigating the issue of limited variability.\n\nTo further illustrate the impact of our generated data, we present additional experimental results:\n\n| Training Data    | maxF1 ↑ | M ↓   | E_φ^M ↑ |\n|------------------|---------|-------|---------|\n| Real only        | 0.742   | 0.081 | 0.848   |\n| Real + Generated | 0.784   | 0.073 | 0.875   |\n\nThese results clearly demonstrate that our generated data significantly improves performance across multiple metrics.\n\n### Q3: Limited evaluation on one dataset\n\nWe acknowledge this limitation in our initial submission. To address this, we have conducted additional experiments on various datasets to demonstrate the generalizability of our approach. These results will be included in the revised manuscript.\n\n### Q4: Applicability to general semantic segmentation datasets\n\nWe have explored the applicability of our method to other semantic segmentation tasks using InSPyReNet [1] as a baseline. Here are some results:\n\n1. ECSSD (Edge detection):\n   | Method         | S_α ↑ | F_max ↑ | MAE ↓ |\n   |----------------|-------|---------|-------|\n   | Baseline       | 0.942 | 0.959   | 0.023 |\n   | Ours (Mix1000) | 0.950 | 0.962   | 0.022 |\n\n2. HKU-IS (Salient object detection):\n   | Method         | S_α ↑ | F_max ↑ | MAE ↓ |\n   |----------------|-------|---------|-------|\n   | Baseline       | 0.931 | 0.952   | 0.022 |\n   | Ours (Mix1000) | 0.943 | 0.954   | 0.021 |\n\n3. UOD (Underwater object detection):\n   | Method         | S_α ↑ | F_max ↑ | MAE ↓ |\n   |----------------|-------|---------|-------|\n   | Baseline       | 0.868 | 0.878   | 0.067 |\n   | Ours (Gen1000) | 0.883 | 0.901   | 0.056 |\n\nThese results demonstrate the effectiveness of our approach across various semantic segmentation tasks. We are currently extending our experiments to more challenging scenarios such as NI-seg and FI, and will include these results in the revised manuscript.\n\n### Reference\n[1] Yu, Q., et al. \"Multi-view Aggregation Network for Dichotomous Image Segmentation.\" CVPR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yzkhAQHDLk", "reviewer": "Reviewer_49D1", "summary": "This paper proposes a generative method for dichotomous image segmentation. In detail, non-rigid and rigid editing techniques are used to generate high-quality synthetic masks. Those masks are leveraged for segmentation model training, which typically requires expensive dichotomous image labeling.", "questions": "Some suggestions\n\n1. The authors use Zero123 for rigid mask editing. I am curious whether ImageDream [1] can be used here. Both of them can generate novel view images. [1] can takes the additional text as the input and I am interested in whether it can be used to directly change the shape of the mask like this paper shows that \"A round table\" can change the shape in Figure 2.\n  \n2. For the topology mentioned in Section 3.2.2, is it easy to show some topology structures of some samples? It would be better to include topology results for better illustration.\n  \n3. For Table 1, the numbers (2500, 5000, ...) are too close to the line.\n  \n4. The authors can include more discussion on related works [2,3]. It is interesting to discuss whether the data filtering strategy in [2] can be combined in this work. And [3], which also takes a dataset as the input and outputs a new one, also includes edited masks for the generation of specially tuned generative models, sharing the similar core idea of this paper.\n\nAs I discussed in Strength, overall this is an interesting paper. And I also love the area of synthetic date generation. However, it seems this paper is finished too rush. There are some improper points in the paper, discussed in the Weakness. I believe the authors should revise the paper thoroughly for precise presentation. And it is very important to decribe the method as how it is implemented in fact. It seems the text, masks, and the canny are all used in the generation according to line 388, then the Figure 2, Section 3.3 including Formula 8, and the algorithm in the appendix should be modified. Considering too much issues that need to be resolved, I prefer rating this paper as \"Borderline reject\" and I am sure to consider raising my rating to \"Borderline accept\" after the rebuttal for my possible misunderstanding and the authors' proper clarification.\n\n[1] Wang, Peng, and Yichun Shi. \"Imagedream: Image-prompt multi-view diffusion for 3d generation.\" *arXiv preprint arXiv:2312.02201* (2023).\n\n[2] Yang, Lihe, et al. \"Freemask: Synthetic images with dense annotations make stronger segmentation models.\" *Advances in Neural Information Processing Systems* 36 (2024).\n\n[3] Zhu, Lingting, et al. \"Generative Enhancement for 3D Medical Images.\" *arXiv preprint arXiv:2403.12852* (2024).", "limitations": "The limitations are discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "This paper introduces an approach for generating high-quality datasets for DIS. In my opinion, the methodology is carefully designed and experiments show the effectiveness. Overall I think this is an interesting and insightful paper.", "weaknesses": "1. There seems to have some typos/errors in the paper.\n  \n   a. In line 140, the (V,E) should be (V, E_s). The E_s does not show in section 3.2.2 except line 139.\n  \n   b. The font should be consistent, e.g., line 139,140 and Appendix. A Algorithm 1 line 7-11.\n  \n   c. The Figure 2 does not show what is discussed in Section 3.3. In the Figure 2, the prompt pool is used for generation. But in Section 3.3, the authors say that the segmentation masks and the edge conditions are used. If the text prompt is not used, it is meaningless to introduce the text in the last row of Figure 2 Stage 1 and Section 3.2.2. Also, here the detailed description of {p^m_i} is missing in the paper, though it is quite clear. Only have P_i shows in Section 3.2.2 and at least {p^m_i} should show in the following sections for consistency.\n  \n   d. \"Topylogy\" should be \"Topology\" in Figure 2.\n  \n   e. Remove full stops in the title of Section 4, 4.1, 4.2.\n  \n2. The authors argue that inspired by ControlNet, canny condition is also used. However, since there is no pretrained ControlNet used, some layers are trained according to line 173 and 174, it is unclear whether the canny condition is useful. Though more conditions present more information intuitively, the canny is already actually included in the informative mask condition. So some ablation studies should be presented to show that the additional condition do helps. But according to Section 5.2, is the pretrained Canny ControlNet used? Need to clarify in the main paper.\n  \n3. The method has limited increase compared with SAM-HQ, which finetunes SAM with additional branch on some prepared data.\n  \n4. It seems the impact of this method itself is limited, especially DIS is not so important. Maybe the authors can think about extending the method to a broader scope. Though I think this is a minor issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their meticulous review and helpful suggestions. We address each point as follows:\n\n### Q1: Typos and Errors\n\nWe apologize for these oversights and will correct all identified issues in the revised version:\n\na. We will change (V,E) to (V, E_s) in line 140 and ensure consistency throughout Section 3.2.2.\nb. We will unify the font style across the paper, including lines 139-140 and Algorithm 1.\nc. We will revise Figure 2 and Section 3.3 to accurately reflect our method's implementation, clarifying the use of text prompts, segmentation masks, and edge conditions. We will also provide a detailed description of {p^m_i} for consistency.\nd. We will correct \"Topylogy\" to \"Topology\" in Figure 2.\ne. We will remove full stops in the titles of Section 4, 4.1, and 4.2.\n\n### Q2: Canny Condition Usage\n\nWe appreciate this insightful observation. While the canny condition is indeed implicitly included in the mask, our experiments show that explicitly including it improves results. We have added an ablation study to demonstrate this:\n\n| Conditions    | maxF1 ↑ | M ↓   | E_φ^M ↑ |\n|---------------|---------|-------|---------|\n| Mask only     | 0.776   | 0.075 | 0.869   |\n| Mask + Canny  | 0.784   | 0.073 | 0.875   |\n\nThese results show that including the Canny condition leads to better performance across all metrics. In Figure 3 of our global PDF, we demonstrate the effect of adding the Canny condition. We found that incorporating Canny edges provides better constraints on the boundaries of generated images, resulting in more precise and detailed outputs.\n\n### Q3: Comparison with SAM-HQ\n\nWhile our improvement over SAM-HQ is modest, it's important to note that SAM-HQ has already incorporated a large amount of additional data during its fine-tuning process. This likely means that SAM-HQ's performance is approaching a plateau, making significant improvements more challenging. In contrast, our method shows more substantial improvements over other baselines, demonstrating its effectiveness, especially in scenarios where performance hasn't yet reached saturation.\n\n### Q4: Suggestions\n\n1. **ImageDream**: We have explored ImageDream and downloaded the weights as suggested. However, we found that it may not be directly suitable for binary image editing without fine-tuning. The results are shown in Figure 1 of our global PDF. Despite this, the method is indeed very inspiring for our work, and we will include a reference to it in our related work section.\n\n2. **Topology visualization**: We agree this would enhance the paper. We have included topology structure visualizations for selected samples in Figure 2 of our global PDF. These visualizations clearly demonstrate how our method preserves and manipulates topological structures during the editing process. We will incorporate these more intuitive visualizations in the next version of our paper.\n\n3. **Table 1 formatting**: We will adjust the spacing in Table 1 for better readability.\n\n4. **Related work discussion**: We appreciate these suggestions and will expand our discussion of [2] and [3]. We will explore incorporating the data filtering strategy from [2] and the dataset augmentation techniques from [3] into our work, and report the results in the revised version.\n\nWe acknowledge that some aspects of the paper need refinement, and we are committed to addressing all these issues in our revision. We will ensure that our method description accurately reflects its implementation, including the use of text prompts, masks, and canny edges in the generation process. We will revise Figure 2, Section 3.3, Formula 8, and the appendix algorithm accordingly.\n\nWe appreciate the opportunity to clarify these points and look forward to submitting a significantly improved version of the paper.\n\nReferences:\n\n[1] Wang, P., & Shi, Y. (2023). Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201.\n\n[2] Yang, L., et al. (2024). Freemask: Synthetic images with dense annotations make stronger segmentation models. NeurIPS 36.\n\n[3] Zhu, L., et al. (2024). Generative Enhancement for 3D Medical Images. arXiv preprint arXiv:2403.12852.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' thoughtful comments and suggestions. To address some of the concerns raised and provide additional support for our claims, we have prepared a global PDF with supplementary visual evidence. This document contains three key figures that demonstrate the effectiveness and novelty of our approach.

## Figure 1: Comparison with ImageDream

Figure 1 presents a comparison between our method and ImageDream, showcasing the effectiveness of our approach in both rigid and non-rigid editing of binary masks. The figure consists of four columns:

1. Original image
2. Result edited by ImageDream
3. Result from our rigid editing method
4. Result from our non-rigid editing method

This comparison clearly demonstrates that while ImageDream is a powerful tool for general image editing, it struggles with the precise requirements of binary mask editing for DIS tasks. Our method, both in its rigid and non-rigid editing capabilities, produces results that are more suitable for DIS applications, maintaining the binary nature of the masks while allowing for meaningful edits.

## Figure 2: Topology Modeling Visualization

Figure 2 provides a visualization of our topology modeling structure, which is crucial for our non-rigid editing process. This figure illustrates how topological constraints are applied during non-rigid editing, ensuring that the structural integrity of the mask is preserved even as complex deformations are applied.

The visualization demonstrates the before and after states of the mask, along with their corresponding topological structures. This clearly shows how our method maintains critical topological features while allowing for significant shape changes, a key innovation in our approach to DIS tasks.

## Figure 3: Impact of Canny Edge Constraints in ControlNet Generation

Figure 3 showcases the effect of incorporating Canny edge constraints in our ControlNet-based image generation process. The figure presents a side-by-side comparison of generation results:

1. Without Canny edge constraints
2. With Canny edge constraints

This comparison vividly illustrates that the addition of Canny edge constraints results in generated images with more defined and accurate boundaries. This improvement is particularly crucial for DIS tasks, where precise edge delineation is essential for accurate segmentation.

Meta-review: All reviewers agree that the paper is well written and provides extensive experiments to validate their approach. reviewers nBLG and were concerned about the novelty of the paper, but after rebuttal they recognized that the approach has some novel concepts such as structure-aware training that is novel.



nBLG borderline reject
+ originality
+ generalization to rigid and non-rigid
+ encouraging experimental results
- brief writing
- limited novelty elaborated but not rejecting other papers did similar parts)
- unclear contribution due to lack of ablation study on l-structure (added this)

AsEU accept
+ well written
+ comprehensive ablation studies
+ quality of results

h8fH borderline accept
+ innovative solution
+ well-written
- novelty; combination of other approaches
- main claim: DIS is not validated; can generated images really help DIS? (yes, have shown)
- evaluation on one dataset (yes, added three more)

49D1 Borderline accept
 +methodology is carefully designed 
+ experiments show the effectiveness
- effect of canny condition unclear (show it works)
- dose not improve SAM-HQ (sam was finetunes already)
Decision: Accept (poster)