FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models
Authors: Lihe Yang
Xiaogang Xu
Bingyi Kang
Yinghuan Shi
Hengshuang Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: learning from synthetic
semantic segmentation
generative models
TL;DR: Leverage densely annotated synthetic images to push the limit of semantic segmentation models.

Abstract: Semantic segmentation has witnessed tremendous progress due to the proposal of various advanced network architectures. However, they are extremely hungry for delicate annotations to train, and the acquisition is laborious and unaffordable. Therefore, we present FreeMask in this work, which resorts to synthetic images from generative models to ease the burden of both data collection and annotation procedures. Concretely, we first synthesize abundant training images conditioned on the semantic masks provided by realistic datasets. This yields extra well-aligned image-mask training pairs for semantic segmentation models. We surprisingly observe that, solely trained with synthetic images, we already achieve comparable performance with real ones (e.g., 48.3 vs. 48.5 mIoU on ADE20K, and 49.3 vs. 50.5 on COCO-Stuff). Then, we investigate the role of synthetic images by joint training with real images, or pre-training for real images. Meantime, we design a robust filtering principle to suppress incorrectly synthesized regions. In addition, we propose to inequally treat different semantic masks to prioritize those harder ones and sample more corresponding synthetic images for them. As a result, either jointly trained or pre-trained with our filtered and re-sampled synthesized images, segmentation models can be greatly enhanced, e.g., from 48.7 to 52.0 on ADE20K.

FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models
Lihe Yang'
Xiaogang 2 2.3
Bingyi Kang*
Yinghuan Shi
Hengshuang Zhao
'The University of Hong Kong
Zhejiang Lab
*Zejiang University
BayteDance
"Nanjing University
https:///ithhu: hwwp:auw.....mmheeeeeeeeeeeeeeee http://gttttb.oooooooooo/////eee
Abstract
Semantic segmentation has witnessed tremendous progress due to the proposal various advanced network architectures. However, they are extremely hungry for delicate annotations train, and the acquisition is laborious and unaffordable. Therefore, we present FreeMask in this work, which resorts to synthetic images from generative models to the burden of both data collection and annotation procedures. Concretely, we first synthesize abundant training images conditioned the semantic masks provided by realistic datasets. This yields extra well-aligned image-mask training pairs for semantic segmentation models. We surprisingly observe that, solely trained with synthetic images, already achieve comparable performance with real ones (e.g., 48.3 vs. 48.5 mloU on ADE20K, and 49.3 vs. 50.5 COCO-Stuff). Then, investigate the role of synthetic images by joint training with real images, or pre-training for real images. Meantime, we design a robust filtering principle suppress incorrectly synthesized regions. In addition, we propose to inequally treat different semantic masks to prioritize those harder ones and sample more corresponding synthetic images for them. As result, either jointly trained or pre-trained with our filtered and re-sampled synthesized images, segmentation models can be greatly enhanced, e.g., 48.7 -> 52.0 on ADE20K.
Introduction
Semantic segmentation aims to provide pixel-wise dense predictions for a whole image. With advanced network architectures, the performance has been steadily boosted the years, and has been applied to various real-world applications, e.g., autonomous driving [40, 3], semantic editing [39, 56], scene reconstruction [36], etc. However, both academic benchmarks and realistic scenarios, suffer from the laborious labeling process and unaffordable cost, leading to limited training pairs, e.g., Cityscapes [16] with only 3K labeled images and ADE20K [76] with only 20K labeled images. We are wondering, whether these modern architectures can be further harvested when more synthetic training data from the corresponding scenes are fed.
Fortunately, the research of generative models has reached certain milestones. Especially, denoising diffusion probabilistic models (DDPMs) [27, 51], are achieving astonishing text-to-image synthesis ability in the recent three years. Can these synthetic images from generative models benefu our semantic segmentation task? There have been some concurrent works [25, 54, 2, 64] sharing similar motivations as ours, i.e., using synthetic data to enhance the semantic understanding tasks. However, they have only explored the classification task, which is relatively easier be boosted even with raw unlabeled images [11, 24, 12, 23). Moreover, they mostly focus on the few-shot scenario, rather than taking the challenging fully supervised scenario into account. comparison, in this work.
"ooresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Real Image
Semantic Mask
Diverse Synthetic Images
Figure 1: Generated synthetic images conditioned on semantic masks. These synthetic images are high-quality and diverse. We wish to utilize these extra training pairs (synthetic image and its conditioned mask) to boost the supervised baseline, which is trained only with real images.
we wish to leverage synthetic images to directly improve the fully-supervised performance of the fundamental dense prediction task-semantte segmentation.
To formulate credible and controllable image synthesis to boost semantic segmentation, we propose to employ the SOTA diffusion models with semantic masks conditional inputs. Unconditional image synthesis requires manual annotations [73, 37], or will suffer from the issue of misalignment between synthetic images and the corresponding dense annotations [20]. Especially,
adopt an
off-the-shelf semantic image synthesis model FreestyleNet [71] to perform mask-to-inet torkk-eo-mmae synthesis.
However, even with such a conditional generator, there are still several challenges: (1) The synthesized image may be of a different domain from our target images. It is necessary to generate in-domain images. (2) The synthesis results cannot be perfect. Some failure cases will deteriorate the subsequent learning process. (3) The performance of our targeted super-suee baseline has almost been
saturated with optimal syper-parameters. Thus, the synthesis should be emphasized on the hard samples that are crucial for improving the generalization performance.
To solve challenge (1), we first prompt the output space of FreestyleNet [71] to be closer to the target datasets, i.e.. FreestyleNet pre-trained with powerful Stable Diffusion model [51], and then fine-tuned on specific mask-to -t-iaae synthesis datasets, e.g., ADE20K [76] and COCO-Stuff [9]. Hence, not only acquires generic knowledge from the large-scale pre-training set of Stable Diffusion, but also is capable of yielding in-domain images conditioned on the semantic masks. As depicted in Figure 1, we can produce diverse high-fidelity synthetic images given existing training masks from our targeted datasets. Then, each synthetic image is paired with its conditioned semantic mask, forming a new training sample with dense annotations.
However, when training models with these synthetic pairs, we observe that it is evidently inferior to the counterpart of training with real images, since there are still some image regions synthesized unsatisfactorily. These poorly synthesized images will have a negative impact on training, even overriding the positive influence brought by the large dataset size, leading to the challenge (2). To this end, we propose to filter synthetic images in a self-adaptive manner at the pixel level according to a class-level evaluation. Intuitively, incorrectly synthesized regions or images will exhibit significant losses, if evaluated under a model tre-trained on real images. Motivated by this, we first obtain the average loss of each semantic class, which is calculated through the whole synthetic set with a real-image pre-trained model. Then we suppress noisy synthetic regions under this principle: if the loss of a synthetic pixel surpasses the average loss of its corresponding class by a certain margin, it will be marked as a noisy pixel and ignored during loss computation. Despite its simplicity, this strategy significantly improves the model performance when trained with synthetic images.
Furthermore, we implement hardness-aware data synthesis to generate more hard samples, effectively solving challenge (3) and prompting the training better. We observe that different semantic masks are of varying hardness. For example, the mask in the first row of Figure 1, which exhibits regular object structures and limited semantic classes, is obviously much easier to learn than the mask in the second row. These over-easy masks and their corresponding synthetic images will dominate the learning process as well as the gradient. Therefore, we propose to inequally treat these semantic masks during conditional image synthesis. Concretely, with recorded class-wise average losses, we can calculate the overall loss semantic mask, which can represent its global hardness. For a semantic mask, we then determine the number of its required synthetic images based on its hardness. The harder it

---

is, the more synthetic images will be generated for it. Consequently, the model can be trained more sufficiently on these challenging layouts. Our contributions lie in four aspects:
We present new roadmap to enhance,ullyy fully-supervised semantic segmentation via generating densely annotated synthetic images with generative models. Our data-centric perspective is orthogonal to the widely explored model-centric (e-g., network architecture) perspective.
We propose robust filtering criterion to suppress noisy synthetic samples at the pixel and class levels. Moreover, we design an effective metric to indicate the hardness of semantic masks, and propose discriminatively sample more synthetic images for harder masks.
With our filtering criterion and sampling principle, the model trained solely with synthetic images can achieve comparable performance with the counterpart of real images, e.g., 48.3 vs. 48.5 mloU on ADE20K and 49.3 vs. 50.5 on COCO-Stuff.
We investigate different paradigms to leverage synthetic images, including (1) jointly training on real and synthetic images, (2) pre-training on synthetic ones and then fine-tuning with real ones. We observe remarkable gains (e.g., 48.7 + 52.0) under both paradigms.
2 Related Work
Semantic segmentation. Earlier semantic segmentation works focus on designing various attention mechanisms [68, 29], enlarging receptive fields [74, 10, 44], and extracting finer-grained informa- tion [38]. With the introduction of the Transformer architecture [21] to CV, advanced semantic segmentation models [70, 75, 14] are mostly Transformer- based. Among them, MaskFormer [15] presents new perspective to perform semantic segmentation via mask classification. Orthogonal to these model-centric researches, our work especially highlights the important role of training data.
Generative models. Generative models have been developed for several years to synthesize photo- realistic images and can be divided into two main categories, i.e., unconditional and conditional models. The input of unconditional generative models is only noise, and the conditional networks take inputs of mulli-modality data (e.g., labels, text, layouts) and require the outputs to be consistent with the conditional inputs. The structures of generative models have evolved rapidly in recent years. Generative Adversarial Networks (GANs) [22]. Variational AutoEncoder (VAE) [35], and Diffusion models [58] are the most representative ones among them. GANs are formulated with the thcoretical basis of Nash equilibrium, where the generator is trained with set discriminator. Several excellent GANs have been proposed in recent years to synthesize high-quality photos, egg., BigGAN [8] and StyleGAN [32, 33, 31, 55]. VAEs belong to the families of variational Bayesian methods and are optimized with an Evidence lower bound (ELBO). VQ-VAE [48] is one famous model which can generate high-resolution images effectively. Diffusion models [58, 28, 42] have recently emerged as a class of promising and effective generative models. It first shows better synthesis ability than GANs in 2021 [19]. and then various diffusion models have been designed and achieved outstanding results, e.g., Stable Diffusson [51], Dalle [49, 46], Imagen [53], etc. Moreover, GANs [43, 63) and Diffusson models [72, 67, 71] can both complete image synthesis from semantic layouts.
Learning from synthetic images. There are three main sources of synthetic images. (1) From computer graphics engines [50, 52]. These synthetic images with dense annotations are widely adopted in domain generalization [65] and unsupervised domain adaptation [17]. However, there exists large domain gap between them and real images, making them struggle to benefit the fully- supervised baseline. (2) From procedural image programs. These inspiring works [7, 34, 6, 62] delicately design abundant mathematical formulas to sample synthetic images. Despite promising results obtained in ImageNet fine-tuning, it is laborious to construct effective formulas, and they still even lagging behind the basic ImageNet pre-training in dense prediction tasks. (3) From generative models. Our work belongs to this research line. We provide a detailed review below.
Early works [1, 59, 73, 37, 30] have utilized GANs to synthesize extra training samples. Recently, there have been some initial attempts to apply Diffusion models [51] to the classification task. For example, He et al. [25] examined the effectiveness of synthetic images in data-scarce settings (zero-shot and fhwtshot). The synthetic images are also found to serve as a promising pre-training data source for downstream tasks. In addition, Sariyildiz et al. [54] manages to replace the whole ImageNet images [18] with pure synthetic ones. However, it still reports huge performance gap compared with its real counterpart. Different from previous works that directly produce non-existing
3

---

Leverage synthetic images
Generative
Pre-training for real images
Model
Joint training with real images
Real Images
Supervision
Segmentor
Diverse Syathetic Images
Figure 2: Illustration of our proposed roadmap to boost fully-supervice semantic segmentation with densely annotated synthetic images. The generative model can synthesize diverse new images conditioned on the semantic mask from target datasels. Synthetic images and conditioned masks form new training pairs. We investigate pre-training and joint training leverage these synthetic images.
images, Trabucco et al. [64] proposes to leverage Stable Diffusion [51] to edit existing training images, e.g., altering the color. The most latest work [2] shares a similar motivation also wishes to enhance the fully-supervised performance. Nevertheless, it only addresses the classification task, which is relatively easier be boosted even directly with unlabeled images [11, 24, 23] or pre-training image-text pairs [45]. Distinguished from it, we aim at the challenging semantic segmentation task which requires more precise annotations than image-text pairs. More importantly, we further propose two strategies to learn from synthetic images more robustly and discriminatively. We also discuss learning paradigms.
in that
Discussion with semi-supervised learning (SSL). Our strategy is different from SSL, i.e., we synthesize images from masks, while SSL predicts pseudo masks for extra unlabeled images. Our proposed roadmap is better than SSL in three aspects: (1) It is not trivial to ensure the extra unlabeled images for SSL of the same domain labeled images, while our target-set fine-tuned generative model can consistently yield in-domain images. (2) is much cheaper and time-saving to synthesize abundant and almost unlimited images from semantie masks, compared with manually collecting appropriate unlabeled images. (3) We observe it more precise in structures to synthesize images from masks than predict masks for images, especially in boundary regions.
3 Method
In this section, we first introduce our synthesis pipeline in Section 3.1. Then, we clarify our filtering criterion for noisy synthetic regions in Section 3.2. Following this, we further present hardness- aware re-sampling principle to emphasize hard-to-learn semantic masks in Section 3.3. Lastly, we discuss two paradigms learn from these high-quality synthetic images in Section 3.4.
3.1 Synthesizing Densely Annotated Images
Semantic image synthesis [66, 61, 63) aims to generate a photo-realistic image conditioned on a given semantic layout. Among these approaches, recent works [67, 71] have shown that large-scale text- to-image diffusion models can trained to synthesize high-quality and diverse results. Motivated by this, we adopt the most recent work FreestyleNet [71], to synthesize additional training images conditioned on the training masks provided by our targeted datasets. Take ADE20K [76] as an example, it only contains 20K image-mask training pairs. However, aided by the mask-to-image synthesis network, we can construct almost unlimited synthetic images from these 20K semantic masks. Conditioned on each semantic mask, if we synthesize K images, will obtain artificial training set whose scale is K x larger than the original real training set. After synthesis, each synthetic image can be paired with the conditioned mask to form new training sample with dense annotations. We provide an illustration of our pipeline Figure 2.
We construct densely annotated synthetic images through mask-to-image synthesis. However, some works adopt a different or even "reversed" approach. We briefly discuss them and make a comparison. DatasetGAN [73] and BigDatasetGAAN [37] first produce synthetic images in an unconditional manner. Their dense annotations will be assigned by human annotators. Then the model will learn from these training pairs and predict pscudo masks for new synthetic images. The
Discussion.

---

Imsage
Somantic
Synthetic Image
Hard
Hardness-aware re-sampling: Harder mask, more synthetic images!
Easy
Figure 3: Visualization of failure cases during synthesis. Incorrect synthetic ro- gions are highlighted with pink boxes.
Figure 4: We propose mask-level hardness-aware re-sampling strategy synthesis. Easy masks are of regular shapes and cover limited and common semantics, while hard masks are the opposite.
main drawback of such methods is the involvement of expensive human efforts. E.g., 5K manual annotations are needed to train classification tasks on ImageNet, which is costly, let alone providing the annotation of dense semantic masks.
To avoid the cost of manual annotations, recent work [20] directly trains segmentation model on real pairs and then uses it to produce pseudo masks for synthetic images. It is very close to the supervisdd scenario (only replacing real unlabeled images with synthetic unlabeled images). However, since there is a domain gap between the real and synthetic images inevitably, we cannot guarantee the correctness of the pseudo masks.
On the other hand, our mask-conditioned image synthesis can avoid the issue of annotation cost and meanwhile produce accurately aligned image-mask pairs, especially in boundary regions. Moreover, our adopted conditional image synthesis is more controllable than unconditioned synthesis. For example, we can better control the class distributions of synthetic images by editing the semantic masks. This has the potential to help alleviate the long-tail distributions. Furthermore, with mask- to-image synthesis, we can synthesize more hard-to-learn scenes (defined by semantic masks) for models to learn, which will be discussed in Section 3.3.
3.2 Filtering Noisy Synthetic Regions
Motivation. As illustrated in Figure 3, although visually appealing, there still exist some artifacts in synthetic images. Blindly using all synthetic samples for training will cause the model to be severely deteriorated by these noisy regions, overshadowing the performance gain brought by additional synthetic images, especially when the synthetic set is large. This will be demonstrated in Section directly scaling up the number of raw synthetic images for training cannot progressively yield better performance on the real validation set, even achieving worse results.
Our filtering strategy. To this end, we propose to adaptively filter harmful synthetic regions via computing hardness score at the class level and conducting filter at the pixel level. Specifically, detrimental synthesized regions denote those regions that do not match corresponding semantic masks well. They usually exhibit large losses if evaluated with a well-trained semantic segmentation model. Motivated by this, we filter synthetic pixels based on the following criterion: if the loss of synthetic pixel belonging to semantic label j has surpassed the recorded average loss of all class-j pixels by a certain margin, this synthetic pixel will be marked as a noisy one and simply ignored during loss computation. To instantiate this idea, we use semantic segmentation model which is pre-trained on real images to calculate dense loss maps { (S, {sk, [1, H W]} where sk is the pixel-wid liee loss value, H and W denote the map height and width, respectively) for all N synthetic images with their semantic masks {M') J Then, we can obtain the average loss hy of class by:
N HW
N HW ME
h2
Σ II =) x ST F0w
1(
j): =
1=1 Aw
where (xx) = if x is True, otherwise 0, and hw indicates the pixel location of (h, w). The hy indeed reflects the learning hardness of semantic class in the corresponding target dataset, i.e
5

---

harder classes tend to have larger h. We will further reuse h as an effective indicator for mask-level hardness in Section 3.3.
With K' average losses of K semantic classes, we can filter noisy synthetic pixels. For a synthetic pixel k with label j, if its loss 8ę hi aa, it will be deemed as a potentially noisy pixel, where here serves as tolerance margin. The smaller a is, the more synthetic pixels will be filtered and ignored during loss computation for better safety. Meanwhile, too small will also lead to the remaining pixels not being informative enough for our model to learn. Fortunately, our filtering strategy is robust a, as the performance stable under different a ranging from 1 to 2.
Effects. Despite the simplicity of this filtering criterion, it significantly boosts the generalization performance of the target model when trained with synthetic images and tested in real images. It is even on par with the performance achieved by real training images (less than 1 %% margin in test mloU). This demonstrates its great potential in replacing real images with synthetic images in the future, especially when dealing with some privacy-sensitive application scenarios.
3.3 Re-sampling Synthetic Images based on Mask-level Hardness
Motivation. We further observe that some semantic masks are easily learnable layouts, while some masks depict relatively messy scenes that are harder to learn, as shown in Figure 4. These simple layouts exhibit small overall losses even if never encountered synthetic image is fed. Thus, the benefit obtained by learning from such layouts is usually trivial. On the other hand, those hard-to-learn scenes require more attention during image synthesis and subsequent training.
Our re-sampling strategy. To address this issue, we present a simple and straightforward principle to quantify the global hardness of each semantic mask. Based on the mask-level hardness, we further propose to discriminatively sample synthetic images from semantic masks, i.e., prioritizing harder masks. Concretely, to obtain the hardness, we reuse the class-wise average loss h defined in Equation 1. A minor difference is that h here calculated on the real training pairs to more precisely reflect the difficulty of training masks. The global mask-level hardness can be viewed an aggregation across the hardness of all its pixels, and the hardness of each pixel is associated with its semantic label. Formally, the hardness p; of the i-th mask can be computed by Ehve /Mj... where M represents the class index of position (h, w) in the semantic mask map.
HW
Following this, we sort totally N semantic masks according to their quantitative hardness. We can then sample more synthetic images for harder masks based on their rankings among all masks. More specifically, for the p-largest-
-aargest-harddesss hardness mask, the number of its synthetic images ?p is determined by:
np = Nuax (N - p)/N,
(2)
where Nmax is our pre-defined maximum number of synthetic images for a single mask. The p ranges from O to (N 1), and ni will be rounded up to an integer, ranging from to Nmax.
Effects. Our hardness-aware synthesis strategy can avoid the learning process being dominated by over-easy image-mask pairs and avoid vanishing gradients. It also enables the challenging layouts to be sufficiently learned and thus achieves better performance. Indeed, to some extent, it shares a similar spirit with online hard example mining (OHEM) [57]. Meanwhile, it fundamentally differs from OHEM in that we synthesize more novel hard samples for the model to learn, rather than just implicitly enlarging the importance of existing hard samples as OHEM does.
3.4 Learning from Synthetic Images
We discuss two paradigms to use the synthetic training set for the learning of semantic segmentation.
Pre-training. The segmentation model first trained with densely annotated synthetic images for the same number of iterations as real images. During synthetic pre-training, we test the transferring performance on the real validation set. By comparing this performance with that obtained by real training images, we gain better intuition on the gap between synthetic and real images. Next, further fine-tune the pre-trained model with real images. This paradigm can effectively overcome the noise caused by synthetic images, especially when synthetic images are of relatively poor quality.
Joint training. The synthetic set usually larger than the real set. Meanwhile, the real training pairs are of higher quality than synthetic ones. Therefore, we first over-sample real images to the

---

Training Data
Training Data
Model
mloU
Model
mloU
Real
Synthetic
Real
Synthetic
48.5
45.8
SegFormer-BA [70]
SegFormer-Be [70]
48.3
44.4
56.0
50.5
Mask2Former [14]
ViT-Adapter [13]
53.5
49.3
Table 1: Transferring performance on ADE20K. The validation set is real images. The Fask2Former uses a
Transferring performance on COCO-Stuff. The validation set is real images. The ViT-Adapter uses UperNet [69] and BEiT-- backhone [5].
Table 2:
Swin-L. [41] backhone pre-trained on ImageNet-22K.
Training Data
Model
Backhone
mloU
A
Real
Synthetic
48.7
Swin-T [41]
3.3%
52.0
51.6
Mask2Former [14]
Swin-S [41]
1.7%
53.3
52.4
Swin-B [41]
1.3%
53.7
45,6
MiT-B2 [70]
2.3%
47.9
SegFormer [70]
48.5
MiT-B4 [70]
1 2.1%
50.6
46.2
VIT-S [21]
1.7%
47.9
Segmenter [60]
49.6
VIT-B [21]
1.5%
51.1
Table 3: Integrating synthetic images by joint training on ADE20K to enhance the real-image bascline.
same number as synthetic images. Then we simply train a segmentation model on the combined set. Compared with pre-training, joint training can better preserve the knowledge acquired from the larger-scale synthetic set. It may more promising when the transferring performance of synthetic images is exceptionally close to that of real images.
4
Experiment
We describe the implementation details Section 4.1. Then we report the transferring performance of synthetic training pairs real validation set in Section 4.2. Further, leverage synthetic images to boost the sully-supervised baseline Section 4.3. Finally, we conduct ablation studies in Section 4.4.
4.1 Implementation Details
We evaluate the effectiveness of synthetic images two widely adopted semantic segmentation benchmarks, i.e., ADE20K [76] and COCO-Stuff [9]. They are highly challenging due to the complex taxonomy. COCO-Stufi is composed of 118,287/5,00 171 semantic classes. In comparison, ADE20K is more limited in training images, containing 20,210/2,000 training/validation images and covering 150 classes.
craiiin/vaalddttin images, spanning over
We use two open-source checkpoints from FreestyleNet [71] for ADE20K and COCO respectively, to produce photo-realistic synthetic images conditioned on training masks. We synthesize at most 20 extra images for each semantic mask on ADE20K, i.e., Nmax = 20 Equation On COCO, considering its large number of semantic masks, we set a smaller Nmax 6 due to the limitation of disk space on our server. By default, we set the tolerance margin a as 1.25 for all experiments. In pre- training, exactly follow the hyper- parameers of regular training. In joint training,

---

Training Data
Model
Backhone
mloU
A
Real
Synthetic
44.5
Swin-T [41]
1.9%
46.4
Mask2Former [14]
46.8
Swin-S [41]
0.8%
47.6
43.5
MiT-B2 [70]
0.7%
44.2
SegFormer [70]
45.8
MiT-B4 [70]
0.8%
46.6
43.5
VIT-S [21]
1.3%
44.8
Segmenter [60]
46.0
VIT-B [21]
1.5%
47.5
Table 4: Integrating synthetic images by joint training on COCO-Stuff to enhance the real-image baseline.
Training Data
Model
Backhone
mloU
Real: Fine-tune
Synthetic: Pre-train
45.6
MiT-B2 [70]
1.4%
47.0
SegFormer [70]
/
48.5
MiT-B4 [70]
1.2.1
/
50.6
46.2
VIT-S [21]
1.0%
47.2
Segmenter [60]
49.6
VIT-B [21]
1 1.0%
50.6
56.0
Mask2Former [14]
Swin-L [41]
0.4%
56.4
Table Integrating
synthetic images pre-training on ADE20K enhance the real-image bascline. f: The Swin-L backbone is pre-trained on the large-scale ImageNet-22K dataset.
over-sample real images to the same number synthetic images. The learning rate and batch size are the same as the regular training paradigm. Due to the actually halved batch size of real images in each iteration, we double the training iterations iterate over real training images for the same epochs regular training. Other hyper-parameters are detailed in the appendix.
4.2 Transferring from Synthetic Training Images to Real Test Images
Existing semantic image synthesis works [61, 71] examine the synthesis ability by computing the FID score [26]. However, we observe this metric is not well aligned with the benefits brought by
augmenting real image datasets with synthetic images, as also suggested in [47]. Therefore, similar to the Classification Accuracy Score proposed in [47]. we exhibit the synthesis quality by transferring the model trained on synthetic images to real test images. The semantic segmentation score (mean Intersection-over- Union) on real test images is robust indicator for synthesized images.
Using this score, we first validate the quality our filtered and re-sampled synthetic images. We train a model using densely annotated synthetic images alone, and compare it with the counterpart of only using real training images. As shown Table of ADE20K, under the widely adopted SegFormer [70] model with MiT-B4 backbone, the performance gap between using synthetic training images and real training images negligible, iee,, 48.3% vs. 48.5% mloU (-0.2%). Then we attempt one of the most powerful semantic segmentation models, i.e., Mask2Former [14] with heavy Swin-L. [41] backbone pre-trained on the large-scale IetgeNet-22K. Although the real-data trained model achieves fantastic result (56.0%), our synthetic training images also yield strong

---

Our Strategies
Our Strategies
Datasct
mloU
Dataset
mloU
Filtering
Re-sampling
Filtering
Re-sampling
43.3
48.0
ADE20K
47.4
COCO
V
48.9
(48.5)
45.1
(50.5)
48.6
48.3
49.3
Table Effectiveness our filtering and re-sampling strategies for transferring performance. The results below the dataset names denote the transferring performance obtained with real training images.
Our Strategies
mloU
48.5
Filtering
Re-sampling
mlot:
48.2
48.0
49.9
49.3
50.6
47.5
tolcrance margin
Table 7: Effectiveness our strategies when jointly training two sources of synthetic and real images.
Figure Ablation study on hyper-parameter a.
model (53.5%). The marginal performance gap (-2.5%) is impressive, especially considering the large gap (around -8%) in the latest work [2] for synthetic classification.
The corresponding experimental results on COCO are shown Table 2, and the observation is consistent. Either with SegFormer [70] the state-of-the-art model ViT-Adapter [13], the transferring performance gap between synthetic and real images is constrained within 1.5%.
We have demonstrated the impressive transferring ability of our filtered and re-sampled synthetic training images toward real test images. indicates they have the great potential completely replace the real training images in the future. They can play a significantly important role in many privacy-sensitive application scenarios. Furthermore, in Section 4.4, we will show that the transferring performance will decrease a lot without our filtering and re-sampling strategies.
4.3 Enhaneing Fully-supervised Baseline with Synthetic Images
We further take full advantage of these synthetic images to enhance the sully-supervised baseline, which is merely trained with real images before. We investigate two training paradigms to integrate synthetic and real training images, i.e., (1) jointly training with both real and synthetic images, and (2) pre-training with synthetic ones, then finettnning with higher-quality real ones.
Under the joint training paradigm, on ADE20K of Table 3, with extra synthetic images, we examine the performance of wide range of modern architectures. All models evidently boosted by larger than 1%. Among them, the widely adopted Mask2Former-Swiie
is improved from 48. 7877% + 52.0% mloU (+3.3%), and SegFormer-B4 is boosted from 48.5% - 50.6% (+2.1%) Also, as demonstrated in Table 4 for COCO-Stuff, although the number of real training images is already large enough (120K), our densely annotated synthetic images further push the upper bound of existing state-of-the-ar art models. The ForneForr--r-in--- model is non-trivially enhanced by 1.9% mloU from 44.5% 46.4%.
Moreover, under the pre-training paradigm, we report results on ADE20K in Table 5. The SegFormer- B4 model is similarly boosted by 2.1 % the joint-training practice. However, observe that in most cases, pre-training is slightly inferior to joint training. We conjecture is due to the high quality of our processed synthetic pairs, making them play a more important role when used by joint training.
4.4 Ablation Studies
Unless otherwise specified, conduct ablation studies ADE20K with SegFormer-B4,
Effectiveness of our filtering and re-sampling strategies. We first demonstrate the indispensable role of our proposed filtering and re-sampling strategies by cxamining the transferring performance

---

Training Scheme
Mask2Former Swin-T
SegFormer MiT-B4
Segmenter + ViT-S
Real Only (1x iters)
48.7
48.5
46.2
Real Only (2x iters)
49.2
48.6
46.7
Real + Synthetic
52.0 (1 2.8%)
50.6 (1 2.0%)
47.9 (1 1.2%)
Table 8: We attempt to double the total iteration real-image training paradigm (2x iters). It evidently inferior to the "real + synthetic" setup under the same training iterations and batch size.
from the synthetic set, the same as the practice in Section 4.2. As shown in Table 6, on ADE20K, the baseline without processing synthetic images, shows considerable performance gap with real images (43.3% vs. 48.5%). Then with our pixel-level filtering criterion, the result is improved tremendously (43.3% + 47.4%). Our hardness-aware re-sampling principle also remarkably boosts the result from 43.3% to 45.1 %. Combining both strategies, the transferring performance achieves 48.3%, outperforming the baseline by 5%. Similar observations also hold true on COCO.
Furthermore, we evaluate our two processing strategies when jointly training synthetic and real images in Table 7. Although the synthetic images look impressive. they can not benefit real images if combined directly without any processing (48.5% + 48.2%). Equipped with our proposed strategies for synthetic images, they successfully boost the fully-supervised baseline from 48.5% to 50.3%.
Training real images at double budget. As aforementioned, due to the halved number of real images in each joint training mini-batch, we double the training iterations. Therefore, we similarly double the training iterations for regular real-image training. As shown in Table of ADE20K, doubling the training iterations under the real-image paradigm (real only 2x iters) does not incur noticeable improvement over the baseline (real only 1x iters). More importantly, it is still significantly inferior to our synthetic joint training (real synthetic) across wide range of network architectures.
Performance with respect to the number of syn- thetic images. It is reported in [2] that when the number of synthetic images surpasses real images, more synthetic images even cause worse perfor- mance. As present Table 9, if not applying our processing strategies for synthetic images, the ob- servation is similar to [2]. However, as long as the two strategies adopted, the performance will keep increasing as the synthetic set enlarges. further proves the necessity of designing appropriate filtering and re-sampling strategies to effectively learn from synthetic images.
N
6
10
20
Filtering & Re-sampling X Fittering& & Re-sampling /
43.7
43.6
43.3
47.2
47.7
48.3
Table 9:
Ablation studies on the number of synthetic images, which is controlled by Nmax- The model is trained with synthetic images only.
Quantitative comparison with online hard ex- ample mining (OHEM). In Section 3.3, we have discussed the commonality and difference be-
Baseline
+ +OHEM
Our Re-sampling
44.0
44.2
45,4
tween our hardness-aware re-sampling strategy
and OHEM. In Table 10, we demonstrate our method is also empirically superior to OHEM.
Table 10: Comparison between our hardness- aware
re-sampling strategy and OHEM with Segmenter-S.
Hyper-parameter tolerance margin a. As shown in Figure 5, our proposed filtering strategy is robust to the tolerance margin a, performing consistently well when a ranges from I to 2.
5 Conclusion
In this work, we present a new roadmap to enhance fully-supervised semantic segmentation via leveraging densely annotated synthetic images from generative models. We propose two highly effective approaches to better learn from synthetic images, i.e., filtering noisy synthetic regions at the pixel and class level, and sampling more synthetic images for hard-to-learn masks. With our two processing strategies, synthetic images alone can achieve comparable performance with their real counterpart. Moreover, investigate two training paradigms, i.e., pre-training and joint training, to compensate real images with synthetic images for better aully-supervised performance.
Acknowledgment. This work is supported by the National Natural Science Foundation of China 662201484, 62222604), HKU Startup Fund, and HKU Seed Fund for Basic Research. We also want to thank FreestyleNet [71] for providing high-quality generative models.
10

---

References
[1] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXi::1711.04440, 2017.
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv:2304.0888 2023. 1, 4, 9, 10
[3] Bhakti Baheti, Shubham Innani, Suhas Gajre, and Sanjay Talbar. Eff-unet: A novel architecture for semantic segmentation in unstructured environment. In CVPR, 2020.
[4] Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust classification via generated datasets. arXiv:2302.025033 2023.
[5] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. In ICLR, 2021.
[6] Manel Baradad, Richard Chen, Jonas Wulff, Tongzhou Wang, Rogerio Feris, Antonio Torralba, and Phillip Isola. Procedural image programs for representation learning. In NeurIPS, 2022.
[7] Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. In NeurlPS, 2021.
[8] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv: 11096, 2018. 3
[9] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018. 2,
[10] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 3
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI., 2020. 1,
[12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 1rXiv:2003.04297 2020. 1
[13] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023. 7.9
[14] Bowen Cheng, Ishan Mísra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022. 3, 7,
[15] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In Advances Neural Information Processing Systems, 2021.
3
[16] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, In CVPR, 2016. 1
[17] Gabriela Csurka, Riccardo Volpi, and Boris Chidlovskii. Unsupervised domain adaptation for semantic image segmentation: comprehensive survey. arXiv:2/12.032: 2021. 3
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[19] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In
NewrIPS, 2021. 3
[20] Kexin Ding, Mu Zhou, He Wang, Olivier Gevaert, Dimitris Metaxas, and Shaoting Zhang. large-scale synthetic pathological dataset for deep learning-enabled segmentation breast cancer. Scientific Data, 2023. 5,

---

[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
3, 7
[22] lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In ICLR, 2015.
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 1,4
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 1,4
[25] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? In ICLR, 2023. 3,3
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NewrIPS, 2017. 8
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 3
[29] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2019.
[30] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. S..iv:2006.0255555 2021.
[31] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurlPS, 2021. 3
[32] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[33] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. CVPR, 2020.
[34] Hirokatsu Kataoka, Ryo Hayamizu, Ryosuke Yamada, Kodai Nakashima, Sora Takashima, Xinyu Zhang, Edgar Josafat Martinez-Noriega, Nakamasa Inoue, and Rio Yokota. Replacing labeled real-image datasets with -enerrteeeedddeee contours. In CVPR, 2022.
[35] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114,02 2013.
3
[36] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In CVPR, 2022.
[37] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations. In CVPR, 2022. 2, 3,4
[38] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinent:: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR, 2017.
[39] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. In NewrIPS, 2021. 1
12

---

[40] Xiaofeng Liu, Yuzhuo Han, Song Bai, Yi Ge, Tianxing Wang, Xu Han, Site Li, Jane You, and Jun Lu. Importance-awart semantic segmentation self-driving with discrete wasserstein training. In AAAI, 2020.
[41] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
7, 8
[42] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[43] Tacsung Park, Ming- Yu Liu, Ting-Chum Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adapttie normalization. In CVPR, 2019.
[44] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kemel matters- improve semantic segmentation by global convolutional network. In CVPR, 2017. 3
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[46] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06115, 2022.
[47] Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In NeurIPS, 2019. 8
[48] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurlPS, 2019.
[49] Mr D Murahari Reddy, Mr Sk Masthan Basha, Mr Chinnaiahgari Hari, and Mr N Penchalaiah. Dall-e: Creating images from text. UGC Care Group I Journal, 2021. 3
[50] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016. 3
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 1, 3, 4
[52] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016. 3
[53] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[54] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In CVPR, 2023. 3,3
[55] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In SIGGRAPH, 2022. 3
[56] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020. I
[57] Abhinay Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- based object detectors with online hard example mining. In CVPR, 2016. 6
[58] Jascha Sohl-Dic,stein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. ICML, 2015. 3
[59] Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised semantic segmentation using generative adversarial network. In ICCV, 2017. 3
13

---

[60] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021. 7,
[61]
Vadim Sushko, Edgar Schönfeld, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In ICLR, 2021. 4, 8
[62] Sora Takashima, Ryo Hayamizu, Nakamasa Inoue, Hirokatsu Kataoka, and Rio Yokota. Visual atoms: Pre-training vision transformers with sinusoidal waves. In CVPR, 2023.
[63] Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Bin Liu, Gang Hua, and Nenghai Yu. Diverse semantic image synthesis via probability distribution modeling. In CVPR, 2021. 3,4
[64] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. arXiv:2302.0794, 2023. 1,4
[65] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generaliza- tion. TKDE, 2022.
[66] Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
[67] Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffusion models. arXiv:2207.00050, 2022, 3,
[68] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[69] Tete Xiao, Yingcheng Liu. Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. ECCV, 2018.
[70] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, 2021. 3, 9,9
[71] Han Xue, Zhiwu Huang. Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. CVPR, 2023. 2, 3, 4, 7, 8, 10, 15
[72] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. 1.Xiv:23022.55433 2023. 3
[73] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Francora Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021. 3,
[74] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, 2017.
[75] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang. Philip HS Torr, al. Rethinking semantic segmentation from Sequence-to-sequeee perspective with transformers. In CVPR, 2021. 3
[76] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. CVPR, 2017. 1, 4,7
14

---

A More Implementation Details
Following FreestyleNet [71], we resize all semantic masks 512x512 for image synthesis. The guid- ance scale of the diffusion model is set as 2.0, and the sampling step is 50. For better reproducibility, we pre-define and fix a sequence of Nmax seeds to synthesize densely annotated images.
For synthetic pre-training, we adopt exactly the same training protocols as real images. Then, during fine-tuning, the base learning rate is decayed to be half of the normal learning rate. Since our whole model parameters are pre-trained with synthetic images, the fine-tuning learning rate is the same throughout the whole model. The model is pre-trained and fine-tuned for the same iterations real images. For joint training with real and synthetic images, real images are over-sampled to the same scale as synthetic images make each mini-batch evenly composed of real images and synthetic images. The batch size is the same as real-imm real-image training. Each real image is iterated for the same number of epochs as the real-image training paradigm.
As for other parperrrrameters.. e.g., data augmentations and evaluation protocols, they are set exactly the same those in regular training paradigms. We adopt the MMSegmentation codebase for our development. We use 8 x Nvidia Tesla V100 GPUs for our training experiments.
B
The Most Improved Classes
We list the most improved ten classes on ADE20K (the gain is measured by IoU): (1) ship: +68. 19, (2) microwave: +48. 72, (3) arcade machine: +45.85, (4) booth: +45.66, (5) oven: +30.86, (6) skyscraper: +23.23, (7) swimming pool: +15.52, (8) armchair: +14.6, (9) hood: +14.43, (10) wardrobe: +13.24.
C Discussions for Future Works and Limitations
Future works. In this work, use the off off-the-shelf semantic image synthesis model to generate densely annotated images. We have validated that the fully-supervised baseline can be remarkably boosted with these synthetie training pairs. We expect more considerable improvements can be achieved in future works by (I) better-trained or larger-scale pre-trained generative models, (2) larger-scale synthetic training pairs, and (3) taking the class distribution into consideration during image synthesis.
Limitations. It is relatively time-consuming produce synthetic training pairs. For example, it takes around 5.8 seconds to synthesize a single image with a V100 GPU. practice, we speed up the synthesis process with 24 V100 GPUs. Therefore can construct the entire synthetic training set for ADE20K and COCO-Stuff in two days. addition, considering the great potential of our densely annotated synthetic images, it will be more practical to apply proposed roadmap to real- world scenarios, e.g., medical image analysis and remote sensing interpretation. We plan to conduct these explorations in future works.
D Visualization of Densely Annotated Synthetic Images and Filtered Regions
We display our diverse densely annotated synthetic images in Figure 6 of ADE20K and Figure of COCO-Stuff. Besides, we also visualize our filtered regions during training for each synthetic image. It can be observed that there exist several patterns of filtered regions, e.g., boundary regions, synthesis failure cases, and small or rare objects. Please refer the following pages for details.
15

---

Real Image
Semantic Mask
Diverse Synthetic Images & Filtered Region (White Region)
Figure 6: Visualization of diverse densely annotated synthetic images ADE20K, well the filtered regions (white regions the semantic mask). Note that the black regions the masks are officially marked "ignored region" by the original ADE20K dataset.
16

---

Real Image
Semantic Mask
Diverse Synthetic Images & Filtered Region (White Region)
Figure 7: Visualization of diverse densely annotated synthetic images on COCO-Stuff, as well as the filtered regions (white regions in the semantic mask).
17


Reviews:
[{"review_id": "Ox7thorSkL", "reviewer": "Reviewer_3GaN", "summary": "This paper proposes a method of synthesizing training images and corresponding semantic masks for training a semantic segmentation network. The off-the-shelf semantic image synthesis model, FreestyleNet, is used to generate images from existing semantic masks. Following the proposed re-sampling technique based on mask-level hardness, harder samples are more frequently generated. During training, to avoid the noisy pixel hampering the model training, pixel-level ignoring technique is used. The generated synthetic images are shown to be effective for model training when they are used together with the existing fully supervised labels.", "questions": "- Do the authors have a plan to release the code? I strongly recommend the authors to make their code publicly available.\n\n", "limitations": "No limitation is discussed.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is overall well-written and easy to understand.\n\n- It is interesting that the synthesized images can improve the performance together with fully supervised dataset. This can be practically utilized for many researchers.", "weaknesses": "- The proposed method heavily depends on trained mask-to-image generative models. The authors showed that naive generation of synthetic images is not sufficient for training a segmentation network, but the filtering and re-sampling techniques are quite naive. Specifically, ignoring uncertain pixels during training is popularly used for label-efficient learning (e.g., weakly and semi-supervised semanic segmentation).\n\n- Closely related references, copy-paste methods (e.g., [ref1]), are missing. They also augment training data by synthesizing images, but unlike the proposed method, they do not require any additional heavy models. The copy-paste method should be discussed and compared.\n\n[ref1] Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation\n\n- In Abstract, the authors mentioned that \"We surprisingly observe that, merely with synthetic images, we already achieve comparable performance with real ones\", but I think it is overstated. To synthesize these images, the trained FreestyleNet is required, but FreestyleNet is already trained with real image-mask pairs. In addition, all the technical design (filtering, re-sampling) and values of hyper-parameters are determined with fully supervised validation data. I recommend the authors to tone down the sentence in Abstract.\n\n- I guess the global hardness in Line 229 do not consider the difficulty of segmenting small objects. Intuitively thinking, small objects of hard class should increase the global hardness, but they actually slightly contribute to the global hardness.\n\n- The authors used only the limited number of synthesized images due to the disk issue. I recommend the two additional experiments: 1) the performance change by varying the number of synthesized images. With this trend, we can infer if more synthesized images can further improve the performance. 2) Increase the total number of synthesized images by saving low-resolution images or high-tolerence polygon of masks. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for your efforts and constructive feedback. We hope the concerns are well addressed.\n\n**Q1-1: Novelty of our filtering and re-sampling strategies.**\n\nPlease refer to our global response for clarification on the filtering strategy.\n\nPlease refer to our response Q2-1 to Reviewer 6xi1 for clarification on the re-sampling strategy. Thank you.\n\n**Q1-2: Discussion with Copy-Paste methods**\n\nThank you for your constructive advice. We agree with you that we should include mixing-based augmentation methods, *e.g.*, Copy-Paste and CutMix, in our related works. We will provide thorough discussions and comparisons in the revised version.\n\n**[Differences]** Existing mixing-based methods fail to generate realistic images, due to ignoring semantic layouts and co-occurrence. Besides, their *re-assembled* images do not contain any novel objects. In contrast, as supported by our visualizations, our method synthesizes extremely realistic novel images and objects, benefiting the perception task significantly.\n\n**Q1-3: Overstatement about \"merely using synthetic images is comparable with real images\"**\n\nThank you for your kind reminder. We will weaken the tone in the revised version.\n\n**Q1-4: Small but hard objects contribute less to mask-level hardness**\n\nThank you for pointing it out. It is true that small but hard objects currently contribute less to mask-level hardness. We conjecture it will be beneficial if these objects can be correctly handled. However, our work aims to propose a *universal* framework to assist real images *across all classes and all objects*. We hope to highlight the value of synthetic data to high-level perception task, especially in the diffusion-model era. We demonstrate the value of simple data filtering and re-sampling principles to synthetic data, *even without taking special cases into account*. In our future works, we will follow your advice to tackle small objects discriminately.\n\n**Q1-5: Performance change with respect to more synthetic images**\n\nAfter submission, we have attempted to increase $N_{\\max}$ from 20 to 40, which means each semantic mask corresponds to at most 40 synthetic images. However honestly, we do not observe an evident improvement. The results under $N_{\\max}=20$ and $N_{\\max}=40$ are similar. We think there are at least three reasons for this phenomenon: \n- **[Extremely challenging scenario]** We aim to boost the fully-supervised baseline, which is significantly more challenging than the few-shot baseline as existing works do, *e.g.*, DiffuMask. We believe it is more practical in the real world to improve such a challenging but widely acknowledged baseline.\n- **[Remarkable improvements have been achieved by $N_{\\max}=20$]** As shown in appendix Tab 1, we remarkably improve the fully-supervised baseline from 48.7% $\\rightarrow$ 52.0% (**+3.3%**) on ADE20K with Mask2Former-Swin-T. As a comparison, the research line of model-centric works, *e.g.*, SegViT [NeurIPS'22], only improves its precedent StructToken from 50.9% $\\rightarrow$ 51.3% (+0.4%). Across our investigated **seven** architectures (appendix Tab 1), we improve the fully-supervised baseline by **+2.0%** on average, which is a much larger gain than existing model-centric works achieve (mostly only 1%, requiring many trials and errors on model designs).\n- **[Small-scale semantic masks]** We synthesize images conditioned on limited semantic masks from the real dataset (20K masks on ADE20K). These masks are small-scale and not diverse enough. Thus, multiple synthetic images from a shared mask may be redundant and not informative enough. This makes extra synthetic images bring limited further gain. However, as a pioneering work to enhance the challenging fully-supervised semantic segmentation with synthetic data, we think it is acceptable that there is still some room for subsequent works to refine these designs. In the future, there may be some ways to first generate novel and diverse semantic masks for later image synthesis.\n\n**Q1-6: Code release**\n\nWe promise to release all our codes, well-trained models, and training logs upon acceptance.\n\n**Q1-7: No limitation is discussed**\n\nWe have indeed discussed it in Appendix Section D. We will prioritize it to the main paper in the revised version.\n"}, {"review_id": "7GqC1gjep7", "reviewer": "Reviewer_6xi1", "summary": "This paper proposes to generate densely annotated synthetic images with generative models to help supervise the learning of fully supervised semantic segmentation frameworks. To improve the effectiveness of synthetic images, the authors further design a robust filtering criterion to suppress noisy synthetic samples at the pixel and class levels and propose an effective metric to indicate the hardness of semantic masks where they sample more synthetic images for harder masks. Ablation studies validate the effectiveness of the proposed method.", "questions": "Questions are listed in the weaknesses.", "limitations": "Limitations have been discussed in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The logic of the article is generally clear, and the method is easy to understand.\n2. The ablation experiments of this paper indicate the effectiveness of the proposed method to a certain extent.", "weaknesses": "1. The proposed strategy termed re-sampling synthetic images based on mask-level hardness is somehow like “Online Hard Example Mining” (OHEM) which is widely used in computer vision area including semantic segmentation, and the adopted metric to measure the sample hardness is also the widely used average losses of all pixels in the input image. Where is the novelty of this part? I don't think generating more hard samples should be the contribution of this section since it is more likely to belong to the contribution of Section 3.1.\n2. The analysis in Section 4.4 for Table 6 is not convincing in my eyes. First, after adopting “Filtering & Re-sampling”, N_max no longer denotes for the number of synthetic images used for training, thus the comparison in Table 6 is unreasonable. If I understand correctly, the number of used synthetic images should be n_p in Eq. (2), which may result in a wrong conclusion in Line 339-348. Second, where is the performance ceiling when setting N_max after adopting “Filtering & Re-sampling”? It looks like setting larger N_max (> 20) will yield better segmentation results. Finally, how to set p in Eq. (2). The reviewer does not find this detail in both paper and the supplement materials.\n3. Why there are no quantitative comparisons between the proposed method and previous methods like DatasetGAN and BigDatasetGAN? The authors argue that “the main drawback of such methods is the involvement of expensive human efforts.” So, how about applying a simple pseudo labeling strategy on these generated images and comparing the results between image-pseudo labeling strategy and your proposed mask-to-image synthesis strategy? If the results are comparable, where are the advantages of using your proposed method? The reviewer believe it is important to compare your method and previous similar methods quantitatively to show the novelty of this paper.\n4. It seems that the performance improvements in Table 3 for Mask2Former is limited. To my knowledge, just simply run Mask2Former for two times may also bring such improvements. Could the author give some analysis about the limited gains for Mask2Former?\n5. The filtering strategy in Section3.2 is naïve and not reliable in my eyes. Specifically, could the authors provide any quantitative results to show that the proposed filter strategy could filter noisy synthetic region rather than some hard samples? \n6. Previous studies like “Focal Loss for Dense Object Detection” indicate that OHEM is unreasonable. Whether the authors compare your re-sampling strategy with some objective-function-based strategy like Focal Loss, weighted cross entropy loss to show the effectiveness of your method? \n7. Why higher mIoU must be a good point is Table 4? For example, if your method could generate some densely annotated synthetic images with other domains which is different from the training and testing image? Whether it would lead to the decrease of mIoU but make the segmentor adapt to broader application scenarios? The reviewer thinks the latter is more important.\n8. Could the authors give the results of performing the methods in Section 3.2 and 3.3 on the real training images? I mean filter the real images with the proposed strategies and re-train the model to see the importance of the generated images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for your efforts and constructive feedback. We hope the concerns are well addressed.\n\n**Q2-1: Our re-sampling strategy is similar to OHEM**\n\nIn L236-L241, we have compared our re-sampling method with OHEM. In semantic segmentation, OHEM ignores high-confidence pixels and only computes the average loss on low-confidence pixels.\n\n**[Relation]** Our motivations are related, *i.e.*, both emphasizing hard samples (however, our hard samples are semantic masks, while OHEM is image pixels). \n\n**[Difference]** Our practices are fundamentally distinguished. We \"generate\" additional hard samples for models to sufficiently learn, while OHEM essentially only performs a one-hot \"re-weighting\" (loss weight 0 for high-confidence pixels, and weight 1 for low-confidence pixels).\n\n**[Superiority]** We *quantitatively* compare our re-sampling method with OHEM below, proving our method is evidently superior to OHEM. (Results below are obtained by training solely on synthetic images with Segmenter-ViT-S, and the filtering strategy is applied to all methods).\n\n|Baseline|+ OHEM (thresh=0.7, min kept=100K)|+ Our re-sampling|\n|:-:|:-:|:-:|\n|44.0|44.2|**45.4**|\n\n**Q2-2: How to set $p$ in Eq 2**\n\nAs mentioned in L233 \"for the $p$-largest-hardness mask\", the $p$ denotes *the rank of a mask* in terms of hardness, ranging from 0 to $(N-1)$ as an integer. According to this $p$ (hardness rank), we then determine the synthesis quota (number of synthetic images) for a mask by Eq 2.\n\n**Q2-3: Tab 6 is not convincing, because $N_{\\max}$ does not denote the number of synthetic images when using re-sampling**\n\nIn the re-sampling case, we actually use fewer synthetic images than the non-re-sampling counterpart, so the better performance from our re-sampling method is convincing. Specifically, \n- when **not using** re-sampling, all semantic masks share the same synthesis quota, *i.e.*, always $N_{\\max}$ synthetic images *from a single mask*.\n- when **using** re-sampling, most semantic masks are equipped with fewer than $N_{\\max}$ synthetic images. The concrete synthesis quota for each mask is determined by Eq 2, which evenly distributes the number of synthetic images for all semantic masks from 1 to $N_{\\max}$. The number of total synthetic images is nearly halved after the re-sampling process. Hence, our superiority over the non-re-sampling counterpart is convincing.\n\n**Q2-4: The performance ceiling when increasing $N_{\\max}$**\n\nPlease refer to our response Q1-5 to Reviewer 3GaN.\n\n**Q2-5: Comparison with DatasetGAN, ***i.e.***, annotating pseudo semantic masks for synthetic images**\n\nThank you for your constructive feedback.\n\n(1) Following your advice, we predict pseudo masks for our synthetic images with the SOTA model (ViT-Adapter-BEiTv2-L-IN22K, ICLR'2023) on ADE20K. The mIoU between predicted pseudo masks and GT masks is 49.79. The final validation mIoU is compared below.\n|Image-to-Pseudo-Mask|Mask-to-Image (Ours)|\n|:-:|:-:|\n|44.0|**45.4**|\n\n(2) Besides, as an extension, we further borrow the real COCO data as an unlabeled source for ADE20K. We validate whether COCO images along with the pseudo labeling strategy can benefit our targeted ADE20K. The results are listed below (\\*: reproduced by us).\n|Real Only\\* (Segmenter-ViT-S)|Real + COCO (Pseudo labeling)|Real + Synthetic (Ours)|\n|:-:|:-:|:-:|\n|45.8|46.2|**48.0**|\n\n**Q2-6: Improvement with Mask2Former-Swin-L-22K is not significant (56.0 $\\rightarrow$ 56.4)**\n\nAs noted in Tab 3 caption, this specific model is sufficiently pre-trained on the extremely large-scale ImageNet-22K. Therefore, its hunger for downstream fine-tuning data is diminished. For other Mask2Former models, our framework performs impressively, *e.g.*, **+3.3%** with Swin-T and **+1.6%** with Swin-S (appendix Tab 1 \\& 3).\n\n**Q2-7: Could the filtering strategy filter noisy synthetic regions rather than hard samples?**\n\nWe present average losses on filtered/non-filtered real/synthetic regions below. First, we compare the loss on non-filtered regions (id 1 and 2), the loss magnitude of real and synthetic regions are close. Then when we compare the loss on filtered regions (id 3 and 4), it is obvious that the synthetic loss becomes much larger (nearly 3$\\times$) than the real loss. Thus, we can conclude there must exist abundant noise in the filtered synthetic regions.\n|Real non-filtered (id: 1) |Syn non-filtered (id: 2)|Real  filtered (id: 3)|Syn filtered (id: 4)|\n|:-:|:-:|:-:|:-:|\n|0.281|0.363|1.100|3.317|\n\n**Q2-8: Compare our re-sampling with objective functions, ***e.g.***, Focal loss and weighted CE loss**\n\nResults are listed below, where our re-sampling strategy performs much better than the mentioned objective functions.\n\n|Baseline|+ Focal loss|+ Weighted (by class frequency) CE |+ Weighted (by val IoU) CE|Our re-sampling|\n|:-:|:-:|:-:|:-:|:-:|\n|44.0|43.9|37.8|43.9|**45.4**|\n\n**Q2-9: Why higher mIoU in Tab 4 is better, what if considering the robustness?**\n\nWe manage to examine the capability of our model to deal with unseen domains. Concretely, we transfer our model trained on ADE20K to COCO. We measure mIoU on the 46 overlapped classes. As proved below, our two strategies are still effective for images from unseen domains.\n|Baseline|+ Filtering |+ Re-sampling|+ Filtering \\& Re-sampling|\n|:-:|:-:|:-:|:-:|\n|37.7|40.3|39.4|**41.3**|\n\n**Q2-10: Apply the two strategies to real images**\n\nAs for our re-sampling strategy, since there is only a single real image available for a semantic mask, we resort to over-sample (repeat) more real images for harder masks, whose number is decided by Eq 2. As shown below, our re-sampling strategy can still boost the real dataset. And it is expected that the filtering practice downgrades the result on real dataset, because the real dataset is free from noise. This further demonstrates that our filtering design is well suited to synthetic data.\n|Baseline|+ Re-sampling|+ Filtering |+ Filtering \\& Re-sampling|\n|:-:|:-:|:-:|:-:|\n|45.8|**46.7**|43.1|43.5|\n"}, {"review_id": "Ve9l89zcmo", "reviewer": "Reviewer_FZD9", "summary": "The Paper talks about using synthetic images generated using generative models as the training set to achieve stronger semantic segmentation models. The efficacy of the model is evaluated on ADE2K and COCO dataset, using SegFormer model. Authors propose pretraining with synthetic and joint images to evaluate which one works the best and in which scenario. Uses sampling and filtering training mechanism to further help the results.", "questions": "1. Would be interesting to know, what is the class frequency for the new generated  dataset used to train the model. Does it follow the similar class frequency distribution as seen in ADE20K and COCO?\n\n2. What are the classes for which the improvement is seen to be significant?", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper demonstrates a clear structure, effectively explaining the fundamental modules and training procedure involved. It successfully addresses the task of scene understanding by leveraging annotations obtained from generative models, thereby enhancing the results of the semantic segmentation task.\n\n2. The inclusion of a self-adaptive module within the training pipeline is a valuable contribution. This module effectively refines erroneous or spurious training examples, leading to improved model performance.\n\n3. The paper incorporates a sampling strategy that focuses on hard-to-learn cases. By prioritizing these challenging instances, the overall performance of the model is enhanced, resulting in better final results.", "weaknesses": "Methods used to improve the performance.\n1. Mining Hard Examples: This is not something novel and (different flavors of it) has been used in many previous work to help in training better examples in case semantic segmentation and object detection.\n\n2. Remove extraneous and harmful samples: Again the filtering mechanism proposed here is not that novel as previous works in the domain of active learning have again used similar techniques to improve training.\n\n3. Use off-the-self models both for generation and training the model, nothing novel in that regard.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for your efforts and constructive feedback. We hope the concerns are well addressed.\n\n**Q3-1: Mining hard examples are not novel**\n\nWe are only related to existing works (*e.g.*, OHEM) *from the aspect of motivation*. It is indeed a widely shared motivation. However, we use totally different approaches. We produce more synthetic images for harder masks, which is fundamentally different from OHEM that simply ignores high-confidence pixels. We allocate synthesis quotas discriminatively, while OHEM only performs selection. Our designed method is well suited to our framework that aims to learn from synthetic data. You may also refer to our response Q2-1 to Reviewer 6xi1 for more details. In Q2-1, we also quantitatively demonstrate the superiority of our re-sampling strategy over OHEM.\n\n**Q3-2: Compare our filtering strategy with the arts in active learning**\n\nOur filtering strategy *discards noisy synthetic regions* (failure cases during synthesis), while active learning aims to find the most informative samples for humans to annotate. The most informative samples in active learning mostly exhibit large or middle losses, while large-loss samples in our scenario are noisy and need to be discarded. Therefore, our motivations are totally different, even contrary to active learning.\n\n**Q3-3: Using off-the-shelf models both for generation and training the model is not novel**\n\nWe hope to highlight that we are indeed a pioneering work to improve the *fully-supervised semantic segmentation performance* with *synthetic data*. Our motivation and our framework are novel in several aspects. We have thoroughly compared our work with previous works in Related Work--\"Learning from synthetic images\". **[Differences]** Briefly summarized here:\n- **[Few-shot *vs*. fully-supervised]** Existing works only focus on a constrained scenario, *e.g.*, few-shot labels, while we address the challenging but widely acknowledged fully-supervised scenario.\n- **[Classification *vs*. semantic segmentation]** Existing works mostly address the classification task, which is cheap to label and even not so urgent for human labels (unsupervised learning performs quite well), while our targeted semantic segmentation task is highly expensive and laborious to annotate.\n- **[Image-to-pseudo-mask *vs*. mask-to-image]** Existing works predict pseudo masks for synthetic images, which is not precise enough, while we adopt a mask-to-image synthesis pipeline to produce better aligned image-mask pairs, as evidenced by our response Q2-5 to Reviewer 6xi1.\n- **[Blindly using *vs*. carefully processing synthetic data]** Existing works of learning from synthetic data ignore the importance of processing synthetic data discriminatively, while our proposed filtering and re-sampling can impressively improve the effectiveness of synthetic data and ultimately yield a much stronger model.\n\nWe provide more discussions in the global response. Please refer to it if you are still concerned. Thank you very much.\n\n**Q3-4: Class frequency of synthetic set and real set**\n\nSince we synthesize images conditioned on semantic masks from the real training set, the class frequency of our synthetic set is exactly the same as the real set if no filtering and re-sampling strategies are applied. Then, if we apply filtering and re-sampling strategies to the synthetic set, its class frequency will be changed. Please refer to our uploaded global PDF for detailed visualizations of the class frequency. Thank you.\n\n**Q3-5: The most improved classes**\n\nWe list the most improved ten classes on ADE20K (the gain is measured by IoU): (1) ship: +68.19, (2) microwave: +48.72, (3) arcade machine: +45.85, (4) booth: +45.66, (5) oven: +30.86, (6) skyscraper: +23.23, (7) swimming pool: +15.52, (8) armchair: +14.6, (9) hood: +14.43, (10) wardrobe: +13.24."}, {"review_id": "7ECBxtBN25", "reviewer": "Reviewer_jaKB", "summary": "The paper introduces an automatic dataset generation with mask-to-image translator. The proposed dataset generator enables to generate controllable and consistent semantic labels in generated images. The labels can be treated as fully supervised teachers from a generator and create pre-trained segmentation models under the (synthetic) supervised learning. Moreover, the dataset generation framework serves difficulty levels inside of the contents in an image. In the framework, the authors utilize FreestyleNet pre-trained with StableDiffusion in order to translate from mask to image (mask-to-image) for the pre-training pairs.", "questions": "- Can the proposed dataset generator with FreestyleNet precisely generate the training images by taking care of object boundaries? As the paper described in l.144-145, \"We observe it is more precise in structures to synthesize images from masks than predict masks for images, especially in boundary regions\". However, the reviewer is concerning that the quality of object boundaries may affect the recognition accuracy even if the process passed as mentioned in Section 3.2 filtering noisy synthetic regions.", "limitations": "There are no negative limitations and societal impacts. On the contrary, the paper alleviate the privacy issues by means of synthetic image (pre-)training. The direction can effectively accelerate the kind of ethical problems in the future.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is clearly written and easy to understand. The presentation to describe the proposed method (e.g., Figures 2, 3, 4) is high quality and convincing in visualization. \n\n- The paper could serve as a good example for data-driven approach in semantic segmentation tasks.\n\n- Experiments and their results cover multiple aspects of the performance of the synthetic pre-training. The paper will serve a good inspiration to others in synthetic pre-training and related topics. \n\n- The two different aspects, 'noise filtering (Section 3.2)' and 'image re-sampling (Section 3.3)' in synthetic pre-training are very reasonable approaches. Indeed, these two have been shown to be effective as shown in Tables 4 and 5. These two approaches are complementarily improve the segmentation performance in terms of mIoU.", "weaknesses": "The reviewer does not find a critical weakness in the paper. However, it could be shown a little weaknesses.\n\n- The paper includes the descriptions as 'controllable' in l.38 and l.179, however, could the proposed approach make more controllable dense annotations? For example, a semantic mask can be manually edited in addition to the image dataset as is. Could the authors consider the kind of flexibly edited approach by humans? As an example of image editing and generation, the reviewer can raise GauGAN from [Park+, CVPR19]. The reviewer doesn't think any image editing should be acceptable, but it would be effective in terms of making the synthetic segmentation dataset more flexible.\n\n[Park+, CVPR19] Taesung Park et al. \"Semantic Image Synthesis with Spatially-Adaptive Normalization\" in CVPR 2019.\n\n- This is not a critical weakness, however, can alone with the synthetic pre-training surpass the real-image dataset? In fact, the single approach with synthetic images can reach comparative scores as shown in Tables 1 and 2. In some cases, the gap is quite close each other (e.g., 48.5 vs. 48.3 in SegFormer-B4 on ADE20k dataset). If it is possible on paper limitation, adding the discussion would make the paper more valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for your appreciation of our work. Many thanks for your efforts and constructive feedback. We hope the concerns are well addressed.\n\n**Q4-1: How can the proposed method be more \"controllable\"?**\n\nThank you. We claim our framework of learning from synthetic images is more controllable, because we believe that in future works, we can edit the semantic masks to construct a targeted synthetic set. For example, considering the class imbalance issue, we may use *Copy-Paste on semantic masks* to produce new layouts that are inclined to rare classes to re-balance the class distribution. During the process, we also need to take the structure and object co-concurrence into account. In addition, we, or other works, may design a new generator for mask synthesis, which is conditioned on class distributions directly.\n\n**Q4-2: Can synthetic images surpass real images alone?**\n\nSince current results achieved by synthetic images are extremely close to real images, we expect synthetic images alone can surpass real images in the future when (1) more powerful image generators are utilized and (2) more effective processing strategies are proposed for synthetic images.\n\n**Q4-3: The quality of boundary alignment**\n\nThank you. As shown in Fig 1 of our main paper and according to our manual check, we find the boundary alignment between synthetic images and conditioned masks is highly precise. The alignment quality is much higher than widely adopted pseudo labeling strategies that predict masks from input images. We safely conjecture that this is because our input semantic masks are encoded in the discrete one-hot format, which is \"sharp\" and much easier to recognize boundaries than the \"smooth\" RGB values that DatasetGAN feeds.\n"}, {"review_id": "PYajaIYVK0", "reviewer": "Reviewer_VTnr", "summary": "The authors propose to use paired synthetic data to train a semantic segmentation model. Specifically, a filtering strategy and a resampling strategy are proposed to control the quality of synthetic data. In this way, the paired synthetic data could further promote the standard segmentation model.", "questions": "See *Weaknesses.", "limitations": "See *Weaknesses.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The overall idea makes sense, and the implementations are reasonable.\n- The paper is well written and easy to follow.", "weaknesses": "- There may be conflict between filtering hard pixels and re-sampling hard masks, because the re-sampled hard samples may be filtered out. More in-depth analysis could be appended to solve this concern.\n- The performance gains  (e.g., 48.5 → 50.6) seems not significant enough, considering that the proposed method introduce too many extra heavy processings. Specifically, 1) fine-tune the generative model on the specific dataset; 2) pre-train a segmentation model, i.e., Line 197; 3) finally train the desired segmentation model.\n- The effectiveness of the proposed method depends on the gap between the generative model and specific dataset, and thus the in-depth analysis about the adaption (fine-tuning) of generative model is necessary. On the one hand, the generative model is hard to adapt when the specific dataset is small. On the other hand, the generative model is harder to provide rich cases when the specific dataset is large. Some in-depth analyses could be appended.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for your efforts and constructive feedback. We hope the concerns are well addressed.\n\n**Q5-1: Conflict between two strategies: the generated additional hard samples are filtered out**\n\nOur filtering strategy does not discard all hard samples. It mainly aims to detect noisy synthetic regions, rather than ignoring all hard samples. As described in L205-207 of our main paper, we set a tolerance margin for filtering, which enables the middle-hardness samples to be kept and mainly removes synthesis failure cases with extremely large losses, as also evidenced by the quantitative results in our response Q2-7 to Reviewer 6xi1. Therefore, the additionally generated images for hard masks are beneficial.\n\n**Q5-2: Improvement (48.5 $\\rightarrow$ 50.6) is not significant enough, especially when the method is sophisticated**\n\n**[Our improvement is much larger than model-centric works]** Actually, the +2.1% gain over the fully-supervised performance on ADE20K is already a tremendous improvement. Across our investigated seven architectures (appendix Tab 1), we improve the fully-supervised baseline by +2.0% on average. **In comparison**, the hot research line of model-centric works, *e.g.*, SegViT [NeurIPS'22], only improves its precedent StructToken from 50.9% $\\rightarrow$ 51.3% (+0.4%). \n\n**[Our pipeline is straightforward]** Moreover, our framework is straightforward to implement and easy to deploy. We do not have to use synthetic images for pre-training and then fine-tuning. We can simply combine them with real images for joint-training. There is only a single stage of training in this case. We demonstrate the strong results under this joint-training scenario in main Tab 1 and appendix Tab 1. It is also worth noting that, once our synthetic images have been generated for a specific application scenario, they can be saved for later use and benefit various types of model architectures. We will also open-source our large-scale ADE20K and COCO synthetic training images for our community to use.\n\n**Q5-3: How to ensure generation quality with small datasets? Will the effectiveness be diminished on large datasets?**\n\n**In case of small datasets:** \n- **[Intuitive analysis]** Owing to the highly capable pre-trained text-to-image diffusion model, the generation quality is still very promising even with limited fine-tuning images. A good example of this is DreamBooth, which can only use a few (3-5) images to personalize a generation model. More importantly, even if the synthesis quality is not promising in some rare scenes, our proposed filtering strategy can safely ensure the remaining synthetic regions are relatively clean. In fact, considering that small datasets are especially hungry for training data, our delicately processed synthetic set will be precious to them and hopefully enhance the fully-supervised baseline remarkably. \n- **[Quantitative results]** To validate this, we select a subset of 1K images from ADE20K. We fine-tune the Stable Diffusion model only with these limited images, and produce 10K synthetic images (already filtered and re-sampled) with this fine-tuned generator. As a result, merely with 1K real images, the validation mIoU is 22.8%. After jointly trained with our 10K synthetic images, the performance is tremendously boosted from 22.8% $\\rightarrow$ 28.6% (**+5.8%**). This improvement indicates our framework can work very well in limited-data scenarios.\n\n**In case of large datasets:** Furthermore, as for the concern about a large real dataset, we have fully demonstrated the effectiveness of our method on the large-scale COCO in main Tab 2, main Fig 5, and appendix Tab 2 (the average improvement is 1.2% across **six** architectures). Note that COCO is one of the largest datasets with challenging taxonomy in semantic segmentation, consisting of 118K training pairs. Lastly, we hope to emphasize that it is extremely difficult and costly to collect million-scale training pairs in semantic segmentation (the latest Segment Anything dataset lacks semantic labels). Therefore, we believe our proposed roadmap of utilizing high-quality synthetic pairs to complement the real dataset is of great value.\n"}]

Author rebuttal:
**[Contributions]** Our technical contributions mainly lie in three folds:
- **[New target \& new roadmap]** We present a new roadmap to enhance *fully-supervised* semantic segmentation via generating *densely annotated* synthetic images with generative models. Our data-centric perspective is orthogonal to the widely explored model-centric (*e.g.*, network architecture) perspective. 
- **[New problem]** We highlight the necessity of designing processing strategies for synthetic images. With our present simple filtering and re-sampling strategies, the model trained with synthetic images can achieve comparable performance with the counterpart of real images, *e.g.*, 48.3 *vs*.  48.5 mIoU on ADE20K and 49.3 *vs*. 50.5 on COCO-Stuff.
- **[Stronger performance]** We achieve 2.0% improvement on average on ADE20K across seven architectures, which is a much larger gap than previous model-centric works achieved (mostly 1%). We believe this will inspire more future works to investigate this promising direction.

**[Uniqueness]** Our work is distinguished from existing works in that:
- **[Few-shot *vs*. fully-supervised]** Existing works only focus on a constrained scenario, *e.g.*, few-shot labels, while we address the challenging but widely acknowledged fully-supervised scenario.
- **[Classification *vs*. semantic segmentation]** Existing works mostly address the classification task, which is cheap to label and even not so urgent for human labels (unsupervised learning performs quite well), while our targeted semantic segmentation task is highly expensive and laborious to annotate.
- **[Image-to-pseudo-mask *vs*. mask-to-image]** Existing works predict pseudo masks for synthetic images, which is not precise enough, while we adopt a mask-to-image synthesis pipeline to produce better aligned image-mask pairs, as evidenced by our response Q2-5 to Reviewer 6xi1.
- **[Blindly using *vs*. carefully processing synthetic data]** Existing works of learning from synthetic data ignore the importance of processing synthetic data discriminatively, while our proposed filtering and re-sampling can impressively improve the effectiveness of synthetic data and ultimately yield a much stronger model.

Lastly, we hope to highlight the filtering strategy only accounts for a (small) portion of our whole work. With this design, we mainly want to emphasize the necessity of processing synthetic images, **which is rarely considered in previous works**. Indeed, we prefer this motivation, compared with concrete instantiations. We think we raise a new problem for future works about *how to better learn from synthetic data*, instead of simply focusing on better synthesis. Besides, we are aware our proposed filtering design is related to existing works (**@3GaN**) in semi-supervised learning, but we adopt similar motivations for totally different scenarios. We will add more discussions in the revised version.

Meta-review: Reviewers missed a discussion of mixing-based strategies and a comparison to mixing-based strategies and prior GAN-based approaches to generating datasets. Reviewers were concerned about the triviality of the proposed filtering & sampling (in particular vs. online hard example mining (OHEM)), the contribution to the results, and missing comparisons to loss-based alternatives to resampling.

The rebuttal addressed these concerns by 
- an additional experiment on mixing-based strategies and promising to add a discussion.
- an additional comparison against a DatasetGAN-based baseline, as requested by [6xi1].
- an ablation study regarding the filtering & sampling vs baseline FreestyleNet.
- a comparison to OHEM

All reviewers acknowledged the rebuttal and found critical concerns addressed. 
In light of the responsive reviewers finding their concerns addressed and the experimental results consistently supporting the value of the proposed contributions, the AC recommends accepting the paper.
Decision: Accept (poster)