SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pre-training. The segmentation model is first trained with densely annotated synthetic images for the same number of iterations as real images. During synthetic pre-training, we test the transferring performance on the real validation set. By comparing this performance with that obtained by real training images, we can gain better intuition on the gap between synthetic and real images. Next, we further fine-tune the pre-trained model with real images. This paradigm can effectively overcome the noise caused by synthetic images, especially when synthetic images are of relatively poor quality.
Joint training. The synthetic set is usually larger than the real set. Meanwhile, the real training pairs are of higher quality than synthetic ones. Therefore, we first over-sample real images to the
Model	Training Data	mIoU
Real	Synthetic
SegFormer-B4 [70]	✓	48.5
✓	48.3
Mask2Former [14]	✓	56.0
✓	53.5
Table 1: Transferring performance on ADE20K. The validation set is real images. The Mask2Former uses a Swin-L [41] backbone pre-trained on ImageNet-22K.
Model	Training Data	mIoU
Real	Synthetic
SegFormer-B4 [70]	✓	45.8
✓	44.4
ViT-Adapter [13]	✓	50.5
✓	49.3
Table 2: Transferring performance on COCO-Stuff. The validation set is real images. The ViT-Adapter uses UperNet [69] and a BEiT-L backbone [5].
Model	Backbone	Training Data	mIoU	Δ
Real	Synthetic
Mask2Former [14]	Swin-T [41]	✓	48.7	↑3.3%
✓	✓	52.0
Swin-S [41]	✓	51.6	↑1.7%
✓	✓	53.3
Swin-B [41]	✓	52.4	↑1.3%
✓	✓	53.7
SegFormer [70]	MiT-B2 [70]	✓	45.6	↑2.3%
✓	✓	47.9
MiT-B4 [70]	✓	48.5	↑2.1%
✓	✓	50.6
Segmenter [60]	ViT-S [21]	✓	46.2	↑1.7%
✓	✓	47.9
ViT-B [21]	✓	49.6	↑1.5%
✓	✓	51.1
Table 3: Integrating synthetic images by joint training on ADE20K to enhance the real-image baseline.
same number as synthetic images. Then we simply train a segmentation model on the combined set. Compared with pre-training, joint training can better preserve the knowledge acquired from the larger-scale synthetic set. It may be more promising when the transferring performance of synthetic images is exceptionally close to that of real images.
We describe the implementation details in Section 4.1. Then we report the transferring performance of synthetic training pairs to real validation set in Section 4.2. Further, we leverage synthetic images to boost the fully-supervised baseline in Section 4.3. Finally, we conduct ablation studies in Section 4.4.
We evaluate the effectiveness of synthetic images on two widely adopted semantic segmentation benchmarks, i.e., ADE20K [76] and COCO-Stuff [9]. They are highly challenging due to the complex taxonomy. COCO-Stuff is composed of 118,287/5,000 training/validation images, spanning over 171 semantic classes. In comparison, ADE20K is more limited in training images, containing 20,210/2,000 training/validation images and covering 150 classes.
We use two open-source checkpoints from FreestyleNet [71] for ADE20K and COCO respectively, to produce photo-realistic synthetic images conditioned on training masks. We synthesize at most 20 extra images for each semantic mask on ADE20K, i.e.,  $N_{\mathrm{max}} = 20$  in Equation 2. On COCO, considering its large number of semantic masks, we set a smaller  $N_{\mathrm{max}} = 6$  due to the limitation of disk space on our server. By default, we set the tolerance margin  $\alpha$  as 1.25 for all experiments. In pre-training, we exactly follow the hyper-parameters of regular training. In joint training, we
Model	Backbone	Training Data	mIoU	Δ
Real	Synthetic
Mask2Former [14]	Swin-T [41]	✓	44.5	↑1.9%
✓	✓	46.4
Swin-S [41]	✓	46.8	↑0.8%
✓	✓	47.6
SegFormer [70]	MiT-B2 [70]	✓	43.5	↑0.7%
✓	✓	44.2
MiT-B4 [70]	✓	45.8	↑0.8%
✓	✓	46.6
Segmenter [60]	ViT-S [21]	✓	43.5	↑1.3%
✓	✓	44.8
ViT-B [21]	✓	46.0	↑1.5%
✓	✓	47.5
Table 4: Integrating synthetic images by joint training on COCO-Stuff to enhance the real-image baseline.
Model	Backbone	Training Data	mIoU	Δ
Real: Fine-tune	Synthetic: Pre-train
SegFormer [70]	MiT-B2 [70]	✓	45.6	↑1.4%
✓	✓	47.0
MiT-B4 [70]	✓	48.5	↑2.1%
✓	✓	50.6
Segmenter [60]	ViT-S [21]	✓	46.2	↑1.0%
✓	✓	47.2
ViT-B [21]	✓	49.6	↑1.0%
✓	✓	50.6
Mask2Former [14]	Swin-L† [41]	✓	56.0	↑0.4%
✓	✓	56.4
Table 5: Integrating synthetic images by pre-training on ADE20K to enhance the real-image baseline. †: The Swin-L backbone is pre-trained on the large-scale ImageNet-22K dataset.
over-sample real images to the same number of synthetic images. The learning rate and batch size are the same as the regular training paradigm. Due to the actually halved batch size of real images in each iteration, we double the training iterations to iterate over real training images for the same epochs as regular training. Other hyper-parameters are detailed in the appendix.
4.2 Transferring from Synthetic Training Images to Real Test Images
Existing semantic image synthesis works [61, 71] examine the synthesis ability by computing the FID score [26]. However, we observe this metric is not well aligned with the benefits brought by augmenting real image datasets with synthetic images, as also suggested in [47]. Therefore, similar to the Classification Accuracy Score proposed in [47], we exhibit the synthesis quality by transferring the model trained on synthetic images to real test images. The semantic segmentation score (mean Intersection-over-Union) on real test images is a robust indicator for synthesized images.
Using this score, we first validate the quality of our filtered and re-sampled synthetic images. We train a model using densely annotated synthetic images alone, and compare it with the counterpart of only using real training images. As shown in Table 1 of ADE20K, under the widely adopted SegFormer [70] model with a MiT-B4 backbone, the performance gap between using synthetic training images and real training images is negligible, i.e.,  $48.3\%$  vs.  $48.5\%$  mIoU  $(-0.2\%)$ . Then we attempt one of the most powerful semantic segmentation models, i.e., Mask2Former [14] with a heavy Swin-L [41] backbone pre-trained on the large-scale ImageNet-22K. Although the real-data trained model achieves a fantastic result  $(56.0\%)$ , our synthetic training images can also yield a strong
Dataset	Our Strategies	mIoU	Dataset	Our Strategies	mIoU
Filtering	Re-sampling	Filtering	Re-sampling
43.3	48.0
ADE20K (48.5)	✓	47.4	COCO (50.5)	✓	48.9
✓	45.1	✓	48.6
✓	✓	48.3	✓	✓	49.3
Table 6: Effectiveness of our filtering and re-sampling strategies for transferring performance. The results below the dataset names denote the transferring performance obtained with real training images.
Our Strategies	mIoU
Filtering	Re-sampling
48.2
✓	49.9
✓	49.3
✓	✓	50.6
Table 7: Effectiveness of our strategies when jointly training two sources of synthetic and real images.
Figure 5: Ablation study on the hyper-parameter  $\alpha$ .
model (53.5%). The marginal performance gap (-2.5%) is impressive, especially considering the large gap (around -8%) in the latest work [2] for synthetic classification.
The corresponding experimental results on COCO are shown in Table 2, and the observation is consistent. Either with SegFormer [70] or the state-of-the-art model ViT-Adapter [13], the transferring performance gap between synthetic and real images is constrained within  $-1.5\%$ .
We have demonstrated the impressive transferring ability of our filtered and re-sampled synthetic training images toward real test images. It indicates they have the great potential to completely replace the real training images in the future. They can play a significantly important role in many privacy-sensitive application scenarios. Furthermore, in Section 4.4, we will show that the transferring performance will decrease a lot without our filtering and re-sampling strategies.
4.3 Enhancing Fully-supervised Baseline with Synthetic Images
We further take full advantage of these synthetic images to enhance the fully-supervised baseline, which is merely trained with real images before. We investigate two training paradigms to integrate synthetic and real training images, i.e., (1) jointly training with both real and synthetic images, and (2) pre-training with synthetic ones, then fine-tuning with higher-quality real ones.
Under the joint training paradigm, on ADE20K of Table 3, with extra synthetic images, we examine the performance of a wide range of modern architectures. All models are evidently boosted by larger than  $1\%$ . Among them, the widely adopted Mask2Former-Swin-T is improved from  $48.7\% \rightarrow 52.0\%$  mIoU  $(+3.3\%)$ , and SegFormer-B4 is boosted from  $48.5\% \rightarrow 50.6\%$ $(+2.1\%)$ . Also, as demonstrated in Table 4 for COCO-Stuff, although the number of real training images is already large enough (120K), our densely annotated synthetic images further push the upper bound of existing state-of-the-art models. The Mask2Former-Swin-T model is non-trivially enhanced by  $1.9\%$  mIoU from  $44.5\% \rightarrow 46.4\%$ .
Moreover, under the pre-training paradigm, we report results on ADE20K in Table 5. The SegFormer-B4 model is similarly boosted by  $2.1\%$  as the joint-training practice. However, we observe that in most cases, pre-training is slightly inferior to joint training. We conjecture it is due to the high quality of our processed synthetic pairs, making them play a more important role when used by joint training.
Unless otherwise specified, we conduct ablation studies on ADE20K with SegFormer-B4.
Effectiveness of our filtering and re-sampling strategies. We first demonstrate the indispensable role of our proposed filtering and re-sampling strategies by examining the transferring performance
Training Scheme	Mask2Former + Swin-T	SegFormer + MiT-B4	Segmenter + ViT-S
Real Only (1× iters)	48.7	48.5	46.2
Real Only (2× iters)	49.2	48.6	46.7
Real + Synthetic	52.0 (↑2.8%)	50.6 (↑2.0%)	47.9 (↑1.2%)
from the synthetic set, the same as the practice in Section 4.2. As shown in Table 6, on ADE20K, the baseline without processing synthetic images, shows a considerable performance gap with real images  $(43.3\%$  vs.  $48.5\%)$ . Then with our pixel-level filtering criterion, the result is improved tremendously  $(43.3\% \rightarrow 47.4\%)$ . Our hardness-aware re-sampling principle also remarkably boosts the result from  $43.3\%$  to  $45.1\%$ . Combining both strategies, the transferring performance achieves  $48.3\%$ , outperforming the baseline by  $5\%$ . Similar observations also hold true on COCO.
Furthermore, we evaluate our two processing strategies when jointly training synthetic and real images in Table 7. Although the synthetic images look impressive, they can not benefit real images if combined directly without any processing  $(48.5\% \rightarrow 48.2\%)$ . Equipped with our proposed strategies for synthetic images, they successfully boost the fully-supervised baseline from  $48.5\%$  to  $50.3\%$ .
Training real images at double budget. As aforementioned, due to the halved number of real images in each joint training mini-batch, we double the training iterations. Therefore, we similarly double the training iterations for regular real-image training. As shown in Table 8 of ADE20K, doubling the training iterations under the real-image paradigm (real only  $2 \times$  iters) does not incur noticeable improvement over the baseline (real only  $1 \times$  iters). More importantly, it is still significantly inferior to our synthetic joint training (real + synthetic) across a wide range of network architectures.
Performance with respect to the number of synthetic images. It is reported in [2] that when the number of synthetic images surpasses real images, more synthetic images even cause worse performance. As present in Table 9, if not applying our processing strategies for synthetic images, the observation is similar to [2]. However, as long as the two strategies are adopted, the performance will
Table 8: We attempt to double the total iteration for real-image training paradigm (2× iters). It is evidently inferior to the "real + synthetic" setup under the same training iterations and batch size.
Nmax	6	10	20
Filtering & Re-sampling X	43.7	43.6	43.3
Filtering & Re-sampling √	47.2	47.7	48.3
keep increasing as the synthetic set enlarges. It further proves the necessity of designing appropriate filtering and re-sampling strategies to effectively learn from synthetic images.
Quantitative comparison with online hard example mining (OHEM). In Section 3.3, we have discussed the commonality and difference between our hardness-aware re-sampling strategy and OHEM. In Table 10, we demonstrate our method is also empirically superior to OHEM.
Table 9: Ablation studies on the number of synthetic images, which is controlled by  $N_{\mathrm{max}}$ . The model is trained with synthetic images only.
Baseline	+ OHEM	+ Our Re-sampling
44.0	44.2	45.4
Table 10: Comparison between our hardness-aware re-sampling strategy and OHEM with Segmenter-S.
Hyper-parameter tolerance margin  $\alpha$ . As shown in Figure 5, our proposed filtering strategy is robust to the tolerance margin  $\alpha$ , performing consistently well when  $\alpha$  ranges from 1 to 2.
In this work, we present a new roadmap to enhance fully-supervised semantic segmentation via leveraging densely annotated synthetic images from generative models. We propose two highly effective approaches to better learn from synthetic images, i.e., filtering noisy synthetic regions at the pixel and class level, and sampling more synthetic images for hard-to-learn masks. With our two processing strategies, synthetic images alone can achieve comparable performance with their real counterpart. Moreover, we investigate two training paradigms, i.e., pre-training and joint training, to compensate real images with synthetic images for better fully-supervised performance.
Acknowledgment. This work is supported by the National Natural Science Foundation of China (62201484, 62222604), HKU Startup Fund, and HKU Seed Fund for Basic Research. We also want to thank FreestyleNet [71] for providing high-quality generative models.
[60] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021. 7, 8
[61] Vadim Sushko, Edgar Schonfeld, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In ICLR, 2021. 4, 8
[62] Sora Takashima, Ryo Hayamizu, Nakamasasa Inoue, Hirokatsu Kataoka, and Rio Yokota. Visual atoms: Pre-training vision transformers with sinusoidal waves. In CVPR, 2023. 3
[63] Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Bin Liu, Gang Hua, and Nenghai Yu. Diverse semantic image synthesis via probability distribution modeling. In CVPR, 2021. 3, 4
[64] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. arXiv:2302.07944, 2023. 1, 4
[65] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. TKDE, 2022. 3
[66] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018. 4
[67] Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffusion models. arXiv:2207.00050, 2022.3,4
[68] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018. 3
[69] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018. 7
[70] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021. 3, 7, 8, 9
[71] Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. In CVPR, 2023. 2, 3, 4, 7, 8, 10, 15
[72] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv:2302.05543, 2023. 3
[73] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021. 2, 3, 4
[74] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017. 3
[75] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, 2021. 3
[76] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 1, 2, 4, 7
Following FreestyleNet [71], we resize all semantic masks to  $512 \times 512$  for image synthesis. The guidance scale of the diffusion model is set as 2.0, and the sampling step is 50. For better reproducibility, we pre-define and fix a sequence of  $N_{\mathrm{max}}$  seeds to synthesize densely annotated images.
For synthetic pre-training, we adopt exactly the same training protocols as real images. Then, during fine-tuning, the base learning rate is decayed to be half of the normal learning rate. Since our whole model parameters are pre-trained with synthetic images, the fine-tuning learning rate is the same throughout the whole model. The model is pre-trained and fine-tuned for the same iterations as real images. For joint training with real and synthetic images, real images are over-sampled to the same scale as synthetic images to make each mini-batch evenly composed of real images and synthetic images. The batch size is the same as real-image training. Each real image is iterated for the same number of epochs as the real-image training paradigm.
As for other hyper-parameters, e.g., data augmentations and evaluation protocols, they are set exactly the same as those in regular training paradigms. We adopt the MMSegmentation codebase for our development. We use  $8 \times$  Nvidia Tesla V100 GPUs for our training experiments.
B The Most Improved Classes
We list the most improved ten classes on ADE20K (the gain is measured by IoU): (1) ship: +68.19, (2) microwave: +48.72, (3) arcade machine: +45.85, (4) booth: +45.66, (5) oven: +30.86, (6) skyscraper: +23.23, (7) swimming pool: +15.52, (8) armchair: +14.6, (9) hood: +14.43, (10) wardrobe: +13.24.
C Discussions for Future Works and Limitations
Future works. In this work, we use the off-the-shelf semantic image synthesis model to generate densely annotated images. We have validated that the fully-supervised baseline can be remarkably boosted with these synthetic training pairs. We expect more considerable improvements can be achieved in future works by (1) better-trained or larger-scale pre-trained generative models, (2) larger-scale synthetic training pairs, and (3) taking the class distribution into consideration during image synthesis.
Limitations. It is relatively time-consuming to produce synthetic training pairs. For example, it takes around 5.8 seconds to synthesize a single image with a V100 GPU. In practice, we speed up the synthesis process with 24 V100 GPUs. Therefore we can construct the entire synthetic training set for ADE20K and COCO-Stuff in two days. In addition, considering the great potential of our densely annotated synthetic images, it will be more practical to apply our proposed roadmap to real-world scenarios, e.g., medical image analysis and remote sensing interpretation. We plan to conduct these explorations in future works.
D Visualization of Densely Annotated Synthetic Images and Filtered Regions
We display our diverse densely annotated synthetic images in Figure 6 of ADE20K and Figure 7 of COCO-Stuff. Besides, we also visualize our filtered regions during training for each synthetic image. It can be observed that there exist several patterns of filtered regions, e.g., boundary regions, synthesis failure cases, and small or rare objects. Please refer to the following pages for details.
Real Image
Semantic Mask
Figure 6: Visualization of diverse densely annotated synthetic images on ADE20K, as well as the filtered regions (white regions in the semantic mask). Note that the black regions in the masks are officially marked as "ignored region" by the original ADE20K dataset.
Diverse Synthetic Images & Filtered Region (White Region)
Diverse Synthetic Images & Filtered Region
Figure 7: Visualization of diverse densely annotated synthetic images on COCO-Stuff, as well as the filtered regions (white regions in the semantic mask).