SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Result of image synthesis with free-form mask. Images in the first row are inverted images in StyleGAN and their edited images with the HairCLIP [36]. Images are converted to the spatial shapes of both masks and images in the first column while preserving styles and identities. We also show a comparison with LinearGAN [38] and OASIS [25] in the last three columns. For comparison we sample the images with random noise for all models
4.2 Free-form image manipulation
In this subsection, we show the flexibility of our proposed method in facilitating free-form image manipulation. Our approach underscores its capability of handling masks that deviate substantially from typical training distributions, such as oversized human ears and big oval eyes. Figure 6 illustrates the image synthesis results when such unconventional masks are used as inputs. Even in cases where the masks deviated from the typical human form, our method successfully synthesized results that corresponded appropriately to the provided masks. In contrast, when LinearGAN [38] and OASIS [25] receive a mask that falls outside the typical human range as a shape guide, they fail to generate results that align with the shape indicated by the mask.
Moreover, the versatility of our method is further emphasized by its compatibility with latent editing techniques such as HairCLIP [36] as shown in Figure 6. While latent editing changes the hairstyle to a yellow color or perm style, our method preserves the shape of images corresponding to given masks. See the Appendix G for additional samples of generated images conditioned with free-form masks.
Figure 7: Image synthesis results when diverse types of input masks are entered, such as scribbles and sketches.
Figure 8: Image generation results with a real image as a guide. The top row consists of images that serve as the style input, while the left column contains target image that determine the desired shape.
4.3 Conditional image synthesis using a variety of conditions.
This part broadens the scope of semantic image synthesis to include just segmentation masks and diverse conditions like sketches and user scribbles. Figure 7(a) demonstrates the outcome when simple scribbles are used as an input mask. Remarkably, even with a scribble mask created with just a single line, our method can generate image synthesis that is well-suited to the indicated shape. Moreover, this user-friendly aspect further enhances the usability of our approach. Additionally, we visualize our results and compare them qualitatively with pSp [23], a GAN inversion approach demonstrating its ability to invert various inputs into latent codes, not solely real images. As illustrated in Figure 7(b), our comparison with pSp, especially in semantic image synthesis using sketch inputs, reveals a significant difference. pSp can generate coarse structures effectively with pSp encoder and style mixing, but it struggles to capture and reproduce the fine details of the input conditions. On the other hand, our method surpasses pSp with more corresponding results with fewer restrictions of matching latent code because of the flexible spatial control of our model using feature maps rearranging. In the Appendix E, we further demonstrate the proficiency of our method in generating images based on various input conditions with implementation details. This additional evidence underscores the versatility of our method and superior performance in conditional image synthesis.
4.4 Exemplar-guided image generation
Beyond the specified conditions, our model produces an output image that has a similar structure to a provided sample. The results of image generation are displayed in Figure 8. The latent codes are derived from real images by utilizing the e4e inversion technique [28] that was trained with the CelebAMask-HQ training set. This inverted latent code enables us to obtain a proxy mask of the sample, thereby aligning the generated images with the real image structure.
In Figure 8, it is evident that our generated results better preserve shape in comparison to other exemplar-guided image generations. The results of Transeditor [39] and DAT [13] do not preserve the original shape as effectively, with DAT even showing a disparity in style. Although competitors use dual latent spaces to portray exemplar structure more accurately, the structural latent code does not always reflect the shape correctly. In contrast, our method more accurately mirrors the exemplar shape because the Rearranger directly forms the feature maps with the proxy mask.
Semantic image synthesis has received significant attention for its ability to reflect intentions on images at a very detailed level, but achieving this requires substantial data. Therefore, we proposed our method as a solution to address this issue. Our method rearranges the features of a pretrained generator to enable detailed control over images, and all elements of our approach are self-supervised, except for a single annotation mapped to human perception. With just one annotation, we achieve precise image control, as demonstrated through experiments in semantic image synthesis with various input conditions. Our method also showed data efficiency in situations where the pixel-level annotation changes. Overall, our approach demonstrated a high level of detail control without the dependence on annotations.
**Limitations** Our one-shot results outperform other competitors, but a slight performance gap remains when compared to full-shot outcomes. Therefore, future work will make it possible to outperform to beat full-shot performance. Also, our approach operates at the feature map level, which, while enabling the natural incorporation of masks, needs to achieve complete pixel-level control. For more details, please refer to the Appendix D.
This work was supported by the National Research Foundation of Korea(NRF) grant (No. 2022R1F1A107624111) funded by the Korea government (MSIT).
[1] Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5134-5142, 2020.
[2] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021.
[3] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8188-8197, 2020.
[4] Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semantics of gans. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5770-5779. IEEE Computer Society, 2020.
[5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139–144, 2020.
[6] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[45] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021.
[46] Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full-resolution correspondence learning for image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11465-11475, 2021.
[47] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5104-5113, 2020.