AniFaceDrawing: Anime Portrait Exploration during Your Sketching
Authors: name: Zhengyu Huang; name: Haoran Xie; name: Tsukasa Fukusato; name: Kazunori Miyata
Published: 2023-06-13 00:43:47

Abstract: In this paper, we focus on how artificial intelligence (AI) can be used to
assist users in the creation of anime portraits, that is, converting rough
sketches into anime portraits during their sketching process. The input is a
sequence of incomplete freehand sketches that are gradually refined stroke by
stroke, while the output is a sequence of high-quality anime portraits that
correspond to the input sketches as guidance. Although recent GANs can generate
high quality images, it is a challenging problem to maintain the high quality
of generated images from sketches with a low degree of completion due to
ill-posed problems in conditional image generation. Even with the latest
sketch-to-image (S2I) technology, it is still difficult to create high-quality
images from incomplete rough sketches for anime portraits since anime style
tend to be more abstract than in realistic style. To address this issue, we
adopt a latent space exploration of StyleGAN with a two-stage training
strategy. We consider the input strokes of a freehand sketch to correspond to
edge information-related attributes in the latent structural code of StyleGAN,
and term the matching between strokes and these attributes stroke-level
disentanglement. In the first stage, we trained an image encoder with the
pre-trained StyleGAN model as a teacher encoder. In the second stage, we
simulated the drawing process of the generated images without any additional
data (labels) and trained the sketch encoder for incomplete progressive
sketches to generate high-quality portrait images with feature alignment to the
disentangled representations in the teacher encoder. We verified the proposed
progressive S2I system with both qualitative and quantitative evaluations and
achieved high-quality anime portraits from incomplete progressive sketches. Our
user study proved its effectiveness in art creation assistance for the anime
style.