# Automatic Colorization of Anime Style Illustrations Using a Two-Stage Generator
## 1 Introduction  
Line-artworks play a crucial role in the initial phases of media industries for making storyboards, games, illustrations, animations, and movies. Since there is a limit in communicating the complicated emotional transitions and atmosphere of a scene using only line-art, it is a customary to use colorized line-art and colorful storyboards to convey ideas. However, coloring line-art is a very tedious and manual labor intensive job. For example, if an animation is two hours long and uses 24 frames per second, then animators have to colorize about 17 thousand frames. If we assume that colorization of line-art takes about 5 min per frame then it would take about 2302 days of working hours without any resting. To automate the colorization of line-art, the research community began to use Generative Adversarial Networks (GANs) [1]. There are also commercialized tools such as PaintsChainer and ClipsStudio that automate the line-art; however, the perceived quality of the results are not as satisfactory as real-world colorized line-arts. Colorization of line-arts are considered as difficult because, unlike black and white images, it does not contain texture or shade information. Although many models make use of a reference image or add few hint pixels on a line-art to aid the colorization process, the result of colorization shows limited color spectrum and monotonous color space. Basic steps included in automatic colorization of line-art are acquiring characteristics or features from a line-art, segmenting the image to generate texture and shades from a limited information, and seamlessly concatenating the segments as a final colorized image. The automatic colorization of line-art can be categorized into three depending on the types of input data. The first category is fully automatic colorization that exploits only the line-art [2, 3]. The second category is semi-automatic colorization which makes use of style transfer based on input data of line-arts and colorized images [4, 5, 6]. The final category is hint-based methods that inform what color to be used in which specific area of an image [7, 8, 9, 10, 11, 12]. All three categories of colorization models have to extract contour of lines from the original line-art. Interestingly, the colorization performance of a generated colored line-art of the artificial line is better than that of real-world line-arts drawn by a artist. We conjecture that the low quality is due to over-fitting of input line-art data and imbalance of distribution of input line style and the distribution of real-world line style. In this work, we provide solution to over-fitting of the line style distribution and colorize a line-art with better color spectrum, shades, and gradation. The proposed scheme is composed of five steps: (1) line-art augmentation with histogram equalization, (2) conversion of original image to grayscale image for subset of line-art, (3) data augmentation with two different line extraction method, (4) Line Detection Model (LDM) loss function for line generalization, and (5) two-stage generator that takes a low resolution image in the draft model and enhances color quality in the colorization stage. Figure 1 shows the result of colored images exploiting the proposed method using real-world images as the input data. Many of the works in the field have used Frechet Inception Distance (FID) score to compare the quality of generated images. However, we believe that simple FID score does not provide a fair comparison in the field of colorization because the generated color spectra are not correlated across the generated images. In this paper, we propose segment-based hinting constraints for consistent and fair comparison of colorized artworks. We named this method as Colorization FID (C-FID). We have  
Figure 1: Proposed method’s real word line-art coloring results. conducted C-FID evaluation on the proposed model with XDoG generated line-art and real-world line-arts.