# A Study on Generating Webtoons Using Multilingual Text-to-Image Models
## 2 Related Work
### Deep Learning-Based Text-to-Image Generation  
Deep learning-based text-to-image generates images using the semantic information of the input sentence. GAN-based text-to-image generation can be divided into two stages: extraction of semantic information from the text and text-to-image generation. First, the sentence is embedded using word embedding, and features are extracted by inputting them into an RNN or transformer structure. In the image-generation stage, the GAN inputs the semantic information of the text, along with noise of the same size as that of the output-generated image, to the generator to create an image. By contrast, the discriminator compares the images generated by the generator with real images. Through the adversarial learning of two neural networks that constantly create fake images and judge whether the generated image is real, they generate fake images that are similar to the real ones. The initial deep-learning-based text-to-image model uses a deep convolutional GAN (DCGAN) [5]. The DCGAN-based text-to-image model generates an image using the conditional vector of a sentence as the condition. Although the generated image is similar in meaning to the input sentence, it does not reflect the contextual meaning of each word in the sentence, thus generating low-quality images. To overcome these disadvantages, the AttnGAN, which introduces an attention mechanism for image generation, was developed. AttnGAN first uses the feature vector of the sentence to create an image, and when generating the next-stage image, it combines the attention map of the word with the image vector to create an improved image step-by-step. AttnGAN can express each word in a sentence in a detailed manner compared with DCGAN; subsequently, StackGAN [7], MirrorGAN [8], DM-GAN [9], R-GAN [10], Stacking VAE and GAN [10], and Dcfgan [11] were developed, which can express the meaning of the input sentence more precisely and generate high-resolution images. Recently, diffusion-model-based text-to-image methods have shown better performance compared with GAN-based methods. GANs suffer from the problem of imbalanced training between the generator and discriminator, which leads to collapse. Whenthe generator creates images that can easily fool the discriminator, it stops learning. By contrast, diffusion models generate images by iteratively adding noise to the training data in the forward process and recovering data from noise in the reverse process [20]. After training, the diffusion models generate images similar to the training data using a reverse process. The DALL-E 2 model released by OpenAI uses contrastive language image pre-training (CLIP) as an encoder and the diffusion model as a decoder to generate images from text [13]. CLIP extracts features from both text and image data and determines the similarity between them using contrastive learning. After training, text and image features with similar characteristics are densely packed in a multimodal space [21]. CLIP extracts an image vector similar to the input text from the pre-trained model and generates an image by inputting it into the diffusion model. Diffusion model-based text-to-image methods have shown superior performance compared with GAN-based methods on benchmark datasets in generative modeling [22; 23]. Examples of diffusion model-based text-to-image generation methods include Dalle-2 [13], DreamBooth [14] and imagegen. GAN and diffusion models have different strengths and weaknesseses [24; 25]. The GAN has the advantage of a shorter training time compared with diffusion models; however, it has a risk of model collapse owing to the learning imbalance between the generator and discriminator. By contrast, diffusion models have the advantage of generating a wider variety of high-quality images than GANs, despite requiring longer training times.