# A Study on Generating Webtoons Using Multilingual Text-to-Image Models
## 3 Text-to-Image Generation Using Multilingual BERT
### Text-to-Image Generation with Multilingual BERT  
A deep learning-based text-to-image model generates images from input text in two stages. The first stage is feature extraction of the input text, and the second stage is the training of a GAN model by combining the extracted text features with noise. Extracting text features that reflect the contextual characteristics embedded in the text is crucial because the quality of the generated images is heavily influenced by the quality of the learned features. Essentially, if the model fails to extract features that capture the essence of the keywords between the text and image, the generated image may deviate from the intended meaning of the text. Therefore, in this study, as shown in Figure 2, we utilized a pre-trained multilingual BERT model [29], which has demonstrated high performance in NLP, to extract the feature vectors of sentences. The sentence feature vectors are obtained using the \"cls\" token in the BERT model as the sentence vector.  
To generate webtoons based on the extracted features of the multilingual treatments, a GAN-based DCGAN model was used. Figure 3 shows the structure of the text-to-image model using DCGAN. The GAN generates images similar to the original data through competition between two neural networks called the generator and discriminator. The generator learns by reflecting the distribution of the input data, generates fake data from random vectors, and compares the data generated by the generator with the original data to determine whether the data are real or fake. To generate webtoons from multilingual treatments, the sentence vectors extracted through the Multilingual BERT are combined with noise and input to the DCGAN for training. The proposed generator model consists of six blocks comprising a convolutional layer, dropout layer, batch normalization layer, ReLU layer, and upsamples as they pass through each block. After passing through the final layer, it generates an image size of 3 \(\times\) 64 \(\times\) 64. The discriminator consists of six blocks, comprising a convolutional layer, batch normalization layer, leaky ReLU layer, and downsamples as it passes through each block. Finally, it combines text features and uses a sigmoid to determine whether the text is real or fake.  
Figure 1: Examples of datasets transformed using CartoonGAN.  
Figure 2: Sentence vector extraction using Multilingual BERT. The sentence on the left side of the figure is a sentence written in Korean with the same meaning as the English.  
The text-to-image loss function using the DCGAN is given by Equation (1), where G represents the generator function, D represents the discriminator function, x is the input vector, z is the random vector, and \(\varphi\)(t) represents the feature vector of the input text. The discriminator is trained to have a larger value for Equation (1) to distinguish between real and fake data, whereas the generator is trained to have a smaller value for Equation (1) to generate fake data similar to the real data. The discriminator of the DCGAN used in this study receives three types of data as inputs and is trained to classify real image-right text as real, real image-wrong text as fake, and fake image-real text as fake. To prevent overfitting, label smoothing and feature matching were used in the experiments. The loss function is as follows:  
\[\begin{split}\min_{D}\max_{G}\mathrm{E(D,G)}=\mathrm{E_{(x,t) \sim P_{data}(x,t)}[\log D(x,\varphi(t))]+E_{z\sim P_{z}(z),t\sim P_{I}(t)}[ \log 1-D(G(z,\varphi(t)))]\\ +\mathrm{E_{x\sim P_{x}(x),\hat{t}\sim pf(t)}[\log 1-D(x,(z, \varphi(\hat{t})))]}\end{split} \tag{1}\]