# TextControlGAN: Text-to-Image Synthesis with Controllable Generative Adversarial Networks
## 3 Methods  
### Text-to-Image Synthesis with Controllable GAN Framework  
In this paper, we present a text-to-image synthesis model based on the ControlGAN [36] framework as a viable alternative to cGAN. One primary advantage of employing a ControlGAN lies in its utilization of an independent classifier, while in cGAN, the discriminator assumes responsibility for classification. This approach enables us to train the classifier with DA methods, incorporating modified and slightly distorted images during the training process. The proposed model, TextControlGAN, builds upon the concept of ControlGAN to condition texts. Given that text encoding vectors possess continuous values, TextControlGAN introduces a regressor, distinguishing it from conventional ControlGAN. The regressor fulfills a role analogous to the classifier in ControlGAN, as it seeks to estimate text encoding vectors from given images. By employing DA methods for training the regressor, which slightly distort input images during the process, the regressor model can be trained with reduced overfitting issues and exhibit enhanced performance. Consequently, TextControlGAN comprises three distinct neural network structures: a generator, a discriminator, and a regressor. Figure 1 illustrates the comprehensive structure of TextControlGAN. The generator generates synthetic images using corresponding text embedding vectors and noise vectors as inputs. These images are then compared to real-world images by the discriminator. Deviating from cGAN and GAN-INT-CLS models, the discriminator is solely responsible for its primary task, which is the binary classification of authentic versus fake imitations. Instead of involving the discriminator in the conditional text learning, conditional texts are learned exclusively by another specialized structure called the regressor. Within the regressor, DA methods are employed to improve performance, enabling the generator to learn the conditional information more effectively than if it relied only on the discriminator in cGAN and GAN-INT-CLS. The objective of TextControlGAN learning can be represented as follows:  
\[ \begin{array}{ll}\min_{G}\max_{D}O(D,G)&=\mathbb{E}_{(x,y)\sim\mathbb{P}_{ \mathbb{z}}(x,y)}[\log D(x)]\\ &+\mathbb{E}_{z\sim\mathbb{P}_{\mathbb{z}}(z),y\sim\mathbb{P}_{\mathbb{y}}}[ \log(1-D(G(z|y)))],\\ &\min_{R}O_{R}(R)=\mathbb{E}_{x\sim\mathbb{P}_{\mathbb{z}}}\Big{[}\sum(y-R(x) )^{2}\Big{]},\end{array}  \tag{3}\]\[\min_{G}\,O_{R}(G)=\mathbb{E}_{z\sim\mathbb{P}_{z},y\sim\mathbb{P}_{y}}\left[\sum(y-R (G(z|y)))^{2}\right] \tag{5}\]  
where \(R\) represents the regressor and \(O_{R}\) is the objective function of the regressor. The TextControlGAN objective as expressed in Equation (3) bears resemblance to that of cGAN; however, a key difference is that, in TextControlGAN, the discriminator (D) does not learn the conditional information with y. As demonstrated in Equations (3) and (5), the generator has dual objectives: it seeks to deceive the discriminator in order to be classified as real, while simultaneously aiming to have the correct text encoding vectors estimated by the regressor, in accordance with the generated images. All structures in TextControlGAN are composed of residual blocks; the generator, discriminator, and regressor each contain four such blocks. Each block comprises two convolutional layers. In the generator, upsampling by a factor of two is executed using the nearest neighbor method within each residual block. The generator is composed of 10 convolutional layers with 96 kernels, where the first and the last two layers are convolutional layers and four residual blocks with two convolutional layers each. Similarly, downsampling is performed on the first three residual blocks of both the dis  
Figure 1: Structure of the proposed TextControlGAN. The \(G\), \(D\), and \(R\) denote the generator, discriminator and regressor, respectively in (**A**). \(z\), \(x\), and \(y\) denote noise variables for the generator, a real image dataset, and corresponding text embedding, respectively.