# Text-Guided Customizable Image Synthesis and Manipulation
## 3 Our Proposed Method
### Customizable and Manipulation Architecture  
**Customizable Image Synthesis.** Figure 2 shows the customizable image synthesis structure. Initially, the text and contour information are encoded into the corresponding feature vectors by the bi-directional long short-term memory (Bi-LSTM) [24] and Inception-V3 [25] models. Then, these two features are combined and processed into the first-stage features through multiple upsampling operations. Afterward, the features are processed into the features of the next stage through the residual block [26] operation and upsampling operation. Simultaneously, the residual block and upsampling operations can be used to continue to obtain the next stage's features. In this paper, the device is limited to only the image features synthesized to the third stage.  
For the contour features, in addition to synthesizing the image features, the affine transformation will be performed with the image features of each stage. As shown below in Figure 2, the contour features obtained at the beginning and the features of the first stage are affinely transformed to form the new features. Then, the new features are affine transformed with the next stage's features, and this process will be repeated until they are affine transformed with the last stage's features to form the final transformed features, which will be used for the final custom image synthesis. The whole affine transformation process is as follows:  
\[h_{i}=\left\{\begin{array}{ll}Up(concat(fea\_t,\ fea\_c))&i=1\\ Up(resnet(h_{i-1}))&i>1\end{array}\right. \tag{3}\]  
\[\begin{array}{c}fea\_c^{new}=conv(fea\_c)\end{array} \tag{4}\]\[h_{1}^{new}=h_{1}\cdot W(fea\_c^{new})+b(fea\_c^{new}) \tag{5}\] \[h_{i}^{new}=h_{i-1}^{new}\cdot W(conv(h_{i-1}^{new}))+b(conv(h_{i-1}^ {new})) \tag{6}\]  
where \(fea\_t\) represents the text feature and \(fea\_c\) represents the contour feature. \(Up\), \(concat\), \(resnet\), and \(conv\) stand for upsampling, concatenation, residual block processing, and convolution operations, respectively. \(W\) and \(b\) are the weights and biases learned using two convolution layers, respectively.  
**Discriminator.** In the T2I research, the image and text matching-aware method [10] is widely used and has shown its effectiveness in enhancing the performance of image synthesis based on text information and improving the semantic consistency between an image and text. Therefore, we also adopted the matching-aware method for our work. Specifically, our discriminator receives three types of input: the generated image (\(I_{g}\)) + matched text (\(T\)), the original image (\(I_{r}\)) + matched text, and the original image + mismatched text (\(T_{mis}\)). The target of the discriminator is to recognize the first and third cases as false and the second case as true. In addition, it also needs to be able to correctly recognize true and false between the synthesized image and the original image.  
**Image Manipulation.** The basic structure and training process of image manipulation is basically the same as that of customizable image synthesis. The difference is that the input in the customizable image synthesis structure is a combination of the text and the contour map, while the input in the image manipulation structure is a combination of the text and the original image. After the image manipulation model is trained, it will be used to modify the result of the customized image synthesis stage. As shown in Figure 2, based on the image manipulation model, new text can be continuously input to modify the content of the previously customized composite image.