# Generating Scenery Images with Larger Variety According to User Descriptions
## 2 Attentional and Imaginative Generative Networks for Scenery Image Generation  
The proposed system framework is illustrated in Figure 1. The framework improves and modifies the architecture of AttnGAN [30] by adding the imagination models in addition to the original attention models. Therefore, the core is named as the attentional and imaginative generative networks. The architecture of the original AttnGAN is displayed in Figure A1 in Appendix A for comparison. We can observe that the main difference between the two architectures is the existence of the imagination model, \(S_{i}^{img}\). The details of the attentional and imaginative generative networks in this work are described below. There are \(m\) generators (\(G_{0},\ G_{1},\ldots,G_{m-1}\)) in the generative model. Each of the \(m\) generators has its corresponding stage (\(S_{0},S_{1},\ldots,S_{m-1}\)) and outputs a hidden layer (\(h_{0},h_{1},\ldots,h_{m-1}\)) to generate images of lower to higher resolutions (\(\hat{x}_{0},\hat{x}_{1},\ldots\hat{x}_{m-1}\)) where \(\hat{x}_{i}=G_{i}(h_{i})\). In Figure 1, the text encoder is a bidirectional LSTM [33], which extracts the global sentence vectors \(\overline{z}\) from the input text descriptions. \(F_{CA}\) is the conditioning augmentation [29] that converts the sentence vector \(\overline{e}\) to the conditioning vector \(c\). The vector \(z\) is a noise vector sampled from a normal distribution. \(S_{i}\) is the upsampling network at each stage. All stages include attention models \(S_{i}^{attn}\) and imagination models \(S_{i}^{img}\), except for \(S_{0}\). Note that \(F_{CA},\ S_{i}^{img},\ S_{i}^{attn},\ S_{i}\), and \(G_{i}\) are all modeled as neural networks. The relationship of hidden layers \(h_{i}\) and the upsampling networks \(S_{i}\) can be expressed as Equations (1) and (2). \[h_{0}=S_{0}(z;F_{CA}(\overline{e})) \tag{1}\]  
\[h_{i}=S_{i}\big{(}h_{i-1};S_{i}^{attn}(e,h_{i-1})\big{)}\ \text{for i}=1,\ 2,\ \ldots m-1 \tag{2}\]  
where \(e\) is the matrix of word vectors. Attention models \(S_{i}^{attn}\) take the word features \(e\) and the image features \(h_{i-1}\) from the previous hidden layer as the inputs. A new perceptron layer is added to convert  
Figure 1: System Framework. the word features \(e\) into the common semantic space of the image features, denoted by \(e_{i}^{\sim}\). Afterwards, a word-context vector \(C_{j}\) is computed for the \(j\)th subregion of the image according to its hidden features \(h\). Each column of \(h\) is a feature vector of the \(j\)th subregion. The computation of the word-context vector \(C_{j}\) is expressed in Equations (3) and (4). In Equation (4), the softmax function converts \(s_{j,i}^{\sim}\) into \(\beta_{j,i}\), which is the weight the attention model learns from the \(i\)th word in the process of generating the \(j\)th subregion of the generated scene. \[s_{j,i}^{\sim}=h_{j}^{T}e_{i}^{\sim} \tag{3}\]  
\[C_{j}=\sum_{i=0}^{T-1}\beta_{j,i}e_{i}^{\sim}\text{, \emph{where }}\beta_{j,i}= \frac{exp\left(s_{j,i}^{\sim}\right)}{\sum_{k=0}^{T-1}exp\left(s_{j,k}^{\sim} \right)} \tag{4}\]  
Imagination models \(S_{i}^{imag}\) are responsible for increasing the variety of the generated images. To elaborate the imagination models, the mechanism of LSTM cells is investigated first. The core concept of an LSTM cell is to add a cell state to enhance the decision in each step. It uses different gates to control the network. The first one is the forget gate, shown in Figure 1(a). The inputs of the forget gate are the current input \(x_{t}\) and the previous output \(h_{t-1}\). This gate uses a sigmoid function, denoted by \(\sigma\), to control if it should forget or remember the information from the previous cell. The second gate is the input gate as shown in Figure 1(b). The input gate determines how much new information should be updated to the new cell. The sigmoid function highlighted in Figure 1(b) determines the update value \(i_{t}\).