# Generating Scenery Images with Larger Variety According to User Descriptions
## 2 Attentional and Imaginative Generative Networks for Scenery Image Generation
Additionally, the tanh function creates the new candidate value \(\hat{C}_{t}\), which is to be added to the state value. When updating the old cell state \(C_{t-1}\) into the new cell state \(C_{t}\), the old state is multiplied by the output of the forget gate \(f_{t}\), and then added by \(i_{t}\times\hat{C}_{t}\), as illustrated in Figure 1(c). The output \(h_{t}\) is based on the cell state but being filtered. The old cell state \(C_{t-1}\) goes through a tanh function and is multiplied by the output of a sigmoid function. To summarize the mechanism, the forget gate decides what should be remembered. The input gate controls what information is relevant and should be added to the state in the current step. Additionally, the output gate decides what the next hidden state should be. Figure 2: Cont. The imagination model utilizes the LSTM cell in the text encoder. In the upsampling processes of \(S_{i}\), it utilizes the hidden features \(h_{i}\) to reinitialize the cell state in the text encoder. The main difference between the proposed framework and the original framework in [30] is that the imagination model extracts the features again from the text descriptions in every upsampling process. Such design enriches the text features given in the corpus description. The loss function for the attentional and imaginative generative networks is defined in Equation (5). \[\mathcal{L}=\mathcal{L}_{G}+\lambda_{1}\mathcal{L}_{C}+\lambda_{2}\mathcal{L }_{DAMSM} \tag{5}\]  
The first term \(\mathcal{L}_{G}\) in Equation (5) is the summation of the loss at each stage \(i\). We can express it as  
\[\mathcal{L}_{G}=\sum_{i=0}^{m}\mathcal{L}_{G_{i}} \tag{6}\]  
At the \(i\)th stage, the adversarial loss for the generator \(G_{i}\) is defined in Equation (7). \[\mathcal{L}_{G_{i}}=-\frac{1}{2}\mathbb{E}_{\hat{x}_{i}\sim p_{G_{i}}}[log(D_ {i}(\hat{x}_{i}))]-\frac{1}{2}\mathbb{E}_{\hat{x}_{i}\sim p_{G_{i}}}[log(D_{i }(\hat{x}_{i},\vec{e}))] \tag{7}\]  
The first part of Equation (7) is the unconditional loss, which discriminates if the image is real or fake. Additionally, the second part is the conditional loss, which determines whether the image and the text description match or not. In Equation (7), \(D_{i}\) denotes the corresponding discriminator for \(G_{i}\). For each discriminator \(D_{i}\), the cross-entropy loss defined in Equation (8) is minimized to classify the input as real or fake. \[ \begin{split}&\mathcal{L}_{D_{i}}=-\frac{1}{2}\mathbb{E}_{x_{i} \sim p_{data_{i}}}[log(D_{i}(x_{i}))]-\frac{1}{2}\mathbb{E}_{\hat{x}_{i}\sim p _{data_{i}}}[log(1-D_{i}(\hat{x}_{i}))]\\ &+-\frac{1}{2}\mathbb{E}_{x_{i}\sim p_{data_{i}}}[log(D_{i}(x_{i },\vec{e}))]-\frac{1}{2}\mathbb{E}_{\hat{x}_{i}\sim p_{data_{i}}}[log(1-D_{i} (\hat{x}_{i},\vec{e}))]\end{split}  \tag{8}\]  
Here, \(x_{i}\) is from the true image distribution \(p_{data_{i}}\) at the \(i\)th scale, and \(\hat{x}_{i}\) is from the model distribution \(p_{G_{i}}\) at the \(i\)th scale. The second term \(\mathcal{L}_{C}\) in Equation (5) is designed to learn the relationship between the hidden state and image text in the imaginative model \(S_{i}^{imag}\). The relationship \(r_{h}\) between each cell memory state \(c_{h}\) and word vector \(e_{j}\) is defined in the following equation. \[r_{h}=c_{h}\times e_{j}^{\,T} \tag{9}\]  
In this way, the relationship between the cell state initialized by the imaginative model \(S_{i}^{imag}\) and the \(j\)th word in the word vector can be learned. Then, the cosine similarity of \(r_{h}\) and the sentence vector \(s_{h}\) is calculated using Equation (10). \[R(r_{h},s_{h})=r_{h}^{\,T}S_{h}/((\|r_{h}\|\|s_{h}\|) \tag{10}\]  
Figure 2: LSTM Cells. (**a**) Forget Gate, (**b**) Input Gate, (**c**) Updating Cell State. For each minibatch, we calculate the matching score of the hidden state \(H\) and description \(D\).