# Generating Scenery Images with Larger Variety According to User Descriptions
###### Abstract  
In this paper, a framework based on generative adversarial networks is proposed to perform nature-scenery generation according to descriptions from the users. The desired place, time and seasons of the generated scenes can be specified with the help of text-to-image generation techniques. The framework improves and modifies the architecture of a generative adversarial network with attention models by adding the imagination models. The proposed attentional and imaginative generative network uses the hidden layer information to initialize the memory cell of the recurrent neural network to produce the desired photos. A data set containing different categories of scenery images is established to train the proposed system. The experiments validate that the proposed method is able to increase the quality and diversity of the generated images compared to the existing method. A possible application of road image generation for data augmentation is also demonstrated in the experimental results. text to image; image generation; generative adversarial networks +
Footnote †: journal: Medical  
The loss function is also modified to increase the probability of correct prediction for the discriminator. In the training process of GAN, mode collapse has always been a problem. Sometimes it is feasible to fine tune the training parameters for better model learning. However, tuning parameters might not always work well. Therefore, Arjovsky et al. [7] used Wasserstein distance to improve the stability of learning and mode collapse. They showed that the Earth-Mover distance could produce better gradient behaviors when performing distribution learning compared with other distance metrics. Metz et al. [8] proposed an unrolled GAN that uses a gradient-based loss function to enhance the generator and applies the Jensen-Shannon (JS) divergence to minimize the loss function of the generator. Nguyen et al. [9] proposed a dual discriminator architecture. One of the discriminators rewards high scores for samples from true data distribution and the other discriminator rewards data from the generator, and thus produces data to fool both two discriminators. The research area of GAN has many applications including art, design, data augmentation, and other applications [10; 11; 12; 13; 14]. For example, the research in [5] generates hand-written numbers. The DRAW system [15] constructs license plates. The work in [16] generates anime characters. The system in [3] demonstrates generating bedroom interiors using deep convolutional generative adversarial networks. PixelCNN [17] generates various different animals and human faces. Ledig et al. [18] proposed a super-resolution GAN to take a low-resolution image as input and generate a high-resolution image. Isola et al. [19] proposed an image-to-image translation framework called pix2 pix based on CGAN to convert an image content from one domain to another. Iizuka et al. [20] proposed to perform image inpainting using two discriminators to monitor local features and global features for both local and global consistency. The development of the GAN techniques also leads to promising results in some novel applications such as 3D object generation, medicine, and autonomous driving [21]. Yu et al. [22] proposed a 3D object inpainting network that can process point cloud data directly without human labeling. Shin et al. [23] proposed a GAN-based method to generate synthetic, abnormal, MRI images with brain tumors. The work in [24] designed a systematic framework to automatically generate test cases to imitate real-world driving scenes. With the progress of computer vision and natural language analysis, generating images according to natural language descriptions has become a popular research topic in recent years [25; 26; 27; 28; 29; 30; 31; 32]. The work in [26] was one of the first works that attempted to generate images according to text descriptions. It translates single-sentence human-written descriptions directly into image pixels.