ESA‐CycleGAN: Edge feature and self‐attention based cycle‐consistent generative adversarial network for style transfer

IET Image ProcessingVolume 16, Issue 1 p. 176-190 ORIGINAL RESEARCH PAPEROpen Access ESA-CycleGAN: Edge feature and self-attention based cycle-consistent generative adversarial network for style transfer Li Wang, Li Wang orcid.org/0000-0002-1056-4913 College of Electronic and Information Engineering, Southwest University, Chongqing, People's Republic of ChinaSearch for more papers by this authorLidan Wang, Corresponding Author Lidan Wang ldwang@swu.edu.cn College of Electronic and Information Engineering, Southwest University, Chongqing, People's Republic of China Correspondence Lidan Wang, College of Electronic and Information Engineering, Southwest University, No.2 Tiansheng Road, Chongqing, People's Republic of China. Email: ldwang@swu.edu.cnSearch for more papers by this authorShubai Chen, Shubai Chen College of Computer & Information Science, Southwest University, Chongqing, People's Republic of ChinaSearch for more papers by this author Li Wang, Li Wang orcid.org/0000-0002-1056-4913 College of Electronic and Information Engineering, Southwest University, Chongqing, People's Republic of ChinaSearch for more papers by this authorLidan Wang, Corresponding Author Lidan Wang ldwang@swu.edu.cn College of Electronic and Information Engineering, Southwest University, Chongqing, People's Republic of China Correspondence Lidan Wang, College of Electronic and Information Engineering, Southwest University, No.2 Tiansheng Road, Chongqing, People's Republic of China. Email: ldwang@swu.edu.cnSearch for more papers by this authorShubai Chen, Shubai Chen College of Computer & Information Science, Southwest University, Chongqing, People's Republic of ChinaSearch for more papers by this author First published: 19 October 2021 https://doi.org/10.1049/ipr2.12342AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Nowadays, style transfer is used in a wide range of commercial applications, such as image beautification, film rendering etc. However, many existing methods of style transfer suffer from loss of details and poor overall visual effect. To address these problems, an edge feature and self-attention based cycle-consistent generative adversarial network (ESA-CycleGAN) is proposed. The model architecture consists of a generator, a discriminator, and an edge feature extraction network. Both the generator and the discriminator contain a self-attention module to capture global features of the image. The edge feature extraction network extracts the edge of the original image and feeds it into the network together with the original image, thereby allowing better processing of details. Besides, a perceptual loss term is added to optimize the network, resulting in better perceptual results. ESA-CycleGAN is applied on four datasets, respectively. The experimental results show that the authors' computed final IS and FID values have good results compared to the results of several other existing models, indicating the superiority of the model in style transfer, which can better preserve the details of the original images with better image quality. 1 INTRODUCTION Image style transfer is an operation that transforms the artistic style of an image without changing the semantics of the original image. Since style is a relatively abstract concept, style transfer is difficult to depict the painting styles of different painters in different periods by computer. The development of style transfer can be divided into three specific stages. The first stage is the traditional image style transfer [1, 2] which mainly uses texture and brushstroke modelling methods to express the characteristics of style through the model established to achieve the effect of style transfer. The traditional image style transfer algorithm is difficult and time-consuming to model, and they can only model for one style, and the modelling methods are not uniform. For each style, the transfer algorithms are independent of each other, and the algorithmic steps are very different, which has great limitations in application. In the second stage, Gatys et al. [3] first proposed a style transfer method based on a convolutional neural network in 2015, and they found that the VGG [4, 5] network can represent the semantic style information and content texture feature information of images. And the style features of the image can be obtained by calculating the Gram [3] matrix, and the output image can be updated continuously by iterative optimization so that the style features of the generated image are similar to those of the style image, and finally the style of the style image and the content of the content image can be reconstructed simultaneously. The neural network-based method can extract global features, no longer just some underlying features, and can have a better transfer effect. However, it is more about matching between a single image to be transferred and a style image to accomplish the style transfer. The third stage, style transfer based on generative adversarial networks [6-10] was first proposed by Goodfellow et al. [11], and this powerful generative model is widely used in many fields, such as image translation [12-16], image drawing [17], super-resolution reconstruction [18] etc. Generative adversarial networks use an adversarial learning approach, which is derived from gaming ideas. With a large amount of training data, the network model can learn the distribution of the sample data and generate data close to the sample distribution. Besides, the GAN (Generative Adversarial Network) based image style transformation method has many derived research applications, such as [19-22]. Among them, CycleGAN [7] uses the idea of duality to transfer image styles from two domains to each other, which can be regarded as a mutual generative process. The advantage of CycleGAN is that it can achieve style transfer between images without annotation and without using a paired training set. The specific operation is to first transfer the image of a domain to the target domain, and then transfer the image back to the initial domain again through the cyclic confrontation of the paired generator and discriminator, to achieve the transfer of image styles between different domains. In the latest research, there are also some other methods with better style transfer results, such as Junho Kim et al. [23] proposed U-GAT-IT, Runfa Chen et al. [24] proposed NICE-GAN, and ZhuoqiMa et al. [25], focusing on the semantic style transfer task, proposed a semantically relevant image style transfer method with dual consistency loss. It makes the output subject-related content and style image pairs semantically and stylistically consistent, increasing the diversity of the generated results. In this paper, we conduct further research work based on the CycleGAN model. According to our observation, the current image style transfer algorithm still faces the following challenges: first, the style transfer network tends to destroy part of the high-frequency detail information, such as edges and textures. Second, in style transfer, there is no attention mechanism to detect the targets that need to be transformed, resulting in not transferring where style transfer is needed and transferring where style transfer is not needed, producing target texture patterns in the background region. This leads to some undifferentiated over-texturing. Further, the loss function cannot capture the perceptual differences between the input and output images. As an extension of previous work, a new structure of style transfer network, ESA-CycleGAN: Edge feature and self-attention based cycle-consistent generative adversarial network for style transfer is proposed, which consists of a generator network, a discriminator network, a self-attention module, and an edge extraction module. In general, the main contributions of our research are summarized as follows. Add a perceptual loss term to the architecture and update the generative network by backpropagation. Continuously optimize the generative network to improve the visual and perceptual effects of the generated images. A self-attention module is added to the CycleGAN network, a structure that allows the generator to focus on the object structure pattern of the input image and try to learn more information to synthesize a more vivid image. The module is combined with the CycleGAN network structure to capture long-range, multi-level correlations in the image. An edge extraction network is added to the original network to solve the problem that the high-frequency parts of the generated images such as edges and small details are partially damaged. Compared with existing methods, ESA-CycleGAN achieves better performance in specific quantitative evaluation indexes. Besides, qualitative methods show that our model has better results in certain localities and pays more attention to the overall style, which both reasonably preserves the contents of the original image and adds the style of the specified painting, and has better results in perception. The rest of this paper is organized as follows: In Section 2, the previous related work is presented. In Section 3, the structure of our proposed ESA-CycleGAN network is described in detail. In Section 4, experiments and results are presented and discussed. In the last section, a summary of this paper is presented. 2 RELATED WORK 2.1 Style Transfer The traditional image style transfer problem uses some nonparametric texture conversion methods [1, 2]. Although these methods obtained better results, they can only extract the underlying features of the image instead of the higher abstract features, so the final results are rougher when dealing with images with more complex colours and textures. Nowadays, methods are more likely to adopt models using parameters to solve the transfer problem. Gatys et al. [3] combined deep learning and texture generative methods, using convolutional neural networks (CNN) to represent the style image as two parts: content and style, with the high-level convolutional layers of the CNN network extracting the global features of the image content and the low-level convolutional layers describing the image style, and this method can generate a more objective artistic style effects. However, this method has a slow convergence speed, long rendering time, and poor stylistic expression. Johnson et al. [26] trained feedforward networks with perceptual loss functions and used VGG trained on ImageNet [5] to simplify the calculation of loss functions, which directly generates stylized images with an efficiency improvement of three orders of magnitude. However, like the problem of Gatys et al. [3], the algorithmic limitations of the Gram matrix for extracting styles prevented the synthesis of realistic effects. 2.2 Generative adversarial network (GAN) GAN is short for Generative Adversarial Network, which has a very important impact on the research in the field of artificial intelligence. The main components of a generative adversarial network are the generator G and the discriminator D, which are trained in an adversarial manner. Generator G learns the data distribution of real image x to generate image y and passes it to discriminator D; discriminator D determines whether image y is real data or fake generated data. Generator G continuously learns and improves its ability to generate image data, while discriminator D gradually improves its ability to discriminate data, they reach a balance in the adversarial game process. The adversarial relationship is shown in Equation (1) m i n G m a x D V D , G = E x ∼ P d a t a x l o g D x + E z ∼ P z x l o g 1 − D G z , (1) where x is the image data, z is the input, G ( z ) is the generated image, P d a t a ( x ) is the distribution of the real image data, P z ( x ) is the noise distribution of the input network G, ∼ is the obedience relationship, and E is the mathematical expectation function. 2.3 Cross-Domain image style transfer model Transforming images from one domain to another but lacking paired training data from the original domain to the target domain can be attributed to the problem of cross-domain image style transformation, as in [27, 28]. Image style transfer tasks that do not need to be performed on paired datasets can be achieved by using a GAN model architecture based on the dual idea. The CycleGAN model is an image style transfer technique based on the idea of duality and is capable of transferring styles between various painting styles. It has good results for colour and texture transfer and can transform between unpaired source and target domains without the need for a one-to-one correspondence between datasets. The architecture of CycleGAN is shown in Figure 1 For example, the mapping from the X domain to the Y domain is to be achieved by mapping G. After mapping G, the picture x in the X domain is mapped to the picture G ( x ) in the Y domain. The discriminator against this generator is D Y , which is used to distinguish whether the generated image is the real data or the generated G ( x ) , which constitutes a single generative adversarial process. Based on the idea of dual learning, the authors of the CycleGAN model propose cycle consistency loss. At this point, another mapping F, which implements the mapping from Y domain to X domain, after the mapping F, the image y in Y domain is mapped to become the image F ( y ) in X domain, and the discriminator against this generator is D X , which is used to distinguish whether the generated image is the real data or the generated F ( y ) . Through continuous mapping and learning, the mutual style transfer between the two domains is achieved so that the final satisfaction: G ( F ( x ) ) ≈ x and F ( G ( y ) ) ≈ y . FIGURE 1Open in figure viewerPowerPoint Architecture of CycleGAN 2.4 Self-Attention As shown in Figure 2 the feature maps are divided into three channels, f ( x ) , g ( x ) , and h ( x ) , and the corresponding feature maps are output by convolving the input with three 1 × 1 convolution kernels for each of the three branches [29]. In which, the feature map output from the f ( x ) channel is transposed and calculated, and then matrix multiplied with the g ( x ) channel to obtain a matrix similar to the covariance matrix, from which the correlation and similarity between pixels are calculated. Subsequently, the attention feature map, that is, the feature weights, is obtained by softmax output. Finally, it is multiplied pixel by pixel with the feature map of the h ( x ) channel to obtain the attention-based weighted feature map, which is calculated as follows. O j = ∑ i = 1 N exp f x i T g x j ∑ i = 1 N exp f x i T g x j ∗ h x i , (2) FIGURE 2Open in figure viewerPowerPoint Self-attention module where h ( x i ) denotes the feature value at position i, f ( x i ) denotes the ith value in channel f, and g ( x j ) denotes the jth value in channel g. f ( x i ) and g ( x j ) run the inner product calculation to derive the attention weights at position i , j . Then a pixel-by-pixel dot product operation is performed with the h ( x i ) obtained from the h ( x ) channel to obtain the feature map based on the attention weighting. Subsequently, after a 1 × 1 convolution, a feature map of the same size as the original input feature information is obtained. 3 ESA-CYCLEGAN In this paper, a self-attention network combining edge features for image style transfer is proposed, named ESA-CycleGAN, that is, edge feature and self-attention based cycle-consistent generative adversarial network for style transfer. "E" stands for "Edge Feature", which is implemented by the edge extraction module, and "SA" stands for "Self-Attention", which is implemented by the self-attention module. First, the edges extracted from the original image input to the style transfer network to obtain stylized edge features to help preserve the details in the image. Secondly, we add the self-attention module to the generator and discriminator to obtain the global feature stylization of the image. Finally, the transferred stylized edge features are combined with the stylized image. 3.1 Network architecture 3.1.1 Edge extraction module The original image x can be divided into low and high-frequency components, x = x L + x H , where the high-frequency component x H contains more details of the image, such as edges and textures. The style transfer of the image, x H is more fragile, easier to be damaged, and difficult to be carried over to the migrated image. As shown in Figure 3 to get a better style transfer effect, we extract the edge features of the input image by a manually preset detector and feed them along with the original image into the network to get the style transferred edge features and combine these features with the stylized input image features, thus obtaining a style transferred image that performs better in terms of details. FIGURE 3Open in figure viewerPowerPoint Schematic diagram of the edge feature extraction module For the edge detector, the Canny [30] algorithm was chosen among multiple edge detection algorithms to extract texture edges for image processing. The main reason is that this algorithm produces more accurate edge detection results with strong resistance to noise interference. Each pixel point can be extracted and positioned directly to the texture image edge, facilitating the next operations on texture edges. The edge features extracted by the edge extractor are fed into the generator with the original image, as in Figure 3 to get the stylized transferred edge features. And these features are combined with the stylized input image features to get the style-transferred image that performs better in terms of details. 3.1.2 Self-attention module A self-attention mechanism [29] is mainly applied in the direction of natural language processing. In recent years, it has gradually been tried and applied a lot in various fields, especially in computer vision, and good results have been achieved in various tasks. Self-attention is a special case of attention mechanism. Unlike the standard attention mechanism, the purpose of the self-attention mechanism is to select the information that is more critical to the current task goal from the global information, so it can make good use of all the feature information of the image. Meanwhile, the self-attention mechanism can focus on itself and extract more relevant information from the input without using other additional information. It can learn the contextual information of the input and thus capture its interrelationship, to better handle the long-distance and multi-level dependencies in the image and coordinate the details of each location in the generated image. The structure diagram of the self-attention module is shown in Figure 2 Since the receptive field of the convolution kernel is localized, many layers must be accumulated to associate regions in different parts of the image. And in convolutional neural networks (CNN), the size of each convolutional kernel is also limited, so each convolutional operation can only cover a very small neighbourhood around a pixel. For features that are far away, they have to go through multiple layers of convolution and pooling operations, and these operations make the height and width of the feature map smaller and smaller, resulting in a larger area covered by the latter layers of the convolution kernel when mapping back to the area corresponding to the original image, and these operations make features that are far away not easy to capture. In contrast, the self-attention module can be viewed as a multiplication of the feature map and its transpose, allowing a direct relationship between pixels at any two locations, so that the dependencies between any two pixels can be learned, and thus, the global geometric features of the image can be acquired in one step, facilitating the learning of specific structural and geometric features. To obtain better-stylized results, a self-attention module is added to the generator and the discriminator, respectively. The exact placement of the modules will be discussed later in the experiment. 3.1.3 Generator The generator network consists of three parts: encoder, transformer, and decoder. The input image is transformed using a generator network, and the specific internal composition of the generator is shown in Figure 4 and Table 1 FIGURE 4Open in figure viewerPowerPoint Structure of the generator of ESA-CycleGAN TABLE 1. The internal structure of the generator Encoder Kernel size Connection method Stride Number of filters 7 × 7 Convolution; instance normalization; ReLU 1 64 3 × 3 Convolution; instance normalization; ReLU 2 128 3 × 3 Convolution; instance normalization; ReLU 2 256 Transformer Residual blocks Residual block1 Residual block2 … Residual block9 Self-attention module Decoder Kernel size Connection method Stride Number of filters 3 × 3 Deconvolution; instance normalization; ReLU 2 128 3 × 3 Deconvolution; instance normalization; ReLU 2 64 7 × 7 Deconvolution; instance normalization; ReLU 1 3 Input and output In the training phase, both the input and output are RGB images of size 256 × 256 with three channels. Encoder The main function of the encoder is to map the high-dimensional input image to the low-dimensional space. By doing so, the computation of the latter is greatly reduced. The input image is downsampled using one stride-1 convolutional layer and two stride-2 convolutional layers. After downsampling, a larger network and a larger effective receptive field can be obtained with the same computational cost, resulting in better style transfer. Transformer 9 residual blocks [31] are used to extract different features of the image and use these features to transform the feature vector of the image from the X domain to the Y domain. Decoder The decoder reconstructs the original size image from the output of the transformer. The decoder process is the opposite of the encoder, using deconvolution layers to reduce the low-level features of the image from the feature vector step by step until the generated image is the same size as the original. Combination Joint frequency domain information. The stylized image features output by the network are combined with the stylized edge features output by the network to generate the final stylized image. 3.1.4 Discriminator The discriminator extracts features from the image and then determines whether the extracted features belong to a specific class by adding convolutional layers to produce a one-dimensional output. In the experiments, a self-attention module is added to the discriminator. The discriminator is trained to discriminate the style images and the generator is trained to generate the style images. The training is considered to have reached maturity when both reach Nash equilibrium, that is, when the generator and discriminator promote each other to reach an optimal adversarial state. The internal composition of the discriminator is shown in Figure 5 and Table 2 FIGURE 5Open in figure viewerPowerPoint Structure of the discriminator of ESA-CycleGAN TABLE 2. The internal structure of the discriminator Discriminator Kernel size Connection method Stride Number of filters 4 × 4 Convolution; leaky ReLU 2 64 4 × 4 Convolution; instance normalization; Leaky ReLU 2 128 4 × 4 Convolution; instance normalization; Leaky ReLU 2 256 Self-Attention Module 4 × 4 Convolution; instance normalization; Leaky ReLU 1 512 4 × 4 Convolution 1 1 3.2 Loss function 3.2.1 Adversarial loss The style transfer model is built on the sample space of two domains, for example, to implement the transfer between X and Y domains. For mapping from the X domain to the Y domain using the CycleGAN model, G is used to represent this mapping. G is the generator in the generative adversarial network, and G is used to transform the image in the X domain to the image in the target Y domain and judge whether it is real data by discriminator D Y . This is a generative adversarial process. For this process, a generative adversarial loss is established as follows: L G A N G , D Y , X , Y = E y ∼ P d a t a y l o g D Y y + E x ∼ P d a t a x l o g 1 − D Y G x . (3) The final objective is: G ∗ = a r g m i n G m a x D Y L G A N G , D Y , X , Y . (4) Conversely, when the CycleGAN training process needs to transform the image in the Y domain into the image in X domain, set the mapping from Y to X as F. F is the generator in the generative adversarial network, and F is used to transform the image in Y domain into the image in the target X domain, judging whether it is the real data by discriminator D X . This is another generative adversarial process and the generative adversarial loss is: L G A N F , D X , Y , X = E x ∼ P d a t a x l o g D X x + E y ∼ P d a t a y l o g 1 − D X F y . (5) The final objective is: F ∗ = a r g m i n F m a x D X L G A N F , D X , Y , X . (6) 3.2.2 Cycle consistency loss CycleGAN introduces cycle consistency loss, and learns both F and G mappings, using G to transform an image x in the X domain space to the Y domain space and then back to x. The loss between x and F ( G ( x ) ) is calculated to avoid the possibility that all images in the X domain are mapped to the same image in the Y domain. Similarly, the loss between y and G ( F ( y ) ) is also computed to avoid the possibility that all images in Y are mapped to the same image in X. The purpose of introducing the cycle consistency loss function is to add more restrictions to preserve the contour features of the input images and reduce the mapping space, as well as to make the process of optimizing GANs more stable. The loss function is as follows: L c y c G , F = E x ∼ P d a t a x F G x − x 1 + E y ∼ P d a t a y G F y − y 1 (7) 3.2.3 Identity loss The generator G is used to generate y style image, so feeding y to G should still generate y. Only then can it be shown that G can generate y style. So, G ( y ) and y should be as close as possible. If this loss is not added, the generator may modify the hue of the image, causing the overall colour to change. L I d e n t i t y G , F = E y ∼ P d a t a y G y − y 1 + E x ∼ P d a t a x F x − x 1 . (8) 3.2.4 Perceptual loss The GAN loss only ensures that the generated image and the target domain image are passed into the discriminating network unable to distinguish between the generated image and the image in the real target domain, but does not make use of supervisory information. The image x in the source domain is expected to be as similar as possible to the paired image y in its corresponding target domain after passing through the generation network G. Therefore, the perceptual loss between images is defined. Similar to the generative description network structure, the image generated by the generative network is passed into the pre-trained VGG-19 network [4] at the same time as the paired image corresponding to the other domain. The high-level features of both are extracted, and the perceptual loss is defined on the high-level features. Inspired by Johnson et al. [26], who used a perceptual loss term to enhance image details, this paper introduces a perceptual loss term L P l to make the images generated by style transfer as unblurred as possible and with less detail loss: L P l G , F = 1 D H W φ G x − φ y 2 2 + φ F y − φ x 2 2 , (9) where φ is the feature extraction function, D, W, and H denote the depth, width, and height of the features, respectively. In this paper, the ReLU activation layer loss of the pre-trained VGG-19 network [4] is used as the perceptual loss. 3.2.5 Total loss Therefore, the total loss for the entire network training is L t o t a l = L G A N G , D Y , X , Y + L G A N F , D X , Y , X + α L c y c G , F + β L I d e n t i t y G , F + γ L P l G , F , (10) where α, β, and γ correspond to the weights of cycle consistency loss, identity loss, and perceptual loss, respectively. 4 EXPERIMENTS 4.1 Discussion of the location of the self-attention module 4.1.1 Data preparation and training details To explore the structure of the network, the exact location of the self-attention module needs to be determined before the experiment. The orange2apple dataset with a small number of photos w