ApprGAN: appearance‐based GAN for facial expression synthesis

IET Image ProcessingVolume 13, Issue 14 p. 2706-2715 Special Section: Adversarial Learning in Image ProcessingOpen Access ApprGAN: appearance-based GAN for facial expression synthesis Yao Peng, Yao Peng School of Electrical and Electronic Engineering, The University of Manchester, Manchester, UKSearch for more papers by this authorHujun Yin, Corresponding Author Hujun Yin hujun.yin@manchester.ac.uk School of Electrical and Electronic Engineering, The University of Manchester, Manchester, UKSearch for more papers by this author Yao Peng, Yao Peng School of Electrical and Electronic Engineering, The University of Manchester, Manchester, UKSearch for more papers by this authorHujun Yin, Corresponding Author Hujun Yin hujun.yin@manchester.ac.uk School of Electrical and Electronic Engineering, The University of Manchester, Manchester, UKSearch for more papers by this author First published: 01 December 2019 https://doi.org/10.1049/iet-ipr.2018.6576Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Facial expression synthesis has drawn increasing attention in computer vision, graphics and animation. Recently, generative adversarial nets (GANs) have become a new perspective for face synthesis and have had remarkable success in generating photorealistic images and image-to-image translation. In this study, the authors present an appearance-based facial expression synthesis framework, ApprGAN, by combining shape and texture and introducing cycle consistency and identity mapping into the adversarial learning. Specifically, given an input face image, a pair of shape and texture generators are trained for synthetic shape deformation and expression detail generation, respectively. Extensive experiments on expression synthesis and cross-database synthesis were conducted, together with comparisons with the existing methods. Results of expression synthesis and quantitative verification on various databases show the effectiveness of ApprGAN in synthesising photorealistic and identity-preserving expressions and its marked improvement over the existing methods. 1 Introduction Facial expression synthesis or transfer aims at altering the expression of a specified person while still preserving the personal identity and facial features of the subject. In spite of considerable effort from academia and industry, expression synthesis and analysis remain challenging due to the subtlety, complexity and variability of human expression [1, 2]. Synthesising photorealistic images with various expressions is of great value in many application scenarios such as human–machine interaction, face editing, emotion analysis, animation and medical care [3]. The past few decades have witnessed numerous techniques being developed for expression synthesis. Existing methods can be divided into two categories: traditional handcrafted methods and recent deep learning-based generative models. The majority of the early studies handle faces by image warping and mapping, facial component editing and face decomposition [4-7]. These methods sometimes can produce natural, high-resolution expressions, but usually require paired training samples and rely highly on the accurate extraction of landmarks. With the recent advances of deep learning, generative adversarial nets (GANs) have achieved remarkable results in several applications [8-10]. The main idea is to simultaneously train paired discriminator and generator, to produce indistinguishable fake faces from real ones. The generator finds low-dimensional representations of faces in a latent space and manipulates them to synthesise new images. Despite their ability to generate impressive or promising results in most cases, synthesised images sometimes lack fine details and tend to be blurry or of low resolution [10]. Besides, both shape and texture attributes are usually encoded in one latent feature space, making it hard to take control of subtle expression changes. In this paper, we propose an appearance-based GAN, ApprGAN, to synthesise expressions by combining the conventional warping-based methods with the generative models. We derive shape and texture attributes of faces and construct two GAN-based networks for geometry deformation and expression detail generation, respectively. We also incorporate reconstruction and identity mapping losses in adversarial learning to stabilise and improve GAN training. Expressions are synthesised from neutral faces on both geometry attributes and expression details. To evaluate the quality of these generated images, correlation coefficients and normalised landmark distances are calculated between synthesised and ground truth images. The effectiveness of the proposed method has been verified on three benchmark databases including the Bosphorus [11], extended (action unit) AU-coded Cohn–Kanade (CK + ) [12, 13] and MUG [14] facial expression databases. The main contributions of the current work are summarised as follows: (i) An appearance-based GAN, ApprGAN, for facial expression synthesis with a combination of shape and texture generative models. (ii) Incorporation of cycle consistency and identity mapping losses into the generative adversarial learning to stabilise and improve image synthesis. (iii) Synthesis of photorealistic and identity-preserving expressions with good generalisation across databases. (iv) Improved quantitative measures of synthesised expressions compared with state-of-the-art methods. The remaining of this paper is organised as follows. Related work is reviewed in Section 2. Section 3 describes the formulation of the objective function, the proposed expression synthesis framework and expression verification method. Section 4 presents experimental results and analysis, followed by conclusions in Section 5. 2 Related work The facial expression describes facial motions and conveys important visual information in communication and understanding human behaviours [15]. Automatic analysis of facial expression has hence attracted a great deal of attention with a wide range of applications such as expression classification [16, 17] and face recognition [18, 19]. As a form of non-verbal communication, facial expressions convey emotion states through muscle movements beneath the skin. In 1872, Darwin first demonstrated the universal nature of expression and linked human emotions with mental states and body movements. Later psychologists proposed and developed the anatomy-based facial action coding system (FACS) [20, 21] that could comprehensively interpret all possible facial expressions. The FACS considers only emotion-related facial motions and describes them by combinations of 44 individual and visually distinguishable action units (AUs). According to the intensity of AUs, expression intensities are annotated in five different levels, from a trace to the maximum. Seven prototypic facial expressions are typically defined as neutral, anger, disgust, fear, happiness, sadness and surprise [22]. In the last two decades, there have been various approaches to generate realistic facial expressions in computer vision [5, 6, 23, 24], most of which are either morph based or example based. Liu et al. [4] proposed to calculate an expression ratio image (ERI) of a subject to capture the illumination change resulting from expression transition. With geometric warping, the ERI could be mapped to a different person's face to obtain facial expression synthesis. Wang and Ahuja [7] used a principal component analysis-based high-order singular decomposition approach to extract separate subspaces (identity, expression and feature subspaces) from a collection of images displaying different expressions. Expressions for a new subject could be synthesised using the learned subspace model. Ghent and McDonald [25] constructed facial expression shape and texture models to generate expression mapping functions. In conjunction with the radial basis function networks, neutral images could be mapped to alternative expressions of the same subjects. Zhang et al. [6] proposed an expression synthesis framework based on geometry to synthesise expression texture from facial landmark movements. Given a face image and its facial landmarks, expressions were synthesised within 14 face subregions and blended along the subregion boundaries. Huang and De la Torre [26] developed a bilinear kernel-reduced rank regression-based method to learn a non-linear mapping from neutral face to various expressions. The input neutral image was further combined with training data for preserving person-specific expression details against untrained illumination conditions. It is often difficult for these warping-based methods to synthesise high-resolutional faces and unseen facial features (such as teeth and inner mouth areas, beard and person-specific expression details). Recently, deep generative models have achieved impressive success in a wide range of applications such as image generation [27], image-to-image style translation [28-30] and super-resolution [31, 32]. The advances have also proved their usefulness in facial expression synthesis and transfer. Susskind et al. [33] trained a deep belief net capable of converting facial attribute labels into realistic face images as well as generating faces displaying specific identities and facial actions. Li et al. [34] built a convolutional neural network to generate facial images with input source images and reference attribute labels. Radford et al. [27] introduced deep convolutional GANs for unsupervised image generation. This approach could learn hierarchical representations of input and be able to generate expressive faces. Choi et al. [8] proposed a multi-domain image-to-image translation approach called StarGAN to synthesise images across domains using a single model, which could also be applied in facial expression transfer. Ding et al. [9] introduced the ExprGAN for facial expression editing with controllable expression intensity. In addition to the network, an expression controller was employed to learn expression code without the need for training data with intensity values. Song et al. [10] built a geometry-guided GAN architecture for synthesising images with continuously adjusting expressions. Geometry information was treated as an additional image channel to be concatenated with the input face to guide the transfer of facial expressions. Similarly, Qiao et al. [35] designed a geometry-contrastive GAN for transferring continuous emotions from one subject to another. Contrastive learning from geometry information was integrated into embedding network. Recently, Pumarola et al. [36] presented an AU-based GAN model for synthetic facial animation, which enables controlling the magnitude of activations of each AU. They also embedded an attention module to allow focusing only on image regions relating to specific expressions, making the network robust to changing backgrounds and illumination conditions. 3 Methods This section describes the formulation of the proposed expression synthesis framework, its objective function and expression verification method. We incorporate reconstruction and identity mapping losses in the adversarial learning to define the objective function and present the pipeline of the proposed ApprGAN. To evaluate the quality of synthesised expressions, we calculate the correlation coefficients and landmark distances. 3.1 Objective function Inspired by CycleGAN [29] and domain transfer network (DTN) [37], we define our loss functions containing three terms: adversarial loss for pushing the distribution of generated images to match those of the target domain; reconstruction loss for preventing unstable and unbalanced GAN training; and identity mapping loss for preserving texture composition and shape identity information. Adversarial loss: As a type of generative models, GANs learn a mapping frominput to target domains based on the min–max game theory. A GANconsists of two adversarial networks: a discriminator that tries toclassify between real and fake samples and a generator thatsimultaneously trained to synthesise realistic fake samples to foolthe discriminator. Given two image domains and , the generator is optimised to map from inputto target domains, and meanwhile the discriminator aims to distinguish betweensamples and synthesised . The adversarial loss isdefined as (1) in which aims to minimise the objectivefunction against attempting to maximise it.Similarly, the adversarial loss for the translation can be expressed as . Reconstruction loss: With the adversarial loss only, training of GANs is still remarkably difficult and suffers from many problems such as non-convergence, unbalance between generator and discriminator, diminished gradient, overfitting and so on. Furthermore, there is still little theoretical explanation of the unstable behaviour of GAN training [38]. To stabilise and improve the learning process, a reconstruction loss is introduced to enforce the learned mapping functions and to be forward–backward cycle consistent [29]. For each image , the forward cycle consistency, , indicates that the reconstructed image should be able to closely match the input. Also for each image , the backward cycle consistency should also be satisfied, . Thus, an -norm-based reconstruction loss can be written as (2) Identity mapping loss: In facial expression synthesis, we need suitable control for expression changes without altering the facial texture and identity information. To preserve face identity and texture composition, we adopt an additional identity mapping loss [37] between input and output as a regulariser. The identity mapping loss encourages generators and to be near identity matrices when real samples from target domains and are fed as input, respectively. The loss term can be introduced as (3) Full objective function: The full objective function for the expression synthesis framework is the sum of all three loss terms as (4)where and are ratios controlling the balance between these three terms. Generators , and discriminators , are trained by solving (5) In practise, to further stabilise the training procedure and generate higher-quality images, we replace the negative log-likelihood adversarial loss by a least-square loss [39] . Instead of solving , we optimise the generator and discriminator by (6) (7)In the , -norm is used instead for the log losses of the original GAN, preventing vanishing gradients when the samples lie in the correct side, but far away from the real data. Owing to that, the objective function contains mapping functions in both directions and , networks trained with loss can be applied to generate images for both input and target domains. Furthermore, since the objective function is defined on two domains rather than image pairs, there is no requirement for the existence of paired training samples in the input and target domains. In expression synthesis, we take neutral expressions as the input domain and train networks for modelling other six basic expressions. Each pair of expressions is separately trained as neutral and anger, neutral and disgust, neutral and fear, neutral and happiness, neutral and sadness and neutral and surprise. The formulation of the loss function for mapping from neutral to happiness images is shown in Fig. 1, and the translation from happiness to neutral shares similar structure and formulation, so are the other expression pairs. Fig. 1Open in figure viewerPowerPoint Formulation of the objective function in the proposed expression synthesis framework 3.2 Expression synthesis framework The active appearance model [40] extracts both shape attributes and texture properties of a face. An unseen face is represented by both shape and texture variations learned from training images. In expression synthesis, the statistical shape model and shape-free texture are commonly used for modelling shape and texture changes caused by skin deformation, respectively [41, 42]. In this paper, we derive shape attributes and texture information from each training sample as a one-dimensional (1D) vector and a 2D image , respectively. Shape and texture for different expressions are then produced and combined to form generated expressions. The general framework of expression synthesis is shown in Fig. 2. Fig. 2Open in figure viewerPowerPoint General process of expression synthesis To reduce landmark differences across subjects caused by scale, pose and viewpoint, shape alignment is necessary. For each face, its shape is represented with a set of landmarks located around key areas (8)where represents coordinates of the landmark along x and y axes, respectively. Translation, rotation and scaling are used to align all the shapes so that eyes are located at the same positions. Given a set of aligned shapes, we split the landmarks of each shape into four groups based on facial features: eyebrows and eyes, nose, mouth and face contour. In addition, for texture attributes, we warp all faces in each expression category so that their landmarks match the mean shape of that expression. According to the distribution of landmarks, Delaunay triangulation divides faces into small triangles based on the landmarks. The piecewise affine transformation is adopted to warp texture from the original shape to the mean shape in corresponding triangles. Given an input neutral image and its shape and texture attributes and , the trained shape and texture generators synthesise and for different expressions. We warp the generated texture to the synthesised shape to obtain the final synthesised expression (see Fig. 3). Fig. 3Open in figure viewerPowerPoint The proposed ApprGAN algorithm In photorealistic expression synthesis, inner mouth region is person specific and cannot be ignored in certain expressions especially happiness. Since there is no landmark labelled within the mouth, synthesised inner mouth region from the training set can hardly produce high-resolution results. Therefore, for an unseen subject, we replace the synthesised inner mouth region with one of the training subjects having the most similar mouth shape. Euclidean distances are calculated between the synthesised mouth shape and each of those from the training set and used as the similarity measure for this replacement. The proposed expression synthesis procedure is summarised in Fig. 3. 3.3 Expression verification on synthesised images To evaluate the quality of synthesised expressions, we compare synthesised images with the ground truth images on both shape and texture attributes [43]. For shape attributes, normalised distance is computed between the ground truth landmarks and the synthesised landmarks (9)where is the absolute function, and and refer to the largest distances between each ground truth shapes in x and y axes, respectively. To measure the texture similarity, we calculate the correlation coefficients between the synthesised texture and its corresponding ground truth texture as (10)where specifies pixel locations in the texture images, and and are the mean values of and , respectively. 4 Experiments and results 4.1 Databases We conducted several experiments on three publicly available databases: Bosphorus, CK + and MUG facial expression databases. The Bosphorus database [11] consists of 105 subjects (60 men and 45 women, aged between 25 and 35) with a total number of 4666 images. Images are of a high resolution of with variations in expression, pose and occlusion. Each subject has up to 35 expressions including seven prototypical expressions (neutral and 6 basic emotions), and each image has 24 manually labelled landmarks per image. The CK + database [12, 13] is a collection of videos of 123 subjects performing various expressions: anger, contempt, disgust, fear, happiness, sadness and surprise. There are totally 593 video sequences starting with neutral images (the first frames) and ending with expressions at the maximum level (the last frames). Video frames are of sizes of or arrays and annotated with 68 facial landmarks. The MUG database [14] contains image sequences of 86 subjects (51 men and 35 women, aged between 20 and 35) performing various expressions (neutral and 6 basic emotions), among which only images of 52 participants are publicly available. Video sequences all start with the neutral images and then go to the apex state before returning to the offset. Image sequences were recorded at a rate of 19 fps with a size of . 4.2 Network architecture and training settings We adopted ten-fold cross-validation for expression synthesis due to a limited number of subjects. We randomly split each database into ten groups and built ten pairs of training and test sets for each database. For example, the happiness face of the subject in group 1 was synthesised using generators trained with images from all other groups 2–10. Therefore, there was no overlapping between the training and test identities and images. In texture synthesis, each input image was resized to 200 × 200. The input vectors for synthesising shapes were made of coordinates of 83 landmarks [http://www.faceplusplus.com/.] (8 points for inner mouth shape), and zeros inserted between four landmark groups, producing vectors with a length of 204. Generator architectures: The generator network has a similar architecture to the image transformation networks [31] which have shown impressive results for both style transfer and image super-resolution. We adopt six residual blocks, fractionally strided convolutions with stride 1/2, reflection padding and instance normalisation. Network architectures used for shape and texture generators are displayed in Table 1. Each residual block contains two 3 × 1 (in shape generator) or 3 × 3 (in texture generator) convolutional layers with the same number of filters and an instance normalisation layer. To reduce artefacts, spatial reflection padding was used in convolutional blocks so that the input and output of each convolutional layer have the same size. Table 1. Network architecture used for shape and texture generators Shape generator Texture generator input 204 × 1 input 200200 3271 conv., stride 1, ReLU 6477 conv., stride 1, ReLU 6431 conv., stride 2, ReLU 12833 conv., stride 2, ReLU 12831 conv., stride 2, ReLU 25633 conv., stride 2, ReLU residual block, 128 filters residual block, 256 filters residual block, 128 filters residual block, 256 filters residual block, 128 filters residual block, 256 filters residual block, 128 filters residual block, 256 filters residual block, 128 filters residual block, 256 filters residual block, 128 filters residual block, 256 filters 6431 conv., stride 1/2, ReLU 12833 conv., stride 1/2, ReLU 3231 conv., stride 1/2, ReLU 6433 conv., stride 1/2, ReLU reflection padding 31 reflection padding 33 171 conv., stride 1, ReLU 177 conv., stride 1, ReLU Discriminator architectures: In constructing discriminators, we adopted a Markovian discriminator – PatchGAN [28]. PatchGAN assumes only local dependency between pixels and models images as Markov random fields. Instead of targeting the whole image, this patch-based discriminator aims to classify whether each overlapping patch in an image is real or fake. We used 34 × 1 and 70 × 70 PatchGANs as discriminators in synthesising shape and texture, respectively. Let 32C4 × 1 represent a 4 × 1 convolutional layer with 32 filters, stride 2 and leaky rectified linear units (ReLUs) having a slope of 0.2, the discriminator architectures are: 32C4 × 1–64C4 × 1–128C4 × 1 for shape discriminator and 64C4 × 4–128C4 × 4–256C4 × 4–512C4 × 4 for texture discriminator. After these convolutional layers, a final layer was added to produce a 1D output. Instance normalisation was used in each convolutional layer, except for the first one. Training details: The standard adversarial training of discriminators only focuses on the latest generated image, which may cause divergence and introduce some artefacts [44]. On the basis of these observations, we instead updated discriminators using a history of previously generated images. During the adversarial training, the size of the image buffer was the same as the number of training images. For all experiments, we set , for shape generation and for texture synthesis, as the shape is more important than texture in regularising expression synthesis, supported by the experimental results. The networks were all trained from scratch using the Adam solver [45] with a mini-batch size of 1, and weights were initialised from a Gaussian distribution with 0 mean and 0.02 standard deviation. The learning rate started at 0.0002 was kept the same for the first 100 epochs and then linearly decreased to 0 over the next 100 epochs. Horizontal flipping was employed to augment data. 4.3 Expression synthesis results For each database, we adopted a ten-fold cross-validation strategy to split training and test pairs and train the networks. For the Bosphorus database, 62 subjects were selected with each having a neutral image and six basic expressions. Fig. 4 displays the input neutral image in the first row and the ground truth images at the bottom row. Synthesised six basic expressions are shown in the second row: anger, disgust, fear, happiness, sadness and surprise, from left to right. Fig. 4Open in figure viewerPowerPoint Synthesised expressions on Bosphorus database when trained using subjects from Bosphorus, CK + , MUG, combined CK + and MUG or combined Bosphorus, CK + and MUG databases For the CK + database, we selected video sequences displaying six basic emotions: anger (33), disgust (46), fear (20), happiness (50), sadness (18) and surprise (45), in which videos in a certain expression were all from different subjects. The first and last frames in each video sequence were considered as the neutral image and the most expressive image, respectively. With input neutral images in the first row, the second and bottom rows of Fig. 5 illustrate the synthesised expressions and the corresponding ground truth images (from left to right: anger, disgust, fear, happiness, sadness and surprise). Fig. 5Open in figure viewerPowerPoint Synthesised expressions on CK + database when trained using subjects from CK + , Bosphorus, MUG, combined Bosphorus and MUG or combined Bosphorus, CK + and MUG databases On the MUG database, video sequences of 51 subjects were selected, from which the first frames were taken as the neutral images and the expression images at its peak intensity were used. Fig. 6 illustrates the input neutral images (in the first row), synthesised anger, disgust, fear, happiness, sadness and surprise expressions (from left to right in the second row), and the ground truth expressions at peak intensity (in the bottom row). Fig. 6Open in figure viewerPowerPoint Synthesised expressions on MUG database when trained using subjects from MUG, Bosphorus, CK + , combined Bosphorus and CK + or combined Bosphorus, CK + and MUG databases 4.4 Expression verification results To measure the quality of the synthesised expressions, we calculated the normalised distances (9) for shape attributes and correlation coefficients (10) for texture attributes between pairs of synthesised and ground truth images. Results for the Bosphorus, CK + and MUG databases are presented in Tables 2–4, respectively. The average correlation coefficients and normalised distances from these tables are , ; , ; ; and , showing the closeness of the synthesised and the ground truth expressions. Table 2. Correlation coefficients and normalised distances between synthesised and ground truth texture, landmarks on the Bosphorus database Expression Expression verification (meanstandard deviation) Correlation coefficients Normalised distances anger 0.9750.012 2.0540.923 disgust 0.9650.015 2.2040.684 fear 0.9720.014 2.1600.639 happiness 0.9700.021 1.8840.525 sadness 0.9770.011 1.9900.695 surprise 0.9740.011 2.0720.622 average 0.9720.014 2.0610.681 Table 3. Correlation coefficients and normalised distances between synthesised and ground truth texture, landmarks on CK + database Expression Expression verification (meanstandard deviation) Correlation coefficients Normalised distances anger 0.9660.010 1.7830.425 disgust 0.9580.020 1.9340.580 fear 0.9550.010 2.0330.408 happiness 0.9540.031 1.5730.260 sadness 0.9610.016 1.8010.517 surprise 0.9230.035 2.1490.543 average 0.9530.020 1.8790.456 Table 4. Correlation coefficients and normalised distances between synthesised and ground truth texture, landmarks on the MUG database Expression Expression verification (meanstandard deviation) Correlation coefficients Normalised distances anger 0.9810.006 2.0190.899 disgust 0.9680.014 1.8840.572 fear 0.9780.007 1.7660.430 happiness 0.9760.010 1.5710.319 sadness 0.9760.009 1.7270.443 surprise 0.9700.010 2.0460.540 average 0.9750.009 1.8350.534 Results of expression verification experiments on va