# EC-GAN: Emotion-Controllable GAN for Face Image Completion
## 1 Introduction  
Image completion [1; 2] (also known as image inpainting) is the task of filling in the missing parts of an image with visually realistic and semantically plausible content. As a low-level vision application, it has been extensively studied and applied in unwanted object removal [3; 4] and image synthesis [5; 6]. As one example, the completion of face images has been an active research topic since it has the potential to benefit a wide range of applications, such as old facial photo restoration, portrait image editing, and face beautification. In order to capture the high-level semantics while maintaining consistency near the borders of the regions to be completed, the current mainstream methods are based on two types of deep learning generative models, namely, Variational Autoencoders(VAEs) [7] and Generative Adversarial Networks (GANs) [8; 9], conditioned on known image regions. A typical strategy is to feed corrupted images into an encoder-decoder network, generating a photorealistic image similar to the ground truth and then filling the corrupted areas with the corresponding content [10]. There have been tremendous advances in image completion with the proper design of networks and loss functions [11; 12; 13]. Due to the diversity in facial expressions and the subjective nature of human vision, one of the main challenges in the completion of face images is that a single corrupted face can produce numerous reasonable results [14; 15; 16]. However, previous research still has its limitations. Concretely speaking, (1) the majority of groundbreaking works, such as GLCIC [17], DeepFill [16], Partial Convolution (PConv) [15], EdgeConnect [18], PENNet [12], and CR-Fill [19], aim to obtain only one desirable result for each image, and thusare incapable of generating a wide variety of completions. (2) Other methods, such as PIC-Net [13], PD-GAN [20], UCTGAN [21], and PUT [22], can produce diverse results by generating from the random samplings of the latent spaces [23]. However, these methods do not have the ability to control and customize the completion results. Thus, imagine that we want an angry face rather than a smiling one. A post-processing step is inevitable after image completion, which means either manually selecting one among numerous pluralistic results or further utilizing an emotion recognition system to pick out the desired one. The two methods outlined above are both labor-intensive and inefficient in practice. In order to bridge the gap between the controllability and diversity during face image completion, we propose an Emotion-Controllable GAN (EC-GAN), a novel face completion model that can infer and customize the emotion based on the information of the unmasked parts. EC-GAN utilizes the CVAE-GAN [23] architecture as the completion network. To embed the expected image prior to the generation process, we first designed an encoder based on the CVAE pipeline since it has excellent performances in learning the structured representation, including expected semantics for real images [24]. Several advanced losses [13; 25] were also introduced to ensure the fidelity and realism of the completed results. We have proposed an emotion inference module to infer approximate emotion from unmasked regions of human faces. Following the usual practice [26], we used seven common categories of facial emotions, namely, _neutral, happy, sad, angry, disgusted, surprised, and fearful_. The emotion inference module is trained to learn the emotion semantics in a supervised manner and it enforces the encoder to disentangle the emotion semantics from the native latent space. To customize a desired emotion, we propose an emotion control module to modify the corresponding emotion semantics in the latent space. In this way, we established our Emotion-Controllable GAN image completion framework. The sub-modules in our model were trained simultaneously in an end-to-end manner, and the model can correctly complete facial images with the emotions designated by users.