# EC-GAN: Emotion-Controllable GAN for Face Image Completion
## 3 Proposed Approach
### The Disentangled Emotion Inference Module  
Since the encoder \(E\) is constructed by multiple convolutional layers with different scales, it mashes up all the facial features in the latent space, which makes the generator \(G\) fail to extract the emotion information from the latent codes. To generate images with customized emotions, we need to disentangle the emotion semantics from the latent space [37]. To address this problem, we designed an emotion inference module which utilizes the mutual information [41] to achieve the independent encoding of emotion semantics in the latent space. The emotion inference module contains two parallel paths. One path is the inference network designed to infer the labels of images and then calculate the entropy of emotions, and the other is the encoder aiming to embed the remaining image information into the latent space. The outputs of the two paths are combined in the emotion control module, which will be interpreted in Section 3.3. The emotion inference network is always trained from scratch for different quantities of emotions. Meanwhile, in order to save calculation costs, the emotion inference module shares the same architecture and parameters with the discriminator \(D\), except for the last output layer. First, the emotion inference network tries to obtain the inferred emotion labels \(c_{gt}^{infer}\) of the ground truth \(I_{gt}\) and the inferred generation labels \(c_{g}^{infer}\) of the constructed image \(I_{g}\). To obtain the emotion semantics, we optimized the inference network in a supervised manner. By using the given emotion label \(c_{gt}\), we can utilize the L-1 loss function to optimize the emotion loss of the inference module:  
\[L_{E}=||c_{gt}^{infer}-c_{gt}||_{1}+||c_{g}^{infer}-c_{gt}||_{1}. \tag{6}\]  
Next, according to information theory, mutual information measures the reduction in uncertainty between two random variables. If the two variables are independent, the mutual information becomes zero. In contrast, if two variables are strongly correlated, then one of them can be predicted from the other. Consequently, we propose to maximize the mutual information \(I(c;G(z,c))\) between the inferred emotion labels \(c\) and the generation distribution \(G(z,c)\) to strengthen their correlation. In the next parts, \(c_{gt}^{infer}\) is replaced with \(c\) for convenience. We set the inferred emotion labels \(c\) as part of the input of generator \(G\) and note this part as the emotion entropy or latent codes of emotion semantics \(z_{emo}^{c}\) in advance. The latent codes of emotions \(z_{emo}^{c}\) are calculated by a replication strategy in which we expand the \(c^{m\times 1}\) dimension to match the latent space dimension, which is noted as the mapping function:  
\[f:c^{m\times 1}\to z_{m\times n\times n}^{c}. \tag{7}\]  
where \(m\) is the number of emotion categories and \(n\times n\) is the dimension of the latent space. By doing so, the generator \(G\) can produce samples containing the information from the inferred emotion labels \(c\) and will change expressions as \(c\) changes. Then, we can calculate the mutual information \(I(c;G(z,c))\). However, the mutual information term \(I(c;G(z,c))\) is hard to maximize directly as it requires the posterior information \(P(c|x)\)[36]. Therefore, we utilize a lower bound of it to approximate \(P(c|x)\) by defining an auxiliary distribution \(Q(c|x)\):  
\[ \begin{split} I(c;G(z,c))&=H(c)-H(c|G(z,c))\\ &=E_{x\sim G(z,c)}[E_{c^{\prime}\sim P(c|x)}[\log P(c^{\prime}|x) ]]+H(c)\\ &=E_{x\sim G(z,c)}[D_{KL}(P(c|x)||Q(c|x))+E_{c^{\prime}\sim P(c|x )}[\log Q(c^{\prime}|x)]]+H(c)\\ &\geq E_{x\sim G(z,c)}[E_{c^{\prime}\sim P(c|x)}[\log Q(c^{ \prime}|x)]]+H(c).\end{split}  \tag{8}\]  
In fact, the entropy of the inferred emotion labels \(H(c)\) can be treated as a constant in that it highly relies on the truth labels \(c_{gt}\).