# A Multi-Input Fusion Model for Privacy and Semantic Preservation in Facial Image Datasets
## 4 Method
### Architecture and Working
#### 4.2.4 Learning Stage  
Inspired by QM-VAE [1], a service-guided generative model that takes several privacy-removed facial images to generate one high-quality image, the framework of this part contains three different parts: the encoder, the decoder, and the embedding space \(e\). Concatenated by the surrogate image and corresponding de-identified images generated by four traditional privacy protection methods (covering the eyes, blurring the faces, adding Laplace noise, and transforming the image into cartoon form), the input \(i\) is fed into the encoder, producing \(z_{e}(i)\). Instead of directly transporting \(z_{e}(i)\) to be finished like an ordinary autoencoder, \(z_{e}(i)\) is transformed into the new embedding vector \(e_{i}\) by searching for the closest embedding. Equation (14) shows the one-hot defined form of posterior categorical distribution \(q(z|i)\) probabilities. \[q(z=k|i)= \begin{cases}1&if\ k=\underset{j}{argmin}\ ||z_{e}(i)-e_{j}||_{2}\\ 0&other\end{cases}  \tag{14}\]  
In the process of achieving the discretization process, we utilize a mapping described in Equation (15) to transform and identify the closest element of \(z_{e}(i)\), which is subsequently passed to the decoder. \[z_{q}(i)=e_{k},\quad\text{where}\ k=\underset{j}{argmin}\ ||z_{e}(i)-e_{j}||_{2} \tag{15}\]Ultimately, \(z_{q}(i)\) is passed through the decoder and becomes the output, \(z_{e}(i)\). Our model utilizes gradients to facilitate the updating of values in Equation (15), which impacts the encoder's discretization and subsequently affects the final output. As the encoder and decoder share the same D-dimensional space, gradients also help adjust the encoder's output to decrease the reconstruction loss. The complete loss function consists of three components. The first part \(L_{q}\) represents the specific machine learning task service quality loss. In this work, we focused on facial semantics (facial expression). In each iteration of training, the expression of generated images is recorded. To make sure the generated expression is the same as the original picture's label, the task-related quality loss, \(L_{q}\), is determined using Equation (16). If the output's facial expression recognition result matches the label, it indicates that features related to facial semantics are maintained and \(L_{q}\) is set to 0. Alternatively, if they do not match, \(L_{q}\) is set to 1. \[L_{q}= \begin{cases}0&if\ E(i)=E(z(i))\\ 1&if\ E(i)\
eq E(z(i))\end{cases}  \tag{16}\]  
In the next part, \(L_{m_{1}}\) quantifies the resemblance between the output images and the original images. The initial component of the function delineates the loss of image reconstruction between the original images and the corresponding identity-concealed images generated by the proposed model. The intention of the second component is to enhance the vector quantization embedding space by continuously updating the dictionary throughout the model training process. Its function is to slow down the update rate of the encoder, keeping the generated output close to the embedding vector. \[L_{m_{1}}=logp(i|z_{q}(i))+||sg[z_{e}(i)]-e||_{2}^{2}+\beta_{m_{1}}||z_{e}(i)- sg[e]||_{2}^{2} \tag{17}\]  
where \(sg\) represents the stop gradient, stopping the gradient flow from flowing through specific parts of the network, and is defined as a constant value in forward computing of the whole model. Changes in the parameter \(\beta_{m_{1}}\) ranging from 0.1 to 2.0 do not significantly affect the outcome. Similar to \(L_{m_{1}}\), the third part \(L_{m_{2}}\) shown in Equation (18) below calculates the degree of resemblance of faces contained in two images. The difference is that \(L_{m_{2}}\) is used to describe the reconstruction loss between output images and the corresponding surrogate facial images generated from the same cluster. \[L_{m_{2}}=logp(i|f(i))+||sg[f(i)]-e||_{2}^{2}+\beta_{m_{2}}||f(i)-sg[e]||_{2} ^{2} \tag{18}\]  
where \(\beta_{m_{2}}\) ranges from 0.1 to 2.0.