# SADG: Self-Aligned Dual NIR-VIS Generation for Heterogeneous Face Recognition
## 3 Method
### Baseline  
As proposed in [8], the baseline network mainly consisted of a dual variational autoencoder. In particular, the autoencoder included two encoder networks \(E_{N}\) and \(E_{V}\), as well as a decoder network \(D\). Given the pairwise NIR-VIS images \(\{x_{N},x_{V}\}\), the encoder \(E_{N}\) plays the role of mapping NIR images \(x_{N}\) to the latent space \(z_{N}\) with a reparameterization trick: \(z_{N}=\mu_{N}+\sigma_{N}\odot\tau\). \(\mu_{N}\) and \(\sigma_{N}\) are defined as the mean and standard deviation of the feature maps from the NIR images in a mini-batch, respectively, \(\tau\) is sampled from a multivariate standard Gaussian and \(\odot\) denotes the Hadamard product. \(E_{V}\) conducts the same operation correspondingly and finally gets the latent code of the VIS images \(z_{V}\). Note that this encoding process can be found in our architecture, as shown in Figure 2. Finally, the latent code \(z_{N}\) and \(z_{V}\) are concatenated to get a joint distribution \(z_{I}\), which is utilized to reconstruct the input images by one decoder network in the baseline network.  
Figure 2: The training process of our proposed self-aligned generation architecture. Given the paired data from the training dataset, we first reverse them by \(E_{N}\), \(D_{N}\), \(E_{V}\) and \(D_{V}\), respectively (the left part). Then, we use a multiscale patch discriminator (the right part) to do the adversarial learning with the generator. The solid-lined rectangle stands for the residual block, and the dotted-lined rectangle stands for the latent code \(z\).  
As with the VAE in the original work [23], DVG constrains the learning process of the posterior distributions \(p_{\varnothing N}(z_{N}|x_{N})\) and \(p_{\varnothing V}(z_{V}|x_{V})\) by a _KL_ divergence:  
\[L_{kl}=\ D_{KL}(p_{\varnothing N}(z_{N}|x_{N})||p(z_{N}))\ +\ D_{KL}(p_{ \varnothing V}(z_{V}|x_{V})||p(z_{V})) \tag{1}\]  
where the prior distributions \(p(z_{N})\) and \(p(z_{V})\) are both the multivariate standard Gaussian distributions. Then, a reconstruction loss is used to force the decoder network to reconstruct the input images \(\{x_{N},x_{V}\}\) from the learned distribution, which is modified in our method for alignment. We will discuss this in Section 3.2.  
Except for \(L_{kl}\), a simplified Wasserstein distance [8] between the two distributions \(p(z_{N}^{(i)})=N(\mu_{N}^{(i)},\sigma_{N}^{(i)})\) and \(p(z_{V}^{(i)})=N(\mu_{V}^{(i)},\sigma_{V}^{(i)})\) is also utilized for posterior distribution learning in latent space, where \(i\) stands for the identity:  
\[L_{dist}=\sum_{i}\frac{1}{2}[\|\mu_{N}^{(i)}\ -\ \mu_{V}^{(i)}\|_{2}^{2}+\| \sigma_{N}^{(i)}\ -\ \sigma_{V}^{(i)}\|_{2}^{2}] \tag{2}\]  
For the image space, DVG uses a pre-trained LightCNNv2 [31] model as the Identity-feature extractor \(F_{id}\) for calculating identification loss. Due to DVG producing a pair of NIR-VIS images for each allotment of time, we can retain the identity information by constraining the feature distance between the reconstructed paired images. The same operation is conducted for the reconstructed image with the original image, and the loss functions are as follows:  
\[L_{id\text{-}pair}\ =\ \|F_{id}\ (\hat{X}_{N})\ -\ F_{id}\ (\hat{X}_{V})\|_{2}^{2} \tag{3}\]  
\[L_{id\text{-}repair}=\|F_{id}\ (\hat{X}_{N})\ -\ F_{id}\ (x_{N})\|_{2}^{2}+\| F_{id}\ (\hat{X}_{V})\ -\ F_{id}\ (x_{V})\|_{2}^{2} \tag{4}\]  
where \(F\ (\cdot)\) denotes the uniformed output of the last fully connected layer in \(F_{id}\). For the purpose of increasing the diversity of the generated images, a diversity loss [32], defined as \(L_{div}\), is adopted as the original model [8].  
After training the network, DVG can generate paired NIR-VIS images with the same identity by the same noise sampled from Gaussian space, which can boost the limited dataset.