# SADG: Self-Aligned Dual NIR-VIS Generation for Heterogeneous Face Recognition
## 3 Method
### Architecture
#### 3.2.1 Self-Aligned Generator  
Inspired by [7; 33; 34], we propose a self-aligned generator for producing ID-consistent aligned NIR-VIS images, including a self-aligned block. As shown in the left part of Figure 2, we utilized two decoders, \(D_{N}\) and \(D_{V}\), with the same structure in our generator, which was different from the baseline. Note that the architecture of our decoders was the same as that in the baseline. As for the training stage, given pairwise NIR and VIS images \(\{x_{N}\), \(x_{V}\}\), we also encoded them to latent code \(z_{N}\) and \(z_{V}\) as a baseline. \(L_{kl}\) and \(L_{dist}\) in Equations (1) and (2) were used for learning the data distribution. Considering the semantic deviation between \(z_{N}\) and \(z_{V}\), we sent the same code to \(D_{N}\) and \(D_{V}\) from one domain at the same time for reconstructing the input paired images \(x_{N}\) and \(x_{V}\), which could align the semantic information in the code level. We further aligned one domain's semantic information to another by a self-aligned block in the feature level. Note that \(z\) in Figure 2 is one code from \(z_{N}\) or \(z_{V}\). We used \(z_{V}\) for exhibition in the experimental part. Figure 3 shows our self-aligned block, mainly made up of a self-attention module [35]. First of all, we fetched out the feature maps from the same layer in decoders \(D_{\alpha}\) and \(D_{\beta}\). Secondly, we fed features from one domain into the self-attention module. Then, the feature maps were transformed into three parts--a, b and c--with two \(1\times 1\) convolutional layers, which halved the channels of features a and b. Next, parts a and b performed elementwise multiplication, followed by a \(1\times 1\) convolutional layer and a softmax operation to get the attention maps m. Finally, \(f_{1}^{(l)}\) was exported from this block through the elementwise multiplication of the feature maps c and attention maps m. The feature maps of other domains in the \(l\)th layer were denoted as \(f_{2}^{(l)}\). We used the Euclidean distance to reduce the semantic gap of the feature maps between two domains:  
\[L_{\alpha}=\sum_{l=n}^{Y}\|f_{1}^{(l)}-f_{2}^{(l)}\|_{2}^{2} \tag{5}\]  
where \(n\) and \(Y\) stand for the number of residual blocks in the two decoders. Note that we used the self-aligned block in the final convolution layer in last three residual blocks. With the help of the self-aligned block, we could extract the most important semantic information in one domain and align another one's semantic information to it. We used it in multiscale feature maps in several layers of the two decoders for better alignment performance. Regarding the reconstructed images, we employed a reconstruction loss [36] with a low weight for domain \(\alpha\) and a normal weight for domain \(\beta\). Specifically, we needed the decoders \(logp_{\theta}(x_{\alpha}|z_{\alpha})\) and \(logq_{\theta}(x_{\beta}|z_{\beta})\) to reconstruct the input images \(\{x_{N},\ x_{V}\}\) to a different extent:  
\[L_{rec}=-\ E_{p_{2\alpha(z_{1}|x_{\alpha})}}logp_{\theta}(x_{\alpha}|z_{\alpha })\ -\rho\ E_{q_{\beta(z_{1}|z_{\beta})}}logq_{\theta}\left(x_{\beta}|z_{\beta}\right) \tag{6}\]  
where \(\rho\) is a very low weight for the images \(x_{\alpha}\) from domain \(\alpha\) and \(x_{\beta}\) is the image to be aligned, whose domain is \(\beta\). Note that the self-attention block and the code z should have been adjusted according to the aligned modality. For example, if we wanted to align the semantic information in the NIR modality to the VIS one, we should send the same VIS code \(z_{V}\) to \(D_{N}\) and \(D_{V}\) first. Then, we extracted the significant semantic information in the VIS modality by our self-attention block and use \(L_{\alpha}\) to constrain the semantic gap between rgw VIS and NIR modalities. Finally, we used the reconstruction loss \(L_{rec}\) to force the NIR images to maintain their domain information at the image level and force the VIS images to be rebuilt faithfully.