Physically-Based Face Rendering for NIR-VIS Face Recognition
Authors: Yunqi Miao
Alexandros Lattas
Jiankang Deng
Jungong Han
Stefanos Zafeiriou
Venue: NeurIPS 2022 Accept (2022)

Abstract: Near infrared (NIR) to Visible (VIS) face matching is challenging due to the significant domain gaps as well as a lack of sufficient data for cross-modality model training. To overcome this problem, we propose a novel method for paired NIR-VIS facial image generation. Specifically, we reconstruct 3D face shape and reflectance from a large 2D facial dataset and introduce a novel method of transforming the VIS reflectance to NIR reflectance. We then use a physically-based renderer to generate a vast, high-resolution and photorealistic dataset consisting of various poses and identities in the NIR and VIS spectra. Moreover, to facilitate the identity feature learning, we propose an IDentity-based Maximum Mean Discrepancy (ID-MMD) loss, which not only reduces the modality gap between NIR and VIS images at the domain level but encourages the network to focus on the identity features instead of facial details, such as poses and accessories. Extensive experiments conducted on four challenging NIR-VIS face recognition benchmarks demonstrate that the proposed method can achieve comparable performance with the state-of-the-art (SOTA) methods without requiring any existing NIR-VIS face recognition datasets. With slightly fine-tuning on the target NIR-VIS face recognition datasets, our method can significantly surpass the SOTA performance. Code and pretrained models are released under the insightface GitHub.

Physically-Based Face Rendering for NIR-VIS Face Recognition
Yunqi Miao"
Alexandros Lattas" Imperial College London and Huawei lattas@@mmeriall.e. ac. uk
Jiankang Deng"
Warwick Manufacturing Group University of Warwick
Huawei and InsightFace
Yunqi. Miao. 1@warwick..o ac uk
@eegg66@imeeral.comm ac. ac.uk
Jungong Han
Stefanos Zafeiriou Department of Computing Imperial College London
Department of Computer Science
Aberystwyth University jungonghan770gmaill com
a. zafeeiiioouiiperrallll. ac
Abstract
Near infrared (NIR) Visible (VIS) face matching is challenging due to the significant domain gaps well a lack of sufficient data for cross cross-modality model training. To overcome this problem, we propose novel method for paired NIR-VIS facial image generation. Specifically, from a large 2D facial dataset and introduce a novel method of transforming the VIS reflectance to NIR reflectance. We then physically- baased renderer to generate a vast, higo-resolution and photorealistic dataset consisting of various poses and identities in the NIR and VIS spectra. Moreover, facilitate the identity feature learning, we propose an IDentity loss, which not only reduces the modality gap between NIR and VIS images at the domain level but encourages the network to focus on the identity features instead facial details, such poses and accessories. Extensive experiments conducted on four challenging NIR-VE VIS face recognition benchmarks demonstrate that the proposed method can achieve comparable performance with the state-of-the-art (SOTA) methods without requiring any existing NIR-VE VIR-VIS face recognition datasets. With slightly fine-tuning on the target NIR- VIS face recognition datasets, our method can significantly surpass the SOTA performance. Code and pretrained models are released under the insightface? GitHub.
reconstruct 3D face shape and reflectance
Maximum Mean Discrepancy (ID-MMD)
Introduction
To overcome the problem that the conventional VISible (VIS) images face recognition methods generally fail achieve satisfactory performance under poor illumination, face recognition across Near InfraRed (NIR) images and VIS images has recently gained increasing attention in the computer vision community [21, 22, 12, 13). However, due the lack of sufficient NIR-VIS data, the training of the NIR-VIS face recognition network is prone to be over-fitting [22].
To alleviate overfitting, previous works attempt to generate large-scale NIR-VIS face images by transferring the VIS images NIR ones [31, 40, 54, 52]. However, the image-to-image translation based methods are limited by the number of data in the source domain and the diversity of generated images [12]. Recently, unconditional generative models are employed to synthesize heterogencous
"Equal contribution. 'Corresponding author. This work is done when Yunqi Miao an intern at Huawei. htteep/nniiiittiiiiiiiiaaaaaee/e
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) DVG-Face [13]
(b) Ours
Figure 1: (a) Multiple VIS (top row) and NIR (bottom row) face images of the same identity generated by DVG-Face [13], and (b) our method, using two subjects from the CelebA dataset [35].
face image pairs from noise [12, 13] and achieve state-of-the-an art performance by adopting the various intra-changes, such as poses and illumination, of the target NIR- -VI datasets during the generation. Although the intra-class diversity is considered, only one VIR-VIS pair is generated for each identity in [13), which limits the potential of synthesized face images the NIR- VIS face recognition task. When generating multiple NIR-VIS image pairs for a given identity by using [13], we observe that the identity consistency cannot be well preserved, as shown Fig. (a). Additionally, the appearance variations of generated images are based on the target NIR-VIS face recognition datasets, which means, different facial images are synthesized to fit different target datasets. Such datasct-specific face generation degrades the generalizability of the NIR-VIS VIS face recognition networks.
To tackle the above problems, we propose novel physically- based facial image generation method, where high-quality NIR-VIL VIS facial image pairs are generated based acquired renderable 3D facial assets. Rendering photorealistic 3D facial datasets enables us acquire paired labelled training data with controllable identity, pose, expression and illumination. In contrast with generative methods, the rendered identity does not change at when varying other parameters, which greatly aids training. However, acquiring human rendering assets requires tremendous manually work, either by scanning systems [37, 18, 27, 28] or artists [47]. The available datasets are either small in size [2] or do not contain relightable reflectance [51], such the diffuse albedo, specular albedo and normals. Recent works [29, 30, 36, 10] have introduced methods that produce high-quality renderable assets from arbitrary facial images. Moreover, Wood al. [47] showed that high-quality synthetic facial data, can be successfully used for computer vision tasks, including landmark localization and facial parsing. However, to our knowledge, there exists no dataset method capable of producing renderable 3D faces, in both VIS and NIR domains.
Utilizing state-of-the-art facial reflectance acquisition method [30], we generate numerous such facial assets and transform them from VIS to NIR, and then render both under the same conditions, in order to generate high-quality training data. Since our novel transformation method is applied per-pixel on high-resolution reflectance maps, the identity of the subject is perfectly preserved in both NIR and VIS. Faces generated by the proposed methods are illustrated in Fig. 1(b). As can be seen, our NIR-VIS face generation outperforms [13] (Fig. 1(a)) in preserving identity consistency as well as retaining the diversity of facial appearances.
The generated high-quality NIR-VIS VIS facial image dataset then used, along with VIS face recognition dataset, to train the NIR-VE VIS face recognition network. To facilitate the identity feature learning as well as reduce the modality discrepancy, an IDentity-based Maximum Mean Discrepancy (ID-MMD) loss is proposed, which pulls the feature centroids of the same identity in the NIR domain and the VIS domain closer. Assisted by the ID-MMD loss, the gap between NIR images and VIS ones is bridged at the domain level, and meanwhile, the network is encouraged to focus on identity

---

features rather than facial details of instances, such as poses and accessories. Overall, our main contributions can be summarized as:
A method capable of generating amounts of paired NIR and VIS facial images, of var- ious identities, poses and illumination via 3D facial reconstruction and a novel VIS-to-NIR
transformation for facial reflectance, is proposed.
To bridge the gap between the NIR images and VIS images, we propose an IDentity-base Maximum Mean Discrepancy (ID-MMD) loss, which not only reduces the modality discrepancy at the domain level but encourages the network to attend to identity features instead of facial details.
Extensive experiments on four NIR-VIE face recognition benchmarks demonstrate that the proposed method achieves comparable performance with the state-of-the-an methods, without requiring any existing NIR-VIS face recognition dataset. By slightly fine-tuning the models on the target NIR-VIS, face recognition datasets, our method surpasses the SOTA performance.
2 Background and Related Work
NIR-VIS Face Recognition. To facilitate the NIR-VIL face recognition, earlier works focuses on learning moodallty-invariinn features [21, 22, 24, 23]. [21] extracts features for NIR and VIS images via a shared feature extractor pretrained on large-scale VIS face images, where the modality -nvaaiityinvariant identity information is then filtered out by an orthogonal constraint. Similarly, DFAL [24] and OMDRA [23] attempt to purify the identity information via decoupling identity-related representa- tions from modality invariint ones. To further reduce the gap between two modalities, WCNN [22] minimizes the Wasserstein distance between feature distributions of NIR and VIS images. WIT [41] treats each modality as whole and squeezes the centers of the two modalities via center maximum mean discrepancy loss. SMCL [46] designs a center-based loss to regulate the relationship between the syncretic modality and the NIR(VIS) one. However, due the limited amount of NIR-VIS data, the aforementioned methods are generally in the mire of the over-fitting problem.
To address the problem, generative models are involved to facilitate the NIR-VIS VIS face recognition task by transferring VIS face images to NIR ones via the image-to-image translation [31, 40, 54] or synthesizing heterogeneous face image pairs [13]. [31] produces VIS faces from NIR images via learning the patch-to-pal pathh mapping between NIR-VIS image pairs. [40] transfers NIR face images the corresponding VIS ones via two-path Generative Adversarial Networks (GAN)-based framework. (54) proposes a multi-stream feature-level fusion technique based on GAN to synthesize visible images from polarimetric thermal images. However, the improvement brought by the "one-to- one" face synthesis strategy is limited by the number of images within the NIR-VIS face recognition datasets, as well as the lack of the attribute diversity of generated images. Henceforth, instead of adopting the conditional image generation, [13] employs the Variational AutoEncoders (VAE) to synthesize paired NIR-VIS face images for given new identity. The identity representations are obtained from an identity sampler, which is pre-trained on a large-scale face recognition dataset. Inspired by [13], generating a large number of paired NIR- -IS face images is beneficial to the NIR-VIS face recognition. However, we notice that, when generating multiple NIR- VIS face image pairs from a given identity representation via [13], the identity consistency not well preserved. Such disadvantage hinders the potential of generated face images to boost the recognition performance since they cannot provide an explicit guidance for identity features learning. Therefore, we propose a novel NIR-VI VIS face image generation method, where a physically- -based renderer is used to generate a vast
dataset.
NIR-VIS Rendering, Synthetic datasets have long been used in face analysis problems with moderate success [55, 15, 14, 39]. DA-GAN [55] and UV-GAN [5] pionecred realistic profile face generation for pose-invariant face recognition. A recent work by Wood et al. [47], achieves state-of saate-of-the-an -htt-artt performance in various facial analysis tasks, by using hand-crafted high-quality shoto-realiste facial avatars. 3D facial appearance methods [29, 30, 36, 10] leverage deep generative models and differentiable rendering to reconstruct facial assets that can be
rendered. Moreover, another work [3] projects incomplete renderings to the latent space of a deep generative model and reconstructs corresponding photo-realistic facial images. Although potent the VIS spectrum, such methods are not directly usable in NIR.
3

---

The literature around NIR rendering from VIS assets remains limited. Wu et al. [50] introduced waaelength-ddeendeer Bidirectional Reflectance Distribution Function (BRDF) for Mid-Waver Wavelength Infrared (MWIR) landscape scene simulation. Moreover, Aguerre et al. [1] simulate urban thermal imaging. While closest to our work, none of the above can directly render photo-realistic faces in NIR-VIS. In that manner, we employ the recent works in photo-realistic synthetic face and combine it with our novel VIS-to-NIR transformation to unlock the potential for NIR-VIS facial matching.
Proposed Method
In this work, in order to overcome the over-fitting problem caused by the available small-scale NIR-VIS face recognition datasets, we generate vast, high-resolution and photo-realistic dataset consisting of large-scale identities with various poses and illumination in the NIR and VIS spectra (Sec. 3.1). Then, the generated NIR-VIS facial image pairs, along with large-scale VIS face recognition dataset, are used to train the NIR-VIS face recognition network, without the of any existing NIR- VIS datasets (Sec. 3.2).
VIS Reflectance
VIS Rendering
RVIS
Input
AD
A
115
1
1
RNIR
NIR Camera Parameters
VIR
HIR
NII
NIR
Reflectance
NIR Rendering
Figure 2: We generate paired NIR-VIS facial images by a) acquiring facial VIS reflectance from the public dataset of facial images (AA), b) transforming the VIS reflectance maps to NIR (T) by exploiting the way human skin reacts different spectra and c) rendering both VIS and NIR facial images in various poses and illumination conditions with the physically-based Rvis and RIR renderers, that model common VIS camera, and NIR camera with a flood illuminator.
3.1 NIR-VIS Facial Rendering
introduce method able to generate vast amounts photo-realistic pairs of NIR-VIS face images, to facilitate the heterogeneous face recognition task. Instead of using 2D generative models, reconstruct a set 3D facial assets capable of photo-realistic rendering in the VIS domain. The assets include facial shape, and VIS spatially-varying reflectance attributes, encoded in UV-space texture maps, which can be sampled during rendering [30]. Meanwhile, we transform these into the NIR domain with novel VIS-NIR transformation method and render paired images, described in the following sections. In this manner, we can create paired labelled NIR- VIS facial images, of numerous identities, in arbitrary poses and illumination conditions.
VIS Reflectance Data Acquisition. Recent works [29, 30, 36, 10] have achieved high-quality render-ready VIS face reconstructions, from arbitrary faces. In that manner, we acquire CelebA [35], dataset of over 200k facial images I,, and fit a textured 3D Morphable Model (3DMM), GANFIT-based [16] fitting F, acquire reconstructed texture and mesh S;. We employ an image-to-iman translation network A, based on AvatarMe+ [30], that disentangles facial reflectance maps from the reconstructed 3DMM texture maps. In that manner, for each image I, acquire the VIS specular albedo A
shape S,, diffuse albedo A AY,,
VIS and surface normals VIIS (also referred to
specular normals [30]), in the VIS domain:
[S,, AVIS
VIS
A
VIS VIS A(F(L))
(I)

---

VIS to NIR Reflectance Transformation. Human skin is a dielectric material and exhibits both diffuse and specular reflectance [37]. In the VIS spectrum, the diffuse albedo A A describes the amount of light emitted per RGB channel when a medium is lit by uniform white illumination. On the other hand, the specular albedo A VIS describes the intensity of the incoming illumination that reflected, at the direction of the normals NVTS, To transform these ppatilly-vvrrying reflectance values to the NIR spectrum, we define the following empirical model, based on the assumption that the reflectance attributes can be linearly described by the wavelength of the incident illumination, in the VIS (380- - 700mmm) and NIR illumination (850mm)
Surface normals can be acquired using a single-sensor input, R,G B [37, 27]. It is well-established that for normals measured under white illumination, shorter wavelength of incident illumination exhibits sharper surface details, due to the subsurface scattering of light in skin [37, 27]. For a green wavelength wD and wavellngtt w we calculate the width a of a Gaussian kernel 9 required to minimize the difference between the green normals N and red normals N (Eq. 2). Such transformations have been shown effective for normals manipulation [27, 28]. For a NIR wavelength NTR, we scale a, based on the distance between the red and NIR wavelength and apply g on VIS normals N VTS (in our case green channel [30], NVTS = NO) to acquire the NIR normals NNRR
NNIR G @(NVIS
WIR W.G W wG 0
N -G(NG, ,0)12
The NIR sensor is monochrome and its response is more similar to that of the VIS red channel. Given that a facial VIS diffuse albedo A is measured under uniform white light, its red channel A R measures the skin's response the red- waveleentntt w". Assuming again relationship between wavelength and spectral response, from red to infrared, we define the NIR diffuse albedo AD as the blurred red channel albedo AB. In contrast to Eq. 2, here we use a Bilateral Filter [42], in order to preserve the edges of the facial details.
Finally, we retain the VIS specular albedo as A = A NIR assuming it is wavelength inde- pendent in the visible spectrum [37, 27]. However, we decrease the overall specular roughness, proportionally to the distance of the target NIR wavelength from the mean of the visible spectrum, following [50]. Following the above, we define the complete NIR-VIS transformation function as VIS A VIS NVIS NIR).
ADIR, A NIR NNIR = T(AY
Paired NIR- VIS Rendering. The importance of explicitly extracting detailed albedo and nor- mal maps, lies in the fact that we can employ photo-realistic rendering algorithms, such as GGX [44] in our case. We collect a set of n environment maps Ei..... E., E E, that define in- coming illumination of various realistic scenes [4]. We define a VIS physically- based renderer RV R M, E) - I E R where h, w is the size of the rendered image, S is the facial shape, VIS and E an environment map. For NIR rendering, we define similar but monochrome renderer + I E 1xxwx1 Finally, as NIR sensors typically rely on flood illumina- tor placed adjacent to the lens, we create an environment map E, with only a flood illuminator placed in the camera direction. Then, for map E, we define the equivalent NIR map E,NIR E; + Ez.
A A A NVIS is the reflectance, M is rotation matrix for the shape
R. RNR M, E)
We then use the generated VIS and NIR facial assets and the VIS and NIR renderers, to generate sets of paired NIR-VIS facial images, under arbitrary illumination and pose. For a given subject i with shape S; and reflectance R, VIS and R NIR,, we randomly sample a) an environment map Ej E E, which is rotated along the Y axis by random angle 0₃ E [0, (0,360]. and b) rotation matrix M,. A NIR-VIS image pair is rendered by 1/7s R! IS (S, RIVSS Mj, Ej) and I R. R,NIR(S RNIR Mj, E, ENIR).
3.2 NIR- Face Recognition
As departure from most NIR- VIS face recognition works, the generated heterogeneous facial dataset is employed, along with a large-scale VIS face recognition dataset, to train a NIR-VIL VIS face recognition network explicitly. Specifically, we present the large-scale VIS face recognition dataset containing C identities as = W and the corresponding identity label as Y = ( Similarly, the synthesized NIR-VIS dataset with C, identities is denoted as X, { I. with label Y. [,, N and N, denote the number of images in the VIS dataset and the synthesized
5

---

MMD
ID-WND
PHIS
Somple
NIR
VES
HER
Medal Center
Identity Center)
The comparisons between the traditional MMD loss, the Pairwise Mean Square Error (PMSE) and the proposed IDentity-basa bbseed MMD (ID-MMD) loss. Different identities are represented by different colors (best viewed in color).
Figure 3:
NIR-VIS dataset, respectively. Note that, since the synthesized identities are from CelebA [35], there are no overlapped identities between the VIS face recognition dataset and the synthesized one. Given the final training set built by X and X,, following [13], widely-used face recognition network [48] is trained under the supervision the identity loss [6] and the proposed (Dentity-basee Maximum Mean Discrepancy (ID-MMD) loss. The NIR-VIS face recognition task is conducted with the identity features derived from the network.
Identity Loss. To improve the discrimination power of the face recognition network, we employ the margin-based softmax loss [6, 45, 33] Lid during training, which is denoted as follows,
e" " (m y2. +22)) mmy)
Cit N. + N.
D log e =fcos(mm () ma) E ylas e con 63.1
where cos0,,1 = W," Ji. is the normalized feature of i-th image, and W is the normalized weight vector of the j-th class. 0z. refers to the angle between Wj and fi- mi, m2, and my is the margin parameters. 8 is the feature scale. N, and N, are the training sample numbers of real VIS faces and synthesized NIR-VIS faces.
ID-MMD LOss. To overcome the main challenge of the NIR-VIS VIS face recognition task, i.e., the cross-modality discrepancy, the Maximum Mean Discrepancy (MMD) loss [17] designed for the transfer learning task is adopted. Formally, given a mini-batch containing M NIR images and N VIS images, the MMD loss Lmndd is formulated as follows,
M
N
Liannd
1
o(1 Σ 1=1
N Σ (antie)
1
(4)
M
where <b-) represents the kernel function that maps the original data to Reproducing Kernel Hilbert Space (RKHS) H. Although the MMD loss reduces the domain discrepancy by aligning NIR- VIS feature distributions, rigidly adopting such domain-leve constraint the NIR-VIS face recognition network training is sub-optimal, since it considers each modality as whole and ignores the identity feature distribution within the modality, as shown in Fig
To solve the problem, an explicit solution is reducing the distance between each NIR-VIS image pair of the same identity in the latent space, i.e., minimizing the Pairwise Mean Square Error (PMSE) Lpmse between the NIR-VIS features. Concretely, random sampling P identities from the heterogeneous dataset, and for each identity, sampling NIR images and K VIS images, to form the mini-batch with 2 x PK images. The Epmse is denoted as follows,
P
K
1 PK p=1k=
Cpmse
- 1
(5)
where pir//ois denotes the normalized feature of k-th NIR/VIS image of p-th identity. Although the identity distribution is considered, such pairwise loss reduces the modality discrepancy the instance level, where the network is highly likely to focus on facial details, such as poses and accessories, rather than identity features. Taking the girl wearing glasses in Fig. 3 as example, the PMSE loss may reduce the feature distance between the NIR- VIS image pair by encouraging the network to attend the frontal pose the glasses, instead of the identity features.
To address the problems of the domain-based (MMD loss) and the instance-based (PMSE) modality discrepancy reduction loss, propose ID-based MMD loss Lidmadd which bridges the modality

---

gap by reducing the distance between the NIR-VIS feature centroids of the same identity in the RKHS. Formally, for the given mini-batch, the proposed Lidmmd is denoted as follows,
K
K
Lidiondd =
Φ K
fnir P.k
o( K
cvis
(6)
p.k
1=1
H
The proposed identity-basse based modality discrepancy reduction loss not only reduces the modality gap between NIR and VIS images, but also encourages the features of the same identity within each modality be compactly distributed, i.e., reducing the intra-modality discrepancy. Overall, the
objective of the NIR-VIS face recognition network is denoted as L = Lid + X Lidmands where is the balancing parameter set as 100 during training.
Experiments
4.1 Databases and Protocols
Four NIR-VIS face recognition datasets are used to evaluate the proposed method. Specifically, the CASIA NIR-VIS 2.0 [32] (725 identities) and the LAMP-HQ [52] (573 identities) are the most challenging NIR-VIS face recognition datasets due the huge facial appearance diversity in poses, illumination, and ages. Following [32. 52], the ten-fold experiments are conducted on both datasets. For each fold, approximately 50% identities are randomly selected as the training set and the rest are adopted as the testing set. Note that, there is no overlap between the training and testing sets. Following [13, 52], the verification rate (VR) @ false accept rate (FAR)=0.011 %, VR FAR=0.1%. and the Rank-1 1 accuracy, are employed for evaluation. In terms of two low-shot NIR-VI HIR-VIS face recognition datasets: the Oulu-CASIA NIR- VIS [26] and the BUAA- VisNir [25], the identities within the datasets are split as 20/20 and 50/100, respectively, for the setting of the training/testing set. Considering the small data scale, the VR@ FAR=0.1 % and the Rank-1 accuracy are adopted as the evaluation metrics.
4.2 Experimental Details
NIR-VIS Face Generation. To acquire the 3D facial assets, we use AvatarMe++[30), trained with RealFaceDB [29] at textures of 1024 x 1024 pixels, on GANFIT [16] base. The dataset used CelebA [35], however, other 2D facial datasets can be used to extend generalization. For the rendering, we employ highly parameterizable commercial rendering software Marmoset Toolbag [38]. For each identity from CelebA, we synthesize 20 VIS and NIR facial image pairs, under different poses and illumination.
NIR-VIS Face Recognition. Following [13], we utilize LightCNN-29 [48] as the NIR-VIL face recognition backbone. To make fair comparison with [13], which adopts about 5 million images from the MS Celeb- 1M [20] dataset for pre-training, we use a subset from the large-scale VIS dataset [56], i.e., WebFace4M [57], for training. WebFace4M contains 4 million images of 200k randomly chosen identities from WebFace260M [57]. Additionally, instead of using 128 x 128 facial images as inputs, all the face images are aligned and cropped to 112 x 112 [7, 19] in the paper.
During training, we first train the network with identity loss Cia for 20 epochs on the WebFace4M and the synthesized dataset. Then, we fine-tune the network on the synthesized images with both identity loss Lia and ID-MMD loss Lidmmd for 5 epochs. The batch size is set as 512. During fine-tuning, 32 identities are randomly selected to form mini-batch, and for each identity, 8 VIS and 8 NIR images are randomly selected. Stochastic gradient descent (SGD) is adopted as the optimizer, where the momentum is set to 0.9 and the weight decay is set to 1e-4. The learning rate is set to le-2 initially and decays by a factor of 0.5 per 10 epochs.
4.3 Ablation Study
NIR Reflectance Generation. We show the importance of each component our proposed NIR reflectance generation method, by reconstructing subjects from LAMP-HQ [52] and showing the effectiveness of our algorithm against alternative approaches. In Fig. 4 and Tab. 1, we compare our method with monochrome rendering of VIS assets or remove some of our transformations. Not only
7

---

(a) GT [52] (b) Gr. RVSS
(c) R RATR
(d) +N
(c) Ours, front (f) Ours, side (g) Ours, side
Figure 4: From left to right: a) Ground Truth sample from LAMP-HQ(52), b) grayscale rendering
with VIS Renderer R and VIS assets, c) rendering with NIR Renderer R. RATR of VIS assets with ENIR d) NIR Renderer RVIR with our NiR-transformed normals NNIR, e) our complete method, including RNIRR NNIR and our transformed diffuse alhedo AD NIR and f), g) additional poses.
method is capable of producing photo-realistic NIR renderings, but can do is in various poses while preserving the subject identity, since it is based on rendering 3D assets.
Gray RVIS
RVIR
+NNIR
Ours
Method
MSt
MISI
FIDJ
PSNR
16.61
17.32
17.34
21.03
Ivi
1vN
VIS-VIS
NIR-VIS
DVG
0.429
0.244
0.149
0.138
0.877
MSE
0.021
0.018
0.018
0.007
Ours
0.641
0.411
0.099
0,084
0.609
Table I: NIR rendering ablation study, showing average MSE and PSNR, for LAMP-HQ [52]
Table 2: Comparisons of the identity consis- tency (MS), 2) the identity diversity (MIS), and 3) the distribution distance between generated and real data (FID) on LAMP-HQ.
reconstructions with our method and the ablation alternatives shown in Fig.
Identity Consistency and Diversity. Following DVG-Face [13], we analyze the identity consistency, the identity diversity and the distribution consistency of the generated NIR-VIS images via evaluation metrics- Mean Similarity (MS), Mean Instance Similarity (MIS), and Frechet Inception Distance (FID), respectively. To make fair comparison, we randomly select 1000 identities from NIR-VIS images generated by DVG-Face and ours. For each identity, 16 NIR images and 16 VIS images are randomly selected for evaluation. The results reported in Tab. 2. Note that, apart from measuring MS between each NIR-VIS image pair as DVG-Face, we also measure the feature similarity across multiple images generated for given identity, which are indicated by 1v1 and 1>N in Tab. 2, respectively. The higher intra-class (identity) similarity (MS) proves the superiority of our generation in preserving identity consistency. Additionally, our method achieves lower inter-class similarity (MIS), which ensures identity diversity. Meanwhile, a lower FID evaluated by LightCNN [48, 13] facilitates the adaptation to weal-world NIR-VIS face recognition datasets.
Effectiveness of Generated Data. To prove the proposed NIR-VI face generation method can significantly facilitate the NIR-VIS face recognition, compare the performances of models trained with the proposed ID-MMD loss under different percentages (0%, 10%, 50%, 100%) of generated data. In Tab. 3, the test results LAMP-HO show that the generated images continuously contribute to performance improvement and the best result is achieved when all generated data are involved.
Comparisons of Domain Adaption Losses. To compare our ID-MMD loss with other modality discrepancy reduction losses, the ablation studies are conducted on the LAMP-HQ dataset [52]. Specifically, PMSE loss, MMD loss, and ID-MMD loss are employed to supervise the learning of the NIR-VIS face recognition network. The corresponding recognition performances are reported in Tab. 4. It can be observed that the network achieves the best performance when optimizing by the proposed ID-MMD loss, surpassing the PMSE loss by 1.56% and the MMD loss by 1.04% when
VR VR@FAR=0.019%
Rato
FAR-000%%
FAR=0.1%
FAR=0.01%
FA
Rank-1
50%
10%
0%
07.28
90.64
82.13
1 1.9
1.7
0..
85.6 100
92.87 1 0.8
97.85
98.64
16 02
à
Catane Land
91.97 + 1.5
90.93 + 1.6
97.96 + 0.3
97.27
98.87 +: 0.3
08.04 07.55+0.3
100%
91.97 1.5
97.96
98.87
+ 0.3
90.11 1.5
0.2
Table 3: Performance comparisons between NIR-
Table 4: The comparisons between the perfor- mances of the backbone network (LightCNN-29) trained with different modality discrepancy re- duction losses on the LAMP-HQ dataset.
VIS face recognition models trained with 0%,
10%, 50%. 100% generated data on the LAMP- HQ dataset.
Effectiveness of IDMMD. To better understand the advantage of the generated NIR-VIS facial images as well as the proposed IDentity- based Maximum Mean Discrepancy (ID-MMD) loss, we visualize the identity feature distributions of the Dulu-CASIA NIR-VE VIS [26] and the LAMP-HO [52]
8

---

Outs- CAS NIR-VIS LC-29916g
Outa-CASIA NIR-VIS ratetry
64.
54-
22
23
23
00
(a) Baseline Oulu-CASIA
(b) Ours Oulu-CASIA
LAMP-NQ-EE LC 2Po
LAMP-HQ-1C 201 Fateree Laaad
58
02-
62-
05
(c) Baseline on LAMP-HQ
(d) Ours on LAMP-HQ
Figure 5: t-SNE [43] visualization of features of 10 identities randomly selected from Oulu-CASIA VIS, and LAMP-HQ. Different identities are denoted by different colors. NIR images; x: VIS images. (Best viewed in color).
dataset. Specifically, for each dataset, we randomly select 10 identities from the testing set. For each identity, we randomly selected 10 NIR images and 10 VIS images. We visualize the distribution of features derived by the baseline network "LC - 29 (Csa)" and the proposed one " LL 291 + Fake (Lid + Lidmma)) with t-SNE [43]. The visualization results are demonstrated in Fig. 5. Different identities are denoted by different colors. As can be seen, after involving the generated NIR-V VIR facial images and the ID-MMD loss, the NIR-VIS features of the same identity are pulled closer. Meanwhile, for each identity, the features within the NIR/VIS domain distribute more compactly. Such visualization results suggest that the proposed method can effectively reduce both intra-modality and inter model-my discrepancies.
4.4 Comparisons with State-of-the-art Methods
We extensively compare our method with the state-of-the-art (SOTA) methods four NIR-VI VIS face recognition benchmarks. The performances are reported in Tab. 5. Following [13]. we set the NIR-VIS face recognition baseline LighICNN-29 [48]. Unlike [13], our bascline model, 1) is trained on the WebFace4M [57], 2) takes 112 x 112 face images as input, and 3) is trained under the margin- based softmax loss [6]. As can be seen Tab. 5, our baseline models achieve a comparable performance with [48] at a lower input resolution.
We show that, with the aid of the synthesized NIR-VIS face images and the modality discrepancy reduction loss, our method greatly improves the baseline performances on four benchmark datasets. To intuitively illustrate the effectiveness of the proposed method, we visualize the feature similarity distribution of positive pairs (belonging to the same identity) and negative pairs (belonging to different identities) of LAMP-HQ Fig. 6. Benefiting from the generated data and ID-MMD loss, the similarities between positive pairs increase while the similarities between negative pairs decrease.
As can be seen, without requiring any existing NIR-VIS face recognition datasets, our method has achieved comparable performances with SOTA methods on the CASIA NIR-VI VIS 2.0 and the Oulu-CASIA NIR- VIS datasets, and has surpassed the SOTA performance by large margin on the other two. Especially, on the challenging LAMP-HQ dataset, our method outperforms the SOTA one [52] by 19.8% in terms of F 1%.
The performances can be further boosted after slightly fine-tuning the models on the target datasets. Specifically, as reported in the last two rows of Tab. 5, by adopting the identity loss (C.a) during

---

Method
NIEVISS
LXMPHO
CASLLC SIEVVTS
BURNNVINNE
TRIVET [54]
71
IDR
W CUCNNE2
94.3
28.42:04
18.7
177-4
ADFL (40)
08.2
73.5 22
05.1 0.5
60.7
05.5
68.0
015.2
RCN 191
98.7 0.2
00.
PUPH (53)
97.7 0.3
70.0
10.3 0.0
56.5
1000
112.4
15.41
MESENIN (N)
10.3 0.1
00:1.:11
PACH
98.3 0.2
02.0
70.3 1.7
95.4
58.2
100.0
50.5
95.6
DVR 1091
58.5 03
5016 0.8
54.9
100.0
06.5
993.2
DVG (12)
50.8-+0.11
1060
90.3
DFAL [24]
18.7
10.1+:0.2
LAMP-HO 152)
0:22
00.2:::00
78.2
97.8
DVG Face
00.2001
$..9::0.0
90.94:
LC
07.4
15.11
LC-29 (L.))
1.9
6..3=110
Fate (Gu))
0.
0.4
08.8 0.3
$10
08.4:::0.3
100.0
00.4
LCC799 Pakn tfee Caterall
65.5-65
10.0
56.0
86.6-11 033
94.5
1000
18.5
100.0
T2239 Realtf..)
10-5-2
14.7-003 0.9
56.7
16.0 03
10:0
19.5
29 Beall Cotered)
100.0
00.9
90.4
Table 5: Comparisons with the state-of-the-art NIR-VIS face recognition methods on the CASIA NIR-VIS 2.0, LAMP-HQ, Oulu-CASIA NIR-VIS, and BUAA-VisNir datasets. LC-29; Adopting the network structure of LighiCNN-29. t: Our baseline model. Fake: The synthesized images are included during training. Lis. Sidmand: The objective(s) used during training/fine-tninng
Fine-tuming tueettunig models on the target NIR- VIS datasets.
(C-)
LC-29' (Fa + Eximma)
neg
neg
pos
pos
Frequency
Frequency
Similarity
Similarity
Figure 6: Feature similarity distribution of positive/negaatve pairs of LAMP-HQ.
fine-tuning, VR VR@FAR-0.01%% increases from 92.0% to 94.5% on LAMP-HQ. After imposing the proposed ID-MMD loss (Lidmma)) during the fine-tuning target NIR-VIS face recognition datasets, the performances are further boosted. Concretely, an improvement of 6.6% on the VR@ FAR==.000 % can be observed on LAMP-HQ. Additionally, that fewer identities are contained, i.e., the Oulu-CASIA NIR-VIS and BUAA-VisNir datasets, we surpass the DVG-Face [13] by 1.8% and 0.8% in terms of VR @ @FA-O.. 1% respectively. In summary, after fine-tuning on the target NIR-VIS face recognition datasets, our method outperforms all other competitors on four benchmarks.
the two low-shot NIR-VIS face recognition datasets
5 Conclusion
To address the problem of insufficient NIR-VIS VIS data for the cross-modality face recognition network training, this paper proposes a novel NIR-VI VIS face generation method, which enables the generation of vast amounts of photo-realistic paired NIR-VIS facial images with various poses and illumination while preserving the identity consistency. Such merit enables the use of the generated dataset, along with large-scale VIS face recognition dataset, to train the NIR-VIS face recognition network, which can achieve comparable performance with the state-of-the-an methods without requiring any existing NIR-VIS face recognition datasets. Additionally, to bridge the domain gap between NIR images and VIS images during training, an IDentity-basee Maximum Mean Discrepancy (ID-MMD) loss is proposed, which reduces the modality discrepancy at the domain level and encourages the network to focus on identity features rather than facial details. Qualitative and quantitative experiment results on four NIR-VIS face recognition benchmarks show the superiority of the proposed method. Finally, understanding the social impacts facial matching, our method is modeled for and limited to mobile phone NIR sensors, and thus its use is aimed at more user-friendly experience for such devices.
Acknowledgements. Stefanos Zafeiriou acknowledges support from the EPSRC Fellowship DE- FORM (EP/S010203/I) and a Google Faculty Fellowship.
10

---

References
[1] José Pedro Aguerre, Elena García-Nevado, Jairo Acuña Paz y Miño, Eduardo Fernández, and Benoit Beckers. Physically based simulation and rendering of urban thermography. In Computer Graphics Forum, 2020.
[2] Oleg Alexander, Mike Rogers, William Lambeth, Yuan Chiang, Wan-Chun Ma, Chuan-Chang Wang. and Paul Debevec. The digital emily project: Achieving a photorealistic digital actor. IEFE Computer Graphics and Applications, 2010.
[3] Prashanth Chandran, Sebastian Winberg, Gaspard Zoss, Jérémy Riviere, Markus Gross, Paulo Gotardo, and Derek Bradley. Rendering with style: combining traditional and neural approaches for high-quality
face rendering. TOG, 2021.
[4] Paul Debevec. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphies with global illumination and high dynamic range photography. ACM Siggraph 2008 Classes. 2008. and Stefanos Zafeiriou. Uv-gan: Adversarial
[5] Jiankang Deng, Shiyang Cheng, Niannan Xue, Yuxiang
facial uv map completion for pose-invariant face recognition. In CVPR, 2018.
[6] Jiankang Deng. Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for
deep face recognition. In CVPR, 2019.
[7] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR, 2020.
[8] Zhongying Deng, Xiaojiang Peng. Zhifeng Li, and Yu Qiao. Mutual component convolutional neural
networks for heterogeneous face recognition. TIP. 2019.
[9] Zhongying Deng, Xiaojiang Peng, and Yu Qiao. Residual compensation networks for heterogeneous face recognition. In AAAI, 2019.
[10] Abdallah Dib, Cedric Thebault, Junghyun Ahn, Philippe- Heliip
Henri Gosselin, Christian Theobalt, and Louis Chevallier. Towards high tidelity monocular face reconstruction with rich reflectance using self- self-supervised
learning and ray tracing. ICCV, 2021.
[11] Boyan Duan. Chaoyou Fu, Yi Li. Xingguang Song, and Ran He. Cross-spectral face hallucination via disentangling independent factors. In CVPR, 2020.
[12] Chaoyou Fu, Xiang Wu, Yibo Hu, Huaibo Huang, and Ran He. Dual variational generation for low shot heterogeneous face recognition. In NeurlPS, 2019.
[13] Chanyou Fu, Xiang Wu, Yibo Hu, Huaiho Huang, and Ran He. Dvg-face: Dual variational generation for heterogeneous face recognition. TPAMI, 2021.
[14] Baris Gecer, Jiankang Deng, and Stefanos Zafciriou. Ostec: One-shot texture completion. In CVPR, 2021.
[15] Baris Gecer, Alexandros Lattas, Stylianos Ploumpis, Jiankang Deng, Athanasios Papaioannou, Stylianos Moschoglou, and Stefanos Zafeiriou. Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks. In ECCV, 2020.
[16] Baris Gecer, Stylianos Ploumpis. Trene Kotsia, and Stefanos Zafciriou. Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction. In CVPR, 2019.
[17] Muhammad Ghifary. W Bastiaan Kleijn, and Mengjie Zhang. Domain adaptive neural networks for object recognition. In Pucific Rim International Conference on Artificial Intelligence, 2014.
[18] Abhiject Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. Multiview face capture using polarized spherical gradient illumination. In SIGGRAPH Asia, 2011.
[19] Jia Guo. Jiankang Deng. Alexandros Lattas, and Stefanos Zafeiriou. Sample and computation redistribution for efficient face detection, In ICLR, 2021.
[20] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong Hc, and Jianfeng Gao. Ms-celeb-Ime A datasct and benchmark for large-scale face recognition. ECCV, 2016.
[21] Ran He, Xiang Wu. Zhenan Sun, and Tieniu Tan. Learning invariant deep representation for NIR-VIS IIR-VIS face recognition. In AAAI, 2017.
[22] Ran He, Xiang Wu, Zhenan Sun, and Tieniu Tan. Wasserstein cnn: Learning invariant features for nir-vis face recognition. TPAMI, 2018.
[23] Weipeng Hu and Haifeng Hu. Orthogonal modality disentanglement and representation alignment network for nir-vis face recognition. TCSVT, 2021.
[24] Weipeng Hu, Wenjun Yan, and Haifeng Hu. Dual face alignment learning network for nir-vis face
recognition. TCSVT, 2021.
[25] Di Huang. Jia Sun. and Yunhong Wang. The buaa-visnir face database instructions, 2012.
[26] Jie Chen, D. Yi, Jimei Yang. Guoying Zhao, S. Z. Li, and M. Pietikainen. Learning mappings for face synthesis from near infrared to visual light images. In CVPR, 2009.
[27] Christos Kampouris, Stefanos Zafeiriou, and Abhijeet Ghosh. Diffuse-specular separation using binary spherical gradient illumination. In EGSR, 2018
[28] Alexandros Lattas, Yiming Lin, Jayanth Kannan, Ekin Ozturk, Luca Filipi, Giuseppe Claudio Guarnera, Gaurav Chawla, and Abhijeet Chosh. Practical and scalable desktop-based high-quality facial capture. In
ECCV, 2022.
[29] Alexandros Lattas, Stylianos Moschoglou, Baris Geeer, Stylianos Ploumpis, Vasilcios Triantafyllou, Abhijeet Ghosh, and Stefanos Zafeiriou. Avatarme: Realistically renderable 3d facial reconstruction in-the-wild. In CVPR, 2020.
[30] Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer, Abhijeet Ghosh, and Ste- fanos P Zafeiiio.. Avatarmet+: Facial shape and brdf' inference with photorealistic rendering aware gans.
aware
TPAMI, 2021.
11

---

[31] José Lezama, Qiang Qiu, and Guillermo Sapiro. Not afraid of the dark: Nir-vis face recognition via cross-spectral hallucination and low-rank embedding. In CVPR, 2017.
[32] Stan Li, Dong Yi, Zhen Lei, and Shengeai Liao. The casia nir-vis 2.0 face database. In CVPR Workshops, 2013.
[33] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017.
[34] Xiaoxiang Liu, Lingxiao Song, Xiang Wu, and Tieniu Tan. Transferring deep representation for nir-vis heterogencous face recognition. In ICB, 2016.
[35] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
[36] Huiwen Luo, Koki Nagano, Han-Wei Kung, Qingguo Xu, Zejian Wang, Lingyu Wei, Liwen Hu, and Hao
Li. Normalized avatar synthesis using stylegan and perceptual refinement. In CVPR, 2021.
[37] Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles- FFelx Chabert, Malte Weiss, Paul E Debevec, et al. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Rendering Téchniques, 2007.
[38] Marmoset. Toolbag 3.07 http://mmammost..cooooooag/////
[39]
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. Sfsnet: Learning shape, reflectance and illuminance of facesin the wild'. In CVPR, 2018.
[40] Lingxiao Song, Man Zhang, Xiang Wu, and Ran He. Adversarial discriminative heterogeneous face recognition. In AAAI, 2018.
[41] Jia Sun, Yanfeng Li. Houjin Chen, Yahui Peng. Xiaodi Zhu, and Jinlei Zhu. Visible-infrared cross- coossmmddlltty person re-identification based on whole-individual training. Neurocomputing. 2021.
[42] Carlo Tomasi and Roherto Manduchi. Bilateral filtering for gray and color images. In ICCV, 1998.
[43] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-snc. JMLR, 2008
[44]
Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. Microfacet models for refraction through rough surfaces. Rendering Techniques, 2007.
[45] Hao Wang. Yitong Wang. Zheng Zhou, Xing Ji. Dihong Gong. Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. CVPR, 2018.
[46] Ziyu Wei, Xi Yang, Nannan Wang, and Xinho Gao. Syncretic modality collaborative learning for visible
infrared person re-identification. In ICCV, 2021.
[47] Erroll Wood, Tadas Baltrušaitis, Charlic Hewitt Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. Fake it till you make it: Face analysis in the wild using synthetic data alone. In ICCV, 2021. [48] Xiang Wu, Ran He, Zhenan Sun, and Tieniu Tan, A light enn for deep face representation with noisy labels.
TIFS. 2018
[49] Xiang Wu, Huaibo Huang, Vishal M Patel, Ran Hc, and Zhenan Sun. Disentangled variational representa-
tion for heterogencous face recognition. In AAAI, 2019.
[50] Xin Wu, Jianqi Zhang, Yang Chen, and Xi Huang. Real-time mid-wweelength infrared scene rendering
with feasible brdf model. Infrared Physicy & Technology. 2015.
[51] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao. Facescape:
large-scale high quality 3d face dataset and detailed riggable 3d face prediction. In CVPR, 2020. [52] Aijing Yu. Haoxue Wu, Huaibo Huang, Zhen Lei, and Ran He. Lamp-hq: A large-scale multi-pose
high-quality database and benchmark for nir-vi vis face recognition. IJCV, 2021.
[53] Junchi Yu, Jie Cao, Yi Li, Xiaofei Jia, and Ran He. Pose- sees-preerrving cross spectral face hallucination. In
IJCAI, 2019.
[54] He Zhang, Benjamin S Riggan, Shuowen Hu, Nathaniel J Short, and Vishal M Patel. Synthesis of high-quality visible faces from polarimetric thermal faces using generative adversarial networks. IJCV, 2019.
[55] Jian Zhao, Lin Xiong, Panasonic Karlekar Jayashree, Jianshu Li, Fang Zhao, Zhecan Wang, Panasonic Su- giri Pranata, Panasomic Shengmei Shen, Shuicheng Yan, and Jiashi Feng. Dual-agent gans for photorealistic and identity preserving profile face synthesis. In NewrlPS, 2017.
[56] Zheng Zhu, Guan Huang, Jiankang Deng. Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang. Dalong Du, Jiwen Lu, et al. Wehface260m: A benchmark for million-scale deep face recognition. TPAMI, 2022.
[57] Zheng Zhu, Guan Huang. Jiankang Deng. Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang. Jiwen Lu. Dalong Du et al. Webface260m: A benchmark unveiling the power of million-scale deep face recognition. In CVPR, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
12

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] The code and model will be submitted as supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Table 5.
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 4.2.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sec.4.
(b) Did you mention the license of the assets? [Yes] See Sec.4.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "qQeilzKJpJm", "reviewer": "Reviewer_Mb2x", "summary": "The work proposes a NIR-VIS face matching dataset constructed with a physically-based renderer. An ID-MMD loss is employed to facilitate the identity feature learning as well as reduce the modality discrepancy. The work achieves state-of-the-art performance on 4 NIR-VIS face recognition benchmarks. ", "questions": "(1)\tIt would be better if the combinations of modality discrepancy reduction losses and id loss, as the combination of losses sometimes can have larger impact than single ones. \n(2)\tIn DVG-Face, the evaluation metrics of generation quality are Mean Similarity, Mean Instance Similarity and Frechet Inception Distance. Why do the authors take different metrics in this work? Table 2 shows that the proposed method holds the smallest Mean Identity feature Distance. However, it can also be interpreted as lack of diversity. \n", "limitations": "See Weaknesses & Questions", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1)\tThe proposed method is capable to automatically generate multiple NIR-VIS image pairs with identity information reserved, which is of great significance. \n(2)\tWith the proposed training scheme, the NIR-VIS face matching dataset helps improve the NIR-VIS face recognition performance by a large margin. \n\nWeaknesses:\n(1)\tThe proposed ID-MMD loss reduces the distance between the NIR-VIS feature centroids of the same identity, which is effective yet not novel [1]. \n[1] Wei, Ziyu, et al. \"Syncretic modality collaborative learning for visible infrared person re-identification.\" ICCV. 2021.\n(2)\tThe training differences from other compared methods should be more detailly stated\n(3).  Some related works are missing, e.g., Dual face alignment learning network for NIR-VIS face recognition, Orthogonal modality disentanglement and representation alignment network for NIR-VIS face recognition, Dual-Agent GANs for Photorealistic and Identity Preserving Profile Face Synthesis\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Sj7wt6KCf94", "reviewer": "Reviewer_Srya", "summary": "This paper proposed a new face rendering technique to synthesize paired NIR-VIS images for improved face recognition in NIR space. Unlike the previous methods, which use image-to-image translation models learned from paired NIR-VIS images, this method use physical-based 3D face rendering. It first reconstructs 3D face meshes and reflectance assets in VIS space, then infers the corresponding reflectance assets in NIR space, and finally synthesizes paired VIS-NIR images at various head poses and illuminations. It also employs a novel ID-MMD loss to close the gap between VIS and NIR features in NIR-VIS face recognition training. The proposed method helped to achieve state-of-the-art face recognition performance on four NIR-VIS benchmarks.", "questions": "- Can we improve the method by using full-head 3D reconstruction, augmented expressions, and augmented accessories?\n- There is a predefined subset of WebFace260M called WebFace4M. The authors said they randomly selected images to create their WebFace4M dataset. Are they the same dataset? If not, better change the dataset name to avoid confusion.", "limitations": "There is no discussion on limitations. The paper limits its application to avoid potential social impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The proposed method allows generating infinite pairs of VIS-NIR facial images covering different head poses and illuminations without learning from any real VIS-NIR image pairs.\n- The paper proposes a sophisticated algorithm to transform a VIS reflectance asset into its NIR version.\n- The paper proposes a novel ID-MMD loss to close the gap between VIS and NIR features in NIR-VIS face recognition training. Ablation studies confirm its effectiveness.\n- By using only the synthesized VIS-NIR images, the trained face recognizers can produce competitive NIR face recognition performance compared with the baseline methods trained on real NIR-VIS datasets. After fine-tuning on real NIR-VIS images, these recognizers provide state-of-the-art and near-perfect performance on four NIR-VIS face recognition benchmarks.\n\n### Weaknesses\n- The synthesized images, particularly the VIS ones, look pretty unrealistic. The 3D models only cover the facial part, leaving some components missing, including hair, facial accessories, and background.\n- The denotations such as \\mathcal{R}^{NIR} should be explained earlier, e.g., in the text or the caption of Table 1, rather than in the caption of Fig. 4.\n- There is a predefined subset of WebFace260M called WebFace4M. The authors said they randomly selected images to create their WebFace4M dataset. Are they the same dataset? If not, better change the dataset name to avoid confusion.\n- It seems the authors did not augment facial expressions; all qualitative figures of synthetic images have neutral expressions. I think the expression is not an important factor in the test benchmarks, but the authors should consider it when developing face recognizers for real-world applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Pn6IESwYeA9", "reviewer": "Reviewer_wS34", "summary": "In this paper, the authors propose a method to synthesize near-infrared faces by transforming them from visible faces. Based on this, the authors can conduct NIR-VIS face recognition without any existing NIR-VIS face datasets. Besides, they also propose an Identity-based Maximum Mean Discrepancy loss to facilitate identity feature learning. The good performance shows the efficiency of their method.", "questions": "Please see my comments above.", "limitations": "The limitations and potential negative societal impact have been adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors utilize a novel 3d-rendering based method to generate large-scale NIR-VIS paired data.\n2. The authors propose an Identity-based Maximum Mean Discrepancy loss to facilitate identity feature learning.\n3. The descriptions of implementations of the proposed synthesis method are detailed.\n\nWeakness:\n1. The motivation for using a 3d-rendering based generating dataset is not clearly illustrated, and it seems that the authors just utilize id-related loss when training. It is suggested to strengthen more use of the generation.\n2. The comparison results are not sufficient. It is suggested to add generated data by different percentages to illustrate the effectiveness of the synthesis method.\n3. I suggest more comparative visualizations to verify the effectiveness of the proposed method. In addition, the authors should discuss how generalizable the proposed method is in practical situations.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept