# Beyond the Visible: A Survey on Cross-spectral Face Recognition

David Anghelone, Cunjian Chen, Arun Ross, Antitza Dantcheva

<figure><img src="image_1.png" /></figure>

## To cite this version:

David Anghelone, Cunjian Chen, Arun Ross, Antitza Dantcheva. Beyond the Visible: A Survey on Cross-spectral Face Recognition. Neurocomputing, 2025. hal-05469800

HAL Id: hal-05469800

https://inria.hal.science/hal-05469800v1

Submitted on 21 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Highlights

## Beyond the Visible: A Survey on Cross-spectral Face Recognition

David Anghelone, Cunjian Chen, Arun Ross, Antitza Dantcheva

* Formalization of the cross-spectral face recognition problem

* Appropriate electromagnetic spectral band for related cross-spectral face recognition

* General components of a cross-spectral face recognition system

* Recent advances based on deep learning in addressing cross-spectral face recognition

* Future research for addressing challenges related to cross-spectral face recognition
# Beyond the Visible: A Survey on Cross-spectral Face Recognition

David Anghelone<sup>a,*</sup>, Cunjian Chen<sup>b,c,*</sup>, Arun Ross<sup>b</sup>, Antitza Dantcheva<sup>a</sup>

<sup>a</sup>Centre Inria d'Université Côte d'Azur, France

<sup>b</sup>Michigan State University, USA

<sup>c</sup>Monash University, Australia

## Abstract

Cross-spectral face recognition (CFR) refers to recognizing individuals using face images stemming from different spectral bands, such as infrared versus visible. While CFR is inherently more challenging than classical face recognition due to significant variation in facial appearance caused by the *modality gap*, it is useful in many scenarios including night-vision biometrics and detecting presentation attacks. Recent advances in deep neural networks (DNNs) have resulted in significant improvement in the performance of CFR systems. Given these developments, the contributions of this survey are three-fold. First, we provide an overview of CFR, by formalizing the CFR problem and presenting related applications. Secondly, we discuss the appropriate spectral bands for face recognition and discuss recent CFR methods, placing emphasis on *deep neural networks*. In particular we describe techniques that have been proposed to extract and compare heterogeneous features emerging from different spectral bands. We also discuss the datasets that have been used for evaluating CFR methods. Finally, we discuss the challenges and future lines of research on this topic.

**Keywords:** Biometrics, Cross-spectral, Face recognition, Deep neural networks, Infrared, Thermal

## 1. Introduction

Face Recognition (FR) has been a very active research field for the last several decades [1, 2, 3, 4]. Advances in deep convolutional neural networks (CNNs) and the associated seminal work on DeepFace [5] have brought significant progress in this area, tackling a number of challenges including variations in pose, illumination and expression (PIE), as well as resolution and unconstrained settings. While such work have predominantly focused on the visible (VIS) spectrum, considering additional spectra allows for increased robustness [6, 7], in particular in the presence of different poses, illumination variations, noise, as well as occlusions. Further benefits include incorporating the absolute size of objects, as well as robustness to presentation attacks such as the use of makeup and masks to circumvent a system [8]. Therefore, comparing RGB face images against face images acquired beyond the visible spectrum, often referred to as Cross-spectral Face Recognition (CFR), is of particular significance in designing FR systems for defense, surveillance, and public safety [6] and falls under the broader category of Heterogeneous Face Recognition (HFR) that addresses the broader concept of recognizing faces across diverse and non-uniform conditions. Specifically, heterogeneous refers to face images being represented in different modalities, e.g., visible, infrared, 2D-3D, resolutions, hand-drawn sketches, and depth (see Figure 1).

The focus of this survey is on CFR, and specifically on comparing visible face images with images acquired beyond the visible spectrum, namely the infrared spectrum. We note that CFR is more challenging than traditional FR for both, human examiners, as well as computer vision algorithms due to the following three factors. Firstly, there is large intra-spectral variation, where within the same modality, face samples of the same subject can exhibit larger appearance variations than face samples of different subjects. Secondly, the *modality gap* is of concern, where appearance variation between two face samples of the same subject can be larger than that of two samples belonging

*Corresponding author.
Email addresses: david.anghelone@inria.fr (David Anghelone), cunjian.chen@monash.edu (Cunjian Chen)
<figure><img src="image_3.png" /><figcaption>Figure 1: Examples of heterogeneous face images of the same subject captured in different modalities, i.e., domains. Images are from the MCXFace dataset [9].</figcaption></figure>

to two different subjects (see Figure 1). This can result in degraded face recognition performance. Finally, the *limited* availability of *training samples* of cross-modality face image pairs can make it difficult to design effective CFR methods using deep neural networks.

<figure><img src="image_4.png" /><figcaption>Figure 2: An overview of cross-spectral face recognition</figcaption></figure>

Recent advances in CNNs and generative adversarial networks (GANs) have allowed for remarkable improvements in CFR [10, 11, 12, 13]. The goal of this survey article is to provide an overview of these advances. Specifically, we describe the proposed methods, used datasets, as well as the results and associated insights. In this context, we consider the electromagnetic spectrum ranging from near infrared (NIR) to more challenging bands such as midwave infrared (MWIR) and longwave infrared (LWIR), as illustrated in Figure 2. Unlike other introductory overview articles including [14, 6, 15], we focus on highlighting recent advances based on *deep learning* in addressing CFR, and techniques that have been developed to bridge the modality gap. Table 1 revisits existing survey papers and highlights novel contributions we present in this survey.

In particular, we showcase the two main approaches in CFR: (i) cross-spectrum feature-based algorithms, which extract and compare features, and (ii) cross-spectrum image synthesis algorithms, which generate images from one spectrum to another for matching and recognition. The former involves comparing infrared probes against visible face image galleries within a common feature subspace. However, following the success of GAN-based image synthesis, the latter attempts to synthesize a “pseudo-visible” images from infrared images. These synthesized face images allow both, academic and commercial off-the-shelf (COTS) FR systems trained on the visible spectrum to be used for comparison.

The remainder of this paper is organized as follows. We formalize CFR in Section 2 and present some applications in Section 3. We discuss the various spectral bands, i.e., sensing modalities, that are relevant to face recognition, in Section 4. In Section 5 we describe the general modules of a CFR system, viz., face image pre-processing and feature extraction. The various deep learning based CFR methods that have been proposed in the literature are explained in Section 6.1 and 6.2, while the relevant datasets used for research in this field are presented in Section 7. Finally, in Section 8, we discuss open research problems that are currently being addressed in the field of CFR.
<table><caption>Table 1: Summary of other related overview articles and the novelty of our work.</caption><thead><tr><td>Survey reference</td><td>Description</td><td>Novelty proposed by our survey</td></tr></thead><tbody><tr><td>R.S. Ghiass et al. [14] (2014) Infrared Face Recognition: A Comprehensive Review of Methodologies and Databases</td><td>Reviews methodologies and databases for infrared face recognition, discussing the robustness of thermal imaging to facial expressions, pose variations, and the use of blood vessel patterns.</td><td><b>Incorporation of Recent Deep Learning Advances:</b> We introduce the latest deep learning techniques, specifically tailored for CFR, surpassing traditional methods reviewed [14]. <b>Focus on Multimodal Integration:</b> We explore the integration of multiple spectral bands, aimed at enhancing recognition accuracy and robustness, a more comprehensive approach than prior reviews.</td></tr><tr><td>S. Ouyang et al. [16] (2016) A Survey on Heterogeneous Face Recognition: Sketch, Infra-red, 3D and Low-resolution</td><td>Provides a detailed description of datasets and benchmarks relevant to evaluation and offers an extensive review of established techniques.</td><td><b>Focus on CFR:</b> Our survey is focused specifically on CFR including NIR, SWIR, MWIR, LWIR and Polar-LWIR, compared to NIR [16]. <b>Deep Learning-based Approaches:</b> We revisit recently proposed deep learning methods for CFR, as opposed to traditional pattern recognition methods [16].</td></tr><tr><td>S. Hu et al. [6] (2017) Heterogeneous Face Recognition: Recent Advances in Infrared-to-Visible Matching</td><td>Focuses on methods for matching infrared images to visible spectrum images, highlighting recent advances and challenges in heterogeneous face recognition.</td><td><b>Bridging the Modality Gap:</b> We provide an in-depth analysis of state-of-the-art techniques for bridging the modality gap using deep learning. <b>Comprehensive Dataset Review:</b> We extensively revisit newly available datasets, enhancing the evaluation framework for CFR methods.</td></tr><tr><td>R. Munir and R.A. Khan [15] (2019) An Extensive Review on Spectral Imaging in Biometric Systems: Challenges and Advancements</td><td>Discusses challenges and advances in spectral imaging for biometric systems, including various spectral bands and their applications.</td><td><b>Detailed Exploration of Cross-Spectral Techniques:</b> We delve into CFR methods with a focus on practical applications. <b>Future Directions and Challenges:</b> We outline future research directions and unresolved challenges, providing a road map for ongoing and future work in the field. <b>Comparison of Spectral Bands:</b> We here compare the effectiveness of different spectral bands, providing insights into optimal spectrum utilization for face recognition.</td></tr></tbody></table>
## 2. Formalization of CFR

Inspired by previous work [17, 18], we firstly formalize the FR-task and then the CFR-task as follows.

Let $\mathcal{M}$ be an electromagnetic spectral space modality associated with a marginal distribution $\mathbb{P}$ over a $d$-dimensional feature space $\mathcal{X} \subset \mathbb{R}^d$ and a label space $\mathcal{Y} \subset \mathbb{N}$.

Given a $n$-face database $X = \{\mathbf{x}_i\}_{i=1}^n$, where $\mathbf{x}_i \in \mathcal{X}$, and their corresponding $n$-identities $Y = \{y_j\}_{j=1}^n$ ($y_j \in \mathcal{Y}$), a FR-model is defined as a parametric function $\mathcal{F}_{FR,\Theta}$ described by the random variables $X$ (feature space), $Y$ (label space) and deep learning model parameters, $\Theta$, which extract features from a CNN.

$$ \mathcal{F}_{FR,\Theta}: \begin{array}{rcl} X & \times & Y \\ (\mathbf{x}_i, y_j) & \mapsto & \mathbb{P}(Y = y_j | X = \mathbf{x}_i, \Theta), \end{array} \qquad (1) $$

where, $i, j \in [1, n]$.

Thus, the FR-model aims to deduce parameters, $\Theta$, that increase the probability of correct recognition to 1, assuming that the correct face is in the gallery. More specifically, for all $k$-index identities $\in [1, n]$

$$ \mathcal{F}_{FR,\Theta}(\mathbf{x}_k, y_k) = \mathbb{P}(Y = y_k | X = \mathbf{x}_k, \Theta) = 1. \qquad (2) $$

However, in the case of CFR, we compare acquired face images in two different modalities (or spectra or domains): the source $\mathcal{M}^s$ and the target $\mathcal{M}^t$. Each of them is associated with a marginal distribution $\mathbb{P}$ over the $d$-dimensional feature space for both source $X^s = \{\mathbf{x}_i^s\}_{i=1}^n \subset \mathcal{X}^s$ and target $X^t = \{\mathbf{x}_i^t\}_{i=1}^n \subset \mathcal{X}^t$, respectively. Note that they share the same set of labels $Y = \{y_j\}_{j=1}^n \subset \mathcal{Y}$.

The CFR-function, $\mathcal{F}_{CFR,\Theta}$, is formalized as follows. Towards finding the deep learning model parameters, $\Theta$, for all $k$-index identities $\in [1, n]$, we have

$$ \mathbb{P}(Y = y_k | X^s = \mathbf{x}_k^s, \Theta) = \mathbb{P}(Y = y_k | X^t = \mathbf{x}_k^t, \Theta) = t, \qquad (3) $$

where, $t \in [0, 1]$ is a predefined threshold.

Algorithms reviewed in this paper seek to provide strategies for determining $\Theta$, formalized in Equation (3).

During enrollment, a *gallery face image* is captured and processed and being stored as *reference template*, representing the biometric information of an individual. At the time of authentication, a *probe face image* is captured and processed in the same way as in the enrollment and compared against a reference template of a claimed identity (verification) or against all stored reference templates (identification) [19].

Mathematically, both heterogeneous verification and identification are formalized as follows. The user is either *genuine* – the claim is true, or an *impostor* – the claim is false. They are represented by the set $\Omega = \{\omega_{true}, \omega_{false}\}$, respectively.

In CFR-verification, an input feature vector $\mathbf{x}_q^t \in \mathcal{X}^t$ stemming from the target modality $\mathcal{M}^t$ and a claimed identity $y_j \in \mathcal{Y}$ are embedded as the pair $(\mathbf{x}_q^t, y_j)$. The comparison algorithm (Section 5.2) analyzes the extracted templates $\mathbf{x}_q^t$, and a function $\mathcal{S}$ (Section 5.3) computes a *similarity score* (11) according to the stored source template $\mathbf{x}_j^s \in X^s$. If the score exceeds a predefined threshold $t$, it is considered a match. Thus,

$$ (\mathbf{x}_q^t, y_j) \in \begin{cases} \omega_{true}, & \text{if } \mathcal{S}(\mathbf{x}_j^s, \mathbf{x}_q^t) \ge t \\ \omega_{false}, & \text{otherwise.} \end{cases} \qquad (4) $$

In CFR-identification, an input feature vector $\mathbf{x}_q^t \in \mathcal{X}^t$ derived from the target modality $\mathcal{M}^t$ is provided to the biometric system. The system attempts to search in the gallery for the appropriate identity $y_k$ correlated with $\mathbf{x}_k^s \in X^s$, where $k \in [1, n]$. The associated template is able to provide the highest similarity score (11). In addition, the highest score exceeding the predefined threshold $t$ leads to a match. Otherwise the user is classified as unknown $y_{unknown}$.

$$ \mathbf{x}_q^t \in \begin{cases} y_k, & \text{if } \max_{k \in [1, n]} \{\mathcal{S}(\mathbf{x}_k^s, \mathbf{x}_q^t) \ge t\} \\ y_{unknown}, & \text{otherwise.} \end{cases} \qquad (5) $$
<figure><img src="image_5.png" /><figcaption>Figure 3: A monitoring system equipped with a thermal sensor applies face and facial landmark detection to a video sequence captured in the wild at night, successfully detecting the person despite challenging conditions, paving the way for subsequent identification steps. Images are from Anghelone *et al.* [20].</figcaption></figure>

### 3. Applications of CFR

CFR-systems have been widely deployed and operate mainly in the visible spectrum [21], due to the ubiquity of low-cost cameras. As motivated in the introduction, CFR-systems have been beneficial in additional applications, for example, in *Intelligence Video Surveillance* (IVS). Beyond *access control* where acquisition is controlled w.r.t. user collaboration, the application of CFR has strong potential to provide new capabilities in *law enforcement, military* and the *AI community*, especially for commercial purposes. In this context, IR sensors offer several advantages over traditional visible light imaging. They enable long-range monitoring and capture biometric modalities at variable standoff distances. IR sensors can also capture images through certain types of obscuring materials and perform well in low-light or nighttime scenarios. However, identifying individuals in real-world situations often involves uncooperative behavior. The unconstrained environment necessitates capturing images of subjects at a distance from the camera, accounting for various facial poses, expressions, and even occlusions.

*Law Enforcement and Homeland Security.* A pertinent application for CFR has been in law enforcement and homeland security [22]. In this context, IR-sensors can be used for long-range monitoring and capturing biometric modalities at variable standoff distances, as well as in low-light and night-time scenarios. These settings are depicted in Figure 3. In such cases, the acquired image often has to be compared against a gallery consisting of visible spectrum face images for identification purposes. This capability enhances the effectiveness of monitoring suspects and ensuring security in public spaces.

*Military Applications.* Similar applications are found in military settings involving identifying individuals in challenging environments [7]. Night-vision devices, which often operate in the infrared spectrum, can capture images in complete darkness. These images can be cross-referenced with visible spectrum databases, facilitating the identification and tracking of persons of interest in combat or surveillance operations.

*Commercial Applications.* In the commercial sector, CFR can improve security measures in places such as shopping malls and airports. AI-powered security systems using CFR can manage access control more effectively, recognizing individuals across different lighting conditions and enhancing overall safety.

*Presentation Attack Detection.* CFR-systems are also beneficial for deflecting presentation attacks [23, 24], which involve attempts to deceive biometric systems using photographs, masks, or other fraudulent methods. By leveraging images from multiple spectral bands, these systems can better discern between genuine and fake biometric data, enhancing security measures against such attacks.
<figure><img src="image_6.png" /><figcaption>Figure 4: **Electromagnetic spectrum.** Modalities and their associated wavelengths, highlighting the visible and infrared radiations.</figcaption></figure>

***Sketch-Visual Identification.*** On a wider scale, HFR (which is a superset of CFR) offers notable application in sketch-visual identification. Law enforcement agencies can use HFR to match a suspect’s sketch with a visible spectrum face image from a mugshot database. This method is particularly useful when photographic evidence is unavailable, and only a hand-drawn sketch of the suspect exists [25].

**4. Different Spectra**

We next proceed to introduce the infrared (IR) spectral bands, which we illustrate in Figure 4, that have been employed in CFR.

Wolff *et al.* [26], Buddharaju *et al.* [27], Kong *et al.* [28], Bhowmik *et al.* [29], and Bourlai and Hornak [7] describe *infrared light* as an invisible, heat-associated energy that can be sensed when radiation or warmth are reflected or emitted from an object. Unlike ultraviolet rays [30], IR-waves penetrate the skin without damage to health. We note that, in principle, IR sensors capture either the *face-reflection* of infrared light or the heat face-emission stemming from the subcutaneous superficial blood vessels. In particular, face contains blood capillaries and forms a subcutaneous network. More physiology-details including the uniqueness of individual’s physiology can be found in the work of Buddharaju *et al.* [27].

IR bands have been defined according to the standard ISO-20473:2007 as near-infrared (NIR) between 0.78–3 µm, mid-infrared (MIR) between 3–50 µm and far-infrared (FIR) 50–1000 µm. Deviating from that, the CIE-International Commission on Illumination has divided the IR-spectrum into the bands IR-A (0.7 µm – 1.4 µm), IR-B (1.4 µm – 3 µm) and IR-C (3 µm – 1000 µm).

In this paper, we use a division of the IR bands based on physical properties, which determine **reflection-dominated** and **emission-dominated** regions. These regions, also denoted as *Passive* and *Active* IR, respectively, are further divided into sub-spectra, which we summarize in Figure 4. We note that a similar scheme has been employed by manufacturers of IR sensors, where specific sensors have been developed, associated to active or passive IR.

**Experiment 1.** In order to demonstrate the modality gap, we conduct the following experiment inspired by Hu *et al.* [6]. We select heterogeneous images pertaining to one subject from the Tufts dataset [31]. Hence, for the HFR scenarios *Visible/Visible*, *Visible/NIR*, *Visible/LWIR* and *Visible/Sketch*, we compute both (i) the associated structural similarity (SSIM) scores and (ii) the cosine similarity measures from facial features extracted with CNNs.

Towards evaluating the modality gap in CFR related to (i), the SSIM metric (6), as proposed by Wang *et al.* [32], has been widely used and offers similarity at the pixel level. In our context, SSIM indicates the similarity between a visible spectrum reference image *x* and a gallery image *y* (in our case infrared) by assessing the perturbation of structural information caused by factors such as luminance and contrast. Specifically,
$$SSIM(x, y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}, \qquad (6)$$

where, $\mu$ and $\sigma$ denote mean and standard deviation, respectively, and $\sigma_{xy}$ represents the covariance of the respective images. Here, $c_1$ and $c_2$ are constants. SSIM (6) ranges between -1 and 1, with 1 being the extreme case, when compared images x and y are identical.

The structural similarity maps and associated SSIM-scores computed in Experiment 1 are shown in Figure 6, highlighting face regions that are most sensitive to infrared energies. While the wavelength in the infrared band increases, the SSIM-difference increases as well. We observe different emission patterns around nose, mouth and eyes (blue panel). Furthermore, the results suggest that the largest challenges (lowest SSIM-scores) have to do with comparing an image in the visible spectrum with a sketch as well as with a thermal image. In both cases, we obtain similar SSIM-scores.

Furthermore, we proceed to showcase the spectral modality gap related to CFR in an additional experiment (ii) by utilizing the cosine similarity measure (formalized below in Equation 11). Specifically, facial biometric features are extracted from associated facial images illustrated in Figure 6 with a state-of-the-art face matcher, ArcFace [33]. ArcFace is used to extract deep features from a visible face, as well as from its counterpart, which are then compared using the cosine similarity measure. The cosine similarity measure is computed based on facial features extracted from distinct spectral modalities, offering valuable insights into the biometric-level similarity of face representations. Table 3 reports scores pertaining to the different scenarios shown in Figure 6. We note that the scores are normalized between 0 to 1, where 1 indicates identical images. As expected, we observe that similarity decreases significantly in cases when facial images sensed in different spectra or modalities (sketch-visual) are compared, indicating that comparison is more challenging in such cases.

<table><caption>Table 3: Computation of cosine similarity S scores using state-of-the-art face matcher ArcFace from a visible spectrum image and its counterpart.</caption><thead><tr><th></th><th>Vis-Vis</th><th>Vis-NIR</th><th>Vis-LWIR</th><th>Vis-Sketch</th></tr></thead><tbody><tr><th>Score of similarity</th><td>0.9545</td><td>0.6652</td><td>0.1475</td><td>0.1024</td></tr></tbody></table>

## 4.1. Active IR

The active IR region comprises of *near infrared* (NIR) and *shortwave infrared* (SWIR), characterized by reflective material properties, which require an (invisible) IR light to reveal the scene.

NIR is placed right next to the red-channel of the visible spectrum, with increased wavelength, which is exhibited in the visual similarities between VIS and NIR images (Figure 1). We observe this in Experiment 1 when comparing VIS-NIR faces, where the SSIM score is 0.479 (Figure 6). NIR is instrumental in CFR, as it remains invariant to lighting-direction and low-light conditions and, hence, is employed for monitoring and night-surveillance [34].

<figure><img src="image_7.png" /><figcaption>Figure 5: **Face beyond the visible.** A comparison of a face in visible (VIS) and infrared bands, viz., NIR, SWIR, MWIR and LWIR. We note the different physiological properties in both **Active IR** (NIR, SWIR) and **Passive IR** (MWIR, LWIR) bands. Figure credit: Hu al. [6]</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 6: **Structure Similarity (SSIM) maps and scores.** A visualization strategy to understand the human and machine visual perception when comparing two images in different scenarios: *Visible* - against *Visible/NIR/LWIR/Sketch*.</figcaption></figure>

**SWIR**-imagery, similar to NIR, is visually close to visible imagery. See Figure 5. The SWIR-band is significantly wider than the NIR-band and enables sensing in atmospherically challenging conditions such as *rain, fog, mist, haze*, and common urban particulates such as *smoke* and *pollution* [29]. In addition, SWIR-sensors are able to capture objects in extremely low-light conditions, which renders SWIR suitable for surveillance, allowing for long-range applications (< one kilometer [35]), as well as for identification purposes [36].

## 4.2. *Passive IR*

Beyond a wavelength of $\lambda = 3\mu m$, the IR band is significantly emissive and aims to acquire heat-sensitive radiation emitted from a human face. Hence, thermal sensors can be designed to capture temperature variations across the facial skin tissue due to the underlying face vasculature [27]. One of the benefits of the emission-dominated regions are that thermal imagery can be passively acquired without any external illumination, in either day or night environments [26]. Passive IR is comprised of Midwave (MWIR) and Longwave (LWIR). Note however that the range between MWIR and LWIR (ranging between $\lambda = 5\mu m – 8\mu m$, see Figure 4) constitutes a strong atmospheric absorption window rendering image acquisition impossible.

**MWIR** is located between SWIR and LWIR, and has both reflective and emissive properties - allowing for the sensing of different facial-skin-features including vein patterns [37]. However, it is the emissive component that is predominantly exploited in CFR, which is why face images captured in MWIR tend to largely resemble heat face signatures, and are visually distinct from visible imagery (see Figure 5).

**LWIR** extends the infrared band up to $\lambda = 14\mu m$ and consists of exclusively emitted radiation. This shift introduces high within-class variations and notably impedes a match with visible images (Figure 5). We observe that
<figure><img src="image_9.png" /><figcaption>Figure 7: Stokes parameters. Example images of a subject captured in the visible spectrum and their corresponding conventional thermal $S_0$, polarization state information $S_1$ and $S_2$, and polarimetric thermal images. Related SSIM-scores are reported in Table 4.</figcaption></figure>

Vis-LWIR is the most challenging scenario as per Figure 6, with the lowest SSIM score of 0.251. LWIR imagery is visually similar to that of MWIR with respect to shape and contrast (Figure 5).

### 4.3. Polarimetric Thermal Imagery

Polarimetric thermal imagery aims to measure the polarization state information of the light contained in the thermal infrared spectrum, as well as in any other spectrum. By performing optical filtering of the light at precise rotational angles, say 0°, 90°, 45° and -45° related to the horizontal axes, more geometrical and textural details are obtained, which is able to enhance the conventional intensity-based thermal images for face recognition purposes. Towards inferring polarimetric-information in the passive IR and capturing the thermal radiance, specific infrared polarimetric sensors have been used. Through light manipulation, the polarization state information is typically expressed in terms of Stokes parameters $\vec{S}$, as firstly introduced by G. Stokes in 1852. For more details, the reader is referred to the work of Pezzaniti and Chenault [38]. $\vec{S}$ is derived from the following linear combination.

$$ \vec{S} = \begin{bmatrix} S_0 \\ S_1 \\ S_2 \\ S_3 \end{bmatrix} = \begin{bmatrix} I_0^\circ + I_{90}^\circ \\ I_0^\circ - I_{90}^\circ \\ I_{45}^\circ + I_{-45}^\circ \\ I_R^\circ + I_L^\circ \end{bmatrix}, \qquad (7) $$

where, $I_0^\circ$ and $I_{90}^\circ$ denote the horizontal and vertical polarized light, respectively, while $I_{45}^\circ$ and $I_{-45}^\circ$ represent the diagonal polarized light and $I_R^\circ$ and $I_L^\circ$, respectively, represent the intensity of the right and left circularly polarized light. The Stokes parameter, $S_0$, represents the conventional thermal images without polarization information, $S_1$ and $S_2$ convey orthogonal polarimetric information and display additional details. Finally, $S_3$ is considered to be 0 in most applications, due to lack of circular information in the thermal spectrum. The degree of linear polarization (DoLP) can then be computed as

$$ DoLP = \frac{\sqrt{S_1^2 + S_2^2}}{S_0}, \qquad (8) $$

and it denotes the spectral radiation window of the electromagnetic spectrum which is linearly polarized. The polarimetric image entails the channel combinations of $S_0$, $S_1$, and $S_2$ (see Figure 7).

**Experiment 2.** We compute SSIM scores between Visible and Thermal images (including each polarization state and degree of linear polarization), which we report in Table 4. The scores confirm the improvement due to the use of polarimetric thermal imagery. We note that SSIM pertaining to polarimetric imagery, outperforms the one due to thermal imagery. Therefore, polarimetric thermal imagery can facilitate CFR, which will be later confirmed in Section 5.3.
<table><caption>Table 4: **Experiment 2.** Computation of cosine similarity scores using state-of-the-art face matchers and structural similarity SSIM for images depicted in Figure 7. The scores are normalized to [0,1].</caption><thead><tr><th rowspan="2">Face Matchers</th><th rowspan="2">Vis/Vis</th><th colspan="3">Stokes parameters</th><th colspan="2">Channel combinations</th></tr><tr><th>S<sub>0</sub></th><th>S<sub>1</sub></th><th>S<sub>2</sub></th><th>DoLP</th><th>Polar</th></tr></thead><tbody><tr><td>SphereFace [39]</td><td>1.0</td><td>0.499</td><td>0.473</td><td>0.486</td><td>0.461</td><td>0.698</td></tr><tr><td>AM-Softmax [10]</td><td>1.0</td><td>0.543</td><td>0.576</td><td>0.555</td><td>0.596</td><td>0.702</td></tr><tr><td>MobileFaceNet [10]</td><td>1.0</td><td>0.505</td><td>0.499</td><td>0.503</td><td>0.507</td><td>0.702</td></tr><tr><td>ArcFace [33]</td><td>1.0</td><td>0.641</td><td>0.658</td><td>0.587</td><td>0.67</td><td>0.684</td></tr><tr><td>SSIM</td><td>1.0</td><td>0.286</td><td>0.236</td><td>0.262</td><td>0.209</td><td>0.300</td></tr></tbody></table>

# 5. Cross-spectral Face Recognition

As in classical face recognition, *face detection* and *alignment* are considered the first and foremost steps of CFR. CFR data is preprocessed by aligning and cropping faces to reference templates based on detected facial landmarks. Then, a standard pipeline proceeds in learning/extracting face features by using optimal network architectures and loss functions.

## 5.1. Face Preprocessing

Very limited work has focused on facial landmark detection beyond the visible spectrum [40, 41, 42, 20]. Since NIR and SWIR face images closely resemble visible spectrum faces, landmark detection methods developed for visible face images are directly applicable.

**Experiment 3.** Towards demonstrating the possibility of adapting landmark detection methods developed for the visible spectrum, we utilize MTCNN [43] to perform face detection and landmark detection experiments on NIR faces. Next, an affine transformation is computed based on the detected landmarks to normalize the detected faces to a frontal pose (see Figure 8).

<figure><img src="image_10.png" /><figcaption>Figure 8: **Experiment 3.** Face and landmark detection experiments conducted on near-infrared faces sampled from [44]. Face normalization was performed by aligning the faces to the reference pose based on the detected landmarks.</figcaption></figure>
As demonstrated, while it might be relatively easy to perform landmark detection on NIR or SWIR face images, it remains a challenge in the context of thermal face images, including MWIR and LWIR. We note that thermal face images tend to have lower contrast, lower resolution and lack texture information [40]. Poster et al. [40] analyzed the performances of three modern deep learning based landmark detection methods on thermal face images, namely *Deep Alignment Network* (DAN), *Multi-Task CNN* (MTCNN), and *multi-class patch based fully CNN* (PBC). These landmark detection algorithms were evaluated on the *ARL polarimetric thermal face* dataset [45] with 5-landmarks. Kopaczka et al. [41] assembled a high-resolution thermal facial image dataset in the LWIR spectral band and manually annotated 68 landmark points. DAN was used as the representative deep learning landmark detector to train and evaluate on the assembled dataset. Chu and Liu [42] proposed to use U-Net as a backbone to detect 68 facial landmarks on the same dataset used in [41]. The landmark detection task was designed as a multi-task learning framework by integrating emotion recognition. Poster et al. [46] used a coupled convolutional network architecture to leverage visible face data to train a thermal-only face landmark detection model. Examples of 68-point ground-truth landmark annotations are depicted in Figure 9.

<figure><img src="image_11.png" /><figcaption>Figure 9: **Landmark detection in thermal imagery.** Example images from a thermal dataset [41] showing 68-point ground-truth landmark annotations.</figcaption></figure>

## 5.2. Deep Feature Extraction

After face images are preprocessed, two different schemes can be leveraged to compute domain-invariant features. Firstly, faces from different spectral bands can be directly ingested by a CNN, aiming to learn a shared feature representation scheme. Secondly, faces can be transformed by GANs into a target domain and features can be extracted from the transformed images.

### 5.2.1. Convolutional Neural Networks (CNNs)

CNNs are a class of neural networks that are highly efficient in learning effective feature representations for FR [39, 10, 33]. For CFR, CNNs operate on specific pairs of data, e.g., NIR-VIS, formed by different spectral bands. Depending on the number of pairs, CNNs can be designed with a two-branch [47, 48, 49] or a three-branch architecture [50]. Weights from individual branches are often shared. An example of such a design is depicted in Figure 10. Visible and NIR images represent respectively input to the network, extracting features. To learn domain-invariant features that are independent of the given spectral bands, a contrastive loss function is applied to enforce the identity constraint for these two extracted feature vectors [47]. For triplet inputs formed by anchor, positive and negative samples, a triplet loss is generally applied to minimize the gap between different spectral bands [50]. There are several prominent CNN-architectures that can be used as backbone for the architecture in Figure 10, some of which we enlist in Table 5.
<figure><img src="image_12.png" /><figcaption>Figure 10: CNNs. Flowchart depicting a two-branch convolutional neural network, learning domain-invariant feature representation.</figcaption></figure>

<table><caption>Table 5: Representative architectures used in CFR for both CNNs and GANs.</caption><thead><tr><th>Year</th><th>Backbone Architecture</th><th>Source</th><th>Used in</th></tr></thead><tbody><tr><td rowspan="2">2014</td><td>VGGNet</td><td>[51]</td><td>[52, 53, 17, 54]</td></tr><tr><td>GoogLeNet</td><td>[55]</td><td>[47]</td></tr><tr><td rowspan="2">2015</td><td>ResNet</td><td>[56]</td><td>[57, 58, 59]</td></tr><tr><td>U-Net</td><td>[60]</td><td>[61], [62], [63], [64]</td></tr><tr><td rowspan="2">2016</td><td>Inception-ResNet</td><td>[65]</td><td>[66, 17, 67]</td></tr><tr><td>DenseNet</td><td>[68]</td><td>[69, 11]</td></tr><tr><td rowspan="2">2017</td><td>Squeeze-and-Excitation Network</td><td>[70]</td><td>[71, 72]</td></tr><tr><td>LightCNN</td><td>[73]</td><td>[50, 48, 49, 74, 12, 17, 59]</td></tr><tr><td>2018</td><td>EfficientNet</td><td>[75]</td><td>-</td></tr><tr><td>2019</td><td>Vision Transformer</td><td>[76]</td><td>[77, 78]</td></tr><tr><td>2020</td><td>SwinTransformer</td><td>[77]</td><td>-</td></tr><tr><td>2021</td><td>SwinTransformer</td><td>[77]</td><td>-</td></tr></tbody></table>

### 5.2.2. Generative Adversarial Networks (GANs)

GANs are deep learning frameworks introduced by Goodfellow et al. [79] (2014), which consist of two sub-networks, a *Generative* model *G* and a *Discriminative* model *D* trained simultaneously. Together, *G* and *D* compete with each other in a minimax game. Specifically, *G* aims to learn the intrinsic distribution $p_G$ over some target data $x \sim p_{\text{data}}$, and associated with a noise prior $z \sim p_z$. *G* draws sample $z$ for creating synthetic data $G(z)$. In the meantime, *D* aids the training by taking $x$ or $G(z)$ as input and performing a binary classification as to whether the input is from the real data distribution $p_{\text{data}}$ or from the generated data distribution $p_G$. *G* attempts to mislead *D* to not be able to distinguish generated images from real, while *D* attempts to make that distinction.

The optimization competition of these two models results in the following minimax game

$$ \min_G \max_D \mathbb{E}_{x \sim p_{\text{data}}(x)}[\log(D(x))] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]. \quad (9) $$

Traditional GANs learn a mapping from a random noise input $z$ to an output image $y$, while conditional GANs
learn the mapping from an observed image x and a random noise z in which both G and D are conditioned to an output image y according to

$$ \min_G \max_D \mathbb{E}_{x \sim \text{data}(x)} [\log(D(x|y))] + \mathbb{E}_{z \sim p_z} [\log(1 - D(G(z|y)))]. \quad (10) $$

Many existing techniques have been developed based on conditional GANs to address the CFR problem (see Figure 11). The generator takes a thermal image as input, seeking to produce a synthesized visible image as output. The discriminator is trained to distinguish between two pairs of images: the real pair, consisting of an input thermal image and a target visible image, and the synthesized pair, consisting of an input thermal image and a synthesized visible image [62, 64, 10, 13]. As noted in these works, there were notable differences in the use of architectures and loss functions. Some variants of architectures such as multi-scale generator [13] and multi-scale discriminator [11] were used in order to account for scale variance. There also exist variants of loss functions, including attribute loss [13], identity loss [10] and shape loss [80].

<figure><img src="image_13.png" /><figcaption>Figure 11: GANs. Flowchart depicting a conditional generative adversarial network for synthesizing visible faces from conditional infrared (thermal) faces.</figcaption></figure>

### 5.2.3. Loss Functions for Cross-spectral Face Recognition

CNNs (Section 5.2.1) and GANs (Section 5.2.2) constitute fundamental deep learning networks. Once such network architectures are designed, suitable loss functions are chosen to minimize prediction errors. Therefore, one active direction in CFR has to do with the development of efficient loss functions [85]. Loss functions are pertinent in deep architectures, as they aim to measure how well algorithms are able to perform correct predictions. In the CFR context, the loss function is specifically designed to mitigate cross-spectral impact, in order to minimize intra-class variation and maximize inter-class variation [47, 50, 57].

Loss functions used in CFR are frequently adapted from traditional FR. We note that the predominantly used loss function in traditional FR remains to be softmax, which is targeted to maximize the probability that the same subject belongs to the target class. Therefore, it encourages the separability of features. Center loss [82] seeks to minimize distances between deep features and their corresponding class centers by simultaneously learning a center for individual classes. However, we note that center loss does not encourage separability of features. Therefore, it is common practice to perform a joint supervision of the softmax loss with center loss. Some other approaches combine euclidean margin-based losses with softmax loss to perform joint supervision. Recently, it has been shown that the features learned by softmax loss have an intrinsic angular distribution [39]. Since then, a flurry of angular-based loss functions have been proposed to impose discriminative constraints on a hypersphere manifold [39, 84, 33]. We summarize loss functions in Table 6.

In most traditional FR systems, samples belonging to the same subject are treated without any differentiation. However, in CFR, samples are divided into different categories based on their spectrum information. This prompts the use of loss functions that can accept a pair of inputs [47], triplet of inputs [50], as well as quadruplet of inputs [57]. These loss functions have achieved impressive performance. Admittedly, directly using loss functions from traditional
<table><caption>Table 6: **Loss Functions.** A comprehensive list of loss functions $\mathcal{L}$ from traditional FR to CFR. Here, $N$ is the number of samples, $T$ is the number of samples in a mini-batch, $W$ is the weight matrix, $b$ is the bias term, $x_i$ and $y_i$ are the $i^{th}$ training sample and the according class label, respectively. $\theta_{y_i} \in [\frac{k\pi}{m}, \frac{(k+1)\pi}{m}]$ with $k \in [0, m-1]$ and $m \ge 1$.</caption><thead><tr><th>Task</th><th>Name</th><th>Notation</th><th>Reference</th><th>Function <i>L</i></th></tr></thead><tbody><tr><td rowspan="6">FR</td><td>Softmax (Cross-Entropy)</td><td><i>L</i><sub>s</sub></td><td>[81]</td><td>-<sup>1</sup>/<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> log (<sup>e<sup>W<sub>yi</sub><sup>T</sup>x<sub>i</sub>+b<sub>yi</sub></sup></sup>/<sub>&sum;<sub>j=1</sub><sup>n</sup> e<sup>W<sub>j</sub><sup>T</sup>x<sub>i</sub>+b<sub>j</sub></sup></sub>)</td></tr><tr><td>Center Loss</td><td><i>L</i><sub>ce</sub></td><td>[82]</td><td><sup>1</sup>/<sub>2</sub> &sum;<sub>i=1</sub><sup>N</sup> ||<i>x</i><sub>i</sub> - <i>c</i><sub>yi</sub>||<sup>2</sup><sub>2</sub></td></tr><tr><td>Marginal Loss</td><td><i>L</i><sub>ma</sub></td><td>[83]</td><td><sup>1</sup>/<sub>m<sup>2</sup> - m</sub> &sum;<sub>i,j,i&ne;j</sub><sup>m</sup> (<i>&xi;</i> - <i>y</i><sub>ij</sub> (<i>&theta;</i> - ||<i>x</i><sub>i</sub>||/||<i>x</i><sub>j</sub>|| - ||<i>x</i><sub>j</sub>||/||<i>x</i><sub>i</sub>||<sup>2</sup>)</td></tr><tr><td>Angular Softmax Loss (SphereFace)</td><td><i>L</i><sub>as</sub></td><td>[39]</td><td>-<sup>1</sup>/<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> <sup>e<sup>||<i>x</i><sub>i</sub>||cos(<i>mt</i><sub>yi</sub>)</sup></sup>/<sub>e<sup>||<i>x</i><sub>j</sub>||cos(<i>mt</i><sub>yi</sub>)</sup> + &sum;<sub>j=1, j&ne;y<sub>i</sub></sub><sup>n</sup> e<sup>||<i>x</i><sub>i</sub>||cos(<i>&theta;</i><sub>j</sub>)</sup></sub></td></tr><tr><td>Large Margin Cosine Loss (CosFace)</td><td><i>L</i><sub>co</sub></td><td>[84]</td><td>-<sup>1</sup>/<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> log (<sup>e<sup>s(cos(<i>&theta;</i><sub>i</sub>) - <i>m</i>)</sup></sup>/<sub>e<sup>s(cos(<i>&theta;</i><sub>i</sub>) - <i>m</i>)</sup> + &sum;<sub>j=1, j&ne;y<sub>i</sub></sub><sup>n</sup> e<sup>s cos(<i>&theta;</i><sub>j</sub>)</sup></sub>)</td></tr><tr><td>Additive Angular Margin Loss (ArcFace)</td><td><i>L</i><sub>aa</sub></td><td>[33]</td><td>-<sup>1</sup>/<sub>N</sub> &sum;<sub>i=1</sub><sup>N</sup> log (<sup>e<sup>s cos(<i>&theta;</i><sub>i</sub>) + <i>m</i></sup></sup>/<sub>e<sup>s cos(<i>&theta;</i><sub>i</sub>) + <i>m</i></sup> + &sum;<sub>j=1, j&ne;y<sub>i</sub></sub><sup>n</sup> e<sup>s cos(<i>&theta;</i><sub>j</sub>)</sup></sub>)</td></tr><tr><td rowspan="3">CFR</td><td>Contrastive Loss</td><td><i>L</i><sub>ct</sub></td><td>[47] - (12)</td><td>{<sup>1</sup>/<sub>2</sub> ||<i>f</i>(<i>x</i><sub>i</sub>) - <i>f</i>(<i>x</i><sub>j</sub>)||<sup>2</sup>, if <i>y</i><sub>i</sub> = <i>y</i><sub>j</sub><br>{<sup>1</sup>/<sub>2</sub> max(0, <i>m</i> - ||<i>f</i>(<i>x</i><sub>i</sub>) - <i>f</i>(<i>x</i><sub>j</sub>)||)<sup>2</sup>, else.</td></tr><tr><td>Triplet Loss</td><td><i>L</i><sub>tp</sub></td><td>[50] - (13)</td><td>&sum;<sub>i=1</sub><sup>N</sup> (<sup>||<i>f</i>(<i>x</i><sub>i</sub><sup>a</sup>) - <i>f</i>(<i>x</i><sub>i</sub><sup>p</sup>)|| - ||<i>f</i>(<i>x</i><sub>i</sub><sup>a</sup>) - <i>f</i>(<i>x</i><sub>i</sub><sup>p</sup>)|| + <i>m</i></sup></td></tr><tr><td>Tetrad Margin Loss</td><td><i>L</i><sub>TML</sub></td><td>[57] - (15)</td><td>{<sup>T</sup>/<sub>&sum;<sub>i&isin;(N,V)</sub> <i>s</i><sub>i</sub><i>t</i><sub>j</sub></sub> (<sup><i>z</i><sub>j</sub><sup>N</sup> &middot; <i>z</i><sub>j</sub><sup>V</sup> - <i>z</i><sub>j</sub><sup>N</sup> &middot; <i>z</i><sub>t</sub><sup>V</sup></sup>/<sub>||<i>z</i><sub>j</sub><sup>N</sup>|| &middot; ||<i>z</i><sub>j</sub><sup>V</sup>||</sub> + <i>m</i><sub>1</sub>)<br>+ <sup><i>z</i><sub>j</sub><sup>N</sup> &middot; <i>z</i><sub>j</sub><sup>V</sup> - <i>z</i><sub>j</sub><sup>V</sup> &middot; <i>z</i><sub>t</sub><sup>V</sup></sup>/<sub>||<i>z</i><sub>j</sub><sup>N</sup>|| &middot; ||<i>z</i><sub>j</sub><sup>V</sup>||</sub> + <i>m</i><sub>2</sub></td></tr></tbody></table>

FR for CFR is based on the premise that facial appearance from other spectral bands is close to the visible spectral band. This is applicable in the case of comparing NIR against VIS faces. With increase in wavelength, difference in the facial appearance also starts to increase. Thus, adversarial loss functions used in GAN can be used to perform the synthesis.

### 5.3. Face Comparison

Trained CNNs or GANs for CFR produce deep feature representations for a given probe-image, which are then compared with those of the gallery images. Using the same notation introduced in Section 2, let $X^s = \{\mathbf{x}_i^s\}_{i=1}^n \subset \mathcal{X}^s$ and $X^t = \{\mathbf{x}_i^t\}_{i=1}^n \subset \mathcal{X}^t$ denote the set of samples from the *Source* (visible) and the *Target* (infrared) modalities, respectively. The corresponding shared set of labels are denoted by $Y = \{y_j\}_{j=1}^n \subset \mathcal{Y}$.

Suppose $\Theta$ denotes the deep process of extracting $d$ features from a CNN, then the similarity score between two templates can be computed as a function $S$ referred to as the *measure of similarity* ranging between 0 and 1, where 1 represents a high similarity. As an example, the *cosine distance* is widely used to calculate such similarity, for all $k$-index identities $\in [1, n]$:

$$ 
\begin{aligned}
S(\mathbf{x}_k^s, \mathbf{x}_k^t) &= \cos(\mathbf{s}, \mathbf{t}) = \frac{\langle \mathbf{s}, \mathbf{t} \rangle}{\|\mathbf{s}\|\|\mathbf{t}\|} \\
&= \frac{\sum_{j=1}^{d} s_j t_j}{\sqrt{\sum_{j=1}^{d} (s_j)^2} \sqrt{\sum_{j=1}^{d} (t_j)^2}},
\end{aligned} 
\quad (11) $$

where, $\mathbf{s} = \Theta(\mathbf{x}_k^s)$ and $\mathbf{t} = \Theta(\mathbf{x}_k^t)$. Table 4 reports cosine similarity scores of state-of-the-art face matchers, when comparing thermal images depicted in Figure 7. Other Euclidean distance based measures such as $L_2$ and $L_1$ can also serve as similarity metrics.
<figure><img src="image_14.png" /><figcaption>Figure 12: Timeline of algorithm development. NIR-to-VIS FR with their corresponding loss functions.</figcaption></figure>

# 6. Algorithms of Different Spectra

FR operating on imagery beyond the visible spectrum can be categorized as *cross-spectrum featured-based algorithms* or *cross-spectrum images synthesis algorithms* (Figure 2). The former involve comparison of infrared probe against a gallery of visible face images, within a common feature subspace. However, after the success of image synthesis based on GANs, cross-spectrum images synthesis algorithms have attempted to synthesize a “pseudo-visible” image from an infrared image. Given the synthesized face image, both academic and commercial-off-the-shelf (COTS) FR-systems trained on visible spectrum, can be utilized for comparison. Furthermore, intensive research has been done to exploit the polarimetry thermal images to further improve CFR.

## 6.1. Reflective IR-to-Visible FR

### 6.1.1. Near Infrared

The appearance difference between NIR and VIS face images is less pronounced compared to other spectral bands and, hence, shared feature representations have often been proposed [50, 47, 48, 49, 74]. CNNs are suitable for NIR-to-VIS CFR scenario, since they seek to automatically extract representative face features. Nevertheless, a challenge remains, which has to do with the illumination-variation between the two modalities. We proceed to discuss a number of recent work, constituting prominent state-of-the-art approaches in matching NIR against VIS face images. These algorithms along with their individual loss functions are summarized in Figure 12. Further, we also list in Table 7 the NIR-to-VIS face comparison performances on public benchmark datasets.

**TRIVET.** Liu *et al.* [50] proposed a CNN with ordinal measures (o-CNN), pre-trained on visible images from the large-scale CASIA WebFace dataset [93] and hence is able to extract general face features, which has been fine-tuned on pairs of NIR-VIS face images, in order to learn a domain-invariant deep representation. To cope with limited image pairs, two types of NIR-VIS triplet loss functions were used, reducing intra-class variations by iteratively setting NIR and VIS images as anchors, such that the network focuses on the identity distinction instead of the spectrum classification.

**IDNet.** Reale *et al.* [47] utilized GoogLeNet with small convolutional filters, pre-trained on the visible CASIA WebFace dataset. For the NIR-to-VIS CFR scenario, they initialized two identical networks based on the pre-trained model, excluding the fully connected and softmax layers. The outputs from these two networks, termed VisNet and NIRNet, were concatenated to form a Siamese network with a contrastive loss function. This coupled deep network design effectively mapped NIR and VIS faces into a spectrum-independent feature space.

**IDR.** He *et al.* [48] presented a CNN-based approach targeted to map both NIR and VIS images into a common subspace by separating the feature space into a shared layer: NIR- and VIS-layer. While the shared layer encoded the modality-invariant identity information, the NIR- and VIS-layers encoded the modality-variant spectrum information, respectively. The NIR and VIS images were respectively input to the two CNN channels with shared parameters. Compared to similar CNN pipelines [50, 47], the proposed method jointly learned identity and spectrum information, leading to substantial performance gains on the CASIA NIR-VIS 2.0 dataset.

**Hallucination.** Lezama *et al.* [52] proposed an approach, which adapts a pre-trained VIS-CNN model towards generating discriminative features for both VIS and NIR face images, without retraining the network. Their approach consisted of two core components, cross-spectral hallucination and low-rank embedding, aiming at modifying both inputs and outputs of the CNN for CFR. Firstly, cross-spectral hallucination was used to transform the NIR image
<table><caption>Table 7: Rank-1 Accuracy and Verification Rate on different datasets for the NIR-VIS face comparison.</caption><thead><tr><th rowspan="2">Year</th><th rowspan="2">Authors</th><th rowspan="2">Method</th><th rowspan="2">Loss</th><th rowspan="2">Dataset</th><th colspan="3">Performance</th></tr><tr><th>Rank-1</th><th>FAR 1%</th><th>FAR 0.1%</th></tr></thead><tbody><tr><td rowspan="4">2016</td><td rowspan="2">Liu et al. [50]</td><td rowspan="2">TRIVET</td><td rowspan="2">Triplet</td><td>CASIA NIR-VIS 2.0 [44]</td><td>95.74</td><td>98.1</td><td>91.03</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>92.2</td><td>67.9</td><td>33.6</td></tr><tr><td rowspan="2">Reale et al. [47]</td><td rowspan="2">IDNet</td><td rowspan="2">Contrastive</td><td>BUAA-VisNir face [87]</td><td>93.9</td><td>93.0</td><td>80.9</td></tr><tr><td>CASIA NIR-VIS 2.0 [44]</td><td>87.1</td><td>96.9</td><td>74.5</td></tr><tr><td rowspan="4">2017</td><td rowspan="2">He et al. [48]</td><td rowspan="2">IDR</td><td rowspan="2">Softmax</td><td>CASIA NIR-VIS 2.0 [44]</td><td>97.3</td><td>98.9</td><td>95.7</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>94.3</td><td>73.4</td><td>46.2</td></tr><tr><td rowspan="2">Lezama et al. [52]</td><td rowspan="2">Hallucination</td><td rowspan="2">Euclidean</td><td>BUAA-VisNir face [87]</td><td>94.3</td><td>93.4</td><td>84.7</td></tr><tr><td>CASIA NIR-VIS 2.0 [44]</td><td>96.41</td><td>-</td><td>-</td></tr><tr><td rowspan="4">2018</td><td rowspan="2">He et al. [49]</td><td rowspan="2">WCNN</td><td rowspan="2">Softmax</td><td>CASIA NIR-VIS 2.0 [44]</td><td>98.7</td><td>99.5</td><td>98.4</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>98.0</td><td>81.5</td><td>54.6</td></tr><tr><td rowspan="2">Wu et al. [74]</td><td rowspan="2">CDL</td><td rowspan="2">Ranking</td><td>BUAA-VisNir face [87]</td><td>97.4</td><td>96.0</td><td>91.9</td></tr><tr><td>CASIA NIR-VIS 2.0 [44]</td><td>98.6</td><td>-</td><td>98.32</td></tr><tr><td rowspan="8">2019</td><td rowspan="2">He et al. [12]</td><td rowspan="2">CFC (GAN)</td><td rowspan="2">UV</td><td>CASIA NIR-VIS 2.0 [44]</td><td>98.6</td><td>99.2</td><td>97.3</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>99.9</td><td>98.1</td><td>90.7</td></tr><tr><td rowspan="2">Cao et al. [57]</td><td rowspan="2">MMDL</td><td rowspan="2">Multi-margin</td><td>BUAA-VisNir face [87]</td><td>99.7</td><td>98.7</td><td>97.8</td></tr><tr><td>CASIA NIR-VIS 2.0 [44]</td><td>99.9</td><td>-</td><td>99.4</td></tr><tr><td rowspan="2">Deng et al. [58]</td><td rowspan="2">MC-CNN</td><td rowspan="2">MCA</td><td>Oulu-CASIA NIR-VIS [86]</td><td>100.0</td><td>-</td><td>97.2</td></tr><tr><td>CASIA NIR-VIS 2.0 [44]</td><td>99.22</td><td>-</td><td>99.27</td></tr><tr><td rowspan="2">Hu et al. [66]</td><td rowspan="2">MDNDC</td><td rowspan="2">Scatter</td><td>CASIA NIR-VIS 2.0 [44]</td><td>98.9</td><td>99.6</td><td>97.6</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>99.8</td><td>88.1</td><td>65.3</td></tr><tr><td rowspan="10">2020</td><td rowspan="2">Cho et al. [59]</td><td rowspan="2">RGM</td><td rowspan="2">Conditional-margin</td><td>CASIA NIR-VIS 2.0 [44]</td><td>99.3</td><td>99.51</td><td>99.02</td></tr><tr><td>BUAA-VisNir face [87]</td><td>99.67</td><td>99.22</td><td>-</td></tr><tr><td rowspan="2">Hu et al. [89]</td><td rowspan="2">DSVN</td><td rowspan="2">Adversarial</td><td>CASIA NIR-VIS 2.0 [44]</td><td>99.0</td><td>99.7</td><td>98.6</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>100.0</td><td>99.3</td><td>95.5</td></tr><tr><td rowspan="2">Iranmanesh et al. [69]</td><td rowspan="2">CpGAN</td><td rowspan="2">Contrastive+Coupled +GAN+Eucl.+Perceptual</td><td>CASIA NIR-VIS 2.0 [44]</td><td>96.63</td><td>-</td><td>87.05</td></tr><tr><td>CASIA HFB [88]</td><td>99.64</td><td>98.4</td><td>89.7</td></tr><tr><td rowspan="2">Hu et al. [67]</td><td rowspan="2">ADCANs</td><td rowspan="2">ASL</td><td>CASIA NIR-VIS 2.0 [44]</td><td>99.1</td><td>99.6</td><td>98.5</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>99.8</td><td>93.2</td><td>78.9</td></tr><tr><td rowspan="2">Hu et al. [90]</td><td rowspan="2">DADRD</td><td rowspan="2">CmM</td><td>BUAA-VisNir face [87]</td><td>99.8</td><td>99.7</td><td>98.4</td></tr><tr><td>CASIA NIR-VIS 2.0 [44]</td><td>99.10</td><td>99.6</td><td>98.6</td></tr><tr><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td>Oulu-CASIA NIR-VIS [86]</td><td>100.0</td><td>98.5</td><td>92.9</td></tr><tr><td>BUAA-VisNir face [87]</td><td>99.9</td><td>99.9</td><td>99.8</td></tr><tr><td rowspan="3">2021</td><td rowspan="3">He et al. [91]</td><td rowspan="3">CAL</td><td rowspan="3">Adversarial + Orthogonal constrain</td><td>CASIA NIR-VIS 2.0 [44]</td><td>99.6</td><td>99.7</td><td>99.4</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>100.0</td><td>98.9</td><td>93.9</td></tr><tr><td>BUAA-VisNir face [87]</td><td>99.9</td><td>99.8</td><td>99.4</td></tr><tr><td rowspan="3">2022</td><td rowspan="3">Luo et al. [78]</td><td rowspan="3">MMTN</td><td rowspan="3">Adversarial + Identity + Content + Style + Cycle</td><td>CASIA NIR-VIS 2.0 [44]</td><td>99.4</td><td>99.8</td><td>99.7</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>100.0</td><td>98.2</td><td>89.0</td></tr><tr><td>BUAA-VisNir face [87]</td><td>99.2</td><td>99.5</td><td>97.2</td></tr><tr><td rowspan="3">2023</td><td rowspan="3">Yang et al. [92]</td><td rowspan="3">RPC</td><td rowspan="3">Domain Adaptation</td><td>CASIA NIR-VIS 2.0 [44]</td><td>99.7</td><td>99.9</td><td>99.6</td></tr><tr><td>Oulu-CASIA NIR-VIS [86]</td><td>100.0</td><td>98.5</td><td>94.4</td></tr><tr><td>BUAA-VisNir face [87]</td><td>100.0</td><td>99.9</td><td>99.8</td></tr></tbody></table>

into the VIS spectrum by learning correspondences between NIR and VIS patches. Next, a low-rank embedding was used to restore a low-rank structure that simultaneously minimized the intra-class variations, while maximizing the inter-class variations. Three pre-trained VIS-CNN models, including VGG-S, VGG-face and COTS, were evaluated on the CASIA NIR-VIS 2.0 dataset.

**WCNN.** He et al. [49] proposed a novel method entitled Wasserstein Convolutional Neural Network (WCNN) streamlined to learn invariant features between NIR and VIS face images. Similar to their previous work [48], the proposed WCNN divided the high-level layer of the shared CNN into two orthogonal subspaces that contain modality-invariant identity features and modality-variant spectrum features. In addition, Wasserstein distance was used to measure the difference between heterogeneous feature distributions to reduce the modality gap. We note that their work represented the first attempt to formulate a probability-based distribution learning for VIS-to-NIR face comparison. WCNN outperformed the work of He et al. [48], improving the performance on the CASIA NIR-VIS 2.0 dataset.

**CDL.** Wu et al. [74] designed a Coupled Deep Learning (CDL) approach, identifying a shared feature space, in order to reduce modality differences in heterogeneous face comparison. The loss function of CDL consisted of two different components: (a) relevance constraints that imposed a trace norm to the softmax loss and a block-diagonal prior to the fully connected layer; and (b) cross-modal ranking that employed triplet ranking regularization to enlarge the size of training data. A semi-hard triplet selection method was also adopted to further improve the NIR-to-VIS comparison performance. Both relevance and ranking losses were jointly optimized.

**CFC.** He et al. [12] proposed Cross-spectral Face Completion (CFC), a GAN-based method, aimed at synthesizing visible spectrum (VIS) images from near-infrared (NIR) images. This heterogeneous face synthesis process comprises two complementary components: (a) texture inpainting, recovering VIS facial textures from NIR images, and (b) pose correction, mapping any pose in NIR images to a frontal pose in VIS images. These components are fused using a warping procedure within an adversarial network, regularized by UV loss, adversarial loss, and L1 reconstruction loss. The main contributions of this work include creating a novel GAN-based end-to-end frame-
work for cross-spectral face synthesis without the need to assemble multiple image patches. The framework includes encoder-decoder structured generators and two novel discriminators to fully account for variations between NIR and VIS images. This approach simplifies the unsupervised heterogeneous face synthesis problem to a one-to-one image translation problem. The decomposition of texture inpainting and pose correction enables the generation of realistic, identity-preserving VIS face images.

**MMDL.** Cao *et al.* [57] combined heterogeneous representation network and decorrelation representation learning in order to design a *Multi-Margin based Decorrelation Learning* (MMDL) framework to extract decorrelated features for both VIS and NIR images. First, the heterogeneous representation network was initialized from a VIS face model and fine-tuned on both VIS and NIR face images. Second, a decorrelation layer was appended to shared feature layer of the representation network to extract decorrelated features for both VIS and NIR images.

**MC-CNN.** Deng *et al.* [58] proposed a *Mutual Component Convolutional Neural Network* (MC-CNN) to extract modality-invariant features. The novelty of MC-CNN was the incorporation of a generative module, i.e., the mutual component analysis (MCA), into the CNN by replacing the fully connected layer with MCA. VIS-NIR image pairs from the same subjects were first input to a shared CNN to extract common feature vectors. Subsequently, these feature vectors were fed into the MCA layer. Finally, MCA loss was included as an additional regularization to the softmax loss.

**MDNDC.** Hu *et al.* [66] proposed *Multiple Deep Networks with scatter loss and Diversity Combination* (MD-NDC). They used three ResNet-v1 networks; one being dedicated to feature extraction, in order to build the architecture of Multiple Deep Networks (MDN), followed by the two other networks in parallel. Jointly with the MDN, a Scatter loss (SL) [94] was used as a loss function, which attempts to maximize distance between the classes and minimize distance within the class to learn highly discriminative features for the CFR task. In addition to the MDN, a joint decision strategy named Diversity Combination (DN) was introduced to auto-adjust weights of all three deep networks of the MDN and make a joint classification decision.

**RGM.** Cho *et al.* [59] were interested in Relational Deep Feature Learning for CFR. Therefore, to bridge the domain gap between visible and invisible spectrum, they proposed a *Relational Graph-structured Module* (RGM) focused on facial relational information. Their graph RGM performed relational modeling from node vectors that represent facial components such as lips, nose and chin. They also performed recalibration by considering global node correlation via *Node Attention Unit* (NAU) to focus on the more informative nodes arising from the relation-based propagation. Furthermore, they suggested a novel conditional-margin loss function *C-Softmax* for efficient projection learning on the common latent space of the embedding vector that adaptively uses the inter-class margin.

**DSVN.** Hu *et al.* [89] proposed *Disentangled Spectrum Variations Networks* (DSVN) to disentangle spectrum variations between VIS and NIR domains and to separate the modality-invariant identity information from modality-variant spectrum information. DSVN is comprised of two major components: spectrum-adversarial discriminative feature learning (SaDFL) and step-wise spectrum orthogonal decomposition (SSOD). The SaDFL was further divided into identity-discriminative subnetwork (IDNet) and auxiliary spectrum adversarial subnetwork (ASANet). The IDNet was used to extract identity discriminative features. The ASANet, on the other hand, was used to eliminate modality-variant spectrum information. Both IDNet and ASANet were designed to extract domain-invariant feature representations via adversarial learning.

**ADCANs.** Hu *et al.* [67] proposed an effective *Adversarial Disentanglement spectrum variations and Cross-modality Attention Networks* (ADCANs) for the VIS-to-NIR CFR scenario. To reduce the gap of cross-modal images and solve the NIR-VIS CFR problem, the authors set-up three key components which are able to learn identity-related and modality-unrelated features. Firstly, they proposed a new objective loss termed *Advanced Scatter Loss* (ASL), aiming at capturing within- and between-class information of the data and embedding them in the network for more effective training, focusing on categories with small inter-class distance and increasing the distance between them. Then, a *Modality-adversarial Feature Learning* (MaFL) including an Identity-Discriminative Feature Learning Network (IDFLN) and a Modality-Adversarial Disentanglement Network (MADN) were incorporated to improve the feature representation of the identity-discriminative component as well as to highlight the spectrum variations through adversarial learning. Finally, a *Cross-modality Attention Block* (CmAB) was introduced, in order to guide the network in selecting pertinent features and suppress noise information. CmAB sequentially applies spatial and channel attention modules to both, IDFLN and MADN, in order to increase the representation ability between them.

**DADRD.** Hu *et al.* [90] proposed the *Dual Adversarial Disentanglement and deep Representation Decorrelation* (DADRD) approach to reduce the gap between NIR-VIS modalities, while enhancing the learning of identity-related
<figure><img src="image_15.png" /><figcaption>Figure 13: **Overview of the DADRD [90] model:** learn identity-discriminative features and disentangle within-class variations. DADRD combines three key components, including *CmM* loss, *DADV* consisting of both *ADMV* and *ADRV* and *DRD* embedded into the three previous *MFR* layers.</figcaption></figure>

features. At the same time, DADRD sought to effectively disentangles the additional residual-related features (i.e. PIE) rather than only extracting modality-related (i.e. NIR and VIS) and identity-related features. In contrast with their prior work, MDNDC [66] and ADCAN [67], the DADRD method combined three key components. Firstly, they proposed a new objective loss termed *Cross-modal Margin* (CmM), and attempted to enhance the learning identity-related features, which captures within- and between-class information of the data (i.e., occlusion, pose, distance, lighting and expressions), while also reducing the modality gap by using a center-variation item. Then, they introduced a *Mixed Facial Representation* (MFR) which is divided into three layers: “(I) Identity-related layer”, “(M) Modality-related layer” and “(R) Residual-related layer”. A *Dual Adversarial Disentanglement Variation* (DADV) was then designed to reduce the intra-class variation with the help of adversarial learning, including both *Adversarial Disentangled Modality Variations* (ADMV) and *Adversarial Disentangled Residual Variations* (ADRV). These adversarial mechanisms disentangle the spectrum variation (in terms of data heterogeneity) and eliminate residual variations such as PIE, respectively. Finally, a *Deep Representation Decorrelation* (DRD) was embedded to the three MFR layers of DADV for the purpose of making them unrelated to each other and enhance feature representations. The DADRD method, combining CmM, DADV and DRD into a unified framework, is depicted in Figure 13.

**DVG-Face.** Fu *et al.* [95] proposed a novel Dual Variational Generation (DVG-Face) framework to learn the joint distribution of paired heterogeneous images. Identity information from large-scale visible face data was also utilized to tackle the problem of identity diversity sampling induced by the small-scale paired heterogeneous data. To ensure identity consistency, a pairwise identity preserving loss was applied to generated paired heterogeneous images. A contrastive learning scheme was employed to train the heterogeneous face matching with positive pairs from generated paired heterogeneous images and negative pairs from different identity samplings. The proposed DVG-Face was an extension from the authors’ previous work DVG [96] that greatly improved the identity diversity of generated face images.

**CAL.** He *et al.* [91] introduced Coupled Adversarial Learning (CAL), a synthesis framework which offers a semi-supervised approach for learning features across spectra, specifically when dealing with a limited number of unpaired training data. Particularly, at the image level, a deep transformation network has been trained to convert NIR images to the VIS domain, addressing appearance discrepancies using carefully designed cycle loss, intensity loss, and texture loss. At the feature level, CAL aimed at creating a deep feature space, where the cross-spectral face matching problem was approximated as a homogeneous face matching problem. Adversarial loss, along with orthogonal constraints, enhanced the ability of cross-spectral feature representation.

**MMTN.** Luo *et al.* [78] presented MMTN, an innovative Memory-Modulated Transformer Network. The ap-
proach followed the recognition via generation strategy, framing the problem as an unsupervised generation task with a "one-to-many" reference-based approach. To delve into prototypical style patterns from the reference domain, authors incorporated a memory module. In addition, a novel style transformer module was designed to facilitate the local fusion of input image content with the reference image's style.

**RPC.** Yang *et al.* [92] introduced RPC, a domain adaptation model which integrated three key components into a unified framework. This integration aimed to uncover intrinsic relationships in cross-domain data and to facilitate the learning of domain-invariant representations. The first component, namely NPS, was employed to accurately estimate pseudo labels based on NIR clusters. These pseudo labels guided the network in learning representations that were invariant across domains. Subsequently, the DCL component was introduced to enhance the discriminate qualities of individual features within each specific domain. Finally, the ICL component was implemented to guide the network in learning robust and domain-independent representations.

## Summary of NIR-to-VIS Face Comparison

Comparing face images in NIR against those in VIS can be coarsely classified into (a) cross-spectrum featured-based methods [50, 47, 48, 49, 74, 57, 58, 59, 89] and (b) cross-spectrum image-synthesis methods [52, 12]. Existing methods (a) predominantly seek to learn a domain-invariant deep representation between NIR and VIS samples, which is motivated by the presence of some similar textures observed in NIR as well as VIS-domains. Such approaches can incorporate two different ways to construct the inputs of the networks. Firstly, NIR and VIS images pertaining to same subjects are related in positive pairs [47, 48, 49, 57, 58, 89], constituting two identical subnetworks with shared weights. A contrastive loss in this context minimizes the intra-class distance

$$ \mathcal{L}_{ct} = \begin{cases} \frac{1}{2} \|f(x_i) - f(x_j)\|^2, & \text{if } y_i = y_j \\ \frac{1}{2} \max(0, m - \|f(x_i) - f(x_j)\|^2), & \text{else.} \end{cases} \qquad (12) $$

Secondly, a triplet can be formed, where a NIR image is set as an anchor, a VIS image of the same subject as a positive sample, and an additional VIS image pertaining to a different subject as a negative sample. Alternatively, a triplet can be formed by setting a VIS image as an anchor and using NIR images from the same and different subjects as positive and negative samples, respectively [50, 74]. This leads to three identical subnetworks with shared weights, where the triplet loss is defined as

$$ \mathcal{L}_{tp} = \sum_{i}^{N} \left( \|f(x_i^a) - f(x_i^p)\| - \|f(x_i^a) - f(x_i^n)\| + m \right). \qquad (13) $$

Here, $x_i^a$ is an anchor image, $x_i^p$ is a positive sample and $x_i^n$ is a negative sample. The goal of triplet loss is to 'push' the negative sample $x_i^n$ away from the anchor $x_i^a$ by a margin $m$ compared to the positive sample:

$$ \|x_i^a - x_i^p\|^2 + m \le \|x_i^a - x_i^n\|. \qquad (14) $$

The triplet loss attempts to minimize the intra-class distance, while maximizing the inter-class distance. Unlike triplet, where a group of three samples is selected, the *Tetrad Margin Loss* (TML) [57] uses a group of four samples to construct tetrad tuples. The designed tetrad sample selection strategy was to choose four heterogeneous decorrelation representations, viz., {$z_j^N, z_j^V, z_k^N, z_l^V$}. Herein, {$z_j^N, z_j^V$} are samples from the same identity, $z_k^N$ denotes the closest NIR sample to $z_j^V$ from another identity, $z_l^V$ represents the closest VIS sample to $z_j^V$ from another identity. The tetrad margin loss is defined as:

$$ \mathcal{L}_{TML}(z_j^N, z_j^V, z_k^N, z_l^V) = \sum_{i \in \{N,V\}} \left( \frac{z_j^N \cdot z_j^V}{\|z_j^N\| \cdot \|z_j^V\|} - \frac{z_j^N \cdot z_k^V}{\|z_j^N\| \cdot \|z_k^V\|} + m_1 \right) + \sum_{i \in \{N,V\}} \left( \frac{z_j^N \cdot z_j^V}{\|z_j^N\| \cdot \|z_j^V\|} - \frac{z_j^V \cdot z_k^V}{\|z_j^V\| \cdot \|z_k^V\|} + m_2 \right). \qquad (15) $$

Subsequently, $\mathcal{L}_{TML}(z_j^N, z_j^V, z_k^N, z_l^V)$ can be defined as

$$ \mathcal{L}_{TML}(z_j^N, z_j^V, z_k^N, z_l^V) = \mathcal{L}_{TML}(z_j^N, z_j^V, z_l^V) + \mathcal{L}_{TML}(z_j^V, z_j^N, z_k^N). \qquad (16) $$
T represents the number of samples in a mini-batch, while m₁ and m₂ are the two margins. The tetrad margin loss can be regarded as the combination of two triplet losses [50, 74]. Intuitively, L<sub>TML</sub>(z<sub>j</sub><sup>N</sup>, z<sub>j</sub><sup>V</sup>, z<sub>i</sub><sup>V</sup>) is very similar to assigning a NIR image z<sub>j</sub><sup>N</sup> as an anchor, a VIS image z<sub>j</sub><sup>V</sup> of the same subject as a positive sample, and another VIS image z<sub>i</sub><sup>V</sup> of a different subject as a negative sample. The difference lies in the use of cosine distance other than the euclidean distance to compare the similarity. Both triplet loss and tetrad margin loss use the hard-sample mining strategy.

In addition to the contrastive loss, triplet loss and tetrad margin loss, other loss functions have notably been proposed in the NIR-to-VIS literature. In [48], a variant of softmax loss was proposed in order to learn modality invariant subspace. The softmax loss function is defined as:

$$
\begin{align}
\mathcal{L} &= \sum_{i \in \{N, V\}} \mathrm{softmax}(F_i, c, \Theta, W, P_i) \nonumber \\
&= - \sum_{i \in \{N, V\}} \left( \sum_{j=1}^{N} \log(\hat{p}_{ij}) \mathbb{1}_{\{y_{ij}=c\}} \right), \tag{17} \\
\text{s.t.} \quad P_i^T W &= 0, \quad i \in \{N, V\}, \nonumber
\end{align}
$$

where, F_i is the fully connected layer of WCNN, c is the class label and Θ is the set of WCNN parameters. W is used to denote the weight matrix of the modality-invariant features (i.e., shared features across spectral domain) and P_i denotes the weight matrix of the spectrum-specific features (i.e., features which are spectral dependent). 1 is an indicator function and p_{ij} denotes the predicted class probability. P_i^T W corresponds to an orthogonal constraint imposed to make the features uncorrelated.

6.1.2. Shortwave Infrared

To date, SWIR-to-visible has received limited attention. This is reflected in the scarcity of deep models and
in the lack of publicly available SWIR face datasets (see Table 11). We proceed to summarize existing prominent
handcrafted methods [7, 97].

**pre-CNN.** Research of Bourlai *et al.* [98] firstly investigated face verification related to the SWIR band. The authors introduced a *geometric* and *photometric normalization* (PN) scheme in order to compensate for the variable environment, coupled with *contrast limited adaptive histogram equalization* (CLAHE). Based on that, three FR methods (commercial and academic) were tested using the *West Virginia University* (WVU) Multi-spectral database.

Kalka *et al.* [99] extended the previous work w.r.t. pre-processing [98] by adding *single scale retinex* (SSR) and also explored the cross-photometric score level fusion rule. For the experiment, three datasets were considered including different environmental setup (fully controlled/semi controlled/uncontrolled). Their finding provided insights such as that pre-processing with PN improved FR performance, obtaining highest results up to Rank-1 accuracy of 100%, when images were acquired under fully controlled conditions.

Bourlai *et al.* [97] pursued the intra-spectral and cross-spectral FR specially focused on (NIR/SWIR/MWIR)-
imaging at various standoff distances and different environmental conditions. They merged previous work [98, 99] in
order to demonstrate that jointly, the use of independent or combined PN, CLAHE, SSR and cross-photometric score
level fusion rule were able to reach better results under different environmental setups.

**CMLD.** Cao *et al.* [100] proposed a local operator called *Composite Multilobe Descriptors* (CMLD) with the aim to extract facial feature through all IR-spectral bands. This operator combined Gaussian function, Local binary patterns (LPB), Weber local descriptor and histogram of oriented gradients (HOG). The experimental results showed that for the SWIR-to-visible scenario, CMLD performed on a private dataset containing SWIR images (at various standoff distances), a Rank-1 accuracy of 78.7% and a verification rate of 99.5% at FAR 10%. Authors also used CMLD to conduct a study on cross-spectral *partial* FR in order to understand the face area which contained useful information for CFR. They found that the eye region is most informative for FR beyond the visible.

**VGGFace.** Bihn *et al.* [53] explored the use of a pre-trained VGG-Face network on visible images to extract features from shortwave infrared (SWIR) images. They hypothesized that the VGG-Face network would perform well on both VIS and SWIR face images since their facial appearance differences are less significant compared to MWIR and LWIR images. SWIR wavelengths at 935 nm, 1060 nm, 1300 nm, and 1550 nm were evaluated, along with a composite image combining 1060 nm, 1300 nm, and 1550 nm. Deep features were extracted from the fc7 layer
<table><caption>Table 8: Rank-1 Accuracy and Verification Rate on different dataset for the MWIR-VIS face comparison.</caption><thead><tr><th>Year</th><th>Authors</th><th>Methods</th><th>Dataset</th><th>Performance %</th></tr></thead><tbody><tr><td>2017</td><td>Sarfraz et al. [103]</td><td>DPM</td><td>NVESD [104]</td><td>98.66 @Rank-1</td></tr><tr><td>2018</td><td>Zhang et al. [62]</td><td>TV-GAN</td><td>IRIS [105]</td><td>19.90 @Rank-1</td></tr><tr><td>2019</td><td>Chen et al. [10]</td><td>SG-GAN</td><td>PCSO [106]</td><td>92.16 @AUC - 15.01 @EER</td></tr><tr><td rowspan="2">2020</td><td rowspan="2">Iranmanesh et al. [69]</td><td rowspan="2">CpGAN</td><td>NVESD [104]</td><td>96.10 @Rank-1</td></tr><tr><td>WSRI [107]</td><td>97.80 @Rank-1</td></tr><tr><td rowspan="3">2021</td><td rowspan="3">Peri et al. [108]</td><td>Pix2Pix-ATC</td><td>MILAB-VTF(B) [108]</td><td>59.3 @AUC - 43.4 @EER</td></tr><tr><td>CycleGAN-ATC</td><td>MILAB-VTF(B) [108]</td><td>54.9 @AUC - 46.4 @EER</td></tr><tr><td>CUT-ATC</td><td>MILAB-VTF(B) [108]</td><td>68.8 @AUC - 36.3 @EER</td></tr></tbody></table>

of the VGG-Face network, producing a 4096-dimensional feature vector. The Euclidean distance between feature representations of VIS and SWIR images was used for similarity comparison.

**GMLM-CNN.** Cao et al. [101] proposed GMLM-CNN, a hybrid method leveraging traditional feature engineering with deep learning techniques for CFR. Repectively, two operators named Nominal Measurement Descriptor and Interval Measurement Descriptor (IMD) extract features at both, nominal and interval measurement levels. These features are then fused into a comprehensive representation using a Gabor Multiple-Level Measurement (GMLM) operator. To further enhance feature selection and recognition performance, GMLM-CNN employed a PCA-based neural network that effectively filtered out redundant information and identified the most informative features.

**SSL.** Nanduri and Chellappa [102] used semi-supervised learning (SSL) and explored the impact of different training datasets with limited data on the generalization capability of the trained network across various test domains, in particular SWIR vs. Visible. Experiments demonstrated that larger models consistently outperformed smaller architectures, even with very small training sets, without compromising generalization performance. When dealing with multiple target domains and data scarcity, prioritizing training data collection for the most challenging domain (farthest from the source domain) showed to be more effective. Wider training datasets (i.e., more classes with fewer samples per class) exhibited better performance compared to deep datasets (i.e., fewer classes with more samples per class). In the presence of noisy unlabeled data (containing samples from classes not represented in the labeled data), back-propagating entropy loss solely on the feature extractor (excluding the classifier) significantly enhanced performance. Designing fine-tuning datasets that solely focus on increasing labeled data without incorporating novel variations (i.e., pose, illumination) was proven a not reliable strategy for consistently improving performance. Instead, authors stated that efforts should be directed towards incorporating data with diverse variations to effectively generalize the network's capabilities.

## 6.2. Emissive IR-to-Visible FR

### 6.2.1. Midwave Infrared

Appearance difference between MWIR and VIS is more pronounced than NIR and SWIR (Figure 5). Therefore, learning a common feature subspace for MWIR-to-VIS is a challenging problem. In most cases, methods developed over the years have relied on GANs to synthesize visible face images from thermal face images. Illustrative examples are shown in Figure 14 and 16. We report the related MWIR-to-VIS face comparison performance in Table 8.

**DPM.** One of the first successful MWIR-to-VIS approaches with deep neural network was introduced by Sarfraz et al. [103], termed *Deep Perceptual Mapping* (DPM). Their approach was targeted to learn a non-linear mapping between visible and thermal domains (including MWIR and LWIR) while preserving the identity information. Specifically, a feed-forward neural network was constructed to regress densely computed features from the visible image to the corresponding thermal image. An MSE loss function was used to measure the perceptual difference between the visible and thermal images. Further, a regularization term with Frobenius norm of the projection matrix was also introduced.

**TV-GAN.** Zhang et al. [62] proposed a *Thermal-to-Visible Generative Adversarial Network* (TV-GAN) that can synthesize visible images from their corresponding thermal images while maintaining identity information during the
reconstruction. To preserve the identity information, a multi-task discriminator was designed. This discriminator served two different purposes. Firstly, one output of this discriminator was used to differentiate whether the generated samples were "real" or "fake". Secondly, the other output of this discriminator was used to perform identity classification in the context of supervised learning, given that the label identities are provided. Note that this identity classification was regarded as a closed-set FR. That means the identity associated with a given sample had already been seen in the training dataset. The proposed TV-GAN is similar to the auxiliary classifier GAN (or AC-GAN). Experiments were conducted on IRIS dataset [105] and compared against three baselines approaches. TV-GAN against other GAN-methods are visualized in Figure 16.

**SG-GAN.** Chen et al. [10] proposed the use of Semantic-Guided Generative Adversarial Network (SG-GAN) to automatically synthesize visible face images from their thermal counterparts. Specifically, semantic labels, extracted by a face parsing network were used to compute a semantic loss function to regularize the adversarial network during training. These semantic cues denoted high-level facial component information associated with each pixel. Further, an identity extraction network was leveraged to generate multi-scale features to compute an identity loss function. To achieve photo-realistic results, a perceptual loss function was introduced during network training to ensure that the synthesized visible face was perceptually similar to the target visible face image. Experiments involving PCSO [106] face dataset showed that the proposed method achieved promising results in both face synthesis and CFR (see Figure 14).

## Summary of MWIR-to-VIS Face Comparison

MWIR-to-VIS CFR scenario has been scarsely studied in scientific literature. Partly this is the case due to lack of public benchmark datasets. Interestingly, early research on comparing MWIR-to-VIS face images adopted a neural network [103], formulated by sequential operations of fully connected layer and non-activation layer. This simple feed-forward neural network does not involve the use of convolution units and can be represented as

$$H(x) = h^N = g(W^N h^{N-1} + b^N), \qquad (18)$$

where, $W$ is the projection matrix, $h$ is the hidden layer and $b$ is the bias. Basically, the output of current hidden layer is a multiplication of project matrix with the output of previous hidden layer. $g$ is a non-activation function to ensure that the mapping is non-linear. The objective function of DPM was formulated as

$$\arg \min_{W,b} \mathcal{L} = \frac{1}{M} \sum_{i=1}^{M} (\bar{x}_i - t_i)^2 + \frac{\lambda}{N} \sum_{i=1}^{N} (\|W^k\|_F^2 + \|b^k\|_2^2). \qquad (19)$$

Here, the first term is the mean square error between the feature vectors of visible $\bar{x}_i$ and thermal $t_i$ image. The second term serves as a regularizer on the weight matrix $W$ and bias $b$. Although the DPM showed promising results on comparing SWIR against VIS images, the performance was still far from satisfactory.

Recent developments of neural networks, specifically in GANs, have pushed the frontiers of SWIR-to-VIS CFR scenario. Both TV-GAN [62] and SG-GAN [10] are inspired by Pix2Pix [109], which involves the use of adversarial loss and $L_1$ loss to regularize the image synthesis process. The $L_1$ loss was used to measure the per-pixel difference between synthesized visible face image and target visible face image. Compared to Pix2Pix, TV-GAN modified the output of the discriminator to perform a closed-set identification. Therefore, a new identity loss was introduced in TV-GAN. SG-GAN, on the other hand, adds more loss functions to the Pix2Pix, including perceptual loss, identity loss and semantic loss. An ablation study was used to demonstrate the individual functionalities of the loss functions (see Figure 14).

### 6.2.2. Longwave Infrared

Face images captured under MWIR and LWIR tend to look very similar (Figure 5). Therefore, algorithms developed to address LWIR-to-VIS closely resembles that of MWIR-to-VIS. Most developed algorithms for LWIR-to-VIS CFR scenario were based on GANs. Here, we give a brief summary of recent representative works for LWIR-to-VIS comparison, related performance are reported in Table 9 and we illustrate the timeline of developments in Figure 15.
<figure><img src="image_16.png" /><figcaption>Figure 14: Synthesizing VIS face images from THM images on the PCSO dataset. Compared to the use of the “$L_{GAN} + L_{R} + L_{P} + L_{I}$” loss function, the output of SG-GAN is semantically more close to the ground-truth VIS image especially around the salient facial regions.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 15: Timeline of developments in algorithms: LWIR-to-VIS face recognition.</figcaption></figure>

**Landmark-based CycleGAN.** Wang et al. [80] proposed a detection network to extract facial landmarks from visible faces and use that to guide the generative network to preserve geometric shapes. Their network was based on *CycleGAN* [114], targeted to perform translation between the thermal and visible face images. The detection network extracted 68-landmarks from visible faces, constructing the shape loss function. To perform the experiment, they established a new database (available online), including 33 subjects with 792 thermal-visible pair images.

**DCA.** Kantarcı et al. [63] introduced a *Deep Convolutional Autoencoder* (DCA) based on the *U-Net* [60] architecture, incorporating Up Convolution and Difference of Gaussians (DoG) to learn the non-linear mapping between visible and thermal face images for CFR. They demonstrated that preprocessing and alignment improved performance by bridging the gap between these domains. The approach was tested on three publicly available thermal-visible datasets: Carl [111], UND-X1 [110], and EURECOM [116]. Additionally, the authors manually annotated six facial landmarks on the Carl and EURECOM datasets to align the faces, which increased performance by approximately 2%. They conducted experiments with three setups: the first two assessed the decoder method without applying the DoG filter as a preprocessing step, while the third tested the DoG filter’s effect on the Up Convolutional Autoencoder Model proposed in this work.

**CRN.** Mallat et al. [117] proposed a solution based on *Cascaded Refinement Network* (CRN). Their approach did not require a large amount of training data, owing to a limited number of training parameters. CRN is a type of CNN that consists of inter-connected refinement modules, which gradually process the image synthesis from lowest resolution (4×4) to the highest resolution (128×128). The refine module includes three different layers, namely input, intermediate and output layers. A contextual loss function was used to train the CRN. The motivation to choose the contextual loss was based on two conditions: (a) robustness to roughly aligned THM-VIS pairs; and (b) invariant to
<table><caption>Table 9: Rank-1 Accuracy and Verification Rate on different dataset for the LWIR-VIS face Comparison.</caption><thead><tr><th>Year</th><th>Authors</th><th>Methods</th><th>Dataset</th><th>Performance %</th></tr></thead><tbody><tr><td rowspan="5">2017</td><td rowspan="3">Sarfraz et al. [103]</td><td rowspan="3">DPM</td><td>UND X1 [110]</td><td>83.73 @Rank-1</td></tr><tr><td>Carl [111]</td><td>71.00 @Rank-1</td></tr><tr><td>NVESD [104]</td><td>97.33 @Rank-1</td></tr><tr><td rowspan="2">Zhang et al. [112]</td><td rowspan="2">GAN-VFS</td><td>Polarimetric thermal [45]-ext. - Therm. Probe</td><td>84.00 @Rank-1</td></tr><tr><td>Polarimetric thermal [45] - Therm. Probe</td><td>85.61 @Rank-1 - 79.30 @AUC - 27.34 @EER</td></tr><tr><td rowspan="5">2018</td><td rowspan="2">Wang et al. [80]</td><td>CycleGAN [114]</td><td>private database</td><td>88.90 @Rank-1 - 98.80 @Rank-3 - 99.80 @Rank-5</td></tr><tr><td>CycleGAN-Based</td><td>private database</td><td>91.60 @Rank-1 - 99.30 @Rank-3 - 99.90 @Rank-5</td></tr><tr><td>Iranmanesh et al. [115]</td><td>CpDCNN</td><td>Polarimetric thermal [45] - Therm. Probe</td><td>88.57 @Rank-1</td></tr><tr><td rowspan="2">Di et al. [64]</td><td rowspan="2">AP-GAN</td><td>Polarimetric thermal [45] - Therm. Probe</td><td>84.16 @AUC - 23.90 @EER</td></tr><tr><td></td><td></td></tr><tr><td rowspan="10">2019</td><td rowspan="3">Kantarcı et al. [63]</td><td rowspan="3">DCA</td><td>UND X1 [110]</td><td>87.20 @Rank-1</td></tr><tr><td>EURECOM [116]</td><td>88.33 @Rank-1</td></tr><tr><td>Carl [111]</td><td>85.00 @Rank-1</td></tr><tr><td>Mallat et al. [117]</td><td>CRN</td><td>EURECOM [116]</td><td>82.00 @LightCNN</td></tr><tr><td>Iranmanesh et al. [118]</td><td>AGC-GAN</td><td>Polarimetric thermal [45] - Therm. Probe</td><td>89.25 @Rank-1</td></tr><tr><td rowspan="2">Di et al. [119]</td><td rowspan="2">SAGAN</td><td>Polarimetric thermal [45] - Therm. Probe</td><td>91.49 @AUC - 15.45 @EER</td></tr><tr><td>ARL-VTF [113]</td><td>99.28 @AUC - 03.97 @EER</td></tr><tr><td rowspan="2">Chen et al. [10]</td><td rowspan="2">SG-GAN</td><td>Polarimetric thermal [45] - Therm. Probe</td><td>93.08 @AUC - 14.24 @EER</td></tr><tr><td></td><td></td></tr><tr><td rowspan="10">2020</td><td>Chatterjee et al. [61]</td><td>U-Net</td><td>Nagoya University</td><td>69.60 @Rank-1</td></tr><tr><td rowspan="2">Kezebou et al. [54]</td><td rowspan="2">TR-GAN</td><td>TUFTS [31]</td><td>80.70 @Resnet50 - 88.65 @VGG16</td></tr><tr><td>UND X1 [110]</td><td>76.40 @Rank-1</td></tr><tr><td rowspan="2">Iranmanesh et al. [69]</td><td rowspan="2">CpGAN</td><td>NVESD [104]</td><td>93.90 @Rank-1</td></tr><tr><td>Polarimetric thermal [45] - Therm. Probe</td><td>89.05 @Rank-1</td></tr><tr><td rowspan="2">Di et al. [13]</td><td rowspan="2">Multi-AP-GAN</td><td>Polarimetric thermal [45] - Therm. Probe</td><td>90.99 @AUC - 17.81 @EER</td></tr><tr><td>Polarimetric thermal [45]-ext. - Therm. Probe</td><td>94.20 @Rank-1</td></tr><tr><td rowspan="2">Fondje et al. [120]</td><td rowspan="2">DAF</td><td>ARL-VTF [113]</td><td>99.76 @AUC - 2.30 @EER</td></tr><tr><td></td><td></td></tr><tr><td rowspan="10">2021</td><td rowspan="2">Immidisetti et al. [121]</td><td>Axial-GAN</td><td>ARL-VTF [113]</td><td>94.4 @AUC - 12.38 @EER</td></tr><tr><td rowspan="2">Anghelone et al. [122]</td><td rowspan="2">LG-GAN</td><td>Polarimetric thermal [45] - Therm. Probe</td><td>93.99 @AUC - 13.02 @EER</td></tr><tr><td>ARL-VTF [113]</td><td>96.96 @AUC - 5.94 @EER</td></tr><tr><td rowspan="4">Peri et al. [108]</td><td>Pix2Pix-ATC</td><td>ARL-VTF [113]</td><td>91.3 @AUC - 16 @EER</td></tr><tr><td>CycleGAN-ATC</td><td>ARL-VTF [113]</td><td>96.8 @AUC - 9.2 @EER</td></tr><tr><td rowspan="2">CUT-ATC</td><td>ARL-VTF [113]</td><td>97.7 @AUC - 6.9 @EER</td></tr><tr><td>TUFTS [31]</td><td>87.4 @AUC - 21.3 @EER</td></tr><tr><td rowspan="10">2022</td><td rowspan="3">Chen et al. [71]</td><td rowspan="3">AG-GAN</td><td>SF dataset [123]</td><td>89.86 @AUC - 17.68 @EER</td></tr><tr><td>ARL-VTF [113]</td><td>98.74 @AUC - 5.56 @EER</td></tr><tr><td>SF dataset [123]</td><td>90.53 @AUC - 17.20 @EER</td></tr><tr><td rowspan="4">Fondje et al. [124]</td><td rowspan="4">DPIF</td><td>ARL-VTF [113]</td><td>99.26 @AUC - 4.30 @EER</td></tr><tr><td>TUFTS [31]</td><td>91.82 @AUC - 16.03 @EER</td></tr><tr><td>Polarimetric thermal [45]-ext.</td><td>93.06 @AUC - 14.97 @EER</td></tr><tr><td>ARL-VTF [113]</td><td>99.99 @AUC - 0.15 @EER</td></tr><tr><td rowspan="4">2023</td><td rowspan="4">Anghelone et al. [72]</td><td rowspan="4">ANYRES</td><td>EURECOM [116]</td><td>93.65 @AUC - 14.05 @EER</td></tr><tr><td>TUFTS [31]</td><td>83.09 @AUC - 24.33 @EER</td></tr><tr><td>SF dataset [123]</td><td>91.39 @AUC - 15.87 @EER</td></tr><tr><td>ARL-VTF [113]</td><td>99.88 @AUC - 1.26 @EER</td></tr></tbody></table>

outlier at the pixel level in the context of per-pixel loss. FR experiments on the EURECOM [116] dataset showed that the proposed method achieves better performance than TV-GAN. In addition, they performed CFR using their synthesized faces (facial variation = Neutral) with two systems, namely *OpenFace* and *LightCNN*.

**U-Net.** Chatterjee and Chu [61] presented a U-Net architecture with a residual network backbone to generate visible face images from thermal face images. The U-Net was modified by using residual blocks with skip-connections instead of the normal convolutional layers as the basic building components. Further, a pixel shuffle upsampling was introduced to replace the transposed convolution layers in the decoder part. A weighted combinations of two different loss functions was used to train the proposed network, namely mean squared loss and perceptual loss. The proposed method was evaluated on thermal face dataset from *Nagoya University*, which contained 900 thermal images and 900 visible images captured simultaneously. A Rank-1 accuracy of 69.60% was achieved.

**TR-GAN.** Kezebou et al. [54] proposed a *Thermal to RGB Generative Adversarial Network (TR-GAN)* to automatically synthesize visible face images captured in the thermal domain. The TR-GAN employed an architecture similar to U-Net with cascade residual blocks for the generator. Basically, it replaced the Resnet blocks in the Cycle-GAN with cascaded-in- cascaded residual blocks. This ensured that the generator synthesizes images with consistent global and local structural information. For the discriminator, the TR-GAN used the same discriminator network architecture as CyleGAN. A pretrained VGG-Face recognition model was used to perform the face comparison after the thermal to visible image translation. The experiments were conducted on TUFTS face dataset and compared against
TV-GAN, Pix2PixHD and CycleGAN. Figure 16 shows samples of synthesized images performed by named models. In comparison, TR-GAN demonstrated superior performance in generating realistic images.

<figure><img src="image_18.png" /><figcaption>Figure 16: **Synthesizing VIS face images** from thermal images on the TUFTS dataset [31] with different visualization results: TR-GAN [54], CycleGAN [11] and TV-GAN [62]. Figure credit: Kezebou *et al.* [54]</figcaption></figure>

**DAF.** Fondje *et al.* [120] proposed a *Domain Adaptation Framework* (DAF) featuring a new feature mapping sub-network with: (a) a *cross-domain identification* loss for effective "co-registration" and "synchronization," and (b) a *domain invariance* loss for cross-domain regularization. Feature representations within each domain were extracted using a truncated pre-trained CNN (VGG16 or ResNet50), aiming to preserve discriminative information across visible and thermal spectra. DAF was embedded in the *Residual Spectral Transform* (RST), a residual block that transforms features with three specific 1x1 convolutions to enhance discriminability. Experiments using three datasets (polarimetric thermal, extended polarimetric thermal with 111 subjects, and a private dataset with 126 paired visible/thermal faces) showed significant improvement compared to DPM [103].

**Axial-GAN.** Immidisetti *et al.* [121] proposed Axial-Generative Adversarial Network (Axial-GAN) to synthesize high-resolution visible images from low-resolution thermal counterparts. The proposed GAN framework designed an axial-attention layer with transformer to model long-range dependencies to facilitate long-distance face matching. Their work can simultaneously address face hallucination and translation for thermal-to-visible face matching. Evaluations on ARL-VTF thermal and multi-modal polarimetric thermal face recognition datasets obtained promising performances compared to SAGAN and HiFaceGAN.

**LG-GAN.** Anghelone *et al.* [122] proposed a Latent-Guided Generative Adversarial Network (LG-GAN) which disentangle the identity from other confounding factors. Hence, an input face image is explicitly decomposed into an *identity latent code* that is spectral-invariant and a *style latent code* that is spectral-dependent. Thermal-to-Visible translation is performed by switching the thermal style code with the opposite visible style code and recombined with the identity code. Interpretation of the identity code offers useful insights in explaining salient facial structures that are essential to generate high-fidelity face images. By using such disentanglement, LG-GAN is able to preserve the identity during the spectral transformation and thus achieve promising face recognition results with respect to visual quality (Figure 17).

**CUT-ATC.** Peri *et al.* [108] presented a comprehensive study on synthesis-based approach for thermal-to-visible face verification. They explored Pix2Pix, CycleGAN and Contrastive Unpaired Translation (CUT) off-the-shelf do-
<figure><img src="image_19.png" /><figcaption>Figure 17: **Synthesizing VIS face images** from thermal images on ARL-VTF dataset [113] using LG-GAN method. Top row is the thermal input, middle row is the LG-GAN generation and bottom row is the ground truth image.</figcaption></figure>

main adaptation algorithms for the task of generating realistic synthetic samples. They further demonstrated that additional custom loss such as Pixel wise correspondence loss and identity loss are instrumental for addressing the thermal-to-visible face verification. Authors highlighted the impact of the face alignment and encompass their method named ATC with above loss functions in a end-to-end thermal-to-visible system comprising ATC: Alignment, spectrum Translation and Classification steps.

**AG-GAN.** Chen et al. [71] offered critical insights with an explainable GAN entitled Attention-Guided Generative Adversarial Network (AG-GAN). Towards enhancing interpretability and investigating specific facial features learned (across spectra), AG-GAN aims at *finding representations* that allow for *extracting* robust biometric features beyond the visible, as well as interpreting the generation process from thermal to visible. While AG-GAN is designed to encode thermal face images into attention maps learnt with supervised attention weights from an auxiliary-domain classifier, an alternative AG-GAN+ version uses *Squeeze and Excitation* to generate in an unsupervised manner attention weights serving to build the attention map.

**DPIF.** Fondje et al. [124] introduced an innovative framework known as the Domain and Pose Invariance Framework (DPIF) for matching off-pose thermal images to frontal visible faces. This framework relies on modified base architectures for extracting image representations. Additionally, a new sub-network is introduced to simultaneously learn *pose* and *domain* invariance using a novel joint-loss function. This function combined the proposed *pose-correction* and *cross-spectrum* losses. Notably, the authors developed a compact and efficient domain adaptation framework. It is worth highlighting that the DPIF framework stands out from previous image-to-image translation approaches by not requiring *a priori* information related to the head pose of probe imagery.

**ANYRES.** Anghelone et al. [72] utilized a pyramidal UNet-based architecture (as related in Figure 18) to translate any (low) resolution thermal face images into high resolution visible face images, placing emphasis on maintaining the cross-spectral identity of depicted individual over any resolution. Authors proposed an algorithm suitable to real world operational scenarios, in which humans are situated at arbitrary distances from the camera, resulting in multi-scale LR thermal face images. Deviating from the state-of-the-art, where resolution is generally fixed with respect to the input, ANYRES is able to operate at any input resolution ranging from low to high.

## Summary of LWIR-to-VIS Face Comparison

LWIR-to-VIS CFR scenario resembles MWIR-to-VIS CFR scenario as face images captured in LWIR and MWIR spectral bands are almost indistinguishable. In the previous works of MWIR-to-VIS comparison, algorithms developed based on Pix2Pix have been used to address the challenge [62, 10]. The Pix2Pix model was designed only to learn forward mapping from one domain to another, while CycleGAN was streamlined to learn both the forward and inverse mappings simultaneously using cycle-consistency loss. In the work of [80], CycleGAN was used to learn such
<figure><img src="image_20.png" /><figcaption>Figure 18: Training of ANYRES [72]. The generator accepts any (low)-resolution thermal face $x_{thm}^{LR}$ as input. It comprises an encoder-decoder bridged by skip connections and gated by Squeeze and Excitation (SE) blocks, which play the role of gate modulator and enable resolution-wise relationships towards bringing a flexible control for balancing encoded features with decoded super resolved features. The discriminators are aimed at distinguishing real images $x_{vis}$ from generated synthetic ones $x_{vis}^{SR}$.</figcaption></figure>

bi-directional mappings for LWIR and VIS face images. The objective function of CycleGAN is defined as

$$ \mathcal{L}(G, F, D_X, D_Y) = \mathcal{L}_{GAN}(G, D_Y, X, Y) \\ + \mathcal{L}_{GAN}(F, D_X, Y, X) \\ + \lambda \mathcal{L}_{cyc}(G, F). \qquad (20) $$

Here, G is used to learn the forward mapping from X to Y, i.e., $G : \{X \to Y\}$. The corresponding discriminator is $D_Y$. Therefore, $\mathcal{L}_{GAN}(G, D_Y, X, Y)$ can be described by

$$ \mathcal{L}_{GAN}(G, D_Y, X, Y) = \mathbb{E}_{y \sim p_{data}(y)}[\log(D_Y(y))] \\ + \mathbb{E}_{x \sim p_{data}(x)}[\log(1 - D_Y(G(x)))]. \qquad (21) $$

Similarly, F is used to learn the inverse mapping from Y to X, i.e., $F : \{Y \to X\}$. The corresponding discriminator is $D_X$. $\mathcal{L}_{GAN}(F, D_X, Y, X)$ can be described by

$$ \mathcal{L}_{GAN}(F, D_X, Y, X) = \mathbb{E}_{x \sim p_{data}(x)}[\log(D_X(x))] \\ + \mathbb{E}_{y \sim p_{data}(y)}[\log(1 - D_X(G(y)))]. \qquad (22) $$

In addition to adversarial losses of $\mathcal{L}_{GAN}(G, D_Y, X, Y)$ and $\mathcal{L}_{GAN}(F, D_X, Y, X)$, a cycle consistency loss was also introduced to ensure that an input image $x$ in domain $X$ can be successfully reconstructed after going through both the forward mapping $G$ and the inverse mapping $F$, i.e., $x \to G(x) \to F(G(x)) \to \bar{x}$. The cycle consistency loss is described by

$$ \mathcal{L}_{cyc}(G, F) = \mathbb{E}_{x \sim p_{data}(x)}\|F(G(x)) - x\|_1 \\ + \mathbb{E}_{y \sim p_{data}(y)}\|G(F(y)) - y\|_1. \qquad (23) $$

Using CycleGAN alone allowed the generation of visually pleasant visible images from thermal images. However, adding more constraints to restrict the mappings is envisioned further improve the synthesis quality. In [80], a 68-point
<table><caption>Table 10: Rank-1 Accuracy and Verification Rate on different dataset for the LWIR Polarimetry-VIS face comparison.</caption><thead><tr><th>Year</th><th>Authors</th><th>Methods</th><th>Dataset</th><th>Performance %</th></tr></thead><tbody><tr><td>2017</td><td>Zhang et al. [112]</td><td>GAN-VFS</td><td>Polarimetric thermal [45] - Polar. Probe</td><td>93.82 @Rank-1 - 79.90 @AUC - 25.17 @EER</td></tr><tr><td rowspan="2">2018</td><td>Di et al. [64]</td><td>AP-GAN</td><td>Polarimetric thermal [45] - Polar. Probe</td><td>88.93 @AUC - 19.02 @EER</td></tr><tr><td>Iranmanesh et al. [115]</td><td>CpDCNN</td><td>Polarimetric thermal [45] - Polar. Probe</td><td>94.08 @Rank-1</td></tr><tr><td rowspan="3">2019</td><td>Di et al. [119]</td><td>SAGAN</td><td>Polarimetric thermal [45] - Polar. Probe</td><td>96.41 @AUC - 10.02 @EER</td></tr><tr><td>Iranmanesh et al. [118]</td><td>AGC-GAN</td><td>Polarimetric thermal [45] - Polar. Probe</td><td>96.54 @Rank-1</td></tr><tr><td>Zhang et al. [11]</td><td>MS-GAN</td><td>Polarimetric thermal [45] -ext. - Polar. Probe</td><td>98.00 @AUC - 7.99 @EER</td></tr><tr><td>2020</td><td>Iranmanesh et al. [69]</td><td>CpGAN</td><td>Polarimetric thermal [45] - Polar. Probe</td><td>95.49 @Rank-1</td></tr><tr><td>2021</td><td>Di et al. [13]</td><td>Multi-AP-GAN</td><td>Polarimetric thermal [45] - Polar. Probe</td><td>93.72 @AUC - 14.75 @EER</td></tr></tbody></table>

landmark detection network was used to extract the landmarks from the visible images. This can assist the CycleGAN to preserve the geometrical shapes of the reconstructed samples. The shape loss was defined as

$$ \mathcal{L}_{\text{shape}} = \frac{1}{68} [(y_s - G(x)_s)^2 + (y_s - F(G(yのs))^2], \quad (24) $$

where, x is a thermal image and G(x) is a synthesize visible image from generator G. y is a visible image and F(G(y)) is a reconstructed visible image.

### 6.2.3. Longwave with polarimetry

Polarimetric imagery in the thermal band has shown several advantages, particularly in bringing more details for both geometrical and textural information. In addition, **Experiment 2** with Table 4 were designed to show the effectiveness of algorithms in comparing face. All LWIR polarimetry-to-VIS face comparison performance are reported on the Table 10 and timeline of developments in algorithms are summarized in the Figure 19.

<figure><img src="image_21.png" /><figcaption>Figure 19: Timeline of developments in algorithms: LWIR Polarimetric-to-VIS face recognition.</figcaption></figure>

**GAN-VFS.** Zhang et al. [112] proposed a Generative Adversarial Network-based Visible face Synthesis (GAN-VFS) algorithm to synthesize visible face images from their corresponding polarimetric images. The proposed GAN-VFS method contained three different modules, including visible feature extraction module, guidance sub-network module and visible image reconstruction module. The visible feature extraction module was used to extract the visible features from the raw polarimetric image. The guidance subnetwork module ensured that extracted visible features were instrumental in reconstructing visible images. The output of the guidance subnetwork constituted the guided features, which were finally used to reconstruct the visible image. This was optimized by the combination of identity loss and perceptual loss.

**AP-GAN.** Di et al. [64] proposed an Attribute Preserved Generative Adversarial Network (AP-GAN) to synthesize the visible image from the polarimetric thermal image. A pre-trained VGG-Face model was first used to extract the attributes from the visible face image. Then, the extracted attributes were used as a prior to ensure that the synthesized visible face image from the polarimetric thermal image would share similar attributes. Finally, a pre-trained FR network was used to extract features from both visible and synthesized visible face images for comparison.

**CpDCNN.** Iranmanesh et al. [115] presented a Coupled Deep Convolutional Neural Network (CpDCNN) to find and hence learn deep global discriminative features between polarimetric LWIR image faces and the corresponding visible faces. They used both thermal and polarimetric state information to bring more detail and then enhance the performance of thermal-to-visible CFR scenario. Their method was based on two VGG-16 coupled architectures, one
<figure><img src="image_22.png" /><figcaption>Figure 20: **Overview of the MS-GAN [11] model:** Synthesize high-quality visible images.</figcaption></figure>

*Vis-DCNN* pretrained on the Imagenet dataset, dedicated to the visible spectrum and another one *Pol-DCNN* initialized with the former one weights, focused to the polarimetric LWIR imagery. Together, Vis-DCNN and Pol-DCNN were linked via a *contrastive* loss function *L*<sub>ct</sub>.

**MS-GAN.** Zhang *et al.* [11] were interested to synthesize high-quality visible images from polarimetric thermal images in order to generate photo realistic visible face images. Note that authors previous work **GAN-VFS** [112] had addressed the same problem. Therefore, they presented a new *Multi-Stream feature-level* (MS-GAN) fusion method based on GAN. The proposed architecture consisted of a *generator* sub-network constructed using an encoder-decoder network based on dense-residual blocks, and a *discriminator* sub-network to capture features at multiple scales. Further, to ensure that the encoded features contain geometric and texture information, a deep-guided sub-network was stacked at the end of the encoding part. Figure 20a illustrates the overview of the proposed architecture and Figure 20b shows corresponding visual results. Moreover, an extended sub-dataset from [45] was collected consisting of visible and polarimetric facial signature from 111 subjects. To be consistent with previous methods [125, 112], two different protocols were introduced. Once the visible images were synthesized via the protocol presented below, the goal was to extract features in order to compare the *synthesized visible* face with a *visible face*. For the feature extraction task, they used the second last fully connected layer of the VGG-face network [126]. Best performances have been achieved with protocol 2.

**SAGAN.** Di *et al.* [119] proposed a synthesis network to convert between polarimetric thermal and visible images based on the *Self-Attention GAN* (SAGAN). The self-attention module was added before the last convolutional layer of the generator and it was used as a weight matrix to refine the feature maps.

**DSU.** Pereira *et al.* [17] suggested that high-level CNN features trained in visible spectral images was domain independent and could be used to encode faces captured in different domains. The low-level CNN features that directly connected to the input signal were domain dependent. These low-level features could be learned by domain specific feature detectors, termed Domain Specific Units (DSU).

**AGC-GAN.** Iranmanesh and Nasrabadi [118] proposed a novel *Attribute-Guided Coupled Generative Adversarial Network* (AGC-GAN) architecture that used facial attributes to improve the thermal-to-visible face recognition performance. The proposed AGC-GAN had two different generators, viz., Vis-GAN and Pol-GAN. The Vis-GAN and Pol-GAN were used to synthesize the corresponding visible image from the visible and polarimetric images, respectively. These two sub-networks were coupled using the contrastive loss function, aiming to find the latent feature vector that was shared by polarimetric face images and their corresponding visible counterparts.

**CpGAN.** Iranmanesh *et al.* [69] proposed a *Coupled Generative Adversarial Network* (CpGAN) depicted in Figure 21, to synthesize visible image from non-visible image such as polarimetric thermal (channel input of $S_0$, $S_1$ and $S_2$) for CFR task.

CpGAN contained two GAN-based sub-networks, with each two appropriate generators and discriminators, dedicated to visible (Vis) and non-visible (NVis) input images. The proposed network was capable of transforming the visible and non-visible modalities into a common discriminative embedding subspace and subsequently synthesizing
<figure><img src="image_23.png" /><figcaption>Figure 21: Flowchart of CpGAN [69] using two GAN based sub-networks (Vis-GAN and NVIS-GAN) coupled by contrastive loss function $\mathcal{L}_{cpl}$.</figcaption></figure>

the visible images from that subspace. In order to efficiently synthesize a realistic visible image from the non-visible modality, a densely connected encoder-decoder structure was used as the generator in each sub-network. The final objective of their proposed CpGAN was to learn a joint deep embedding that captured the interrelationship between the visible and non-visible facial imagery for spectrally invariant face recognition.

**Multi-AP-GAN.** Di et al. [13] proposed a novel Multi-Scale Attribute Preserved Generative Adversarial Network (Multi-AP-GAN) to synthesize the visible image from the polarimetric thermal image with preserved attributes. This work was an extension to their previous work on AP-GAN [64]. The attributes were extracted from visible face images by a pre-trained VGG-Face model. The Multi-AP-GAN consisted of two major components: a multimodal compact bilinear (MCB) pooling-based generator and a generator with the multi-scale architecture. The MCB pooling method was used to concatenate image feature vector and semantic attribute vector, rather than using a simple concatenation. The multi-scale generator was designed to generate multiple intermediate outputs at different resolution scales. The discriminator was applied to individual multi-scale resolution outputs. The proposed Multi-AP-GAN was regularized by a multi-scale adversarial loss, a perceptual loss, an identity loss and a reconstruction loss.

## Summary of Polar-LWIR to VIS Face Comparison

Comparing polarimetric thermal to visible face images remains a challenge problem due to large domain discrepancy. Nearly all works have proposed GANs to synthesize visible images from the polarimetric thermal images [11, 119, 69, 13], rather than using CNNs to learn a domain-invariant feature representation. Associated GANs were based on conditional GANs, where mapping was learned from an observed image $x$ and a random noise vector $z$ to an output $y: G: \{x, z\} \leftrightarrow y$. The objective function of a conditional GAN is defined [109] as

$$ \mathcal{L}_{GAN}(G, D) = \mathbb{E}_{x,y \sim p_{data}(x,y)}[\log D(x, y)] + \mathbb{E}_{x \sim p_{data}(x)}[\log(1 - D(x, G(x))], \quad (25) $$

where $G$ is the generator, $D$ is the discriminator, $x$ is the thermal image and $y$ is the target visible image. $p_{data}(x)$ indicates that $x$ is from the true data distribution and $p_{data}(x, y)$ indicates that both $(x, y)$ are from the true data distribution.
The objective of generator G is to synthesize visually realistic visible face images from thermal face images, while the discriminator D is structured to distinguish the target visible face images from the synthesized ones, conditioned on the input thermal image. In other words, the input to a discriminator D is a pair of samples (x, y). The adversarial loss function $\mathcal{L}_{GAN}(G, D)$ only ensures that we can generate synthesized visible images G(x) that are "real", but it does not provide information whether the generated sample is close to the target visible image at the per-pixel level. This can be remedied by using a per-pixel loss function, $L_1$

$$ \mathcal{L}_1 = \mathbb{E}_{x,y \sim p_{\text{data}}(x,y)} \|G(x) - y\|_1. \qquad (26) $$

The final objective function is described by

$$ G^* = \arg \min_G \max_D \lambda_G \mathcal{L}_{GAN}(G, D) + \lambda_1 \mathcal{L}_1(G). \qquad (27) $$

Using jointly, adversarial loss $\mathcal{L}_{GAN}(G, D)$ and $\mathcal{L}_1(G)$ serves as a foundation for these proposed methods [11, 119, 69, 13]. To further improve the comparison performance, more loss functions are employed to constrain the mapping space.

## 7. Datasets

Most significant research achievements made over the past decade have heavily relied on new datasets and on designing benchmarks for evaluating CFR-algorithms. As deep learning approaches are known to be data-hungry, training data and hence publicly available datasets are a pertinent key stone in the advancement of CFR. We proceed to review such existing datasets, grouping them with respect to spectral bands, in which images were acquired. We summarize discussed datasets in Table 11. Figure 22 illustrates samples from representative CFR-datasets.

**NIR-VIS.** The CASIA HFB [88] is an early benchmark dataset developed for CFR. CASIA HFB comprises of 57 male and 43 female individuals (100 in total). Per subject 4 NIR and 4 VIS face images, as well as a 3D depth image have been acquired. CASIA NIR-VIS 2.0 [44] is an extension to the CASIA HFB dataset and constitutes one of the first large NIR-VIS face dataset, containing 17,580 NIR-VIS face images of 725 subjects with variations in pose, expressions, eyeglasses, and distances. Compared to HFB, NIR-VIS 2.0 has several important features: (a) the number of subjects has a threefold increase; (b) the age distribution of the subjects is significantly larger; and (c) an evaluation protocol was proposed for benchmarking. Based on these dataset properties, CASIA NIR-VIS 2.0 has been the most widely used benchmark dataset for evaluating NIR-VIS CFR. In addition to CASIA HFB and NIR-VIS 2.0 datasets, there also exist benchmark datasets that exhibit other properties. The Oulu-CASIA NIR-VIS dataset [86] is composed of 80 subjects with six expression variations (anger, disgust, fear, happiness, sadness, and surprise). As the facial images of this database are captured under different environments from two institutes, the related illumination conditions differ slightly. We note that compared to CASIA NIR-VIS 2.0, Oulu-CASIA NIR-VIS is more suitable to study NIR-VIS CFR under facial expression changes. In addition, per subject 48 VIS samples and 48 NIR samples are provided. The BUAA-VisNir dataset [87] is often used for domain adaptation evaluation. This dataset contains 150 subjects, with 9 VIS images and 9 NIR images of each subject that correspond to 9 distinct poses or expressions. The training set and testing set are composed of 900 images of 50 subjects and 1800 images of the remaining 100 subjects, respectively. This evaluation protocol is more challenging due to the larger pose and illumination variations observed between the training and test sets. To further expand the range of variability of training samples, a new Large-Scale Multi-Pose High-Quality database LAMP-HQ [127] containing over 73000 images from 573 individuals with large diversities in pose, illumination, attribute, scene, and accessory was proposed. Nevertheless, these datasets have yet to address the problem of outdoor CFR at a long distance. The LDHF-DB dataset [128] offers subjects captured at distances of 60 meters, 100 meters, and 150 meters, in both VIS and NIR spectral bands. Therefore, LDHF-DB is highly instrumental in investigating cross-distance and cross-spectral NIR-VIS FR.

**SWIR-VIS.** The PRE-TINDERS [129] dataset is composed of 48 frontal face subjects and a total of 384 images captured at two wavelengths: visible spectrum and SWIR spectrum at 1550 nm. 4 images per subject are available for each spectral band: 2 images have neutral expression and 2 images depict the person talking (open mouth). All images in this dataset are acquired at close distance from the camera (about 2 meters) in a single session. A light source at 1550 nm is illuminating the face of the subjects for images captured in the SWIR spectral band. The original
resolutions of the acquired images before normalization are 640 × 512 for SWIR images and 1600 × 1200 for visible images. The *TINDERS* [129] dataset is composed of 48 frontal face subjects having images at the same two spectral bands. The SWIR images are acquired at two long ranges (50 and 106 meters). At both distances, 4 or 5 images per class are available: two/three entail neutral expression and two have talking expression. A total of 478 images are available in the SWIR band. The visible (color) images were collected at a short distance and in two sessions (3 images per session), and all of them have neutral expression, resulting in a total of 288 images. The original resolutions of images before normalization are 640 × 512 for SWIR images and 640 × 480 for visible images.

**MWIR-VIS.** The *WSRI* dataset [107] comprises of 1615 visible and 1615 MWIR images from 64 subjects. There are 25 images per individual including different facial expressions. Visible images were captured at resolution 1004 × 1004, while MWIR images were captured at resolution 640 × 512. Face images from both modalities were preprocessed, being resized to 235 × 295 pixels. Training and testing set were split randomly into a set of 10 subjects and 54 subjects, respectively. *MILAB-VTF(B)* [108] is the largest collection of long-range unconstrained paired MWIR-Visible face images, including 400 different identities with both, indoor and outdoor acquisitions.

**LWIR-VIS.** The *UND X1* dataset [110] contains LWIR and visible images related to 241 subjects with different variations in lighting, expression and time lapse. The original resolutions of the images are 1600x1200 pixels for the visible modality and 320x240 pixels for the LWIR modality. Both modalities are resampled to 150x110 pixels after preprocessing. The training set composed of 159 subjects was captured in the visible and LWIR modalities, with only one image per subject. On the other hand, the test set contained remaining 82 subjects with multiple images per subject. This database is challenging due to the low resolution and noise present in the LWIR imagery. This leads to significant difference between the two modalities in this dataset. The *TUFTS* [31] dataset contains over 10,000 images from 113 individuals. Images were acquired in various modalities, including visible, thermal images, near infrared images of individual face. The *EURECOM* [116] dataset contains seven different facial variations, five different illumination conditions, five different types of occlusions, and four different head poses that belong to 50 subjects. In total, there are 2100 images in the dataset. The *ARL-VTF* [113] dataset is the newest and largest collection of time-synchronized visible-thermal face images presented in this survey. The dataset includes 395 subjects and 549,712 images with variations in expression, pose and eyewear. ARL-VTF is endowed with annotations, metadata as well as standardized protocols for fair evaluation.

**Polar-VIS.** The *Polarimetric Thermal Face* dataset [45] (also called ARL-MMFD) contains polarimetric LWIR and visible face images of 60 subjects, later extended to 111 subjects [11]. Data was collected at three different distances: Range 1 (2.5 m), Range 2 (5 m), and Range 3 (7.5 m). At each range, baseline and expressions data were collected and annotated. In the baseline condition, the subject was asked to maintain a neutral expression while looking at the polarimetric thermal sensor. On the other hand, in the expression condition, the subject was asked to count numerically upwards from one, resulting in different expressions in the mouth to eye regions. Each subject has 16 images of visible and 16 polarimetric LWIR images in which four images are from the base-line condition and the remaining 12 images are from the expression condition.

**Multiple-Spectral.** The *NVESD* dataset [104] was collected jointly by the U.S. Army NVESD in 2013 from 50 different subjects in different scenarios and settings including physical exercise (fast walk) and different ranges (1m, 2m and 4m). The dataset is composed of 450 images in each modality. The images were captured simultaneously from different identities with the original resolution of 640 × 480 pixels for all of the modalities. After preprocessing, the image resolution is resampled to 174 × 174 and the dataset is split into training and testing sets.

**In-the-Wild Dataset.** The *Thermal Faces in the Wild* (TFW) dataset [130] was the first dataset collected in both indoor and outdoor environments under controlled and in-the-wild conditions, respectively. The TFW dataset includes 145 subjects and a total of 9202 images with variation in pose. In addition, images are associated with manually labeled bounding boxes and five facial landmarks.

# 8. Open Research Challenges

Despite significant achievements, CFR still faces a number of challenges. In this section, we proceed to identify and discuss some key challenges that are essential for further success of CFR.

**Scarcity of datasets.** Unlike traditional FR, where millions of faces can be leveraged to train deep learning based models, CFR remains significantly impeded by lack of large-scale heterogeneous face datasets. In addition, very few
<table><caption>Table 11: A quick reference summary of the main databases of face images acquired in the infrared spectrum.</caption><thead><tr><th rowspan="2">Year</th><th rowspan="2">Name</th><th rowspan="2">Ref.</th><th colspan="6">Spectrum</th><th colspan="2">Number of</th></tr><tr><th>Vis</th><th>NIR</th><th>SWIR</th><th>MWIR</th><th>LWIR</th><th>Pola.</th><th>Subjects</th><th>Images</th></tr></thead><tbody><tr><td>2003</td><td>UND X1</td><td>[110]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>241</td><td>4584</td></tr><tr><td>2006</td><td>IRIS-M3</td><td>[131]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>82</td><td>2624</td></tr><tr><td rowspan="2">2009</td><td>Oulu-CASIA NIR-VIS</td><td>[86]</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>80</td><td>7680</td></tr><tr><td>CASIA HFB</td><td>[88]</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>100</td><td>5097</td></tr><tr><td rowspan="3">2010</td><td>TINDERS</td><td>[129]</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td>48</td><td>1255</td></tr><tr><td>USTC-NVIE</td><td>[132]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>103</td><td>7460</td></tr><tr><td>PolyU-NIR</td><td>[133]</td><td></td><td>✓</td><td></td><td></td><td></td><td></td><td>335</td><td>34000</td></tr><tr><td>2011</td><td>PRE-TINDERS</td><td>[129]</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td>48</td><td>576</td></tr><tr><td rowspan="4">2012</td><td>BUAA-VisNir</td><td>[87]</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>150</td><td>2700</td></tr><tr><td>LDHF-DB</td><td>[128]</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>100</td><td>-</td></tr><tr><td>Equinox</td><td>[134]</td><td>✓</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td><td>90</td><td>25000</td></tr><tr><td>IRIS</td><td>[105]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>32</td><td>8456</td></tr><tr><td rowspan="4">2013</td><td>NVESD</td><td>[104]</td><td>✓</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td><td>50</td><td>900</td></tr><tr><td>CASIA NIR-VIS 2.0</td><td>[44]</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>725</td><td>17580</td></tr><tr><td>Carl Database</td><td>[111]</td><td>✓</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td>41</td><td>7380</td></tr><tr><td>PCSO</td><td>[106]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>1003</td><td>3011</td></tr><tr><td>2015</td><td>ND-NIVL</td><td>[135]</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>574</td><td>24605</td></tr><tr><td>2016</td><td>Polarimetric thermal</td><td>[45]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td>✓</td><td>60</td><td>14400</td></tr><tr><td rowspan="2">2018</td><td>EURECOM</td><td>[116]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>50</td><td>2100</td></tr><tr><td>UL-FMTV</td><td>[136]</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td></td><td>238</td><td>-</td></tr><tr><td>2019</td><td>IJB-MDF</td><td>[137]</td><td>✓</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td><td>251</td><td>-</td></tr><tr><td rowspan="3">2020</td><td>LAMP-HQ</td><td>[127]</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td><td>573</td><td>73616</td></tr><tr><td>Tufts</td><td>[31]</td><td>✓</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td>113</td><td>10000</td></tr><tr><td>HQ-WMCA</td><td>[24]</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>✓</td><td></td><td>51</td><td>58080</td></tr><tr><td rowspan="4">2021</td><td>ARL-VTF</td><td>[113]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>395</td><td>549712</td></tr><tr><td>MILAB-VTF(B)</td><td>[108]</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td></td><td>400</td><td>-</td></tr><tr><td>TFW</td><td>[130]</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td>145</td><td>9202</td></tr><tr><td>Speaking faces</td><td>[123]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>142</td><td>7668</td></tr><tr><td rowspan="2">2022</td><td>SF-TL54</td><td>[138]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>142</td><td>2256</td></tr><tr><td>MCXFace</td><td>[9]</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>✓</td><td></td><td>51</td><td>7406</td></tr><tr><td rowspan="2">2023</td><td>BYDB</td><td>[139]</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>✓</td><td></td><td>412</td><td>1476292</td></tr><tr><td>LVT</td><td>[140]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td></td><td>52</td><td>612</td></tr><tr><td>-</td><td>WSRI</td><td>[107]</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td></td><td>64</td><td>3230</td></tr></tbody></table>
<figure><img src="image_24.png" /><figcaption>Figure 22: Illustration of heterogeneous face recognition datasets.</figcaption></figure>

existing datasets include annotations of face bounding box and landmarks [41]. Such annotations are essential for
developing automated face and landmark detection for CFR.

**Large standoff distance.** Limited work has focused on CFR at large standoff distances [128]. Most existing studies assume that faces are captured at short standoff distances. However, capturing faces at large standoff distances results in face image quality degradation, including low-resolution, blur and noise. Early attempts towards addressing this challenge have sought to restore high-quality images from degraded ones [128, 141]. Such methods were developed by learning a locally linear mapping between low-quality patches and high-quality patches. The linear mapping assumption has been relaxed at the patch-level, while in practice, the mapping is often considered non-linear. In this context, it has been demonstrated that GANs are highly suitable to model non-linear mappings.

**Modality gap.** The key problem with CFR is the apparent disparity between visible and invisible spectrum, which strongly impacts the comparison performance (see Experiment 2). Although cross-spectrum featured-based methods, such as *transfer learning* or *domain adaptation* have achieved good results, especially in the *Active IR* band, they remain less efficient in the *Passive IR* band. An obvious solution is to incorporate details related to both geometrical and textural information. Unsupervised-processing methods with deep generative models, such as cross-spectrum images synthesis, appear to be a promising trend, particularly with improvements in *GANs*. So far, "synthetic data" has not yet outperformed the state-of-the-art in CFR, but we foresee promising results in the near future.

**Privacy.** Infrared imaging is a keystone biometric solution in terms of security and privacy [6]. Firstly, humans are unable to recognize individuals from IR images, as IR appearances do not resemble RGB images (Figure 5). Secondly, IR imaging ensures pseudo-anonymized recognition and compliance with legal requirements regarding personal data and template protection. Finally, IR is robust against attempted diversion and spoofing attacks, as artifacts are directly identifiable by appearing differently on face images. While intuitive, the above has not been studied rigorously in scientific literature.

**Presentation Attacks.** FR systems have shown to be vulnerable to presentation attacks, where adversaries use artifacts such as prints, video replays, makeup, and 3D masks to undermine the system. Reflection difference between live faces and spoof faces under NIR [142, 143], SWIR [144] or LWIR [8] spectral bands can be exploited to detect such presentation attacks. CFR can provide a unique insight, complementing RGB-imagery, as it imparts liveness detection, as well as shape determination [24].

**Next generation of synthesis models for CFR.** As the field of deep learning continues to evolve, new and innovative methods for generating synthetic images are emerging. While GANs have been popular in recent years for translating thermal face images into visible-like face images [108, 122, 145, 10, 72], they still face several limitations such as *mode collapse* and *instability* during training. In contrast, models based on the diffusion process, such as the Diffusion Probabilistic Models (DPMs), offer a promising alternative for generating high-quality synthetic images
with greater stability and less distortion, preserving identity better. Hence, transitioning from GANs to DPMs opens
up exciting new possibilities for generating interspectral images with improved realism and accuracy, rendering them
a worthwhile avenue of future research with new scientific keystones. In this context, a pioneer work has explored
thermal to visible face translation by proposing a Denoising Diffusion Probabilistic Model [146]. By exploring and
developing these new models, the field of computer vision will advance, towards creating more sophisticated and
reliable systems for a wide range of applications related to CFR.

9. Conclusions

In this survey, we reviewed methods that have been developed for cross-spectral face recognition (CFR), placing emphasis on recently proposed deep CNN-architectures. We discussed associated benefits and limitations, as well as the datasets used. CFR is endowed with the ability to identify people in challenging illumination settings, as well as to detect presentation attacks. Such benefits have been exploited in applications such as *defense*, *surveillance* and *public safety*.

While traditional FR is restricted to comparing images in the visible spectrum, CFR is inherently more challenging, as it must address both subject identification and spectral variation. In order to separate spectral variation from identity, approaches such as domain-invariant feature representation have been proposed. Despite such CNN-based approaches usually performing well in minimizing spectral variation between NIR and visible spectral bands, the large modality gap introduced by capturing face images in the thermal spectral band can be effectively addressed using GANs. In CNN-based approaches designed to learn domain-invariant features, the *triplet* loss function has demonstrated better performance than the *contrastive* loss function, since the former considers both intra-class minimization and inter-class maximization. For GAN-based approaches, the *identity* loss function is essential to ensure that synthesized faces preserve the same identity as the ground-truth face. Other loss functions such as *attribution* loss, *shape* loss, *perceptual* loss or *semantic* loss are utilized as auxiliary loss functions.

We here conclude that the NIR-to-visible CFR scenario has been extensively studied in the literature, achieving
high accuracy. However, SWIR and MWIR sub-bands remain widely unused due to their limited practicality and
high equipment costs. We note that the COVID-19 pandemic has brought to the fore a significant decrease in the cost
of LWIR sensors, rendering such sensors more accessible for a wider range of applications. This, coupled with the
increasing demand for surveillance and protection solutions, e.g., against spoofing attacks, has made LWIR imaging
an appealing option for various industries.

In terms of spectral band selection, NIR is preferred for reflective IR-to-Visible FR in the majority of applications.
There is not much difference between MWIR or LWIR spectral bands in emissive IR-to-Visible face recognition.
Nevertheless, the use of polarimetry for LWIR has been attributed to better face comparison results. While NIR and
SWIR spectral bands can produce relatively good face resolutions, this remains a challenge for MWIR and LWIR
sensors. Towards improving the poor resolution associated with thermal images, there is a necessity to develop
high-resolution sensors by integrating super-resolution post-processing schemes. CFR has been impeded by lack of
large publicly available face datasets. With the emergence of more comprehensive and large-scale datasets, such as
ARL-VTF [113] and MILAB-VTF(B) [108], we foresee further improvements in CFR performance.

To conclude, CFR has several challenging issues, and this survey has reiterated the significant need for systematic theoretical research in this field, rather than empirical experimentation with different loss functions and networks. Such concerns necessitate the introduction of robust and reliable methods to identify forged images and videos.

References

[1] W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld. Face recognition: A literature survey. *ACM Computing Surveys*, 35(4):399–458, 2003.

[2] A. F. Abate, M. Nappi, D. Riccio, and G. Sabatino. 2D and 3D face recognition: A survey. *Pattern Recognition Letters*, 28(14):1885–1906, 2007.

[3] S. Z. Li and A. K. Jain, editors, *Handbook of Face Recognition*. Springer London, 2011.

[4] Mei Wang and Weihong Deng. Deep face recognition: A survey. *Neurocomputing*, 2020.

[5] Y. Taigman, M. Yang, M.A. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1701–1708, 2014.

[6] S. Hu, N. Short, B.S. Riggan, M. Chasse, and M.S. Sarfraz. Heterogeneous face recognition: recent advances in infrared-to-visible matching. In *IEEE International Conference on Automatic Face & Gesture Recognition*, pages 883–890, 2017.
[7] T. Bourlai and L.A. Hornak. Face recognition outside the visible spectrum. Image and Vision Computing, 55:14-17, 2016.

[8] L. Spinoulas, M. E. Hussein, D. Geissbühler, J. Mathai, O. G. Almeida, G. Clivaz, S. Marcel, and W. Abdalmageed. Multispectral biometrics system framework: Application to presentation attack detection. IEEE Sensors Journal, 21(13):15022-15041, 2021.

[9] Anjith George, Amir Mohammadi, and Sébastien Marcel. Prepended domain transformer: Heterogeneous face recognition without bells and whistles. IEEE Transactions on Information Forensics and Security, 2022.

[10] C. Chen and A. Ross. Matching thermal to visible face images using a semantic-guided generative adversarial network. In IEEE International Conference on Automatic Face & Gesture Recognition, pages 1-8, 2019.

[11] H. Zhang, B.S. Riggan, S. Hu, N. J. Short, and V. M. Patel. Synthesis of high-quality visible faces from polarimetric thermal faces using generative adversarial networks. International Journal of Computer Vision, 127(6-7):845-862, 2019.

[12] R. He, J. Cao, L. Song, Z. Sun, and T. Tan. Adversarial cross-spectral face completion for nir-vis face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(5):1025-1037, 2019.

[13] Xing Di, Benjamin S. Riggan, Shuowen Hu, Nathaniel J. Short, and Vishal M. Patel. Multi-scale thermal to visible face verification via attribute guided synthesis. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(2):266-280, 2021.

[14] R.S. Ghiass, O. Arandjelović, A. Bendada, and X. Maldague. Infrared face recognition: A comprehensive review of methodologies and databases. Pattern Recognition, 47(9):2807-2824, 2014.

[15] R. Munir and R.A. Khan. An extensive review on spectral imaging in biometric systems: Challenges & advancements. Journal of Visual Communication and Image Representation, 65, 2019.

[16] Shuxin Ouyang, Timothy M. Hospedales, Yi-Zhe Song, and Xueming Li. A survey on heterogeneous face recognition: Sketch, infra-red, 3D and low-resolution. Image and Vision Computing, 56:28-48, 2016.

[17] T. de Freitas Pereira, A. Anjos, and S. Marcel. Heterogeneous face recognition using domain specific units. IEEE Transactions on Information Forensics and Security, 14(7):1803-1816, 2019.

[18] K. Weiss, T.M. Khoshgoftaar, and D. Wang. A survey of transfer learning. Journal of Big data, 3(1):9, 2016.

[19] A.K. Jain, A. Ross, and S. Prabhakar. An introduction to biometric recognition. IEEE Transactions on circuits and systems for video technology, 14(1):4-20, 2004.

[20] David Anghelone, Sarah Lannes, Valeriya Strizhkova, Philippe Faure, Cunjian Chen, and Antitza Dantcheva. Tfld: Thermal face and landmark detection for unconstrained cross-spectral face recognition. In 2022 IEEE International Joint Conference on Biometrics (IJCB), pages 1-9. IEEE, 2022.

[21] A. Prati, C. Shan, and K. I-K Wang. Sensors, vision and networks: From video surveillance to activity recognition and health monitoring. Journal of Ambient Intelligence and Smart Environments, 11(1):5-22, 2019.

[22] M. Krišto and M. Ivasic-Kos. An overview of thermal face recognition methods. In International Convention on Information and Communication Technology, Electronics and Microelectronics, pages 1098-1103, 2018.

[23] A. George and S. Marcel. Can your face detector do anti-spoofing? face presentation attack detection with a multi-channel face detector, 2020.

[24] G. Heusch, A. George, D. Geissbühler, Z. Mostaani, and S. Marcel. Deep models and shortwave infrared information to detect face presentation attacks. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2020.

[25] S. Ouyang, T. Hospedales, Y.Z. Song, X. Li, C.C. Loy, and X. Wang. A survey on heterogeneous face recognition: Sketch, infra-red, 3D and low-resolution. Image and Vision Computing, 56:28-48, 2016.

[26] L.B. Wolff, D.A. Socolinsky, and C.K. Eveland. Face recognition in the thermal infrared. In Computer Vision Beyond the Visible Spectrum, pages 167-191. 2005.

[27] P. Buddharaju, I. T. Pavlidis, P. Tsiamyrtzis, and M. Bazakos. Physiology-based face recognition in the thermal infrared spectrum. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(4):613-626, 2007.

[28] S.G. Kong, J. Heo, B.R. Abidi, J. Paik, and M.A. Abidi. Recent advances in visual and infrared face recognition review. Computer Vision and Image Understanding, 97(1):103-135, 2005.

[29] M.K. Bhowmik, K. Saha, S. Majumder, G. Majumder, A. Saha, A.N. Sarma, D. Bhattacharjee, D. K. Basu, and M. Nasipuri. Thermal infrared face recognition—a biometric identification technique for robust security system. Reviews, refinements and new ideas in face recognition, 7, 2011.

[30] N. Narang, T. Bourlai, and L.A. Hornak. Can we match ultraviolet face images against their visible counterparts? In Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXI, page 94721Q, 2015.

[31] K. Panetta, Q.W. Wan, S. Agaian, S. Rajeev, S. Kamath, R. Rajendran, S.P. Rao, A. Kaszowska, H.A. Taylor, A. Samani, and X. Yuan. A comprehensive database for benchmarking imaging systems. 42(3):509-520, 2020.

[32] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.

[33] J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2019.

[34] S. Z. Li, R. Chu, S. Liao, and L. Zhang. Illumination invariant face recognition using near-infrared images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(4):627-639, 2007.

[35] F. Nicolo and N. A. Schmid. Long range cross-spectral face recognition: Matching SWIR against visible light images. IEEE Transactions on Information Forensics and Security, 7(6):1717-1726, 2012.

[36] S. Heist, M. Landmann, M. Steglich, Y. Zhang, P. KÃŒhmstedt, and G. Notni. Pattern projection in the short-wave infrared (SWIR): accurate, eye-safe 3D shape measurement. In Dimensional Optical Metrology and Inspection for Practical Applications VIII, volume 10991, pages 99-108, 2019.

[37] T. Bourlai. Mid-wave IR face recognition systems. SPIE Newsroom Magazine-Defense & Security, 2013.

[38] J L. Pezzaniti and D.B. Chenault. Polarization imaging for facial recognition enhancement system and method, 2020. US Patent App. 16/431,374.

[39] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song. Sphereface: Deep hypersphere embedding for face recognition, 2017.
[40] D. Poster, S. Hu, N. Nasrabadi, and B. Riggan. An examination of deep-learning based landmark detection methods on thermal face imagery. In *IEEE Conference on Computer Vision and Pattern Recognition Workshops*, pages 980–987, 2019.

[41] M. Kopaczka, R. Kolk, J. Schock, F. Burkhard, and D. Merhof. A thermal infrared face database with facial landmarks and emotion labels. *IEEE Transactions on Instrumentation and Measurement*, 68(5):1389–1401, 2019.

[42] W. Chu and Y. Liu. Thermal facial landmark detection by deep multi-task learning. In *IEEE International Workshop on Multimedia Signal Processing*, pages 1–6, 2019.

[43] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. *IEEE Signal Processing Letters*, 23(10):1499–1503, 2016.

[44] S.Z. Li, D. Yi, Z. Lei, and S. Liao. The CASIA NIR-VIS 2.0 face database. In *IEEE Conference on Computer Vision and Pattern Recognition Workshops*, 2013.

[45] S. Hu, N.J. Short, B.S. Riggan, C. Gordon, K.P. Gurton, M. Thielke, P. Gurram, and A.L. Chan. A polarimetric thermal database for face recognition research. In *IEEE Conference on Computer Vision and Pattern Recognition Workshops*, 2016.

[46] Domenick D. Poster, Shuowen Hu, Nathan J. Short, Benjamin S. Riggan, and Nasser M. Nasrabadi. Visible-to-thermal transfer learning for facial landmark detection. *IEEE Access*, 9:52759–52772, 2021.

[47] C. Reale, N.M. Nasrabadi, H. Kwon, and R. Chellappa. Seeing the forest from the trees: A holistic approach to near-infrared heterogeneous face recognition. pages 320–328, 2016.

[48] R. He, X. Wu, Z. Sun, and T. Tan. Learning invariant deep representation for nir-vis face recognition. In *AAAI Conference on Artificial Intelligence*, page 2000–2006, 2017.

[49] R. He, X. Wu, Z. Sun, and T. Tan. Wasserstein cnn: Learning invariant features for nir-vis face recognition. *IEEE transactions on pattern analysis and machine intelligence*, 41(7):1761–1773, 2018.

[50] X. Liu, L. Song, X. Wu, and T. Tan. Transferring deep representation for NIR-VIS heterogeneous face recognition. pages 1–8, 2016.

[51] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In *International Conference on Learning Representations*, 2015.

[52] J. Lezama, Q. Qiu, and G. Sapiro. Not afraid of the dark: NIR-VIS face recognition via cross-spectral hallucination and low-rank embedding. In *IEEE Conference on Computer Vision and Pattern Recognition*, 2017.

[53] M. Bihn, M. Gunther, D. Lemmond, and T. Boult. Evaluating a convolutional neural network on short-wave infra-red images. In *IEEE Winter Applications of Computer Vision Workshops*, pages 18–27, 2018.

[54] L. Kezebou, V. Oludare, K. Panetta, and S. Agaian. TR-GAN: thermal to RGB face synthesis with generative adversarial network for cross-modal face recognition. In *Mobile Multimedia/Image Processing, Security, and Applications*, volume 11399, pages 158 – 168. SPIE, 2020.

[55] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In *IEEE conference on Computer Vision and Pattern Recognition*, pages 1–9, 2015.

[56] Ka. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In *IEEE conference on Computer Vision and Pattern Recognition*, pages 770–778, 2016.

[57] B. Cao, N. Wang, X. Gao, J. Li, and Z. Li. Multi-margin based decorrelation learning for heterogeneous face recognition. In *International Joint Conference on Artificial Intelligence*, pages 680–686, 2019.

[58] Z. Deng, X. Peng, Z. Li, and Y. Qiao. Mutual component convolutional neural networks for heterogeneous face recognition. *IEEE Transactions on Image Processing*, 28(6):3102–3114, 2019.

[59] M. Cho, T. Kim, I.J. Kim, and S. Lee. Relational deep feature learning for heterogeneous face recognition. *IEEE Transactions on Information Forensics and Security*, 16:376–388, 2020.

[60] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In *International Conference on Medical Image Computing and Computer Assisted Intervention*, pages 234–241, 2015.

[61] S. Chatterjee and W.T. Chu. Thermal face recognition based on transformation by residual U-Net and pixel shuffle upsampling. In *International Conference on Multimedia Modeling*, pages 679–689, 2020.

[62] T. Zhang, A. Wiliem, S. Yang, and B. Lovell. TV-GAN: Generative adversarial network based thermal to visible face recognition. In *2018 International Conference on Biometrics*, pages 174–181, 2018.

[63] A. Kantarcı and H.K. Ekenel. Thermal to visible face recognition using deep autoencoders. In *2019 International Conference of the Biometrics Special Interest Group (BIOSIG)*, pages 1–5. IEEE, 2019.

[64] X. Di, H. Zhang, and V. M. Patel. Polarimetric thermal to visible face verification via attribute preserved synthesis. In *IEEE International Conference on Biometrics Theory, Applications and Systems*, pages 1–10, 2018.

[65] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In *Conference on Computer Vision and Pattern Recognition*, pages 2818–2826, 2016.

[66] W. Hu, H. Hu, and X. Lu. Heterogeneous face recognition based on multiple deep networks with scatter loss and diversity combination. *IEEE*, 7:75305–75317, 2019.

[67] W. Hu and H. Hu. Adversarial disentanglement spectrum variations and cross-modality attention networks for nir-vis face recognition. *IEEE Transactions on Multimedia*, pages 1–1, 2020.

[68] G. Huang, Z. Liu, L. Van Der Maaten, and K.Q. Weinberger. Densely connected convolutional networks. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 4700–4708, 2017.

[69] S.M. Iranmanesh, B. Riggan, S. Hu, and N.M. Nasrabadi. Coupled generative adversarial network for heterogeneous face recognition. *Image and Vision Computing*, page 103861, 2019.

[70] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 7132–7141, 2018.

[71] Cunjian Chen, David Anghelone, Philippe Faure, and Antitza Dantcheva. Attention-guided generative adversarial network for explainable thermal to visible face recognition. In *2022 IEEE International Joint Conference on Biometrics (IJCB)*, pages 1–8, 2022.

[72] David Anghelone, Sarah Lannes, and Antitza Dantcheva. Anyres: Generating high-resolution visible face images from low-resolution
thermal face images. In *IEEE International Conference on Multimedia and Expo*, 2023. ICME 2023. IEEE Computer Society, 2023.

[73] X. Wu, R. He, Z. Sun, and T. Tan. A light cnn for deep face representation with noisy labels. *IEEE Transactions on Information Forensics and Security*, 13(11):2884–2896, 2018.

[74] X. Wu, L. Song, R. He, and T. Tan. Coupled deep learning for heterogeneous face recognition. 2018.

[75] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In *International conference on machine learning*, pages 6105–6114. PMLR, 2019.

[76] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. *arXiv preprint arXiv:2010.11929*, 2020.

[77] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 10012–10022, October 2021.

[78] Mandi Luo, Haoxue Wu, Huaibo Huang, Weizan He, and Ran He. Memory-modulated transformer network for heterogeneous face recognition. *IEEE Transactions on Information Forensics and Security*, 17:2095–2109, 2022.

[79] I.J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks, 2014.

[80] Z. Wang, Z. Chen, and F. Wu. Thermal to visible facial image translation using generative adversarial networks. *IEEE Signal Processing Letters*, 25(8):1161–1165, 2018.

[81] Y. Sun, Y. Chen, X. Wang, and X. Tang. Deep learning face representation by joint identification-verification. In *Advances in neural information processing systems*, pages 1988–1996, 2014.

[82] Y. Wen, K. Zhang, Z. Li, and Y. Qiao. A discriminative feature learning approach for deep face recognition. In *European conference on computer vision*, pages 499–515. Springer, 2016.

[83] J. Deng, Y. Zhou, and S. Zafeiriou. Marginal loss for deep face recognition. In *IEEE Conference on Computer Vision and Pattern Recognition Workshops*, pages 60–68, 2017.

[84] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu. Cosface: Large margin cosine loss for deep face recognition. In *IEEE Conference on Computer Vision and Pattern Recognition*, 2018.

[85] I. Masi, Y. Wu, T. Hassner, and P. Natarajan. Deep face recognition: A survey. In *2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)*, pages 471–478, 2018.

[86] J. Chen, D. Yi, J. Yang, G. Zhao, S.Z. Li, and M. Pietikainen. Learning mappings for face synthesis from near infrared to visual light images. In *IEEE Conference on Computer Vision and Pattern Recognition*, pages 156–163, 2009.

[87] D. Huang, J. Sun, and Y. Wang. The buaa-visnir face database instructions. *School Comput. Sci. Eng., Beihang Univ., Beijing, China, Tech. Rep. IRIP-TR-12-FR-001*, 2012.

[88] S. Z. Li, Z. Lei, and Meng Ao. The hfb face database for heterogeneous face biometrics research. In *IEEE Conference on Computer Vision and Pattern Recognition Workshops*, pages 1–8, 2009.

[89] W. Hu and H. Hu. Disentangled spectrum variations networks for nir-vis face recognition. *IEEE Transactions on Multimedia*, 22(5):1234–1248, 2020.

[90] W. Hu and H. Hu. Dual adversarial disentanglement and deep representation decorrelation for nir-vis face recognition. *IEEE Transactions on Information Forensics and Security*, 2020.

[91] Ran He, Yi Li, Xiang Wu, Lingxiao Song, Zhenhua Chai, and Xiaolin Wei. Coupled adversarial learning for semi-supervised heterogeneous face recognition. *Pattern Recognition*, 110:107618, 2021.

[92] Yiming Yang, Weipeng Hu, Haiqi Lin, and Haifeng Hu. Robust cross-domain pseudo-labeling and contrastive learning for unsupervised domain adaptation nir-vis face recognition. *IEEE Transactions on Image Processing*, 2023.

[93] D. Yi, Z. Lei, S. Liao, and S.Z. Li. Learning face representation from scratch. *ArXiv preprint*, 2014.

[94] W. Hu and H. Hu. Discriminant deep feature learning based on joint supervision loss and multi-layer feature fusion for heterogeneous face recognition. *Computer Vision and Image Understanding*, 184:9–21, 2019.

[95] Chaoyou Fu, Xiang Wu, Yibo Hu, Huaibo Huang, and Ran He. Dvg-face: Dual variational generation for heterogeneous face recognition. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2021.

[96] Chaoyou Fu, Xiang Wu, Yibo Hu, Huaibo Huang, and Ran He. Dual variational generation for low-shot heterogeneous face recognition. *CoRR*, abs/1903.10203, 2019.

[97] T. Bourlai and B. Cukic. Multi-spectral face recognition: Identification of people in difficult environments. In *IEEE International Conference on Intelligence and Security Informatics*, pages 196–201, 2012.

[98] T. Bourlai, N. Kalka, A. Ross, B. Cukic, and L. Hornak. Cross-spectral face verification in the short wave infrared (swir) band. In *International Conference on Pattern Recognition*, pages 1343–1347, 2010.

[99] N. D. Kalka, T. Bourlai, B. Cukic, and L. Hornak. Cross-spectral face recognition in heterogeneous environments: A case study on matching visible to short-wave infrared imagery. In *International Joint Conference on Biometrics*, 2011.

[100] Z. Cao, N.A. Schmid, and T. Bourlai. Composite multilobe descriptors for cross-spectral recognition of full and partial face. *Optical Engineering*, 55(8):1–15, 2016.

[101] Zhicheng Cao, Natalia A Schmid, Shufen Cao, and Liaojun Pang. Gmlm-cnn: A hybrid solution to swir-vis face verification with limited imagery. *Sensors*, 22(23):9500, 2022.

[102] Anirudh Nanduri and Rama Chellappa. Semi-supervised cross-spectral face recognition with small datasets. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops*, pages 588–596, January 2024.

[103] M. Sarfraz and R. Stiefelhagen. Deep perceptual mapping for cross-modal face recognition. *International Journal of Computer Vision*, 01 2017.

[104] K.A. Byrd. Preview of the newly acquired nveds-arl multimodal face database. In *SPIE*, volume 8734, pages 8734–8734, 2013.

[105] University of Tennessee. Iris thermal/visible face database. http://www.cse.ohio-state.edu/otcbvs-bench/, 2012.
[106] B.F. Klare and A.K. Jain. Heterogeneous face recognition using kernel prototype similarities. *IEEE transactions on pattern analysis and machine intelligence*, 35(6):1410–1422, 2013.

[107] WRIGHT STATE Dataset. https://wsri.wright.edu/.

[108] N. Peri, J. Gleason, C. D. Castillo, T. Bourlai, V. M. Patel, and R. Chellappa. A synthesis-based approach for thermal-to-visible face verification. In *2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)*, pages 01–08, 2021.

[109] P. Isola, J. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In *IEEE Conference on Computer Vision and Pattern Recognition*, pages 5967–5976, 2017.

[110] University of Notre Dame. University of notre dame biometric data set collection c. https://cvrl.nd.edu/projects/data/nd-2006-data-set, Last accessed 2012.

[111] V. Espinosa-Duró, M. Faundez-Zanuy, and J. Mekyska. A new face database simultaneously acquired in visible, near-infrared and thermal spectrums. *Cognitive Computation*, 5:119–135, 2013.

[112] H. Zhang, V.M. Patel, B.S. Riggan, and S. Hu. Generative adversarial network-based synthesis of visible faces from polarimetrie thermal faces. In *International Joint Conference on Biometrics*, pages 100–107. IEEE, 2017.

[113] D. Poster, M. Thielke, R. Nguyen, S. Rajaraman, X. Di, C. N. Fondje, V.M. Patel, N.J. Short, B.S. Riggan, N.M. Nasrabadi, and S. Hu. A large-scale, time-synchronized visible and thermal face dataset. In *IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pages 1559–1568, 2021.

[114] J.Y. Zhu, T.g Park, P. Isola, and A.A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In *IEEE international conference on computer vision*, pages 2223–2232, 2017.

[115] S.M. Iranmanesh, A. Dabouei, H. Kazemi, and N.M. Nasrabadi. Deep cross polarimetric thermal-to-visible face recognition. In *International Conference on Biometrics*, pages 166–173. IEEE, 2018.

[116] K. Mallat and J. Dugelay. A benchmark database of visible and thermal paired face images across multiple variations. In *International Conference of the Biometrics Special Interest Group (BIOSIG)*, pages 1–5, 2018.

[117] K. Mallat, N. Damer, F. Boutros, A. Kuijper, and J. Dugelay. Cross-spectrum thermal to visible face recognition based on cascaded image synthesis. In *International Conference on Biometrics*, pages 1–8, 2019.

[118] S.M. Iranmanesh and N.M. Nasrabadi. Attribute-guided deep polarimetric thermal-to-visible face recognition, 2019.

[119] X. Di, B.S. Riggan, S. Hu, N.J. Short, and V.M. Patel. Polarimetric thermal to visible face verification via self-attention guided synthesis. In *International Conference on Biometrics*, pages 1–8. IEEE, 2019.

[120] C. N. Fondje, S. Hu, N. J. Short, and B.S. Riggan. Cross-domain identification for thermal-to-visible face recognition. In *International Joint Conference on Biometrics*, 2020.

[121] Rakhil Immidisetti, Shuowen Hu, and Vishal M. Patel. Simultaneous face hallucination and translation for thermal to visible face verification using axial-gan. CoRR, abs/2104.06534, 2021.

[122] David Anghelone, Cunjian Chen, Philippe Faure, Arun Ross, and Antitza Dantcheva. Explainable thermal to visible face recognition using latent-guided generative adversarial network. In *IEEE International Conference on Automatic Face & Gesture Recognition, FG 2021, Jodhpur, India, Dec 15-18, 2021*. IEEE, 2021.

[123] Madina Abdrakhmanova, Askat Kuzdeuov, Sheikh Jarju, Yerbolat Khassanov, Michael Lewis, and Huseyin Atakan Varol. Speakingfaces: A large-scale multimodal dataset of voice commands with visual and thermal video streams. *Sensors*, 21(10):3465, 2021.

[124] Cedric Nimpa Fondje, Shuowen Hu, and Benjamin S Riggan. Learning domain and pose invariance for thermal-to-visible face recognition. *IEEE Transactions on Biometrics, Behavior, and Identity Science*, 5(1):15–28, 2022.

[125] B. S. Riggan, N. J. Short, S. Hu, and H. Kwon. Estimation of visible spectrum faces from polarimetric thermal faces. In *IEEE International Conference on Biometrics Theory, Applications and Systems*, 2016.

[126] O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In *British Machine Vision Conference*, 2015.

[127] A. Yu, H. Wu, H. Huang, Z. Lei, and R. He. LAMP-HQ: A large-scale multi-pose high-quality database for nir-vis face recognition. *arXiv preprint*, 2019.

[128] H. Maeng, S. Liao, D. Kang, S.W. Lee, and A.K. Jain. Nighttime face recognition at long distance: Cross-distance and cross-spectral matching. In *ACCV*, 2012.

[129] WVHTC Foundation Technologies Group. Pre-tinders and tinders face database. 2011.

[130] A. Kuzdeuov, D. Aubakirova, D. Koishigarina, and H. A. Varol. Tfw: Annotated thermal faces in the wild dataset. 2021.

[131] H. Chang H. Harishwaran M. Yi A. Koschan B. Abidi and M. Abidi. An indoor and outdoor, multimodal, multispectral and multi-illuminant database for face recognition. In *Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops*, page 54, 2006.

[132] S. Wang, Z. Liu, S. Lv, Y. Lv, G. Wu, P. Peng, F. Chen, and X. Wang. A natural visible and infrared facial expression database for expression recognition and emotion inference. *Transactions on Multimedia*, 12(7):682–691, 2010.

[133] B. Zhang, L. Zhang, D. Zhang, and L. Shen. Directional binary code with application to polyu near-infrared face database. *Pattern Recognition Letters*, 31:2337–2344, 10 2010.

[134] M. Li J. Wang, S. Lai. Equinox corpotation. equinox face database. available online: http://www.equinoxsensors.com/products/hid.html (accessed on 25 april 2012). *Sensors*, 2014.

[135] J. Bernhard, J. Barr, K. W. Bowyer, and P. Flynn. Near-ir to visible light face matching: Effectiveness of pre-processing options for commercial matchers. In *IEEE International Conference on Biometrics Theory, Applications and Systems*, pages 1–8, 2015.

[136] R. S. Ghiass, H. Bendada, and X. Maldague. Université laval face motion and time-lapse video database (ul-fmtv).

[137] Nathan D. Kalka, James A. Duncan, Jeremy Dawson, and Charles Otto. Iarpa janus benchmark multi-domain face. In *2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS)*, pages 1–9, 2019.

[138] Askat Kuzdeuov, Darina Koishigarina, Dana Aubakirova, Saniya Abushakimova, and Huseyin Atakan Varol. Sf-tl54: A thermal facial landmark dataset with visual pairs. In *2022 IEEE/SICE International Symposium on System Integration (SII)*, pages 748–753. IEEE, 2022.

[139] David Anghelone. *Computer vision and deep learning applied to face recognition in the invisible spectrum*. PhD thesis, Université Côte d'Azur, 2023.

[140] Nelida Mirabet-Herranz and Jean-Luc Dugelay. Lvt face database: A benchmark database for visible and hidden face biometrics. In *2023*
*International Conference of the Biometrics Special Interest Group (BIOSIG)*, pages 1–6, 2023.

[141] D. Kang, H. Han, A.K. Jain, and S.W. Lee. Nighttime face recognition at large standoff: Cross-distance and cross-spectral matching. *Pattern Recognition*, 47(12):3750–3766, 2014.

[142] F. Jiang, P. Liu, and X. Zhou. Multilevel fusing paired visible light and near-infrared spectral images for face anti-spoofing. *Pattern Recognition Letters*, 128:30–37, 2019.

[143] F. Jiang, P. Liu, X. Shao, and X. Zhou. Face anti-spoofing with generated near-infrared images. *Multimedia Tools and Applications*, 2020.

[144] H. Steiner, A. Kolb, and N. Jung. Reliable face anti-spoofing using multispectral swir imaging. In *International Conference on Biometrics*, pages 1–8, 2016.

[145] Cunjian Chen and Arun Ross. Matching thermal to visible face images using a semantic-guided generative adversarial network. In *IEEE International Conference on Automatic Face & Gesture Recognition, FG 2019, Lille, France, May 14-18, 2019*. IEEE, 2019.

[146] Nithin Gopalakrishnan Nair and Vishal M Patel. T2V-DDPM: Thermal to visible face translation using denoising diffusion probabilistic models. In *2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)*, pages 1–7. IEEE, 2023.