# On Estimating Uncertainty of Fingerprint Enhancement Models

Indu Joshi, Ayush Utkarsh, Riya Kothari, Vinod K Kurmi, Antitza Dantcheva, Sumantra Dutta Roy, Prem Kumar Kalra

<figure><img src="image_1.png" /></figure>

## To cite this version:

Indu Joshi, Ayush Utkarsh, Riya Kothari, Vinod K Kurmi, Antitza Dantcheva, et al. On Estimating Uncertainty of Fingerprint Enhancement Models. Digital Image Enhancement and Reconstruction, Chapter 2, Elsevier, pp.29-70, 2023, Hybrid Computational Intelligence for Pattern Analysis series, 978-0-323-98370-9. 10.1016/B978-0-32-398370-9.00009-3. hal-04391813

HAL Id: hal-04391813

https://hal.science/hal-04391813v1

Submitted on 15 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# On Estimating Uncertainty of Fingerprint Enhancement Models

Indu Joshi<sup>a,e,*</sup>, Ayush Utkarsh<sup>b</sup>, Riya Kothari<sup>c</sup>, Vinod K. Kurmi<sup>d</sup>, Antitza Dantcheva<sup>a</sup>, Sumantra Dutta Roy<sup>e</sup> and Prem Kumar Kalra<sup>e</sup>

<sup>a</sup>Inria Sophia Antipolis, France

<sup>b</sup>Independent Researcher, India

<sup>c</sup>University of Southern California, USA

<sup>d</sup>KU Leuven, Belgium

<sup>e</sup>Indian Institute of Technology Delhi, India

## ARTICLE INFO

Keywords:

Uncertainty Estimation
Fingerprint Enhancement
Fingerprint Segmentation
Bayesian Deep Learning
Fingerprint Preprocessing

## ABSTRACT

The state-of-the-art models for fingerprint enhancement are sophisticated deep neural network architectures that eliminate noise from fingerprints by generating fingerprints image with improved ridge-valley clarity. However, these models perform fingerprint enhancement like a black box and do not specify whether a model is expected to generate an erroneously enhanced fingerprint image. Uncertainty estimation is a standard technique to interpret deep models. Generally, uncertainty in a deep model arises because of uncertainty in parameters of the model (termed as model uncertainty) or noise present in the data (termed as data uncertainty). Recent works showcase the usefulness of uncertainty estimation to interpret fingerprint preprocessing models. Motivated by these works, this chapter presents a detailed analysis of the usefulness of estimating model uncertainty and data uncertainty of fingerprint enhancement models. Furthermore, we also study the generalization ability of both these uncertainties on fingerprint ROI segmentation. A detailed analysis of predicted uncertainties presents insights into the characteristics learnt by each of these uncertainties. Extensive experiments on several challenging fingerprint databases demonstrate the significance of estimating the uncertainty of fingerprint enhancement models.

## 1. Introduction

The ubiquitous nature of fingerprints and the robust performance of Automated Fingerprint Recognition Systems (AFRS) promotes its application in forensics, law enforcement, access control, and a wide range of other applications. An AFRS typically constitutes of five modules: acquisition, region of interest (ROI) segmentation, enhancement, feature extraction, and matching. The performance of an AFRS majorly depends on feature extraction and matching modules. ROI segmentation and enhancement modules are typically designed to improve the performance of the fingerprint extraction module. While acquiring a fingerprint image, the quality of obtained fingerprint image depends on various factors such as fingerprint sensing technology used, the physical condition of the fingertip (fingertip skin quality, aging, scars, cuts, wrinkles, dry or wet skin), and physical contact between fingertip and sensor, i.e., exertion of excessive or too low pressure. Poor skin conditions lead to spurious minutiae or fading away of genuine minutiae. Dry fingertips are difficult to image and result in blurred ridges. Wet fingertips often lead to thickened ridges, unclear valleys and sometimes may lead to false minutiae due to false joining of ridges. Similarly, inappropriate contact with fingerprint sensors leads to fingerprint images with poor contrast. Various studies report the correlation between fingerprint image quality and performance of AFRS and the challenges of state-of-the-art AFRS on poor quality fingerprints. Figure 1 presents sample poor quality fingerprints used for the experiments and analysis presented in this chapter. To alleviate the limitation of AFRS on poor quality fingerprint images, the fingerprint enhancement module plays a key role.

A fingerprint enhancement module improves the contrast of a fingerprint image, removes background noise, improves ridge-valley clarity, and predicts the missing ridge information due to poor skin condition, overlapping text, or overlapping fingerprints in the background (usually found in case of latent fingerprints). As a result, the quality of a fingerprint image is generally improved after fingerprint enhancement. The improved quality of the fingerprint image improves its suitability for authentication by promoting accurate minutiae extraction. Accurate minutiae extraction,

<figure><img src="image_3.png" /></figure>

indu.joshi@inria.fr, indu.joshi@cse.iitd.ac.in, indujoshi1992@gmail.com (I. Joshi)

ORCID(s): 0000-0001-7511-2910 (I. Joshi)

<footnote>*Corresponding author</footnote>
<figure><img src="image_4.png" /><figcaption>**Figure 1:** Sample poor quality fingerprints used for the experiments and analysis presented in this chapter. Rural Indian Fingerprint Database Puri, Narang, Tiwari, Vatsa and Singh (2010) comprises of distorted fingerprints with unclear ridges due to warts, scars and creases along with dry and wet fingerprint images (see the first row). IIITD- MOLF Database Sankaran, Vatsa and Singh (2015) comprises of samples with unclear ridge details, background noise and overlapping fingerprints in the background (see the second row).</figcaption></figure>

by consequence, improves the authentication performance of AFRS. Although fingerprint enhancement module is an essential part of an AFRS, however, enhancement is especially critical for non-ideal fingerprints such as fingerprints of aging and elderly population, people involved in excessive manual work, such as the rural population of undeveloped or developing nations, fingerprints with cosmetics such as henna and fingerprints found at crime scenes (latent fingerprints). State-of-the-art fingerprint enhancement models are mostly based on deep neural networks that often perform enhancement like a black box and do not specify cases for which the model is expected to fail. Failure of fingerprint enhancement models and subsequently of AFRS on genuine samples often leads to undue inconvenience to genuine users. Estimating *uncertainty* of a predictive model allows interpreting the confidence of a model in its prediction. A reliable measure of uncertainty of a fingerprint enhancement model can be especially useful to identify the samples for which model is highly likely to generate an erroneous enhanced image. Such cases can be segregated for manual evaluation. Uncertainty can also be useful for a human operator to understand what is causing the model to fail. Thus, to interpret the otherwise black-box behaviour of standard fingerprint enhancement models, the notion of uncertainty is introduced in fingerprint enhancement models.

Uncertainty originating in a fingerprint enhancement model can be broadly categorized as either *model uncertainty* or *data uncertainty*. Model uncertainty signifies the uncertainty in model parameters due to the limited availability of training data. Model uncertainty can asymptotically vanish in the limit of infinite training data. Model uncertainty is usually higher for out-of-distribution samples, i.e., fingerprints with noise patterns that are not seen during training. As a result, the fingerprint samples with noise patterns unseen during training and for which the fingerprint enhancement generates an erroneous enhanced image, the predicted model uncertainty is usually high. While data uncertainty is the uncertainty originating due to noise present in a given fingerprint sample. Occlusion due to text or overlapping fingerprints in the background, sensor noise, unclear ridge information due to injuries, aging, or creases results in high data uncertainty. Data uncertainty cannot be explained away even in the limit of infinite training data as it captures the inherent noise in a fingerprint sample. As a result, both these uncertainties extract different but complementary information that imparts interpretability to fingerprint enhancement models. Figure 2 illustrates both model uncertainty and data uncertainty observed during fingerprint enhancement by a state-of-the-art fingerprint enhancement model FP-E-GAN. We observe that introducing either type of uncertainty improves the performance of the baseline fingerprint enhancement model. Additionally, the predicted spatial uncertainty maps impart interpretability to the fingerprint enhancement model.

## Research Contributions

The research contributions in this chapter are as follows:

* This chapter is based on recent researches in the fingerprints domain Joshi, Anand, Dutta Roy and Kalra (2021a), Joshi, Kothari, Utkarsh, Kurmi, Dantcheva, Dutta Roy and Kalra (2021b) and Joshi, Utkarsh, Kothari, Kurmi,
<figure><img src="image_5.png" /><figcaption>**Figure 2:** Visualization of uncertainty estimated for a fingerprint enhancement model. MU-GAN and DU-GAN represent the fingerprint enhancement model obtained by introducing model uncertainty and data uncertainty respectively into the baseline fingerprint enhancement model FP-E-GAN. The visualization of both the uncertainties illustrates that higher model uncertainty is predicted for the pixels where the enhancement model generates spurious patterns (first row) or erroneously enhances background noise (last row). The fingerprint enhancement model predicts higher data uncertainty for noisy and background pixels which in turn, helps the model to minimize erroneous predictions on noisy pixels.</figcaption></figure>

Dantcheva, Dutta Roy and Kalra (2021c), and discusses in detail, how to interpret model and data uncertainty from deep learning-based fingerprint enhancement models.

* A rigorous analysis of model parameters, computation time, and model performance is presented to understand the effect of both the kind of uncertainties.

* A detailed analysis of predicted uncertainties is provided to share insights on model and data uncertainty.

* For analyzing the effect of uncertainty on fingerprint enhancement, both data uncertainty and model uncertainty are introduced into two different state-of-the-art fingerprint enhancement models.

* Three challenging databases corresponding to the fingerprints acquired from the rural Indian population and latent fingerprints are used to evaluate the effectiveness of modelling model and data uncertainties.

* To evaluate the generalization ability of modelling data and model uncertainties on fingerprint preprocessing models, their effect on fingerprint segmentation is analyzed. Experiments on fingerprint ROI segmentation are conducted on twelve challenging publicly available fingerprints databases.

# 2. Related Work

## 2.1. Fingerprint Enhancement

### 2.1.1. *Classical Image Processing Techniques for Enhancement*

Traditional approaches for fingerprint enhancement perform *filtering* either in spatial or frequency domain to exploit cues such as ridge orientation or ridge continuity to recover the corrupted or unclear regions of a fingerprint image Hong, Wan and Jain (1998); Gottschlich and Schönlieb (2012); Turroni, Cappelli and Maltoni (2012); Ramos, de Lima Borges, Andrezza, Primo, Batista and Gomes (2018); Wang, Li, Huang and Feng (2008); Gottschlich (2011); Chikkerur, Cartwright and Govindaraju (2007); Ghafoor, Taj, Ahmad and Jafri (2014). Hong et al. (1998) work on a normalized fingerprint image to compute ridge orientation through gradients and ridge frequency by modelling a sinusoid wave along a direction normal to the ridge orientation. The enhanced image is obtained by applying Gabor filters tuned to the computed orientation and frequency. Gottschlich and Schönlieb (2012) perform locally adaptive
fingerprint enhancement through anisotropic filtering along the direction of local ridge orientations. Turroni et al. (2012) exploit context defined by local orientation, frequency, and quality to adapt the filter characteristics for a specific fingerprint region. The authors propose to iterative apply contextual filtering starting from high-quality regions to low-quality regions so that the contextual information from low-quality regions can be reliably predicted. Ramos et al. (2018) apply adaptive Gabor filtering such that the standard deviation of Gaussian function in the Gabor filter is adapted based on the signal frequency. Wang et al. (2008) argue that Gabor filters have limited bandwidth, whereas the log-Gabor filters can be designed with any arbitrary bandwidth. Consequently, log-Gabor filters are better suited for oriented-textured patterns such as fingerprints. The authors calculate the orientation and ridge frequency as defined in Hong et al. (1998), whereas the curvature information is used to find the angular bandwidth for filtering with log-Gabor filter. Gottschlich (2011) argues that the curved Gabor filter is highly suited for fingerprints due to their inherent curvature and uses it for filtering fingerprint ridges. Chikkerur et al. (2007) exploit Short Time Fourier Transform (STFT) to estimate ridge orientation and ridge frequency. Context filtering using the estimated filters is used to obtain the enhanced image. Ghafoor et al. (2014) propose a method that offers the advantages of filtering in both frequency and spatial domain. The authors propose band-pass filtering in the frequency domain so that only ridge patterns are filtered. Furthermore, directional filtering is applied in the spatial domain to smooth ridges along the ridge directions.

Hsieh, Lai and Wang (2003) observe that the approaches presented above focus only on local information such as ridge orientation and ignore global information. The authors propose to perform wavelet decomposition to obtain different spatial or frequency sub-images. These sub-images are converted into texture spectrum domain and use global information to improve under or over-inked regions. Directional filtering followed by wavelet reconstruction is used to obtain the enhanced fingerprint image. Jirachaweng and Areekul (2007) argue that the Gabor filters result in ridge discontinuity and blocking artifacts around regions with high curvatures. Furthermore, the computational complexity of enhancement algorithms due to the computation of Fast Fourier Transform (FFT) is very high. To address these limitations, the authors propose to apply filtering on the Discrete Cosine Transform (DCT) domain. Yoon, Feng and Jain (2010) estimate finger-print rotation and skin distortion model to estimate orientation field. Furthermore, they also estimate orientations from singular points using the zero-pole method. Both the orientation information are exploited to obtain the orientation field for a given input fingerprint. The orientation field is then used to tune the Gabor filter and obtain an enhanced fingerprint image.

A general shortcoming of classical image processing-based fingerprint enhancement methods is that their performance is heavily dependent upon the quality of contextual information, i.e., ridge orientation and ridge frequency. Quite often, the contextual information extracted by these methods around fingerprint regions with high distortions is not reliable. Consequently, these methods generally obtain poor performance on highly distorted fingerprint regions. To address the above-mentioned limitation, learning-based fingerprint enhancement methods are proposed.

### 2.1.2. Learning Based Enhancement Models

Many algorithms use dictionary based approach for approximating orientation field Feng, Zhou and Jain (2013); Yang, Feng and Zhou (2014); Chen, Feng and Zhou (2016); Liu, Liu and Yang (2017); Chaidee, Horapong and Areekul (2018). Feng et al. (2013) argue that the orientation estimation is analogous to spelling correction in a sentence. They propose to create a dictionary of orientation patches and claim that it helps to eliminate non-words errors, i.e., prediction of such orientations that cannot exist in real life. To begin with, the authors compute an initial estimate of the orientation field using STFT and compare the initial estimate with each dictionary element to identify potential candidates. Furthermore, they use compatibility between the neighbouring patches to find the optimal candidate. Orientation information of all orientation patches is then summarized to obtain the final orientation field. Yang et al. (2014) utilize spatial locality information to improve orientation estimation. The authors claim that only specific orientations occur at a given location. In order to exploit this information, they introduce localized dictionaries, i.e., create a dictionary for every location in a fingerprint. Due to this, each dictionary contains only a limited number of orientations leading to faster dictionary look-ups and fewer non-word errors. Chen et al. (2016) observe that the average size of noise varies depending upon the quality of fingerprints. For a poor quality image, one can obtain better results by using a dictionary with bigger patch size and vice-versa. Motivated by this observation, the authors create multi-scale dictionaries, i.e., dictionaries of different patch sizes, and use compatibility between neighbours across different scales to find the optimal orientation patch for a given estimate. Liu et al. (2017) propose sparse coding for denoising of orientation patches. Authors create multi-scale dictionaries from good quality fingerprints. After computing the initial estimate, they then reconstruct the orientation using the dictionary of the smallest size with sparse coding. The quality of an orientation patch is then estimated based on compatibility with the neighbours. If the quality is below a certain
threshold, then the orientation patch is reconstructed using a dictionary of bigger patches. This process is continued
until the quality of the reconstructed orientation patch is satisfactory. Chaidee et al. (2018) propose sparse-coded
dictionary learning in the frequency domain which fuses responses from Gabor and curved filters. The dictionary is
constructed from the frequency response. During testing, spectral response is computed and encoded by the spectral
encoder. The sparse representation of the spectral code is computed and decoded by the spectral decoder to reconstruct
the Fourier spectrum. A weighted sum of the reconstructed image is obtained from both the filters is computed to obtain
the final enhanced image.

A common shortcoming of dictionary-based orientation estimation approaches is that the dictionary is constructed
from good quality fingerprint patches, due to which the orientation is not reliably predicted for noisy fingerprint regions.
To address this limitations, *orientation prediction networks* are proposed Cao and Jain (2015); Qu, Liu, Liu, Guan,
Yang and Zhang (2018). Cao and Jain (2015) pose orientation field estimation as a classification problem and exploit
a CNN-based classification model. K-means clustering is performed on orientation patches of good quality images to
select 128 representative orientation patch classes. The authors extract 1000 orientation patches for each orientation
class and train the network with the corresponding simulated poor quality fingerprint. During testing, for each patch in
the input fingerprint, an orientation class is predicted by the model. Qu et al. (2018) propose a deep regression neural
network to predict orientation angle values. The input fingerprint image is first preprocessed using total variation
decomposition and Log-Gabor filtering. The preprocessed image is presented as input to the network and orientation
is estimated. Boosting is performed to further improve the prediction accuracy. Later, researchers make a paradigm
shift in fingerprint enhancement by working towards *directly constructing the enhanced image* Sahasrabudhe and
Namboodiri (2014); Schuch, Schulz and Busch (2016); Rama and Namboodiri (2011); Svoboda, Monti and Bronstein
(2017); Qian, Li and Liu (2019); Wong and Lai (2020); Li, Feng and Kuo (2018) rather than predicting the orientation
field and filtering through the Gabor filter tuned at the predicted orientation.

Sahasrabudhe and Namboodiri (2014) propose a deep belief network for the enhancement of fingerprints. The
network is trained using a greedy strategy such that the first layer learns oriented ridges while the second layer learns
higher-level features. Hierarchical probabilistic inference from the second layer is used for reconstructing enhanced
fingerprints. Schuch et al. (2016) propose a deconvolutional autoencoder (DeConvNet) for enhancement of fingerprint
images. Svoboda et al. (2017) propose an autoencoder network that is trained to minimize gradient and orientation
between the output and target enhanced image. Qian et al. (2019) exploit DenseUnet to enhance poor quality fingerprint
patches. Wong and Lai (2020) propose a multi-task learning model that is trained to generate not only an enhanced
image but also perform orientation correction. Li et al. (2018) propose a multi-task learning based enhancement
algorithm that accepts texture component of fingerprint image (preprocessed using Total Variation decomposition)
as an input for the proposed model. The proposed solution is based on encoder-decoder architecture trained with a
multi-task learning loss. One branch enhances the fingerprint, and the other branch predicts orientation. Recently,
Joshi, Utkarsh, Singh, Dantcheva, Dutta Roy and Kalra (2022 [accepted]) propose channel level attention model for
improving the generalization ability of fingerprint enhancement models. Schuch, Schulz and Busch (2017) present
rigorous comparisons between several relevant fingerprint enhancement methods.

To summarize, most of the learning based fingerprint enhancement methods utilize auto-encoder architectures
at their core and improve their representations either through predicting related information or using an adversarial
discriminator. We also observe that all the models proposed so far are black-box models and do not provide any
information on the model's confidence in the prediction. All these observations motivate the discussions in this chapter.

## 2.2. Uncertainty Estimation

Uncertainty estimation techniques can be broadly categorized into four different categories: *single network deterministic techniques*, *ensemble techniques*, *test-time augmentation techniques* and *Bayesian techniques*. Single network deterministic techniques for uncertainty estimation are the methods in which only a single forward pass is required to make a prediction. The uncertainty is either directly predicted by the model or an external network is exploited to predict it. Ensemble technique combine the predictions of several deterministic models and estimate the predictive uncertainty. Test-time augmentation techniques also exploit a single deterministic model to make a prediction. However, during testing, these techniques augment the test input to obtain various predictions and later exploit these predictions to estimate predictive uncertainty. Bayesian techniques are the methods that exploit stochastic deep models to make a prediction. Because of the stochastic nature of the model, different predictions are obtained for the same input during different inferences. These different predictions are then used to estimate the predictive uncertainty.
2.2.1. Uncertainty Estimation through Single Network Deterministic Techniques

A deterministic model is characterized by fixed model weights. Subsequently, in a deterministic network, the pre-
diction is always the same after forwarding the same input. The uncertainty for a deterministic model is either obtained
by using an external network Raghu, Blumer, Sayres, Obermeyer, Kleinberg, Mullainathan and Kleinberg (2019); Ra-
malho and Miranda (2020) or directly predicted by the network itself Sensoy, Kaplan and Kandemir (2018); Malinin
(2019); Możejko, Susik and Karczewski (2018). Priors networks and Gradient penalty based methods are among the
most widely used methods in this category. In the Prior networks Malinin and Gales (2018), the Dirichlet distribution
is modeled for a network's output to obtain the distribution of uncertainty. To train the model, Kullback-Leibler di-
vergence (KLD) is minimized between the predictions of in-distribution data with a sharp Dirichlet distribution. For
the out-of-distribution data, the prediction is KLD is minimized between a flat Dirichlet distribution and the predic-
tions. On the other hand, the gradient penalty-based method Van Amersfoort, Smith, Teh and Gal (2020) enforces
the detectability of changes to obtain the out-of-distribution (OOD) data. Similarly, methods proposed in Oberdiek,
Rottmann and Gottschalk (2018); Lee and AlRegib (2020) use the gradient metric such as norms to define the uncer-
tainty of the prediction. A major limitation of this category of techniques is high sensitivity to initial model weights
Gawlikowski, Tassi, Ali, Lee, Humt, Feng, Kruspe, Triebel, Jung, Roscher et al. (2021).

2.2.2. Uncertainty Estimation through Ensemble Techniques

An ensemble model combines the prediction from different models. Ensemble techniques for uncertainty esti-
mation model network parameters as random variables of some prior distribution. For each forward pass, the model
parameters are sampled from this distribution. As a result, stochastic model outputs are obtained and uncertainty of
these predictions is quantified. An ensemble model can be obtained in different ways such as data shuffle Lakshmi-
narayanan, Pritzel and Blundell (2017), random initialization, bagging, boosting Achrack, Kellerman and Barzilay
(2020), and employing different architectures. Other ensemble based approaches for uncertainty estimation include
sub-sample Valdenegro-Toro (2019) and batch samples Wen, Tran and Ba (2019). These methods aim towards lower-
ing the memory and computational requirements through sharing of parts among the single members. Deep ensem-
ble Fort, Hu and Lakshminarayanan (2019) is one of the most frequently used ensemble based techniques to obtain the
uncertainty in deep learning models. Gustafsson, Danelljan and Schon (2020) show comparisons between ensemble
and Monte Carlo dropout sampling based uncertainty. Ovadia, Fertig, Ren, Nado, Sculley, Nowozin, Dillon, Lakshmi-
narayanan and Snoek (2019) evaluate different uncertainty methods for a task in which the test set has a distributional
shift. Vyas, Jammalamadaka, Zhu, Das, Kaul and Willke (2018) show that the OOD detection is improved in case of
ensemble models. High memory requirements for training and testing is a key limitation of this category of approaches
for uncertainty estimation Gawlikowski et al. (2021).

2.2.3. Uncertainty Estimation through Test-time Augmentation Techniques

It is an easy to implement category of uncertainty estimation techniques. The basic working principle behind these
techniques is to exploit different views from a given test data by augmenting it. All the augmented samples are used
to obtain the final prediction and the associated uncertainty. The techniques in this category are especially useful for
applications with very limited data such as medical image processing. Shanmugam, Blalock, Balakrishnan and Guttag
(2020) argue that several factor such as size of the training set, nature of problem, network architecture and kind of
augmentation must be taken into consideration while choosing a test-time augmentation strategy. In particular, the
authors propose an aggregation function to aggregate the predictions obtained for the various augmented inputs. Kim,
Kim and Kim (2020) propose to learn a loss predictor such that the test-time augmentations with the minimum loss
for an input sample is selected. Lyzhov, Molchanova, Ashukha, Molchanov and Vetrov (2020) propose greedy policy
search to design a test-time augmentation policy that selects augmentations to be performed. High inference time is a
major limitation of this class of uncertainty estimation techniques Gawlikowski et al. (2021).

2.2.4. Uncertainty Estimation through Bayesian Techniques

Bayesian techniques for uncertainty estimation model a deterministic deep model into a Bayesian neural network (BNN). These methods estimate posterior distribution over model parameters (conditioned on the training data) is obtained using a prior distribution. The BNN posterior can capture the uncertainty in model parameters, which can obtain the uncertainty in the predictions. This uncertainty occurs due to uncertainty in model parameters and called model uncertainty. There are many works proposed to estimate and model the uncertainty in the deep learning models. In the classification problem, Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow and Fergus (2014) propose to treat
softmax output as one source of uncertainty. However, imperceptible perturbations to a real image can change a deep network's softmax output to arbitrary values due to which softmax output is not a good estimate of the uncertainty. Another way is to obtain uncertainty through OOD. Some of the works proposed methods for OOD are likelihood test ratio Ren, Liu, Fertig, Snoek, Poplin, Depristo, Dillon and Lakshminarayanan (2019), density estimation, and bias detection. Gal (2016) model two types of uncertainties in deep learning models, namely epistemic and aleatoric uncertainty. Traditionally, BNNs have been computationally complex and far more complicated than non-Bayesian NNs. Monte Carlo dropout is a practical way to approximate BNNs Gal and Ghahramani (2016). Concrete dropout Gal, Hron and Kendall (2017) explores the dropout probabilities to obtain the variation inference using dropout. A major advantage of these approximation based methods is that these only require ensemble of prediction at test time which makes these methods widely applied in many downstream tasks Kendall and Gal (2017); Mukhoti and Gal (2018). Marin (2019) propose data uncertainty using the entropy measurement. Low memory requirements and low sensitivity to choice of initial model weights Gawlikowski et al. (2021) encourage the use of Bayesian techniques for estimating uncertainty of fingerprint enhancement models.

## 3. Model Uncertainty Estimation

Uncertainty in a fingerprint enhancement can originate either because of uncertainty in model parameters termed as model uncertainty or uncertainty as a result of noise in the input fingerprint image. State-of-the-art deep learning based fingerprint enhancement models are deterministic models that output point estimates of prediction. To exploit Bayesian techniques for approximating uncertainty from a deterministic model, it is converted into a probabilistic model so that the statistical analysis of model's prediction can be conducted. Estimating model uncertainty in a fingerprint enhancement model aims at understanding what the fingerprint enhancement model does not know. State-of-the-art deep models for fingerprint enhancement do not output model uncertainty. Studies indicate that the predictive probabilities output by deep models cannot correctly indicate the model's confidence in the prediction. Gal and Ghahramani (2016) demonstrate that an uncertain deep model may also output a high predictive probability through the output of softmax. Therefore, a high predictive probability cannot be regarded as a reliable measure to infer the model's confidence. Bayesian deep learning offers a practical mechanism to infer uncertainty from deep learning based fingerprint enhancement models. Estimating model uncertainty allows to know the confidence with which output of the fingerprint enhancement can be trusted. To formalize, model uncertainty is modelled as placing a prior distribution over the weights of a model and then estimating how much these weights vary given the training data. Monte Carlo dropout is a computationally efficient mechanism to estimate model uncertainty from a deep fingerprint enhancement model. The spatial uncertainty maps output during inference indicate the fingerprint enhancement model's per-pixel confidence. We now share details on Monte Carlo dropout based model uncertainty estimation.

### 3.1. Bayesian Neural Networks

Assuming that the training set of input fingerprint images is denoted by $X = \{x_1, x_2...x_M\}$, the set of corresponding enhanced images is denoted as $Y = \{y_1, y_2...y_M\}$ and the baseline fingerprint enhancement model is represented as $y = f^\theta(x)$. A Bayesian neural network (BNN) aims to infer the distribution over model weights $\theta$ that are likely to have generated the set of enhanced images, $Y$ for a given training set of input fingerprint images $X$ and baseline fingerprint enhancement model. The probability distribution of an enhanced image given an input fingerprint is denoted as $p(y|x, \theta)$. Training of a BNN implies learning the posterior distribution $p(\theta|X, Y)$ and finding the most probable model weights given the set of input fingerprints and corresponding enhanced fingerprints images. For a given test input fingerprint image $x_{test}$, the output probability is defined as:

$$p(y_{test}|x_{test}, X, Y) = \int p(y_{test}|x_{test}, \theta) p(\theta|X, Y) d\theta \\ p(\theta|X, Y)) = \frac{p(X, Y|\theta)p(\theta)}{\int p(X, Y|\theta)p(\theta)d\theta}$$

Obtaining the posterior probability distribution, however, requires marginalizing the product of the likelihood ($p(X, Y|\theta)$) and the prior ($p(\theta)$) over all the possible combinations of model weights. Deep models have an extremely high number of model weights, due to which marginalizing is impractical. As a result, posterior probability distribution $p(\theta|X, Y)$ of a deep model is intractable. However, in order to conduct from inference a BNN, the posterior probability is required. In order to overcome this limitation of BNNs, the variational inference is a widely used technique to approximate the
<figure><img src="image_6.png" /><figcaption>**Figure 3: Model uncertainty estimation in a fingerprint enhancement model.** At test time, dropout is introduced into the baseline fingerprint enhancement model. The average of stochastic outputs obtained for the Monte Carlo samples serves as the enhanced fingerprint. Variance of these Monte Carlo samples is the estimated model uncertainty of the fingerprint enhancement model.</figcaption></figure>

posterior of model weights. It requires defining an approximating variational distribution $q_w(\theta)$ such that $q_w(\theta)$ is close to the true posterior of model weights. Here $w$ denotes the variational parameters of the approximated distribution.

$$p(y_{test}|x_{test}, X, Y) \approx \int p(y_{test}|x_{test}, \theta) q_w(\theta) d\theta$$

## 3.2. Approximating Inference via Monte Carlo Dropout

For a given neural network of any arbitrary depth and non-linearities, Gal and Ghahramani (2016) in their seminal work show that by introducing a Bernoulli distribution over approximating variational distribution placed over every layer of model weights, the model effectively minimizes the KLD between the posterior and the approximate distribution. This enables approximation of variational inference from a deterministic deep neural network by reparameterizing the variational distribution as a Bernoulli distribution. Dropout allows sampling of a binary variable for each input fingerprint image and every output unit in each layer of the model. Assume $y_k$ represents the output of layer k. After the introduction of dropout, the output is modified as:

$$y_k = p_k * y_k$$

$$y_k = \text{Bernoulli}(p_k) y_k$$

This result signifies that the introduction of dropout is essentially placing a Bernoulli variational distribution over model weights. Therefore, for any deterministic deep model, approximation of Bayesian inference can be obtained by applying dropout before every layer of model weights. Monte Carlo integration is a practical approximation mechanism for evaluating the integral for variational inference. Here, the Monte Carlo integration of the model outputs after introducing dropout is carried out, due to which method is termed as *Monte Carlo dropout*.

$$p(y_{test}|x_{test}, X, Y) \approx \int p(y_{test}|x_{test}, \theta) q_w(\theta) d\theta$$
$$= \frac{1}{S} \sum_{s=1}^{S} p(y_{test} | x_{test}, \tilde{\theta}_{s})$$

where $\tilde{\theta}_s \sim q_{iw}(\theta)$ and $q_{iw}(\theta)$ is the dropout distribution. The above result implies that the output of the model is approximated by averaging the output of S stochastic outputs (a stochastic output is the model output obtained after forward pass over the model with dropout).

### 3.3. Estimating Model Uncertainty

Model uncertainty is quantified as the variance of the model predictions. Model uncertainty through Monte Carlo dropout based deep Bayesian model can be computed as:

$$var(y_{test}) = \frac{1}{S} \sum_{s=1}^{S} (p(y_{test} | x_{test}, \tilde{\theta}_s))^T p(y_{test} | x_{test}, \tilde{\theta}_s) - E(y_{test})^T E(y_{test})$$

where

$$E(y_{test}) = \frac{1}{S} \sum_{s=1}^{S} (p(y_{test} | x_{test}, \tilde{\theta}_s))$$

In order to estimate Bayesian inference from the baseline fingerprint enhancement model, dropout is introduced before each layer of the model. Consequently, as a result of introducing dropout, the output of a model may change for each iteration. Due to this, the model output is now referred to be a stochastic output. For a test input fingerprint image $x_{test}$, the model output ($E(y_{test})$) is approximated as the mean of S stochastic samples. Model uncertainty is the variance of S stochastic outputs ($var(y_{test})$) (see Figure 3).

## 4. Data Uncertainty Estimation

Noise in a fingerprint image can arise due to various reasons such as sensor noise, dust or grease on the fingerprint sensing device, unclear or blurred boundaries (due to wet or dry fingertips), or false traces from previously acquired fingerprints. Data uncertainty of a fingerprint enhancement model accounts for uncertainty due to noise in a fingerprint image. As a result, data uncertainty, unlike model uncertainty, cannot be reduced even if an infinitely large training database is available to train the fingerprint enhancement model. Separately quantifying model and data uncertainties is a good idea as it helps to understand which uncertainties can be reduced and which are unlikely to be reduced. Data uncertainty can be approximated as either *homoscedastic* or *heteroscedastic* uncertainty. Homoscedastic uncertainty assumes that the noise observed by the model is uniformly distributed across the input fingerprint image. In other words, homoscedastic uncertainty implies that the data uncertainty is invariant to the choice of input to the model. However, the assumption of a uniform noise model may not always be satisfied. To address the limitation of homoscedastic uncertainty estimation, heteroscedastic uncertainty estimation assumes that the noise is a random variable whose mean and variance need to be estimated. To put it simply, heteroscedastic uncertainty means that the data uncertainty or the noise in the input is dependent on the input to the model. Various regions in a fingerprint image observe the different levels of noise. For instance, background regions are expected to be noisier than the foreground fingerprint region. Therefore, data uncertainty is modelled as heteroscedastic uncertainty for fingerprint enhancement models. Bayesian deep learning can be successfully utilized to estimate data uncertainty in fingerprint enhancement models.

To formalize, data uncertainty is modelled by placing a distribution over the model of the output. We assume that the output of the fingerprint enhancement model is corrupted by zero-mean Gaussian noise. Data uncertainty estimation, in this case, aims at approximating the variance of the noise. As data uncertainty is modelled as heteroscedastic, the noise variance is learnt as a function of the input fingerprints rather than a constant value for all fingerprint images. Furthermore, the uncertainty is learned as spatial maps with pixel level correspondence to the input fingerprint image. To estimate heteroscedastic data uncertainty originating in fingerprint enhancement, network architecture of the baseline fingerprint enhancement model is modified. In the modified architecture, not only enhanced fingerprint but also the data uncertainty is learnt as a function of the input fingerprint image Gal (2016). To achieve this, both the functions are tied together by splitting the last layer into two branches. Both the branches are trained in a multi-task manner. First task is to predict the enhanced fingerprint image, while the other task is to predict the data uncertainty
<figure><img src="image_7.png" /><figcaption>**Figure 4:** Data uncertainty estimation in a fingerprint enhancement model. An additional branch is introduced into the final layer of the baseline fingerprint enhancement model. One branch of the final layer predicts the enhanced fingerprint. The newly introduced branch is dedicated to learn data uncertainty prediction.</figcaption></figure>

(see Figure 4). The branch predicting the enhanced fingerprint image is trained in a fully supervised manner, using noisy synthetic fingerprints and the corresponding good quality binarized fingerprint images. The data uncertainty branch, on the other hand, is trained in an unsupervised manner. A prior over the weights of noise variance (data uncertainty) is placed. To ensure training of the resulting architecture, the loss function is also modified, depending upon the functional form of the loss function. The loss function of a fingerprint enhancement model can be either regression based or cross-entropy based. In case of a regression based loss function, change in the model output can be directly computed. While, for cross-entropy loss based fingerprint enhancement models, change in the logs is computed instead of change in output of softmax (output probabilities). In the following subsections, we describe how the loss function of fingerprint enhancement models should be adapted to also estimate data uncertainty in addition to predicting enhanced fingerprint image.

## 4.1. Estimating Data Uncertainty for Regression Loss based Fingerprint Enhancement Models

Assuming loss function of a fingerprint enhancement model is originally defined to be $\frac{1}{P} \sum_{k=1}^{P} \|y_k - f_e(x_k)\|^2$. Where P represents the total number of training pixels and $x_k$ signifies pixel k in the input fingerprint image x. $y_k$ and $f_e(x_k)$ denote the ground annotation and output of fingerprint enhancement model for pixel k. We assume that $f_e(x_k)$ is adulterated with a zero mean Gaussian noise whose variance is $\sigma(x_k)$. Estimating data uncertainty aims at learning the noise variance $\sigma(x_k)$. To learn $\sigma(x_k)$ as a function of input fingerprints, the loss function is adapted as:

$$ \frac{1}{P} \sum_{i=1}^{P} \frac{1}{2\sigma(x_k)^2} \|y_k - f_e(x_k)\|^2 + \frac{1}{2} \log \sigma(x_k)^2 $$

Intuitively, adapting the loss function as above allows the fingerprint enhancement model to alleviate the residual error term by a factor of $\sigma(x_k)$. As $\sigma(x_k)$ is a function of input fingerprint, the adaptation of loss function enables the fingerprint enhancement model to attenuate the residual error obtained on noisy pixels. To achieve this, the fingerprint enhancement model outputs high data uncertainty ($\sigma(x_k)$) on noisy pixels. This, in effect, makes the fingerprint enhancement model achieve *noise-aware* enhancement and predict high uncertainty on noisy pixels to promote a lower effect on the overall loss. However, to ensure that predicted uncertainty is indeed dependent on the input fingerprint, the fingerprint enhancement model is discouraged from predicting high uncertainty on all input pixels. This is achieved by
introducing log σ(x_k)² term in the new loss function. High predicted uncertainty leads to higher contribution from the log σ(x_k)² term in the loss function while lower predicted uncertainty leads to high loss due to σ(x_k)²||y_k - f_e(x_k)||². As a result, the fingerprint enhancement model learns to predict higher uncertainty on noisy and erroneous pixels while lower uncertainty on the rest of the pixels in the input fingerprint image.

## 4.2. Estimating Data Uncertainty for Cross-Entropy Loss based Fingerprint Enhancement Models

As explained above, predicting heteroscedastic uncertainty in regression loss based fingerprint enhancement models enables them to attenuate loss function on noisy pixels. It is a desirable characteristic for cross-entropy loss based fingerprint enhancement models as well. However, directly quantifying change in model output for cross-entropy loss based fingerprint enhancement models is not possible. Therefore, the idea of predicting heteroscedastic uncertainty in cross-entropy loss based fingerprint enhancement models is extended in the logit space. To implement this idea, heteroscedastic regression uncertainty is placed over the logit space and marginalized. We now provide details about this method. Let f_e(x_k) denotes the vector of binary variable (logit) for a pixel i of the input fingerprint image (x). This vector is adulterated with a random zero-mean Gaussian noise. The adulterated logit is denoted by ˆx_k. The corrupted logit is then converted to a probability score p_k by the softmax function.

$$ \hat{x}_k \sim \mathcal{N}(f_e(x_k), \sigma(x_k)^2) $$

$$ \hat{p}_k = \mathrm{Softmax}(\hat{x}_k) $$

In order to learn σ(x_k)² and the parameter of feature extractor (f_e) of the fingerprint enhancement model, Monte Carlo integration of softmax probabilities of sampled logits is carried out. In order to achieve this, the loss function is adapted to be:

$$ \hat{x}_{k,s} = f_e(x_k) + \sigma(x_k) \alpha_s, \quad \alpha_s \sim \mathcal{N}(0, I) \\ \frac{1}{P} \sum_{k=1}^{P} \log \frac{1}{S} \sum_{s=1}^{S} \exp(\hat{x}_{k,s,\hat{l}} - \log \sum_{\hat{l}} \exp(\hat{x}_{i,s,\hat{l}})) $$

$\hat{x}_{k,s}$ represents corrupted logit obtained for input x_k and stochastic sample s. $\hat{x}_{k,s,\hat{l}}$ represents the $\hat{l}$ element in the logit vector $\hat{x}_{k,s}$. $\hat{l}$, S, and P represent the total number of class labels, the number of Monte Carlo samples, and the total number of pixels in training images, respectively. Similar to the adapted loss function for regression, the above-mentioned modified cross-entropy loss, this loss function can also be interpreted as learning an attenuation of loss.

# 5. Experimental Evaluation

## 5.1. Databases

Poor quality fingerprint images generally result either due to poor skin conditions or structured noise in the background. To study the effect of uncertainty estimation on fingerprint enhancement models, we evaluate the performance of the resulting models on three challenging fingerprint databases. Two rural Indian fingerprint databases are analyzed to examine the effect on poor quality fingerprints. On the other hand, performance on latent fingerprints is investigated to evaluate the performance on noisy backgrounds. Please note that the training of the all the fingerprint enhancement models studied in this chapter is executed on the same training database as performed in Joshi et al. (2021a). Details about the testing datasets are presented next.

1. Rural Indian Fingerprint Database: This is a publicly available fingerprint database that has fingerprint samples acquired from the rural Indian population. The volunteers constitute farmers, carpenters, and housewives who are rigorously involved in manual work. This database has total of 1631 fingerprint samples.

2. A private rural Indian fingerprint database which has challenging samples of elderly population and poor quality fingertips. This database has 1000 fingerprint images.

3. IIITD-MOLF: This is the largest latent fingerprint database available in the public domain. This database has 4400 latent fingerprints and corresponding fingerprints obtained from optical sensors.
<figure><img src="image_8.png" /><figcaption>**Figure 5:** Sample cases demonstrating the enhanced fingerprints generated by MU-GAN and the corresponding predicted model uncertainties. Results indicate that high model uncertainty is predicted at the enhanced fingerprint regions with spurious ridge structure. While low uncertainty is predicted on enhanced regions with smooth and clear ridge details.</figcaption></figure>

In Section 6.3, we analyze the generalization ability of the Bayesian deep learning for uncertainty estimation of fingerprint ROI segmentation methods. The experiments on fingerprint ROI segmentation are conducted on fingerprint verification competition (FVC) databases: FVC 2000, FVC 2002, and FVC 2004. Each of these databases constitutes of four sub-databases, out of which fingerprints in three databases (DB1-DB3) are collected from fingerprint sensors with varying sensing technologies. DB4 for each of these databases contains synthetic fingerprints obtained from Anguli A.H.Ansari (2011), an open-source implementation of SFinGe Cappelli, Maio and Maltoni (2004). Each sub-database is divided into two parts: set A and set B. Set B is used for training the ROI segmentation model, while set A is used to evaluate the ROI segmentation performance. In total, 960 images are used for training, while 9600 images are used for testing. The ground truth constituting of manually marked segmentation masks of ROI are taken from Thai and Gottschlich (2016).

## 5.2. Evaluation Metrics

We now the evaluation metrics used to assess the effectiveness of fingerprint enhancement models after adapting them to also estimate model and data uncertainty.

1. *Ridge Structure Preservation Ability:* Biometric identity of a person lies in the ridge details of his/her fingerprints. Therefore, it is critical that a fingerprint enhancement model must be able to preserve the finest ridge details while enhancing the fingerprint. To evaluate the ridge preservation ability of fingerprint enhancement models, we generate some noisy synthetic fingerprints and calculate the peak signal-to-noise-ratio (PSNR) Ndjah, Kikuchi, Yukawa, Watanabe and Muramatsu (2011) between enhanced fingerprint and ground truth binarized image. Synthetic (noisy) test samples are generated by introducing varying backgrounds and noise into good quality synthetic fingerprints. The ground truth binarized fingerprints images are obtained using NBIS.

2. *Fingerprint Quality Assessment:* The goal of a fingerprint enhancement model is to improve the quality of an input fingerprint. The quality of a fingerprint image is characterized by various factors such as uniform contrast, clarity of ridges and valleys, the flow of ridge orientations, number and quality of minutiae, etc. We adopt the NFIQ module of NBIS (NIST Biometric Image Software) NIST to evaluate the quality of enhanced fingerprints. NFIQ outputs a score for each input fingerprint image. The score value is within the range of one to five. One represents the best quality, while five signifies the worst quality. Ideally, fingerprint quality score should improve (value should reduce in this case) after enhancement.

3. *Fingerprint Matching Performance:* An enhanced fingerprint image is expected to obtain a higher match score and an accurate fingerprint matching performance. To assess fingerprint matching performance obtained on enhanced fingerprints, the fingerprint matching performance is evaluated under both verification and identification setting. Under verification mode, a fingerprint matching system is required to output whether the input fingerprint matches with any fingerprint in the gallery or not. While, during the identification mode of opera-
<figure><img src="image_9.png" /><figcaption>**Figure 6:** Test examples quantifying the progression in the ridge preservation proficiency after introducing Monte Carlo dropout. Higher PSNR scores between the enhanced fingerprints and the ground truth binarized fingerprints are obtained for MU-GAN as compared to the baseline FP-E-GAN. The improvement by MU-GAN is attributed to the model averaging effect introduced by the Monte Carlo dropout.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Figure 7:** Enhancement performance on the Rural Indian Fingerprint database after introducing model uncertainty estimation: (a) histogram of NFIQ scores; DET curves obtained using (b) Bozorth (c)MCC</figcaption></figure>

tion, a fingerprint matching system is required to identify a list of potential matches from the gallery. The rural Indian fingerprint databases are evaluated under verification mode. For each of these databases, the average equal error rate (EER) is computed, and the detection error tradeoff (DET) curve is plotted. On the other hand, performance on latent fingerprints is evaluated for the identification mode. Rank-50 accuracy and cumulative matching characteristics (CMC) curve is computed. Please note that Bozorth NIST and MCC Cappelli, Ferrara and Maltoni (2010b), Cappelli, Ferrara and Maltoni (2010a) and Ferrara, Maltoni and Cappelli (2012) are used for conducting fingerprint matching.

## 5.3. Effect of Estimating Model Uncertainty

## 6. Results and Analysis

In order to analyze the effect of estimating model uncertainty of a fingerprint enhancement model, we introduce Monte Carlo dropout into the following fingerprint enhancement models: DeConvNet Schuch et al. (2016) and FP-E-GAN Joshi, Anand, Vatsa, Singh, Dutta Roy and Kalra (2019). The resulting architectures are termed as MU-
<table><caption>**Table 1**</caption><thead><tr><th>Enhancement Model</th><th>Avg.NFIQ Score (↓)</th></tr></thead><tbody><tr><td>Raw Image</td><td>2.94</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>1.95</td></tr><tr><td><strong>MU-DeConvNet</strong></td><td><strong>1.68</strong></td></tr><tr><td><strong>MU-GAN</strong></td><td><strong>1.33</strong></td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td><strong>1.31</strong></td></tr></tbody></table>

Average fingerprint quality scores attained on the Rural Indian Fingerprint database. Competitive fingerprint quality of enhanced images is obtained after incorporating model uncertainty estimation.

<table><caption>**Table 2**</caption><thead><tr><th>Enhancement Model</th><th>Matching Algorithm</th><th>Avg. (↓)</th></tr></thead><tbody><tr><td>Raw Image</td><td>Bozorth</td><td>16.36</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>Bozorth</td><td>10.93</td></tr><tr><td><strong>MU-DeConvNet</strong></td><td>Bozorth</td><td><strong>8.48</strong></td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>Bozorth</td><td><strong>7.30</strong></td></tr><tr><td><strong>MU-GAN</strong></td><td>Bozorth</td><td>7.46</td></tr><tr><td>Raw Image</td><td>MCC</td><td>13.23</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>MCC</td><td>10.86</td></tr><tr><td><strong>MU-DeConvNet</strong></td><td>MCC</td><td>7.56</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>MCC</td><td>5.96</td></tr><tr><td><strong>MU-GAN</strong></td><td>MCC</td><td><strong>5.06</strong></td></tr></tbody></table>

Verification performance obtained on the Rural Indian Fingerprint Database. Reduced average EER indicates improved matching performance on enhanced images generated after incorporating model uncertainty estimation.

<figure><img src="image_11.png" /><figcaption>**Figure 8:** Enhancement performance on the private fingerprint database after introducing model uncertainty estimation: (a) histogram of NFIQ scores; DET curves obtained using (b) Bozorth (c)MCC</figcaption></figure>

DeConvNet and MU-GAN respectively. The modified architectures are expected to achieve better performance as compared to the baseline due to the model averaging effect introduced by Monte Carlo Dropout. During visual inspection of enhanced samples (see Figure 5), we find that the ridge-valley clarity and smoothness of ridges are improved after introducing Monte Carlo dropout. Furthermore, the ridge prediction ability has also improved. To quantify these observations, we quantitatively analyze the enhancement performance of both these architectures. Firstly, we evaluate the effect on the ridge preservation ability of the fingerprint enhancement model after introducing Monte Carlo dropout. Sample synthetic test cases and the obtained enhanced images generated by MU-GAN are compared with the baseline FP-E-GAN and the ground truth in Figure 6. It is visually evident even that the ridge structure ability of MU-GAN is far better in contrast to FP-E-GAN. To quantify this improvement, we calculate PSNR between the ground truth and the enhanced images. Higher PSNR is attained by the model with Monte Carlo dropout, which illustrates that the improved ridge preservation ability.

Next, we evaluate the fingerprint quality of the enhanced images output by both these models. We find that the average NFIQ fingerprint quality score has improved (or at least competitive with the baseline) after introducing Monte
<table><caption>**Table 3**</caption><thead><tr><th>Enhancement Model</th><th>Avg.NFIQ Score (↓)</th></tr></thead><tbody><tr><td>DeConvNet Schuch et al. (2016)</td><td>4.12</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>2.28</td></tr><tr><td>MU-GAN</td><td>1.92</td></tr></tbody></table>

Average fingerprint quality scores attained on the private fingerprint database. Fingerprint quality of enhanced images has improved significantly after incorporating model uncertainty estimation.

<table><caption>**Table 4**</caption><thead><tr><th>Enhancement Model</th><th>Matching Algorithm</th><th>Avg. EER (↓)</th></tr></thead><tbody><tr><td>DeConvNet Schuch et al. (2016)</td><td>Bozorth</td><td>28.75</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>Bozorth</td><td>17.06</td></tr><tr><td>MU-GAN</td><td>Bozorth</td><td>12.75</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>MCC</td><td>26.80</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>MCC</td><td>15.85</td></tr><tr><td>MU-GAN</td><td>MCC</td><td>11.55</td></tr></tbody></table>

Verification performance obtained on the private Indian Fingerprint Database. Reduced average EER indicates improved matching performance on enhanced images generated after incorporating model uncertainty estimation.

<figure><img src="image_12.png" /><figcaption>**Figure 9:** Comparison of results on IIITD-MOLF database by DeConvNet Schuch et al. (2016), FP-E-GAN and MU-GAN: (a) histogram of NFIQ scores; CMC curve comparing the identification performance obtained using (b) Bozorth (c) MCC</figcaption></figure>

Carlo dropout (see Table 1, Table 3 and Table 5). For better understanding of distribution of obtained NFIQ quality scores, the histogram of scores is provided in Figure 7 (a), Figure 8 (a) and Figure 9 (a). All these results indicate that the fingerprint quality of enhanced images is indeed improved after introducing Monte Carlo dropout as a model uncertainty estimation technique. Lastly, we assess the fingerprint matching performance on the enhanced images generated by MU-DeConvNet and MU-GAN. For both the rural Indian fingerprints databases, we compute the average EER and find that the average EER on the enhanced images has reduced significantly (see Table 2 and Table 4). The corresponding DET curves are presented in Figure 7 (b)-(c) and Figure 8 (b)-(c) respectively. These results illustrate the fact that for the rural Indian fingerprints databases, incorporating Monte Carlo dropout results in a significant increase in the matching performance obtained for the enhanced images. For evaluating the matching performance obtained on the enhanced images generated for latent fingerprints from IIITD-MOLF database, we calculate Rank-50 accuracy. Different from the rural Indian fingerprint databases, we find that instead of improvement, rather the fingerprint matching performance decreases on enhanced fingerprints obtained after introducing model uncertainty. Table 6 reports the obtained Rank-50 accuracies. The corresponding CMC curves are presented in Figure 9 (b)-(c). The obtained results indicate that the performance in enhancement performance does not generalize on latent fingerprints. We understand that generalization on latent fingerprints with varying complex background noise poses a significant challenge on fingerprint enhancement models. These challenges are increased more after introducing model uncertainty as the model is trained on synthetic training examples and able to perform well on similar noise patterns
<table><caption>**Table 5**
Comparison of average quality scores obtained using NFIQ on IIITD-MOLF database.</caption><thead><tr><th>Enhancement Model</th><th>Avg.NFIQ Score (↓)</th></tr></thead><tbody><tr><td>Raw Image</td><td>4.96</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>4.09</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>1.91</td></tr><tr><td>MU-GAN</td><td>1.48</td></tr></tbody></table>

<table><caption>**Table 6**
Comparison of identification performance obtained on IIITD-MOLF database when matched across Lumidigm gallery.</caption><thead><tr><th>Enhancement Model</th><th>Bozorth (↑)</th><th>MCC (↑)</th></tr></thead><tbody><tr><td>Raw Image</td><td>5.45</td><td>6.06</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>14.02</td><td>14.27</td></tr><tr><td>Svoboda et al. Svoboda et al. (2017)</td><td>N.A.</td><td>22.36</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td><strong>28.52</strong></td><td><strong>34.43</strong></td></tr><tr><td>MU-GAN</td><td>25.09</td><td>28.61</td></tr></tbody></table>

<figure><img src="image_13.png" /><figcaption>**Figure 10:** Sample cases demonstrating the enhanced fingerprints generated by DU-GAN and the corresponding predicted data uncertainties. Results indicate that high data uncertainty is predicted at the poor quality and noisy fingerprint regions in the input fingerprint image. While low uncertainty is predicted on good quality fingerprint regions with ridge-valley clarity.</figcaption></figure>

but not the ones as seen in real latent fingerprints.

## 6.1. Effect of Estimating Data Uncertainty

Having analyzed the effect of estimating model uncertainty, we now shift attention towards analyzing the effect of data uncertainty estimation on fingerprint enhancement models. To achieve this, we modify the architecture of De-ConvNet Schuch et al. (2016) and FP-E-GAN Joshi et al. (2019) (as suggested in Section 4). The modified fingerprint enhancement models are termed as *DU-DeConvNet* and *DU-GAN* respectively. Modifying a fingerprint enhancement model to estimate data uncertainty enables it to attenuate the loss function such that noise-aware enhancement can be conducted. As a result, DU-DeConvNet and DU-GAN are expected to attain superior enhancement performance than the baselines DeConvNet and FP-E-GAN, respectively. Figure 10 presents sample enhanced fingerprints generated by DU-GAN. DU-GAN outperforms the baseline in predicting the missing ridge details. Furthermore, the enhanced fingerprints generated by DU-GAN have smoother ridges and higher ridge-valley clarity. Additionally, we also observe that DU-GAN predicts higher data uncertainty on noisy pixels. These results help to qualitatively evaluate the enhancement performance of DU-GAN.

To quantitatively evaluate the fingerprint enhancement performance obtained using DU-GAN, we first assess the
<figure><img src="image_14.png" /><figcaption>**Figure 11:** Test examples quantifying the progression in the ridge preservation proficiency after introducing data uncertainty estimation. Higher PSNR scores between the enhanced fingerprints and the ground truth binarized fingerprints are obtained for DU-GAN as compared to the baseline FP-E-GAN. The improvement by DU-GAN is attributed to the noise adaptive enhancement achieved as a result of learning of loss attenuation.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>**Figure 12:** Enhancement performance on the Rural Indian Fingerprint database after introducing data uncertainty estimation: (a) histogram of NFIQ scores; DET curves obtained using (b) Bozorth (c)MCC</figcaption></figure>

ridge preservation ability of DU-GAN. Figure 11 illustrates the enhanced fingerprints obtained for synthetic test cases. The sample cases present in the figure demonstrate that DU-GAN enhances unclear ridge details while preserving them. These results qualitatively signify that DU-GAN has better ridge preservation ability than baseline FP-E-GAN. We also calculate the PSNR value between ground truth binarized image and the enhanced fingerprint to quantitatively assess the ridge preservation ability. High PSNR values are obtained for DU-GAN as compared to FP-E-GAN, which indicates that ridge preservation ability is improved after incorporating data uncertainty estimation into fingerprint enhancement models. Next, we assess the fingerprint quality of enhanced fingerprints generated after introducing data uncertainty estimation into baseline fingerprint enhancement models. Table 7 and Table 9 demonstrate that for the rural Indian fingerprints, we observe that the fingerprint quality improves after enhancement. The corresponding histograms of NFIQ values are plotted in Figure 12 (a) and Figure 13 (a). However, for latent fingerprints, the fingerprint quality rather degrades after enhancement, as reported in Table 11. Figure 14 presents the corresponding histogram of NFIQ scores.

Lastly, we evaluate the fingerprint matching performance obtained on enhanced fingerprints generated after incorporating data uncertainty estimation in fingerprint enhancement models. For both the rural Indian fingerprints databases, we compute the average EER and find that the average EER on the enhanced images has reduced significantly (see Table 8 and Table 10). The corresponding DET curves are presented in Figure 12 (b)-(c) and Figure 13 (b)-(c) respectively. These results illustrate the fact that for the rural Indian fingerprints databases, the matching per-
<table><caption>**Table 7**</caption><thead><tr><th>Enhancement Model</th><th>Avg. NFIQ Score (↓)</th></tr></thead><tbody><tr><td>Raw Image</td><td>2.94</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>1.95</td></tr><tr><td><strong>DU-DeConvNet</strong></td><td><strong>1.84</strong></td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>1.31</td></tr><tr><td><strong>DU-GAN</strong></td><td><strong>1.26</strong></td></tr></tbody></table>

Comparison of average quality scores attained using NFIQ on the Rural Indian Fingerprint database. Fingerprint quality of enhanced images improves after incorporating data uncertainty estimation.

<table><caption>**Table 8**</caption><thead><tr><th>Enhancement Model</th><th>Matching Algorithm</th><th>Avg. EER (↓)</th></tr></thead><tbody><tr><td>Raw Image</td><td>Bozorth</td><td>16.36</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>Bozorth</td><td>10.93</td></tr><tr><td><strong>DU-DeConvNet</strong></td><td>Bozorth</td><td><strong>8.71</strong></td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>Bozorth</td><td>7.30</td></tr><tr><td><strong>DU-GAN</strong></td><td>Bozorth</td><td><strong>7.13</strong></td></tr><tr><td>Raw Image</td><td>MCC</td><td>13.23</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>MCC</td><td>10.86</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>MCC</td><td>5.96</td></tr><tr><td><strong>DU-DeConvNet</strong></td><td>MCC</td><td><strong>5.36</strong></td></tr><tr><td><strong>DU-GAN</strong></td><td>MCC</td><td><strong>5.13</strong></td></tr></tbody></table>

Verification performance obtained on the Rural Indian Fingerprint Database. Reduced average EER indicates improved matching performance on enhanced images generated after incorporating data uncertainty estimation.

<figure><img src="image_16.png" /><figcaption>**Figure 13:** Enhancement performance on the private fingerprint database after introducing data uncertainty estimation: (a) histogram of NFIQ scores; DET curves obtained using (b) Bozorth (c)MCC</figcaption></figure>

formance on the enhanced images has increased significantly after introducing data uncertainty. For latent fingerprints from IIITD-MOLF database, we find that the fingerprint matching performance decreases on enhanced fingerprints obtained after introducing data uncertainty. Table 12 reports the obtained Rank-50 accuracies. The corresponding CMC curves are presented in Figure 14 (b)-(c). The obtained results indicate that the performance in enhancement performance does not generalize on latent fingerprints. We understand that generalization on latent fingerprints with varying complex background noise poses a significant challenge on fingerprint enhancement models. The fingerprint enhancement model is trained on synthetic training examples and able to perform well on similar noise patterns but not the ones as seen in real latent fingerprints. These challenges are increased more after introducing data uncertainty. As a result, matching performance degrades for latent fingerprints.

## 6.2. Comparison of Model and Data Uncertainty

This section provides a detailed comparison between model and data uncertainty estimation in fingerprint enhancement models. We compare the model complexity and computational cost to estimate model and data uncertainty. The comparisons are provided in terms of model parameters and inference time. All the relevant discussions are presented
<table><caption>**Table 9**
Average fingerprint quality scores attained on the private fingerprint database. Fingerprint quality of enhanced images has improved significantly after incorporating data uncertainty estimation.</caption><thead><tr><th>Enhancement Algorithm</th><th>Avg. NFIQ Score (↓)</th></tr></thead><tbody><tr><td>DeConvNet Schuch et al. (2016)</td><td>4.12</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>2.28</td></tr><tr><td>DU-GAN</td><td>1.79</td></tr></tbody></table>

<table><caption>**Table 10**
Verification performance obtained on the private Indian fingerprint database. Reduced average EER indicates improved matching performance on enhanced images generated after incorporating model uncertainty estimation.</caption><thead><tr><th>Enhancement Algorithm</th><th>Matching Algorithm</th><th>Avg. EER (↓)</th></tr></thead><tbody><tr><td>DeConvNet Schuch et al. (2016)</td><td>Bozorth</td><td>28.75</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>Bozorth</td><td>17.06</td></tr><tr><td>DU-GAN</td><td>Bozorth</td><td>11.24</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>MCC</td><td>26.80</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>MCC</td><td>15.85</td></tr><tr><td>DU-GAN</td><td>MCC</td><td>11.50</td></tr></tbody></table>

<figure><img src="image_17.png" /><figcaption>**Figure 14:** Comparison of results on IIITD-MOLF database by DeConvNet Schuch et al. (2016), FP-E-GAN and DU-GAN: (a) histogram of NFIQ scores; CMC curve comparing the identification performance obtained using (b) Bozorth (c) MCC.</figcaption></figure>

next.

### 6.2.1. Comparison of Model Complexity

To give insights on the computational complexity introduced by model and data uncertainty estimation, we demonstrate whether any additional model complexity is introduced by uncertainty estimation. In this research, uncertainty estimation is introduced in the generator sub-network of the baseline fingerprint enhancement model FP-E-GAN. Therefore, in Table 13, we compare the model parameters in the generator sub-network of the FP-E-GAN, MU-GAN, and DU-GAN. As expected, the model parameters of FP-E-GAN and MU-GAN are the same because the architecture of MU-GAN is similar to FP-E-GAN, just with an additional introduction of dropout. As a result, the model parameters are exactly the same as the baseline. In contrast, data uncertainty is learnt as a function of the input and requires the introduction of an additional branch in the model architecture. As a result, few additional parameters are introduced without significantly increasing the model complexity. Subsequently, DU-GAN has more parameters than the baseline FP-E-GAN. These results help us to conclude that model uncertainty estimation does not increase the model complexity while data uncertainty estimation increase it, but the increase is insignificant as compared to the baseline.
<table><caption>**Table 11**</caption><thead><tr><th>Enhancement Algorithm</th><th>Avg. NFIQ Score (↓)</th></tr></thead><tbody><tr><td>Raw Image</td><td>4.96</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>4.09</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td><strong>1.91</strong></td></tr><tr><td>DU-GAN</td><td>3.01</td></tr></tbody></table>

Comparison of average quality scores obtained using NFIQ on IIITD-MOLF database.

<table><caption>**Table 12**</caption><thead><tr><th>Enhancement Algorithm</th><th>Bozorth (↑)</th><th>MCC (↑)</th></tr></thead><tbody><tr><td>Raw Image</td><td>5.45</td><td>6.06</td></tr><tr><td>DeConvNet Schuch et al. (2016)</td><td>14.02</td><td>14.27</td></tr><tr><td>Svoboda et al. Svoboda et al. (2017)</td><td>N.A.</td><td>22.36</td></tr><tr><td>FP-E-GAN Joshi et al. (2019)</td><td><strong>28.52</strong></td><td><strong>34.43</strong></td></tr><tr><td>DU-GAN</td><td>23.16</td><td>27.21</td></tr></tbody></table>

Comparison of identification performance obtained on IIITD-MOLF database when matched across Lumidigm gallery.

<table><caption>**Table 13**</caption><thead><tr><th>Enhancement Model</th><th>Parameters</th></tr></thead><tbody><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>11376129</td></tr><tr><td>MU-GAN</td><td>11376129</td></tr><tr><td>DU-GAN</td><td>11386178</td></tr></tbody></table>

Comparison of model parameters introduced by model and data uncertainty estimation. The reported parameters are the total number of model parameters in the generator sub-network of the model. Model uncertainty estimation using Monte Carlo dropout does not introduce any additional model parameters while data uncertainty estimation introduces only a few additional parameters. These results indicate that uncertainty estimation does not significantly affect the model complexity.

<table><caption>**Table 14**</caption><thead><tr><th>Enhancement Model</th><th>Time (ms.)</th></tr></thead><tbody><tr><td>FP-E-GAN Joshi et al. (2019)</td><td>5.13</td></tr><tr><td>DU-GAN</td><td>5.87</td></tr><tr><td>MU-GAN</td><td>49.32</td></tr></tbody></table>

Comparison of inference time for estimating model and data uncertainty. Data uncertainty estimation requires the introduction of one more branch in the final layer of the baseline network. Therefore, inference time is not significantly increased. However, model uncertainty estimation using Monte Carlo dropout requires averaging of stochastic output obtained for each Monte Carlo sample. In this study, ten Monte Carlo are used. As a result, inference time of MU-GAN is about ten folds in contrast to FP-E-GAN.

### 6.2.2. Comparison of Inference Time

Next, we compare the inference time for model and data uncertainty estimation. For this, we compute the inference time of baseline FP-E-GAN, MU-GAN, and DU-GAN on a Tesla T4 GPU. The inference time is reported in Table 14. We find that the inference time after introducing data uncertainty is comparable to the baseline model. In contrast, the inference time is significantly increased after incorporating Monte Carlo dropout based model uncertainty. As the model uncertainty estimation requires averaging the stochastic outputs, thus the inference time is directly dependent on the number of samples using which the Monte Carlo integration is performed. For MU-GAN, we have used ten Monte Carlo samples. Subsequently, the inference time of MU-GAN is about ten folds in contrast to that of the baseline FP-E-GAN. On the other hand, inference time of DU-GAN is competitive to that of FP-E-GAN. These results indicate that the high computational time is required to estimate model uncertainty while data uncertainty can be estimated without significant increase in inference time.

## 6.3. Generalization on Fingerprint ROI Segmentation

This section investigates whether the usefulness of estimating model and data uncertainty generalizes to fingerprint preprocessing. In this direction, we study the effect of estimating model and data uncertainty in a fingerprint ROI
<figure><img src="image_18.png" /><figcaption>**Figure 15:** Analysis of the efficacy of the predicted model uncertainty. Model uncertainties are predicted for all the FVC Databases, from FVC 2000 DB1 (represented as D1) to FVC 2004 DB4 (represented as D12). A comparison of mean model uncertainties is provided for (a) foreground versus background pixels (b) correctly versus incorrectly classified pixels. Lower model uncertainty values are estimated for foreground and correctly classified pixels as compared to background and incorrectly classified pixels respectively. These results indicate the usefulness and efficacy of predicted model uncertainty.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>**Figure 16:** Analysis of the efficacy of the predicted data uncertainty. Data uncertainties are predicted for all the FVC Databases, from FVC 2000 DB1 (represented as D1) to FVC 2004 DB4 (represented as D12). A comparison of mean data uncertainties is provided for (a) foreground versus background pixels (b) correctly versus incorrectly classified pixels. Lower data uncertainty values are estimated for foreground and correctly classified pixels as compared to background and incorrectly classified pixels respectively. These results indicate the usefulness and efficacy of predicted data uncertainty.</figcaption></figure>

segmentation model. As motivated in Joshi, Utkarsh, Kothari, Kurmi, Dantcheva, Dutta Roy and Kalra (2021d), we work with Recurrent Unet (RUnet) as the baseline fingerprint ROI segmentation model. We introduce Monte Carlo dropout and data uncertainty estimation into RUnet, and the resulting fingerprint ROI segmentation models are termed as *MU-RUnet* and *DU-RUnet* respectively. We analyze both predicted uncertainty values and segmentation performance after introducing model and data uncertainty. The detailed analysis is presented next.

### 6.3.1. Predicted Uncertainty

Theoretically, model uncertainty should be high on pixels where a fingerprint ROI segmentation model is probable to predict an incorrect output. In contrast, data uncertainty, is expected to be high on noisy pixels. To analyze the uncertainty predicted on these pixels, we calculate the average uncertainty predicted for foreground as opposed to background pixels. We also compare the average predicted uncertainty for correctly classified pixels. The annotation of foreground and background is the same as provided in the ground truth annotations. Similarly, a pixel is regarded correctly or incorrectly classified based on manual annotations. Please note that a similar analysis of predicted uncertainty values is not conducted for fingerprint enhancement models due to the unavailability of ground
<figure><img src="image_20.png" /><figcaption>**Figure 17:** Sample cases illustrating the estimated model and data uncertainty. Results show that estimating both the kind of uncertainties promotes improved segmentation efficacy in contrast to baseline RUnet. Both these uncertainties provide different but complementary information. Model and data uncertainties capture similar information for a segmentation model as observed for an enhancement model. Here also model uncertainty signifies the confidence of the model in its output. As a result, lower uncertainty is attained on correctly segmented pixels as compared to incorrectly segmented pixels. Likewise, data uncertainty signifies noise in the input fingerprints. Therefore, lower uncertainty values are predicted for foreground pixels as compared to pixels in the boundaries and background.</figcaption></figure>

truth annotations. Figure 15 (a) compares predicted model uncertainty for foreground and background pixels. In general, background pixels are far more challenging for segmentation as compared to the foreground. Therefore, as expected, predicted model uncertainty is higher for background pixels in contrast to foreground pixels. Likewise, by definition, model uncertainty should be high for pixels that are likely to be incorrectly classified. As presented in Figure 15 (b), the predicted model uncertainty is significantly higher for incorrectly classified pixels in contrast to the correctly classified pixels. The obtained results demonstrate the efficacy of predicted model uncertainty.

Next, we analyze the predicted data uncertainty. Figure 16 (a) compares the predicted data uncertainty for foreground and background pixels. By definition, data uncertainty should be high for noisy pixels. Generally, background pixels in a fingerprint image are noisier than the foreground. Therefore, as expected, we observe that significantly lower data uncertainty is predicted for foreground pixels in contrast to the background pixels. Likewise, a fingerprint ROI segmentation model is more likely to be incorrectly classifying noisy pixels. In Figure 16 (b), we compare the predicted data uncertainty values for correctly classified and incorrectly classified pixels. We find that much lower data uncertainty values are output by the model for correctly classified pixels in contrast to incorrectly classified pixels which confirm the efficacy of predicted data uncertainty. Lastly, in Figure 17, we illustrate the sample cases demonstrating the segmentation obtained after introducing uncertainty estimation and predicted uncertainties. We find that segmentation performance improves after introducing either kind of uncertainty. These results also show that predicted model uncertainty is higher for pixels that are incorrectly segmented and low for correctly segmented pixels. Furthermore, higher data uncertainty is predicted for noisy background pixels as compared to the foreground. These results demonstrate that model and data uncertainties represent different and useful information for improving the performance of the fingerprint ROI segmentation model.
<table><caption>**Table 15**</caption><thead><tr><th rowspan="2">Database</th><th colspan="2">Jaccard Similarity (↑)</th><th colspan="2">Dice Score (↑)</th></tr><tr><th>RUnet</th><th>MU-RUnet</th><th>RUnet</th><th>MU-RUnet</th></tr></thead><tbody><tr><td>2000DB1</td><td><strong>88.15</strong></td><td>87.97</td><td><strong>93.34</strong></td><td>93.14</td></tr><tr><td>2000DB2</td><td>86.40</td><td><strong>88.43</strong></td><td>92.39</td><td><strong>93.58</strong></td></tr><tr><td>2000DB3</td><td>93.74</td><td><strong>95.39</strong></td><td>96.50</td><td><strong>97.57</strong></td></tr><tr><td>2000DB4</td><td>94.28</td><td><strong>94.89</strong></td><td>97.04</td><td><strong>97.36</strong></td></tr><tr><td>2002DB1</td><td><strong>96.95</strong></td><td>96.83</td><td><strong>98.44</strong></td><td>98.38</td></tr><tr><td>2002DB2</td><td>94.88</td><td><strong>95.13</strong></td><td>97.28</td><td><strong>97.40</strong></td></tr><tr><td>2002DB3</td><td>91.83</td><td><strong>93.87</strong></td><td>95.53</td><td><strong>96.73</strong></td></tr><tr><td>2002DB4</td><td>91.17</td><td><strong>91.53</strong></td><td>95.32</td><td><strong>95.54</strong></td></tr><tr><td>2004DB1</td><td>98.78</td><td><strong>98.98</strong></td><td>99.38</td><td><strong>99.49</strong></td></tr><tr><td>2004DB2</td><td>93.94</td><td><strong>95.98</strong></td><td>96.69</td><td><strong>97.93</strong></td></tr><tr><td>2004DB3</td><td>94.62</td><td><strong>95.29</strong></td><td>97.17</td><td><strong>97.55</strong></td></tr><tr><td>2004DB4</td><td>94.73</td><td><strong>96.18</strong></td><td>97.21</td><td><strong>98.03</strong></td></tr></tbody></table>

Quantification of improved segmentation performance obtained after incorporating Monte Carlo dropout. MU-RUnet attains higher Jaccard similarity and Dice scores in contrast to RUnet which highlight the improvement after estimating model uncertainty.

### 6.3.2. Segmentation Performance

We also assess the fingerprint ROI segmentation performance obtained after introducing the model and data uncertainty into the baseline fingerprint ROI segmentation model. To quantify the segmentation ability, we compute standard segmentation metrics: Jaccard Similarity Choi, Cha and Tappert (2010) and Dice score Dice (1945). Table 15 compares the segmentation ability of MU-RUnet as opposed to RUnet. We find that both Dice score and Jaccard similarity values have significantly improved after introducing model uncertainty. Figure 18 presents the sample cases illustrating the improved efficacy of segmentation attained by MU-RUnet in contrast to RUnet. The improved efficacy of segmentation attained by MU-RUnet is attributed to the model averaging effect introduced by Monte Carlo dropout. This averaging effect reduces model over-fitting for MU-RUnet. This justifies the improvement in segmentation efficacy.

At last, we evaluate the segmentation performance obtained after introducing data uncertainty. Table 16 compares the Jaccard similarity and Dice scores attained by DU-RUnet in contrast to RUnet. Higher values for both the evaluation metrics are attained by DU-RUnet as opposed to RUnet. Figure 19 presents sample cases that highlight that improved segmentation performance is obtained after the introduction of data uncertainty into the baseline fingerprint ROI segmentation model. The improved performance obtained after introducing data uncertainty is attributed to the fact that data uncertainty estimation enables DU-RUnet to identify noisy pixels. As a result, it makes better predictions on noisy pixels, which results in overall improved segmentation performance.

## 7. Conclusion

In this chapter, we share details on uncertainty estimation for fingerprint enhancement models. We discuss estimating both model and data uncertainty for a fingerprint enhancement model. Through extensive experimentation, we find that modeling both kinds of uncertainties is useful. Both these uncertainties capture different but complementary information, which helps to improve the enhancement performance. We also demonstrate that the proposed uncertainty estimation techniques generalize on fingerprint ROI segmentation. Analysis of predicted uncertainty is performed, which indicates the efficacy and usefulness of predicted uncertainty values. The model complexity is analyzed, and we find that model uncertainty estimation does not increase the model complexity at all while data uncertainty estimation adds only a few parameters, which are insignificant compared to the model complexity of the baseline fingerprint enhancement model. We also compare the inference time for model and data uncertainty and observe that model uncertainty estimation significantly increases the inference time by a factor of the number of Monte Carlo samples. In contrast, the inference time for data uncertainty estimation is comparable to that of the baseline fingerprint enhancement model. The discussions presented in this chapter show the efficacy and usefulness of uncertainty estimation as an interpretability mechanism for a fingerprint enhancement model. We understand that model interpretability is a desired characteristic for all the components of an automated fingerprint recognition system. In the future, the techniques
<figure><img src="image_21.png" /><figcaption>**Figure 18:** Sample cases illustrating the improvement in the segmentation performance after introduction of Monte Carlo dropout uncertainty into the baseline RUnet architecture.</figcaption></figure>

<table><thead><tr><th rowspan="2">Database</th><th colspan="2">Jaccard Similarity (↑)</th><th colspan="2">Dice Score (↑)</th></tr><tr><th>RUnet</th><th>DU-RUnet</th><th>RUnet</th><th>DU-RUnet</th></tr></thead><tbody><tr><td>2000DB1</td><td>88.15</td><td><strong>88.52</strong></td><td>93.34</td><td><strong>93.62</strong></td></tr><tr><td>2000DB2</td><td>86.40</td><td><strong>88.07</strong></td><td>92.39</td><td><strong>93.42</strong></td></tr><tr><td>2000DB3</td><td>93.74</td><td><strong>95.36</strong></td><td>96.50</td><td><strong>97.55</strong></td></tr><tr><td>2000DB4</td><td>94.28</td><td><strong>94.97</strong></td><td>97.04</td><td><strong>97.40</strong></td></tr><tr><td>2002DB1</td><td>96.95</td><td><strong>97.07</strong></td><td>98.44</td><td><strong>98.50</strong></td></tr><tr><td>2002DB2</td><td>94.88</td><td><strong>95.43</strong></td><td>97.28</td><td><strong>97.60</strong></td></tr><tr><td>2002DB3</td><td>91.83</td><td><strong>93.06</strong></td><td>95.53</td><td><strong>96.25</strong></td></tr><tr><td>2002DB4</td><td>91.17</td><td><strong>91.89</strong></td><td>95.32</td><td><strong>95.74</strong></td></tr><tr><td>2004DB1</td><td>98.78</td><td><strong>99.00</strong></td><td>99.38</td><td><strong>99.50</strong></td></tr><tr><td>2004DB2</td><td>93.94</td><td><strong>96.37</strong></td><td>96.69</td><td><strong>98.14</strong></td></tr><tr><td>2004DB3</td><td>94.62</td><td><strong>95.47</strong></td><td>97.17</td><td><strong>97.65</strong></td></tr><tr><td>2004DB4</td><td>94.73</td><td><strong>95.61</strong></td><td>97.21</td><td><strong>97.70</strong></td></tr></tbody></table>

**Table 16**

Quantification of improved segmentation performance obtained after estimating data uncertainty. Higher Jaccard similarity and Dice scores are attained by DU-RUnet in contrast to RUnet which highlight the improvement after estimating data uncertainty.

presented in this chapter can be extended to other modules of a fingerprint matching system as well.

## 8. Acknowledgement

The authors acknowledge the support from the HPC services of Inria Sophia Antipolis and IIT Delhi for the computational resources used in this research. The authors sincerely thank Prof. Phalguni Gupta from IIT Kanpur and Prof. Kamlesh Tiwari from BITS Pilani for sharing the private rural Indian fingerprints database used in this research.

## References

* Achrack, O., Kellerman, R., Barzilay, O., 2020. Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation. arXiv preprint arXiv:2010.01917.

* A.H.Ansari, 2011. Generation and Storage of Large Synthetic Fingerprint Database. Technical Report. Indian Institute of Science Bangalore.

* Cao, K., Jain, A.K., 2015. Latent Orientation Field Estimation via Convolutional Neural Network, in: Proc. International Conference on Biometrics (ICB), pp. 349 – 356.
<figure><img src="image_22.png" /><figcaption>**Figure 19:** Sample cases illustrating the improvement in the segmentation performance after introduction of data uncertainty estimation into the baseline RUnet architecture.</figcaption></figure>

Cappelli, R., Ferrara, M., Maltoni, D., 2010a. Fingerprint Indexing Based on Minutia Cylinder-Code. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 1051–1057.

Cappelli, R., Ferrara, M., Maltoni, D., 2010b. Minutia Cylinder-Code: A New Representation and Matching Technique for Fingerprint Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 2128–2141.

Cappelli, R., Maio, D., Maltoni, D., 2004. Sfinge: An Approach to Synthetic Fingerprint Generation, in: Proc. International Workshop on Biometric Technologies, pp. 147–154.

Chaidee, W., Horapong, K., Areekul, V., 2018. Filter Design Based on Spectral Dictionary for Latent Fingerprint Pre-Enhancement, in: Proc. International Conference on Biometrics (ICB), pp. 23–30.

Chen, C., Feng, J., Zhou, J., 2016. Multi-Scale Dictionaries Based Fingerprint Orientation Field Estimation, in: Proc. International Conference on Biometrics (ICB), pp. 1–8.

Chikkerur, S., Cartwright, A.N., Govindaraju, V., 2007. Fingerprint Enhancement using STFT Analysis. Pattern Recognition 40, 198–211.

Choi, S.S., Cha, S.H., Tappert, C.C., 2010. A Survey of Binary Similarity and Distance Measures. Journal of Systemics, Cybernetics and Informatics 8, 43–48.

Dice, L.R., 1945. Measures of the Amount of Ecologic Association Between Species. Ecology 26, 297–302.

Feng, J., Zhou, J., Jain, A.K., 2013. Orientation Field Estimation for Latent Fingerprint Enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 925–940.

Ferrara, M., Maltoni, D., Cappelli, R., 2012. Noninvertible Minutia Cylinder-Code Representation. IEEE Transactions on Information Forensics and Security 7, 1727–1737.

Fort, S., Hu, H., Lakshminarayanan, B., 2019. Deep Ensembles: A Loss Landscape Perspective. arXiv preprint arXiv:1912.02757.

Gal, Y., 2016. Uncertainty in Deep Learning. Ph.D. thesis. University of Cambridge.

Gal, Y., Ghahramani, Z., 2016. Dropout As a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, in: Proc. International Conference on Machine Learning (ICML), pp. 1050–1059.

Gal, Y., Hron, J., Kendall, A., 2017. Concrete dropout, in: NIPS.

Gawlikowski, J., Tassi, C.R.N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al., 2021. A Survey of Uncertainty in Deep Neural Networks. arXiv preprint arXiv:2107.03342.

Ghafoor, M., Taj, I.A., Ahmad, W., Jafri, N.M., 2014. Efficient 2-Fold Contextual Filtering Approach for Fingerprint Enhancement. IET Image Processing 8, 417–425.

Gottschlich, C., 2011. Curved-Region-Based Ridge Frequency Estimation and Curved Gabor Filters for Fingerprint Image Enhancement. IEEE Transactions on Image Processing 21, 2220–2227.

Gottschlich, C., Schönlieb, C.B., 2012. Oriented Diffusion Filtering for Enhancing Low-Quality Fingerprint Images. IET Biometrics 1, 105–113.

Gustafsson, F.K., Danelljan, M., Schon, T.B., 2020. Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision, in: Proc. IEEE International Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 318–319.

Hong, L., Wan, Y., Jain, A., 1998. Fingerprint Image Enhancement: Algorithm and Performance Evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 777–789.

Hsieh, C.T., Lai, E., Wang, Y.C., 2003. An Effective Algorithm for Fingerprint Image Enhancement Based on Wavelet Transform. Pattern Recognition 36, 303–312.

Jirachaweng, S., Areekul, V., 2007. Fingerprint Enhancement Based on Discrete Cosine Transform, in: Proc. International Conference on Biometrics (ICB), pp. 96–105.

Joshi, I., Anand, A., Dutta Roy, S., Kalra, P.K., 2021a. On Training Generative Adversarial Network for Enhancement of Latent Fingerprints, in:
AI and Deep Learning in Biometric Security, pp. 51–79.

Joshi, I., Anand, A., Vatsa, M., Singh, R., Dutta Roy, S., Kalra, P., 2019. Latent Fingerprint Enhancement using Generative Adversarial Networks, in: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 895–903.

Joshi, I., Kothari, R., Utkarsh, A., Kurmi, V.K., Dantcheva, A., Dutta Roy, S., Kalra, P.K., 2021b. Explainable Fingerprint ROI Segmentation using Monte Carlo Dropout, in: IEEE Winter Conference on Applications of Computer Vision Workshops (WACVW), pp. 60–69.

Joshi, I., Utkarsh, A., Kothari, R., Kurmi, V.K., Dantcheva, A., Dutta Roy, S., Kalra, P.K., 2021c. Data Uncertainty Guided Noise-Aware Preprocessing of Fingerprints, in: International Joint Conference on Neural Networks (IJCNN), pp. 1–8.

Joshi, I., Utkarsh, A., Kothari, R., Kurmi, V.K., Dantcheva, A., Dutta Roy, S., Kalra, P.K., 2021d. Sensor-Invariant Fingerprint ROI Segmentation using Recurrent Adversarial Learning, in: International Joint Conference on Neural Networks (IJCNN), pp. 1–8.

Joshi, I., Utkarsh, A., Singh, P., Dantcheva, A., Dutta Roy, S., Kalra, P.K., 2022 [accepted]. On Restoration of Degraded Fingerprints. Multimedia Tools and Applications.

Kendall, A., Gal, Y., 2017. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, in: Proc. Advances in Neural Information Processing Systems (NIPS), pp. 5574–5584.

Kim, I., Kim, Y., Kim, S., 2020. Learning Loss for Test-Time augmentation. arXiv preprint arXiv:2010.11422.

Lakshminarayanan, B., Pritzel, A., Blundell, C., 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems 30.

Lee, J., AlRegib, G., 2020. Gradients As a Measure of Uncertainty in Neural Networks, in: Proc. IEEE International Conference on Image Processing (ICIP), pp. 2416–2420.

Li, J., Feng, J., Kuo, C.C.J., 2018. Deep Convolutional Neural Network for Latent Fingerprint Enhancement. Signal Processing: Image Communication 60, 52–63.

Liu, S., Liu, M., Yang, Z., 2017. Sparse Coding Based Orientation Estimation for Latent Fingerprints. Pattern Recognition 67, 164–176.

Lyzhov, A., Molchanova, Y., Ashukha, A., Molchanov, D., Vetrov, D., 2020. Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation, in: Proc. Conference on Uncertainty in Artificial Intelligence (UAI), pp. 1308–1317.

Malinin, A., 2019. Uncertainty Estimation in Deep Learning with Application to Spoken Language Assessment. Ph.D. thesis. University of Cambridge.

Malinin, A., Gales, M., 2018. Predictive uncertainty estimation via prior networks, in: Proc. Advances in Neural Information Processing Systems (NIPS), pp. 7047–7058.

Możejko, M., Susik, M., Karczewski, R., 2018. Inhibited Softmax for Uncertainty Estimation in Neural Networks. arXiv preprint arXiv:1810.01861.

Mukhoti, J., Gal, Y., 2018. Evaluating Bayesian Deep Learning Methods for Semantic Segmentation. arXiv preprint arXiv:1811.12709.

Ndajah, P., Kikuchi, H., Yukawa, M., Watanabe, H., Muramatsu, S., 2011. An Investigation on the Quality of Denoised Images. International Journal of Circuit, Systems, and Signal Processing 5, 423–434.

NIST, . NBIS- NIST Biometric Image Software. http://biometrics.idealtest.org/.

Oberdiek, P., Rottmann, M., Gottschalk, H., 2018. Classification Uncertainty of Deep Neural Networks Based on Gradient Information, in: Proc. IAPR Workshop on Artificial Neural Networks in Pattern Recognition, pp. 113–125.

Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., Snoek, J., 2019. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift, in: Proc. Advances in Neural Information Processing Systems (NIPS), pp. 13991–14002.

Puri, C., Narang, K., Tiwari, A., Vatsa, M., Singh, R., 2010. On Analysis of Rural and Urban Indian Fingerprint Images, in: Proc. International Conference on Ethics and Policy of Biometrics, pp. 55–61.

Qian, P., Li, A., Liu, M., 2019. Latent Fingerprint Enhancement Based on DenseUNet, in: Proc. International Conference on Biometrics (ICB), pp. 1–6.

Qu, Z., Liu, J., Liu, Y., Guan, Q., Yang, C., Zhang, Y., 2018. Orienet: A Regression System for Latent Fingerprint Orientation Field Extraction, in: Proc. International Conference on Artificial Neural Networks, pp. 436–446.

Raghu, M., Blumer, K., Sayres, R., Obermeyer, Z., Kleinberg, B., Mullainathan, S., Kleinberg, J., 2019. Direct Uncertainty Prediction for Medical Second Opinions, in: Proc. International Conference on Machine Learning (ICML), pp. 5281–5290.

Rama, R.K., Namboodiri, A.M., 2011. Fingerprint Enhancement using Hierarchical Markov Random Fields, in: Proc. IEEE International Joint Conference on Biometrics (IJCB), pp. 1–8.

Ramalho, T., Miranda, M., 2020. Density Estimation in Representation Space to Predict Model Uncertainty, in: Proc. International Workshop on Engineering Dependable and Secure Machine Learning Systems, pp. 84–96.

Ramos, R.C., de Lima Borges, E.V.C., Andrezza, I.L.P., Primo, J.J.B., Batista, L.V., Gomes, H.M., 2018. Analysis and Improvements of Fingerprint Enhancement from Gabor Iterative Filtering, in: SIBGRAPI Conference on Graphics, Patterns and Images, pp. 266–273.

Ren, J., Liu, P.J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., Dillon, J., Lakshminarayanan, B., 2019. Likelihood ratios for out-of-distribution detection. Advances in Neural Information Processing Systems 32, 14707–14718.

Sahasrabudhe, M., Namboodiri, A.M., 2014. Fingerprint Enhancement using Unsupervised Hierarchical Feature Learning, in: Proc. IAPR- and ACM-sponsored Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), pp. 1–8.

Sankaran, A., Vatsa, M., Singh, R., 2015. Multisensor Optical and Latent Fingerprint Database. IEEE Access 3, 653–665.

Schuch, P., Schulz, S., Busch, C., 2016. De-Convolutional Auto-encoder for Enhancement of Fingerprint Samples, in: Proc. International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1–7.

Schuch, P., Schulz, S., Busch, C., 2017. Survey on the Impact of Fingerprint Image Enhancement. IET Biometrics, 102–115.

Sensoy, M., Kaplan, L., Kandemir, M., 2018. Evidential deep learning to quantify classification uncertainty. Advances in Neural Information Processing Systems 31, 3179–3189.

Shanmugam, D., Blalock, D., Balakrishnan, G., Guttag, J., 2020. When and Why Test-Time Augmentation Works. arXiv preprint arXiv:2011.11156.
Svoboda, J., Monti, F., Bronstein, M.M., 2017. Generative Convolutional Networks for Latent Fingerprint Reconstruction, in: Proc. IEEE International Joint Conference on Biometrics (IJCB), pp. 429–436.

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R., 2014. Intriguing properties of neural networks, in: 2nd International Conference on Learning Representations, ICLR 2014.

Thai, D.H., Gottschlich, C., 2016. Global Variational Method for Fingerprint Segmentation by Three-Part Decomposition. IET Biometrics 5, 120–130.

Turroni, F., Cappelli, R., Maltoni, D., 2012. Fingerprint Enhancement using Contextual Iterative Filtering, in: Proc. International Conference on Biometrics (ICB), pp. 152–157.

Valdenegro-Toro, M., 2019. Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classification. arXiv preprint arXiv:1910.08168.

Van Amersfoort, J., Smith, L., Teh, Y.W., Gal, Y., 2020. Uncertainty Estimation using a Single Deep Deterministic Neural Network, in: Proc. International Conference on Machine Learning (ICML), pp. 9690–9700.

Vyas, A., Jammalamadaka, N., Zhu, X., Das, D., Kaul, B., Willke, T.L., 2018. Out-of-Distribution Detection using An Ensemble of Self Supervised Leave-Out Classifiers, in: Proc. European conference on computer vision (ECCV), pp. 550–564.

Wang, W., Li, J., Huang, F., Feng, H., 2008. Design and Implementation of Log-Gabor Filter in Fingerprint Image Enhancement. Pattern Recognition Letters 29, 301–308.

Wen, Y., Tran, D., Ba, J., 2019. Batchensemble: an alternative approach to efficient ensemble and lifelong learning, in: International Conference on Learning Representations.

Wong, W.J., Lai, S.H., 2020. Multi-Task CNN for Restoring Corrupted Fingerprint Images. Pattern Recognition 101, 107203–107213.

Yang, X., Feng, J., Zhou, J., 2014. Localized Dictionaries Based Orientation Field Estimation for Latent Fingerprints. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 955–969.

Yoon, S., Feng, J., Jain, A.K., 2010. On Latent Fingerprint Enhancement, in: Biometric Technology for Human Identification VII, pp. 766707–766716.