1

Facial Deblur Inference using Subspace Analysis
for Recognition of Blurred Faces
Masashi Nishiyama, Abdenour Hadid, Hidenori Takeshima, Jamie Shotton, Tatsuo Kozakaya, Osamu Yamaguchi

Abstract— This paper proposes a novel method for recognizing
faces degraded by blur using deblurring of facial images. The
main issue is how to infer a Point Spread Function (PSF)
representing the process of blur on faces. Inferring a PSF froma single facial image is an ill-posed problem. Our method uses
learned prior information derived from a training set of blurred
faces to make the problem more tractable. We construct a feature
space such that blurred faces degraded by the same PSF are
similar to one another. We learn statistical models that represent
prior knowledge of predefined PSF sets in this feature space. A
query image of unknown blur is compared with each model andthe closest one is selected for PSF inference. The query image
is deblurred using the PSF corresponding to that model, andthus ready for recognition. Experiments on a large face database(FERET) artificially degraded by focus or motion blur show that
our method substantially improves the recognition performancecompared to existing methods. We also demonstrate improved
performance on real blurred images on FRGC 1.0 face database.
Furthermore, we show and explain how combining the proposedfacial deblur inference with the local phase quantization (LPQ)
method can further enhance the performance.
Index Terms— Face Recognition, Inference, Point Spread Function, Deblur

I. I NTRODUCTIOND

UE to its many potential applications, face recognition has
become one of the most active topics in computer visionresearch [1]. However, despite the significant progress in the last
decade, the design of recognition algorithms that are effective
over a wide range of viewpoints, occlusions, aging of subjects andcomplex outdoor lighting is still a major area of research. While
there is a significant number of works addressing these issues,
problems caused by image degradations due to other factors suchas blur, noise and sampling are mostly overlooked. This is particularly surprising as such image degradations also significantly
affect the performance of face recognition systems and are often
present in images and videos in real-world applications such as
watch-list monitoring and video surveillance. Only recently has
research community started to look at facial image degradations
e.g. through facial denoising [2]. The focus of this paper istherefore coping with blur and, in particular, automatic deblurringof face images for enhancing the recognition performance.
Blur affects the appearance of faces in images, causing two
main problems for face recognition: (i) the facial appearance ofan individual changes drastically due to blur as Figure 1 (a)and (b) depict; and (ii) different individuals tend to appear
more similar when blurred (for example, the difference in theappearance between the two blurred faces in Figure 1 (b) and (d)is much smaller than that between the corresponding sharp faces
in Figure 1 (a) and (c)).
A few existing methods attempt to handle these problems.
However, they are not yet satisfactory when facing the significant
amount of blur that is common in many real-world settings.

Person A(a) Sharp(b) Blurred

Feature spacefor
PSF inference

Person B(c) Sharp(d) Blurred

Set of PSF2
Set of PSF1

Fig. 1. Variation in facial appearance caused by blur. We construct a new
frequency magnitude-based feature space in which faces with similar level ofblur are close to each other, regardless of identity. These sets are used to find
an appropriate PSF to deblur query images for accurate face recognition.

For instance, Stainvas & Intrator [3] match a query image toartificially blurred copies of the original sharp target images
registered for identification. The method can alleviate the firstproblem of dissimilarity caused by blur, but the second problem
of similarity remains. Moreover, the target images may alreadybe blurred themselves. Our approach is based on removing theblur from facial appearances using blind image deconvolution [4].
The deblurred images can then be used to perform more robust
recognition. Obviously, such an approach can solve both problems(i) and (ii) simultaneously, but requires a Point Spread Function(PSF) that represents the blurring process.
In the field of blind image deconvolution, many methods havebeen proposed for deblurring from a single image [5]. For instance, Chan and Wong [6] simultaneously infer a PSF and deblur
an image using total variation regularization. Other methods
attempt to model the smoothness of intensity changes aroundedges. A PSF is inferred using information derived from thissmoothness using the variation of Gaussian scale [7] [8], wavelet
coefficient [9] [10], the summation of image derivatives [11], or
alpha values representing the object boundary transparency [12].
These methods have to solve an ill-posed problem because they do
not adequately exploit the prior knowledge of the image content,
and so the quality of the deblurred image using an inferred PSF is
often poor. As our experiments will reveal, the deblurred images
using these methods are insufficient for accurate face recognition.
Yuan et al. [13] and Ancuti et al. [14] infer a PSF using multiple
images captured from the same scene. This setting limits facerecognition applications. Fergus et al. [15] infer a PSF using
heavy-tailed natural image priors, but these priors are very genericand so fairly weak, and the PSF inference is computationallyexpensive. It appears then that the majority of previous methods
for blind image deconvolution use the smoothness of intensitychanges around edges to infer the PSF from a single image
without prior knowledge of the image contents. These methods

2thus often infer a poor quality PSF as it is difficult to distinguishbetween blurred edges and smooth object surfaces.
Our idea is to exploit prior knowledge of how facial appearances are changed by blur. We thus introduce a new method called
FAcial DEblur INference (FADEIN) for inferring PSFs using
learned statistical models of the variation in facial appearance
caused by blur. In our method, we define a representative set of
PSFs for a particular application during training. We artificiallyblur the sharp training images for each PSF in the set and use themto build a model of facial appearance under that PSF. The method
infers a PSF by comparing the query image (with unknown
identity and amount of blur) to each model during testing.

Training
imagesPSF
H1

HN

Blurred
image

PSF
Estimation
1. Compare
φ 1 , H1

*
*
PCA

PCA

φ1
φN
Variation inappearance(a) Learning

Restoration

φ N , HN

2. Select
HsIdentification(b) Recognition

Fig. 2. FADEIN comprises two steps: (a) learning statistical models of facial
blur appearance, and (b) using these models to recognize individuals in query
images of unknown blur.

A. Overview of FADEIN
We briefly describe the overview of our FADEIN algorithm.
We map blurred images to a feature space for learning statisticalmodels. Simply vectorizing the images does not generalize wellfor the feature space, since the vectorized images are not clearly
separated and yields in poor performance as our experiments
in Section III show. We instead design a new ‘magnitude-based
feature space’ in the frequency domain, as illustrated in Figure 1,
in which the variation of facial appearance caused by blur islarger than the variation of facial appearance between individuals.
Faces blurred by the same PSF are similar in the new feature
space. We learn the statistical models by approximating each set
as a low-dimensional linear subspace using principal component
analysis (PCA). We compare query images to each subspace
during testing. The most similar subspace gives an accurateinferred PSF that is used to deblur the query image. The resulted
image can then be fed to a standard face recognition algorithm.
Our extensive experiments on real and artificially blurred face
images show high PSF inference and significant face recognition
performance improvements in comparison to existing methods.
B. Extension of FADEIN
The preliminary results of this research have been publishedin [16]. In this article, we provide improved results including amore thorough investigation and extended experimental evaluationof the algorithm.
We also take the following method into consideration. Recently,
Ahonen et al. [17] proposed another interesting approach tothe problem of recognizing blurred faces. Instead of performing
deblurring, the authors aimed to directly derive blur invariant
features from the original face images using the phase information in the frequency domain. The method, called Local Phase
Quantization (LPQ), showed promising results. In our extended
evaluation, we compare our proposed facial deblur inference
against the LPQ approach and demonstrate the superiority of our
proposed approach. Furthermore, we describe a combined scheme
exploiting the advantages of both methods for further enhancing
the face recognition performance.
The rest of this paper is organized as follows. Section II
describes our method for PSF inference, and Section III demonstrates its effectiveness through extensive experiments and analysis. We summarize our contributions and conclude in Section IV.
II. FACIAL DEBLUR INFERENCE
In this section we present our new approach (FADEIN) for
inferring PSFs using learned statistical models of the variation in

No blurHard blurFig. 3. Examples of detected faces from blurred images using the facedetector proposed in [18].

facial appearance caused by blur. The degradation process causedby blur is defined asg = Hf + n ,(1)

where vector g represents the blurred image g(u, v), matrix H the
PSF, vector f the original sharp image, and vector n the noise.
Equation (1) represents an explicit appearance model for blur.
Note that g and f consist of only the facial region given by aface detector. Blur is not a major problem to face detection as theexisting method [18] can detect small faces even at 15×15 pixels.
Figure 3 shows facial regions detected from blurred images.
If the PSF H and noise n are known, a sharp image f can beexactly recovered from the blurred image g. Our work focuseson accurately inferring a PSF H from a blurred facial imageg, though we also compare in our experiments how, given ourinferred PSF, two existing deblurring methods cope with the noise.
As illustrated in Figure 2, our approach consists of first learning
statistical models of facial blur appearance and then using thesemodels to recognize individuals in query images of unknown blur.
Let us assume that a representative fixed set Ω = {Hi }N
i=1
of N PSFs are constituted as described in [19]. These PSFs arechosen for a particular application. We represent statistical models
for PSF inference as subspaces in the frequency magnitude-based
feature space (see Section II-A). We denote a set of statisticalmodels asΦ = { (Hi , φi ) }N
(2)
i=1 ,
where φi represents the subspace modeling the variation in facial
appearance induced by PSF Hi . We learn subspaces φi from a setof M training images Ψ = {fk }M
k=1 . A training image fk is a sharp
image artificially blurred by each Hi to learn the subspace φi(see Section II-B). Note that the subjects in Ψ can be completely
different to ones for identification. This means that the training
images for PSF inference do not depend on the target images
registered for identification. The method we are proposing in thisarticle not only overcomes both problems cited in Section I but

3

g

|g|

log |g|u(a) Raw pixel feature space

0

ξ

(b)

0

ξ

(c) Magnitude-based feature space

Fig. 4. Comparison of feature vectors. The blue curves are feature vectors generated from the top-left sharp image and the red curves are ones generated
from the bottom-left blurred image. The average of images of 200 persons is shown. The vectors in our feature space (c) are much more clearly separated
than in (a) using raw pixels. (a) One scan-line through a simple vectorization of the image. (b) One row of the 2D Fourier transform of the images. (c) The
same row after taking logarithms. The transformations create our feature space. The horizontal axis of (a) represents dashed lines in the face images. The onesof (b), (c) represent frequency on dashed lines in the Fourier domain, with low frequency in the center and higher frequencies further away from the center.

also needs only a single target example and works even for reallife capture settings where the target images are blurred.
A query image g of unknown blur and identity is comparedto each subspace using the subspace method [20]. This method
is easy to implement and gives an accurate and stable similarity
measure (see Section II-C). The closest subspace φs is selected
and the corresponding PSF Hs is the result of our PSF inference.
The query image g is deblurred using Hs (see Section II-D), andfinally identification of the deblurred image is performed.
A. Frequency magnitude-based feature space
We construct a feature space that is sensitive to the appearance
variations of different blurs but insensitive to the difference
between individuals. We base the feature space on frequency domain amplitude, because of the phenomenon that high-frequency
amplitudes for blurred images become smaller than those forsharp images [21]. Since we want a feature space invariant to
identity, we deliberately ignore phase information (note that,without blur, phase information is, of course, a good feature forface recognition [22]).
Inferring a PSF using frequency domain amplitude is a wellknown technique [19], [23]–[25]. The method in [19] selects
a PSF from a representative fixed set Ω using amplitude of a
sharp image and noise. But, it is difficult to correctly estimate
these amplitudes from the blurred image because the sharp
image is unknown. Other methods [23]–[25] use the relationshipbetween PSF parameters and the positions of the zero crossingsof amplitude. However, detecting these positions is difficult forreal blurred images because the zero crossings are affected by
the noise amplitude. Our method instead uses the whole frequency
amplitude domain for constructing the feature space. The variation
caused by noise is included in the subspaces φi learned in our
feature space by adding noise to training images (see Section IIB).
We emphasize discriminative high-frequency amplitude to improve the PSF inference accuracy. Our method first transforms
a blurred image g(u, v) in the space domain to a feature imagex(ξ  , η  ) asx(ξ  , η  ) = [ C(|g(ξ, η)|) ] ↓ ,(3)where g(ξ, η) is the 2D Fourier transform of g(u, v), | | takes theamplitude, C takes dynamic range compression, and [ ] ↓ represents downsampling. Dynamic range compression is performed

to emphasize high-frequency values because non-discriminative
low-frequency values are much larger than discriminative highfrequency values. For the function C , we simply take logarithms
that shift from a large value to a small value while maintaining
order relation. Downsampling helps reduce noise and accelerate
subspace learning. Bilinear interpolation is used for downsampling. We choose the procedures C and [ ] ↓ from the viewpointsof computation time and inference accuracy. The feature vector x
is generated by raster scanning x(ξ , η  ). Finally, as preprocessing
for the subspace method, vector x is normalized so that x2 = 1.
Figure 4 compares feature vectors generated from a sharp
image and its blurred version. The vectors in Figure 4 (a) to (d)are the outputs of each step in Equation (3) and the normalization.
Note that the waveforms of the vectors in Figure 4 (a) are almost
the same, but the difference between sharp and blurred images
appears most cleanly in our feature space in Figure 4 (d).
Further, Figure 5 uses PCA to visualize two feature spaces containing images blurred by three different PSFs. Raw pixel based
feature space (center) directly vectorizes the blurred facial images.
Observe that our frequency magnitude-based feature space (right)
gives sets that are much better separated. Our experiments below
compare these two feature spaces and confirm that our feature
space gives more accurate PSF inference and face recognition.

B. Learning subspaces
We generate subspaces for the statistical models to represent
sets in the feature space. Training images are blurred by applying
each PSF Hi to sharp training images fk in a training set Ψ as
gik
= Hi fk + n , where n is the noise that we assume to bewhite Gaussian.
 M
Next, we apply PCA to blurred training images {gik}k=1 intoour feature space, as follows. Having transformed image gik
feature vector xik , the correlation matrix [26] is defined as Ai =

 T
 M
1 M
M
k=1 xik (xik ) . Note that the mean vector of {xik }k=1
is not subtracted for the correlation matrix because our implementation of the subspace method uses an angle for similarity
measure (see Equation (4)). Eigenvectors and eigenvalues of thecorrelation matrix Ai are calculated. The first D eigenvectors,
sorted by decreasing eigenvalue, form the basis vectors of thesubspace φi = {bij }D
j=1 . The low-dimensional subspace spannedby the basis vectors φi is defined by maximizing the variance of
training feature vectors {x ik }M
k=1 in this subspace.

Third eigen axis

Third eigen axis

4Second eigen axis

Second eigen axis

First eigen axis

First eigen axis

Raw pixel feature space

Magnitude-based feature space

Fig. 5. Visualization of sets in two feature spaces. Artificially blurred facial images (left) are vectorized either directly (center, raw pixels) or using thefrequency domain amplitude (right, our method). Colors represent the particular PSFs used to blur the faces. Our evaluation highlights the importance of themuch cleaner separation of the PSF sets using our method.

C. Determining the PSF
Given a blurred query image g of unknown blur, we find theclosest subspace φs in a set of statistical models Φ ass==

arg max cos2 θiiarg maxiD(bij T x)2 ,(4)(a) No blur(b) σ = 2(c) σ = 4(d) σ = 6(e) σ = 8

Fig. 6. Examples of synthesized images of camera focus blur from the subset‘fb’ of FERET. Blurred images (b), (c), (d), and (e) are synthesized from theoriginal image (a) using Gaussian PSFs with given standard deviations σ.

j=1

where x is the feature vector of g, θi is the angle between vector
x and subspace φi , and i ∈ {1, . . . , N }. A small angle θi indicatesthat the blurred query image g is similar to a set of the training
images blurred by PSF Hi that generated the subspace. The
maximization of cos2 θi over i thus infers the most appropriate
PSF Hs .
D. Restoration
Given PSF Hs , we can then remove blur from the degraded
query image g. Simple matrix inverse-based deconvolutions givea poor quality result because of existing noise n in Equation(1). We thus use the BTV regularization from [27]. The detail of
BTV is described in the Appendix of [16]. Deconvolution using
the BTV regularization recovers not only a sharp image but alsoa noiseless image.
III. E XPERIMENTAL ANALYSIS ON FACE RECOGNITION USING
FACIAL DEBLUR INFERENCE

To demonstrate the effectiveness of FADEIN in recognizing
blurred (and also sharp) faces, we performed extensive experiments on two publicly available databases, namely FERET [28]and FRGC 1.0 [29]. We investigate camera focus blur by artificially degrading sharp images. Camera focus blur arises when thecamera focal length is not correctly adjusted to the subject. We
also investigate camera motion blur that arises when the camera
is moved while capturing an image. Furthermore, we test our
method on real blurred images. In the experiments with artificial
blur, only the face regions are blurred while in the experimentswith real blurred images, the entire images are blurred andautomatic face detection is used for detecting the facial regions.
Facial deblur inference can be used as preprocessing before
feature extraction for face recognition. Thus, it can be usedwith any traditional face recognition algorithm. In our previous
work [16], we evaluated the performance using only raw pixel
values as features which are computed by simply raster scanningthe deblurred image. For extensive experimentation, this paper
considers baseline face recognition techniques for feature extraction from the deblurred images. We use the local binary pattern‘LBP’ [30], the eigenfaces ‘EF’ [31], and the laplacianfaces‘LF’ [32]. LBP is a face descriptor which makes histograms
derived by counting labels assigned to every pixel. The label of
LBP is computed by thresholding the neighborhood of each pixel
with the center pixel value. EF is a coefficient vector derived
by projecting the deblurred image to a linear subspace computed
using PCA. LF is a coefficient vector computed by PCA andlocality preserving projections that keeps the local structure ofneighborhood samples in the feature space.
Once the features are extracted, we adopt in all our experiments
the nearest-neighbor (NN) classifier with the Euclidean distancefor face recognition. This choice is motivated by the fact that, inmost cases, only one query image per person is available.
A. PSF inference accuracy
To gain insight into the PSF inference accuracy, we analyzed
synthesized images by blurring sharp query faces from FERET
database. Sharp faces are blurred by shift-invariant Gaussian−(u2 +v 2 )1, where σ is the standard
PSFs as H(u, v) = Z exp
2σ2H(u, v)dudv = 1.
deviation, and Z is a normalization term as
This PSF is defined in the space domain. White Gaussian noiseof 30dB is added to the synthesized images. Examples are shownin Figure 6. The facial images are registered using determinedfacial feature points. Image g(u, v) is of size 128 × 128, anddownsampled x(ξ , η  ) is of size 64 × 64. For the representative
fixed set Ω of PSFs, we use N = 18 PSFs, including 17 Gaussians
and 1 ‘no blur’ delta function. The parameters of the Gaussian
PSFs are set from σ = 1 to 9 in increments of 0.5. The databaseincludes three subsets: ‘bk’, ‘fa’, and ‘fb’. Each subset contains asingle image per person. Subset ‘bk’ is used as training set Ψ tolearn the subspaces in a set of statistical models Φ. The dimensionof each subspace is D = 20. The number of training images is

5

85

4

(ii) Hu & Haan
Recognition Rate(%)

Average error of PSF inference(i) Raw pixel feature space(iii) Magnitude-based feature space

3

2

1

80

EF

75

FADEIN + EF

70

LF
65

FADEIN + LF
60
55

LBP

50

FADEIN + LBP

45

0No blur

2

4

6

8

Standard deviation of Gaussian PSF for blurred query set

Fig. 7.

PSF inference accuracy on FERET of artificially camera focus blur.

TABLE IR ESTORATION PERFORMANCE OF ARTIFICIAL CAMERA FOCUS BLUR .

MethodRaw pixel feature space
Hu & Haan [7]Magnitude-based feature space

PSNR
28.8dB
28.4dB
30.4dBM = 200. We evaluate inference accuracy on the 1001 images

that remain in subsets ‘fa’ and ‘fb’ after removing the individuals
present in subset ‘bk’. The identification target set is ‘fa’ and thequery set is ‘fb’.
Figure 7 shows PSF inference accuracy on the query images‘fb’. Average error is defined as σc − σs 1 where σc is the standard deviation of the PSF used for synthesizing the blurred image(groundtruth), and σs is that of the inferred PSF. We compare ourmethod ‘magnitude-based feature space’ against two alternatives:‘raw pixel feature space’ which uses a directly vectorized feature
space (cf. Figure 5) and ‘Hu & Haan [7]’ which estimates thestandard deviation of Gaussian PSF from the smoothness of intensity changes around edges. Hu & Haan’s method can handle only
camera focus blur. From the results in Figure 7, we can clearly
see that our PSF inference method performs much better than
the alternatives. An interesting anomaly is that the PSF inference
accuracy of ‘raw pixel feature space’ increases with blur. Thisis probably because increasing the blur reduces variation, which
can thus be more adequately represented by a low-dimensional
subspace. In Table I, we demonstrate restoration performance interms of peak signal-to-noise ratio (PSNR) computed between theoriginal image and the deblurred image. We report the average
PSNR over different values of σc . Our method (30.4dB) performs
better than ‘raw pixel feature space’ method (28.8dB) and Hu &Haan’s method (28.4dB).
B. Face recognition performance with and without FADEIN
To analyze the effectiveness of our proposed facial deblur
inference (FADEIN), we consider different face recognition methods (EF, LF, and LBP) and compare the recognition results
obtained with and without the use of FADEIN. We evaluatedthe performance on the target set ‘fa’ and the artificially blurred
query set ‘fb’ of FERET, which are synthesized using the samesettings described in Section III-A.
Figure 8 reports identification accuracy as recognition rate:
the probability that a query image is matched to the correct
target image of the same individual. We demonstrate the averageFig. 8. Identification performance with and without FADEIN on FERET for
artificial camera focus blur. Query set are blurred with Gaussian PSF.
TABLE III DENTIFICATION PERFORMANCE ON FERET OF ARTIFICIAL CAMERA
FOCUS BLUR USING FACE RECOGNITION METHODS CONSIDERING BLUR .

MethodHu & Haan [7]LPQ
FADEIN

Recognition rate
73.4%
78.0%
83.4%

recognition rates over different values of σc of blur PSFs. It isclearly shown that FADEIN significantly enhances identification
performance for all considered methods. For instance, LBP withand without FADEIN yields in identification rates of 82.6%
and 67.2%, respectively. FADEIN overcomes the sensitively of
LBP to blur which was reported in [17] and also confirmed
in our experiments. This very significant gain in recognition
performance demonstrates the effectiveness of our proposed facial
deblur inference.
In our unoptimized implementation on a single-core 3.4GHz
processor, learning in FADEIN takes about 5 minutes and PSF
inference takes about 0.5 seconds per image. The BTV regularization [27] takes up to 1.5 seconds per image.
C. Comparing FADEIN-based face recognition to state-of-theart methods addressing the blur problems
How does our FADEIN-based face recognition technique compare to state-of-the-art methods addressing the blur problem?
To investigate this question, we evaluate and compare FADEIN
against two alternative methods designed for identifying blurred
faces: deconvolution using existing PSF inference before identification and extraction of blur invariant features from blurred faces
for identification. For the deconvolution approach, we used the
PSF inference of Hu & Haan [7] and the restoration of the BTV
regularization. After the deconvolution, we used raw pixel values
as features for identification. For the invariant feature approach,
we used the local phase quantization (LPQ) method [17] [33].
LPQ works like LBP for extracting face descriptors by assigning
quantized label to phase value in the frequency domain. Ahonen
et al. have recently adopted LPQ for identifying blurred faces
yielding in very good results [17]. Note that LPQ was originallyproposed by Ojansivu and Heikkila for blurred texture analysis
in [33]. In our approach, we adopted FADEIN for deconvolution
and raw pixel values as features. We evaluated the performanceon the same database as in Section III-B. The parameters for
FADEIN are the same as in Section III-A.
Table II shows identification performance of our method andthe two alternatives. We demonstrate the average recognition rates

6

TABLE IIII DENTIFICATION PERFORMANCE OF ARTIFICIAL CAMERA FOCUS BLUR
WITH RANDOM STANDARD VARIATION OF G AUSSIAN PSF.

Recognition Rate (%)

90

80

70
(i)

LPQ

(ii)

FADEIN(iii)

FADEIN + LPQ

MethodLPQ
Hu & Haan [7] + LPQ
FADEINFADEIN + LPQ

60

50

40
No blur

2

4

6

Recognition rate
84.0%
83.8%
82.8%
88.3%

8

Standard deviation of gaussian PSF for blurred query set

Fig. 9. Identification performance by combining PSF inference methods with
LPQ on FERET for artificial camera focus blur.

TABLE IVI DENTIFICATION PERFORMANCE FOR ARTIFICIAL CAMERA MOTION BLUR .

MethodLPQ
Yitzhaky & Kopeika [11] + LPQ
FADEINFADEIN + LPQ(a)Recognition rate
59.3%
49.1%
82.0%
82.3%

(b)

Fig. 10.
Examples of images artificially blurred with random standard
variation of Gaussian PSF in (0, 8] on FERET. (a) Target images from thesubset ‘fa’. (b) Query images for the subset ‘fb’. Note that both target andquery images are blurred to simulate real-life face capture conditions.

on the same setting described in Section III-B. Though LPQ
shows better performance compared to Hu & Haan’s method(78.0% versus 73.4%), FADEIN significantly outperforms bothmethods, yielding an identification rate of 83.4%.
D. Combining FADEIN and LPQ
While our FADEIN for face recognition is based on deblurring
the face images before recognition, the LPQ method uses a
different strategy as it aims at extracting blur invariant featuresfrom the original face images using the phase information in
the frequency domain. Although the results clearly demonstrate
the superiority of FADEIN over LPQ (as shown in Table II and
Figure 9), the experiments revealed very interesting properties:
while LPQ works very well with small amount of blur andsuffers when the amount of blur increases, the FADEIN approachsignificantly decreases the amount of blur in the face images
but uses simple raw pixels as features for recognition. Thismeans that one can look at FADEIN and LPQ as complementary
methods for further enhancing the recognition performance. Thus,
to exploit the advantages of FADEIN and LPQ, we derived a
new approach combining both methods: we extract the LPQ
features from the deblurred face images. Figure 9 shows thenthe results of this combination ‘FADEIN + LPQ’. As expected,
the combined approach enhances the recognition performancecompared to the use of FADEIN and LPQ separately. We alsostudied other combination schemes such as fusing FADEIN with
LPQ and/or LBP at feature level but the results were not better
than ‘FADEIN + LPQ’.
E. Experiments with different artificial blur settings1) Random focus blur: We tested identification performance on
FERET when the query set images were blurred by an unknown
random Gaussian sigma in the range (0, 8]. Figure 10 shows

examples used in this experiment. Some faces are blurred only intarget images and other faces are blurred only in query images.
Note that facial deblur inference is performed on both target andquery images. From Table III, an improvement over the othermethods by combining our deblurring and LPQ is again evident
and consistent performance is demonstrated for unknown blursthat do not precisely coincide with the training PSFs.
2) Camera motion blur: In the second experiment, tests were
performed on synthesized images by blurring sharp query faces
from FERET using shift-invariant linear motion blur PSFs as
H(u, v) = 1/Z if (u, v)2 < b and v = u tan θ, otherwiseH(u, v) = 0,where b is the length of camera motion, θ is theangle, and Z is a normalization term. White Gaussian noise of
30dB is added to the synthesized images. Identification query
images are blurred by the PSF in the range b = 5, 10, 15, 20 andθ = 0, 0.25π, 0.5π, 0.75π . Examples are shown in Figure 11. For
the representative fixed set Ω of PSFs, we use N = 41 PSFs,
including 10 × 4 = 40 motion blur function and 1 ‘no blur’ delta
function. The parameters are set from b = 2.5 to 25 in incrementsof 2.5 and from θ = 0 to 0.75π in increments of 0.25π . The
other parameters for learning statistical models are the same as
in Section III-A.
Table IV reports average recognition rate. We compare ourmethods to two alternatives. The first one is LPQ as in Section IIIC. While the PSF inference of Hu & Haan [7] cannot be applied
because it is only considers focus blur, we instead use themethod of Yitzhaky & Kopeika [11]. As Table IV shows, ourapproaches yield again in significantly better results compared tothe two alternatives (LPQ and Yitzhaky & Kopeika’s method).
The average errors between the correct parameters of blurred
query images and the inferred ones using Yitzhaky & Kopeika’s
method [11] were b = 6.2, θ = 0.1 while FADEIN yielded insignificantly better performance (b = 0.7, θ = 0.0).
3) Putting 2 PSF types together: For simultaneously handlingboth types of blur (out-of-focus and camera motion), we conducted experiments by putting the two types of PSF together.
We used 18 PSFs of focus blur described in Section III-A and
40 PSFs of motion blur described in Section III-E.2 to learn thestatistical models Φ. Given an unknown PSF image, FADEIN
selects a PSF from the 18 + 40 = 58 PSFs. Target set is no blur7

b = 5, θ = 0

b = 5, θ = 0.25π

b = 5, θ = 0.5π

b = 5, θ = 0.75π

b = 10, θ = 0

b = 10, θ = 0.25π

b = 10, θ = 0.5π

b = 10, θ = 0.75π

(a)

(b)

Fig. 12. Example images from FRGC 1.0. (a) Target image. (b) Query image
in Exp4. Real camera focus blur arises in (b).
b = 15, θ = 0

b = 15, θ = 0.25π

b = 15, θ = 0.5π

b = 15, θ = 0.75π

b = 20, θ = 0

b = 20, θ = 0.25π

b = 20, θ = 0.5π

b = 20, θ = 0.75π

Fig. 11. Examples of synthesized images of camera motion blur from thesubset ‘fb’ of FERET. Blurred images are synthesized from Figure 6 (a) using
linear motion blur PSFs with given b, θ.
TABLE VI DENTIFICATION PERFORMANCE FOR PUTTING 2 PSF TYPES TOGETHER( FOCUS AND MOTION BLURS ).

MethodLPQ
FADEINFADEIN + LPQ

Recognition rate
66.6%
79.6%
82.9%

‘fa’ and query set is ‘fb’ artificially blurred by focus blur (thesame setting described in Section III-B) and motion blur (thesame one described in Section III-E.2).
Table V reports the average recognition rates over all thequery images. The obtained results show that our proposed
approach (FADEIN) yields better performance than LPQ. Miss
classification rate of the two PSF types (i.e. confusion between
motion and focus blurs) was only 1.0%.
F. Evaluation on real blurred face images
The final experiment used real blurred images from FRGC 1.0.
We evaluated identification performance in terms of verification
rate under the setups ‘Exp4’ that is evaluated on uncontrolled still
query images including blurred faces. Each query set consists of
608 images of 152 individuals. We count that 366 query images
are degraded by camera focus blur. Target images are collectedunder a controlled still condition. A single image is captured perperson for target. The number of target images is 152. Figure 12
shows example images from FRGC 1.0. Camera focus blur arisesin Figure 12 (b) as the camera focal length is adjusted to thebackground. The amount of blur in both the target and query
images is not constant and is unknown. Note that facial deblur
inference is performed on both target and query images, sincereal face target databases already contain considerable focus blur.
For the representative fixed set Ω of PSFs, we use N = 8
PSFs, including 7 Gaussians and 1 ‘no blur’ delta function. The
parameters of the Gaussian PSFs are set from σ = 1 to 4 inincrements of 0.5. Three subsets ‘bk’, ‘fa’, and ‘fb’ in FERET
are used as training set Ψ to learn the subspaces in a set of
statistical models Φ. The dimension of each subspace is D = 20.
The number of training images is M = 2591.

(a) Faces with real blur(b) Faces deblurred using FADEIN

Fig. 13. (a) Example images from FRGC 1.0 with real blur. (b) Faces
deblurred using FADEIN.

We show some deblurred images in Figure 13. The deblurred
images are sharp compared with the blurred images. We evaluated
ROC curves in terms of verification rate and false acceptance rate.
As shown in Figure 14, the use of FADEIN for deblurring real
face images also enhances the performance of LPQ and outperforms the PSF inference of Hu & Haan [7], confirming our earlier
findings. We can also notice in Figure 14 that the performanceenhancement using only FADEIN is not as significant as with theartificially blurred faces in the previous experiments e.g. Figure 6.
This is mainly due to the fact that these real blurred faces containless blur than the artificially blurred faces. Figure 15 shows
the inferred amount of blur in these real blurred face images.
The median value is σ = 2.5 excluding no blurred faces. With
additional blur, we expect much more performance enhancement
using FADEIN.
IV. C ONCLUSION
We proposed a novel approach for recognizing blurred faces
using facial deblur inference. Our algorithm inferred PSF using
learned models of facial appearance variation under different
amounts of blur. The inferred PSFs were used to sharpen bothquery and target images. Our extensive experiments on both real
and artificially blurred face images demonstrated substantiallymore accurate PSF inference and face recognition than existing
methods. The salient contributions of this paper are the following:• Learning statistical models of the variation in facial appearance caused by blur to accurately infer PSFs from blurred
images.
• A new frequency magnitude-based feature space in whichthese statistical models are learned. This feature space isdesigned to emphasize the different appearances of different
levels of blur but be invariant to facial identity.
• A demonstration that these statistical models generalize well,maintaining PSF inference accuracy: the models are trainedon faces that are completely different to target images for
identification.
• A new scheme combining facial deblur inference with LPQ

8100(i) Hu & Haan(ii) LPQ(iii) FADEIN

95

Verification rate (%)(iv) FADEIN+LPQ

90

85

80
0

5

10

15

20

False acceptance rate (%)

Fig. 14.

ROC curves on FRGC 1.0 (real blurred images).

Num. of face images

300

200

100

0

Fig. 15.

No blurr

1
2
3
Inferred amount of blur4

Histogram of inferred amount of blur on FRGC 1.0.

(which is one of the state-of-the-art methods) is introduced,
yielding in excellent results.
We believe that our deblurring method has relevance not only
for face recognition, but also for other restricted classes ofimages, such as recognizing textual character, hand and body
postures under blur. However, our method may not work well forother objects consisting of uniform texture, e.g. plaster sphere,plastic cup, and metal fork. Note also that our proposed approach(FADEIN) has not yet been proven for images blurred with multi
unknown factors or with severe blur such as camera shake. As
future work, we are planning to explore and improve the toleranceof our approach to such factors. A promising direction consistsof modeling the PSF inference as a regression problem rather
than a classification problem, which is likely to scale better when
multiple factors are considered together.
R EFERENCES[1] S. Z. Li and A. K. Jain, Handbook of Face Recognition. Springer,
2005.
[2] T. Bourlai, A. Ross, and A. K. Jain, “On matching digital face images
against scanned passport photos,” Proc. IEEE Int’l Conf. Biometrics,
Identity and Security, 2009.
[3] I. Stainvas and N. Intrator, “Blurred face recognition via a hybrid
network architecture,” Proc. Int’l Conf. Pattern Recognition, vol. 2, pp.
805 – 808, 2000.
[4] D. Kundur and D. Hatzinakos, “Blind image deconvolution,” IEEE
Signal Processing Magazine, pp. 43 – 64, May 1996.
[5] P. Campisi and K. Egiazarian, Blind Image Deconvolution : Theory andApplications. Taylor & Francis, 2007.
[6] T. F. Chan and C.-K. Wong, “Total variation blind deconvolution,” IEEE
Trans. Image Processing, vol. 7, no. 3, pp. 370 – 375, 1998.
[7] H. Hu and G. de Haan, “Low cost robust blur estimator,” Proc. IEEE
Int’l Conf. Image Processing, pp. 617 – 620, 2006.

[8] J. H. Elder, “Local scale control for edge detection and blur estimation,”
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 7,
pp. 699 – 716, 1998.
[9] F. Rooms, A. Pizurica, and W. Philips, “Estimating image blur in the
wavelet domain,” Proc. Fifth Asian Conf. Computer Vision, pp. 210 –
215, 2002.
[10] H. Tong, M. Li, H. Zhang, and C.Zhang, “Blur detection for digital
images using wavelet transform,” IEEE Proc. Int. Conf. Multimedia andExpo, vol. 1, pp. 17 – 20, 2004.
[11] Y. Yitzhaky and N. S. Kopeika, “Identification of blur parametersfrom motion blurred images,” Graphical Models and Image Processing,vol. 59, no. 5, pp. 310 – 320, 1997.
[12] J. Jia, “Single image motion deblurring using transparency,” Proc. IEEE
Conf. Computer Vision and Pattern Recognition, pp. 1 – 8, 2007.
[13] L. Yuan, J. Sun, L. Quan, and H. Y. Shum, “Image deblurring withblurred/noisy image pairs,” ACM Trans. Graphics, vol. 26, no. 3, pp. 1– 10, 2007.
[14] C. Ancuti, C. O. Ancuti, and P. Bekaert, “Deblurring by matching,”
Computer Graphics Forum, vol. 28, no. 2, pp. 619 – 628, 2009.
[15] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman,“Removing camera shake from a single photograph,” ACM Trans.
Graphics, vol. 25, no. 3, pp. 787 – 794, 2006.
[16] M. Nishiyama, H. Takeshima, J. Shotton, T. Kozakaya, and O. Yamaguchi, “Facial deblur inference to improve recognition of blurred faces,”
Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1115–
1122, 2009.
[17] T. Ahonen, E. Rahtu, V. Ojansivu, and J. Heikkila, “Recognition of
blurred faces using local phase quantization,” Proc. Int’l Conf. Pattern
Recognition, pp. 1 – 4, 2008.
[18] T. Mita, T. Kaneko, B. Stenger, and O. Hori, “Discriminative feature cooccurrence selection for object detection,” IEEE Trans. Pattern Analysis
and Machine Intelligence, vol. 30, no. 7, pp. 1257–1269, 2008.
[19] A. E. Savakis and H. J. Trussell, “Blur identification by residual spectral
matching,” IEEE Trans. Image Processing, vol. 2, no. 2, pp. 141 – 151,
1993.
[20] S. Watanabe and N. Pakvasa, “Subspace method of pattern recognition,”
Proc. Int. Joint Conf. Pattern Recognition, pp. 25 – 32, 1973.
[21] R. C. Gonzalez and R. E.Woods, Digital Image Processing. Prentice
Hall, 2007.
[22] M. Savvides, B. V. K. V. Kumar, and P. K. Khosla, “Eigenphases vs.
eigenfaces,” Proc. Int’l Conf. Pattern Recognition, vol. 3, pp. 810 – 813,
2004.
[23] D. B. Gennery, “Determination of optical transfer function by inspectionof the frequency-domain plot,” J. Opt. Soc. Amer., vol. 63, pp. 1571 –
1577, 1973.
[24] M. Cannon, “Blind deconvolution of spatially invariant image blurs withphase,” IEEE Trans. Acoust. Speech Signal Process, vol. ASSP-24, pp.
58 – 63, 1976.
[25] M. M. Chang, A. M. Tekalp, and A. T. Erdem, “Blur identification
using the bispectrum,” IEEE Trans. Signal Processing, vol. 39, no. 10,
pp. 2323 – 2325, 1991.
[26] E. Oja, “Subspace methods of pattern recognition,” Research Studies
Press, 1983.
[27] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust
multiframe super resolution,” IEEE Trans. Image Processing, vol. 13,
no. 10, pp. 1327 – 1344, 2004.
[28] P. J. Phillips, H. Moon, P. J. Rauss, and S. Rizvi, “The feret evaluation
methodology for face recognition algorithms,” IEEE Trans. PatternAnalysis and Machine Intelligence, vol. 22, no. 10, pp. 1090 – 1104,
2000.
[29] P. J. Phillips, P. J. Flynn, T. Scruggs, K. W. Bowyer, J. Chang,
K. Hoffman, J. Marques, J. Min, and W. Worek, “Overview of the facerecognition grand challenge,” Proc. IEEE Conf. Computer Vision and
Pattern Recognition, vol. 1, pp. 947 – 954, 2005.
[30] T. Ahonen, A. Hadid, and M. Pietikainen, “Face description with local
binary patterns: application to face recognition,” IEEE Trans. PatternAnalysis and Machine Intelligence, vol. 28, no. 12, pp. 2037 – 2041,
2006.
[31] M. Turk and A. Pentland, “Eigenfaces for recognition,” J. Cognitive
Neurosicence, vol. 3, pp. 71 – 86, 1991.
[32] X. He, S. Yan, Y. Hu, and P. Niyogi, “Face recognition using laplacianfaces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27,
no. 3, pp. 328 – 340, 2005.
[33] V. Ojansivu and J. Heikkila, “Blur insensitive texture classification using
local phase quantization,” Proc. Int’l Conf. Image and Signal Processing,pp. 236 – 243, 2008.