Recognition of Blurred Faces via Facial Deblurring Combined with
Blur-Tolerant Descriptors
Abdenour Hadid
Machine Vision Group
University of Oulu, Finland

Masashi Nishiyama, Yoichi SatoInstitute of Industrial ScienceThe University of Tokyo, JapanAbstract
Blur is often present in real-world images and significantly affects the performance of face recognition
systems. To improve the recognition of blurred faces,
we propose a new approach which inherits the advantages of two recent methods. The idea consists offirst reducing the amount of blur in the images via
deblurring and then extracting blur-tolerant descriptors for recognition. We assess our analysis on real
blurred face images (FRGC 1.0 database) and also on
face images artificially degraded by focus blur (FERETdatabase), demonstrating significant performance enhancement compared to the state-of-the-art.

1. Introduction
Current face recognition systems perform well underrelatively controlled environments. However, the design of algorithms that are effective over a wide range ofviewpoints, occlusions, aging of subjects and complex
outdoor lighting is still a major area of research, despite
the great deal of progress during the recent years [4].
Other factors which also affect the performance of facerecognition systems include blur which may cause significant image degradations (see examples of blurred
faces in Fig. 1). Blur is unfortunately often present inreal-world face images and is usually originated fromcamera motion or misfocused optics. It affects the appearance of faces in images, causing two main problems for face recognition: (i) the facial appearance ofan individual changes drastically due to blur; (ii) anddifferent individuals tend to appear more similar whenblurred [5].
Surprisingly, while there has been an enormous
amount of research on face recognition under poseand illumination changes, problems caused by blur are
mostly overlooked. The focus of this paper is thus on

Figure 1. Examples of blurred face images

blur, and particularly on how to improve the recognition of blurred faces.
Basically, there are two main approaches in existing
techniques for face recognition under blur. The first direction consists of deblurring the face images via deconvolution with an aim of recovering sharp images readyfor processing using any conventional face recognition
algorithm. Another way of handling the recognition of
blurred faces consists of deriving blur-invariant features
directly from the blurred images without having to recover the sharp images which, anyway, are not necessarily needed for recognition [2]. We noticed that the twoapproaches depict different and complementary characteristics. This observation is behind our proposed approach which is presented in this paper.
The rest of this paper is organized as follows. Section 2 briefly describes a recent approach using facial deblurring [5] while Section 3 discusses anotherpromising technique based on blur-invariant descriptors [2]. These two methods can be considered as stateof-the-art in recognition of blurred faces. Section 4 reveals the pros and cons of each method, explains themotivations and introduces an adequate fusion scheme
for combining the advantages of both methods. We
perform extensive experimental analysis in Section 5,
demonstrating significant performance enhancement.
We also compare the results against those of various
fusion schemes and other state-of-the-art methods like
local binary patterns (LBP) [1]. A conclusion is drawnin Section 6.

or π. Furthermore, the shape of H for a regular PSF isclose to a Gaussian or a sinc-function, which makes atleast the low frequency values of H to be positive. At
these frequencies,  H = 0 causes  G =  F to be a blurinvariant property. This phenomenon is the basis of thelocal phase quantization (LPQ) method.
For a given image, each pixel is labeled with a blurinvariant LPQ code. The occurrences of these LPQ
codes are collected into a histogram. The classificationis then performed using the χ2 (Chi-square) dissimilarity metric for comparing a target histogram S to a model2li)
histogram M : χ2 (S, M ) = i=0 (SSii−M+Mi , where l is
the length of feature vector used to represent each face.
For an efficient representation, facial images are firstdivided into several local regions from which LPQ histograms are extracted and concatenated into an enhanced feature histogram which is used as the blurinvariant face descriptor. The experiments showed thatsuch LPQ descriptors are highly discriminative and produce good face recognition results for both sharp andblurred face images. We therefore adopt this approachas a baseline method for face recognition using blurinvariant features.

2. Facial Deblurring
As mentioned above, one way to deal with the recognition of blurred faces is to restore the sharp images
via image deconvolution and then use conventional facerecognition algorithms for matching. Many methods
have indeed been proposed for deblurring from a single image [3]. However, it appears that most the existing methods attempt to infer the point spread function (PSF) representing the process of blur without priorknowledge of the image contents. This generally yields
in deblurring results that are often insufficient for accurate face recognition.
Very recently, Nishiyama et al. [5] introduced an approach exploiting face prior information for facial deblurring, yielding in very promising results. The idea
is to exploit prior knowledge of how facial appearances
are changed by blur. The method first chooses a representative set of PSFs. Then, sharp training images
are artificially blurred using each PSF to build a statistical model of facial appearance under that PSF. Givena query image (with unknown identity and amount ofblur), the method infers a PSF by comparing the image
to each model during testing. The inferred PSF is thenused to deblur the query image and perform face recognition. Experiments on real and artificially blurred face
images showed high PSF inference and good recognition performance. We therefore adopt Nishiyama et al.’s
approach as a baseline method for face recognition using deblurring.

4. Proposed Approach
Our proposed approach to deal with the recognitionof blurred faces is described in Algorithm 1.
G IVEN A QUERY IMAGE ( WITH UNKNOWN IDENTITY AND AMOUNTOF BLUR ) TO RECOGNIZE :1. E STIMATE THE PSF BY COMPARING THE QUERY IMAGE

3. Blur-Invariant Descriptors

TO THE MODELS ( BUILT DURING THE TRAINING AS IN [5])An alternative to deal with the recognition of blurred
faces is to extract features which are invariant or tolerant to the presence of blur. This is an emerging direction
and not much work has been done before. To the best of
our knowledge, the only notable work on explicit construction of blur-invariant descriptors for face recognition is that of Ahonen et al. [2] who derived blur invariant features from the blurred face images using thephase information in the frequency domain [6]. The
method, called local phase quantization (LPQ), showedvery promising results.
In the LPQ approach, the spatial blurring is represented by a convolution between the image intensity anda PSF. In the frequency domain, this results in a multiplication G = F · H, where G, F, and H are the Fouriertransforms of the blurred image, the sharp image, andthe PSF respectively. Considering only the phase of thespectrum, the relation turns into a sum  G =  F +  H.
If the PSF is centrally symmetric, the transform H becomes real valued and the phase angle  H must equal 0

2. U SE THE INFERRED PSF TO DEBLUR THE QUERY IMAGE
3. D IVIDE THE DEBLURRED IMAGE INTO LOCAL BLOCKS
4. E XTRACT BLUR TOLERANT HISTOGRAMS FROM EACH LOCALBLOCK USING LPQ [2]5. C ONCATENATE THE LOCAL HISTOGRAMS INTO AN ENHANCEDFEATURE HISTOGRAM

6. P ERFORM HISTOGRAM MATCHINGT HE NEAREST- NEIGHBOR (NN) DETERMINES THE IDENTITY OF THEQUERY IMAGE .

Algorithm 1: Steps of the proposed approach
The main advantage of deblurring approaches is thatonce sharp images are restored, any conventional facerecognition technique can be chosen and applied. However, our experiments have showed that the deblurring
process is rarely perfect as it can only reduce (thoughsignificantly) the amount of blur while avoiding notable2

artifacts. So, in practice, some amount of blur still remains after deblurring. On the other hand, our experiments also revealed that blur-invariant descriptors generally work very well with small amount of blur butsuffer when the amount of blur increases. These observations are behind our idea of proposing a hybrid approach to deal with the recognition of blurred faces. The
proposed approach, sketched in Algorithm 1, consists offirst reducing the amount of blur via image deblurringand then extracting blur-tolerant descriptors for robust
face recognition.

deteriorates when the amount of blur increases. The facial deblurring approach seems to handle large amountof blur much better than LBP and LPQ. As expected,
our proposed approach (DeblurLPQ) significantly enhances the recognition performance compared to theuse of LPQ and deblur separately, yielding in the bestresults.
100

Recognition Rate (%)

90

5. Experimental Analysis
To assess the performance of the proposed approachin recognizing blurred (and also sharp) faces, we performed extensive experiments on two publically available face databases namely FERET [8] and FRGC 1.0[7], considering both artificial and real blurred images.
We first analyzed synthesized images by blurring
sharp query faces from FERET database as in [5]. Sharp
faces are blurred by Gaussian PSFs (σ = 2,4,6,8). White
Gaussian noise of 30dB is also added to the synthesizedimages. Examples are shown in Figure 2. The databaseincludes three subsets: ‘bk’, ‘fa’, and ‘fb’ having 200,1196, and 1195 images, respectively. Each subset contains a single image per person. Subset ‘bk’ is used as atraining set. We evaluated the recognition accuracy on
the 1001 images that remain in subsets ‘fa’ and ‘fb’ after removing the individuals present in subset ‘bk’. The
identification target set is ‘fa’ and the query set is ‘fb’.

8070605040

LBP [TPAMI 2006]
LPQ [ICPR 2008]
Deblur [CVPR 2009]
DeblurLPQ [This paper]

3020No blur

2468

Standard deviation of gaussian PSF for blurred query set

Figure 3. Recognition performance on
FERET for artificial camera focus blur

Recognition Rate (%)

10090807060

DeblurLBP
Feature Fusion: LPQ + DeblurLPQ
Feature Fusion: LPQ + DeblurLBP
DeblurLPQ

5040No blur

2468

Standard deviation of gaussian PSF for blurred query set
No blurσ=2σ=4σ=6σ=8Figure 4. Results of different fusionschemes for artificial camera focus blurFigure 2. Examples of synthesized images of focus blur from FERETOur proposed approach consists of associating LPQ
and deblurring in a complementary manner: it extracts blur-tolerant descriptors from the deblurred face
images. Thus, it inherits the advantages of bothmethods. To gain insight into the proposed approach, we also studied other combination schemes
such as (i) extracting LBP histograms from the deblurred face images (DeblurLBP); (ii) fusing LPQ descriptors with DeblurLBP representation at feature level(LPQ+DeblurLPQ); (iii) and fusing LPQ descriptors
with DeblurLBP at feature level (LPQ+DeblurLPQ).
The results of these combination schemes are shown

Fig. 3 compares the results of four different approaches including LBP [1] as one of the state-of-the-art
in recognition of sharp face images, LPQ [2] as a baseline for blur-invariant descriptors for face recognition,
deblur [5] as a representative of facial deblurring andDeblurLPQ which is the proposed method in this work(Section 4). From the results, we can notice that LBP
is very sensitive to blur although it is one of the stateof-the-art methods in face recognition. We also noticethat the blur-invariant descriptor approach (LPQ) works
very well with small amount of blur but its performance3

in Fig. 4, demonstrating that the proposed association(DeblurLPQ) yields in the best performance.

the recognition performance. The results show againthat the proposed approach (DeblurLPQ) yields in thebest performance, confirming our earlier findings.

Recognition Rate (%)

96

6. Conclusion

9492

The salient contributions of this paper are: (i) anovel fusion scheme for improving the recognitionof blurred faces is proposed; (ii) state-of-the-art inface recognition under blur is reviewed and pros/consof each method are revealed; (iii) various fusion
schemes are studied and evaluated; and (iv) extensive
experimental analysis on artificial and real blurred
face images is conducted, demonstrating significant
performance enhancement using the proposed approachcompared to the state-of-the-art.
As a future work, it is of interest to expand the evaluation of the proposed approach by explicitly considering
severe motion blur caused, for instance, by camera
shake.

908886

DeblurLPQ with empirical parameter adjustments
DeblurLPQ with automatically learned parameters

8482
No blur2468

Standard deviation of gaussian PSF for blurred query set

Figure 5. Empirical vs. automatic parameter tuning
An important parameter in the proposed approachis the size of the local neighborhood (K-by-K) that isused to compute the LPQ label [2] at each image pixel.
So far, the value of K is empirically determined during
training and then kept constant in testing. However, wenoticed that the optimal value of K is function of theamount of blur (σ) in every blurred query which is variable. We therefore trained a linear regression model tolearn the relationship between K and σ. Fig. 5 shows
the results when K is automatically determined by theregression model given σ which is estimated during thedeblurring stage. The result shows further performanceenhancement, indicating the importance of the choiceof the local neighborhood size when using LPQ.

ACKNOWLEDGMENT
The financial support of the Academy of Finland,
Infotech Oulu and the University of Oulu is gratefully
acknowledged.

References[1] T. Ahonen, A. Hadid, and M. Pietikäinen. Face description with local binary patterns: Application to face recognition. IEEE TPAMI, 28(12):2037–2041, 2006.
[2] T. Ahonen, E. Rahtu, V. Ojansivu, and J. Heikkila.
Recognition of blurred faces using local phase quantization. Proc. ICPR, pages 1–4, 2008.
[3] P. Campisi and K. Egiazarian. Blind Image Deconvolution : Theory and Applications. Taylor & Francis, 2007.
[4] S. Z. Li and A. K. Jain, editors. Handbook of Face Recognition. SpringerVerlag, New York, 2005.
[5] M. Nishiyama, H. Takeshima, J. Shotton, T. Kozakaya,and O. Yamaguchi. Facial deblur inference to improverecognition of blurred faces. Proc. CVPR, pages 1115–1122, 2009.
[6] V. Ojansivu and J. Heikkila. Blur insensitive texture classification using local phase quantization. Proc. Int’l Conf.
Image and Signal Processing, pages 236 – 243, 2008.
[7] P. J. Phillips, P. J. Flynn, T. Scruggs, K. W. Bowyer,J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek.
Overview of the face recognition grand challenge. Proc.
CVPR, 1:947 – 954, 2005.
[8] P. J. Phillips, H. Moon, P. J. Rauss, and S. Rizvi. The feretevaluation methodology for face recognition algorithms.
IEEE TPAMI, 22(10):1090 – 1104, 2000.
[9] X. Tan and B. Triggs. Enhanced local texture featuresets for face recognition under difficult lighting conditions. Proc. Analysis and Modeling of Faces and Gestures, pages 168 – 182, 2007.

Table 1. Recognition rates on FRGC 1.0
for real blurred images
MethodRate(%)

LBP
33.4

DeblurLBP
33.7LPQ
74.8

DeblurLPQ
79.6

In addition to artificial blur, we also considered real
blurred images from FRGC 1.0. face database. We evaluated identification performance under the setup ‘Exp4’
that is evaluated on uncontrolled still query images including blurred faces. Each query set consists of 608
images of 152 individuals. We count that 366 query images are degraded by camera focus blur. Target images
are collected under a controlled still condition. A single
image is captured per person for target. The number oftarget images is 152. The amount of blur in both thetarget and query images is not constant and is unknown.
We applied the illumination normalization preprocessing [9] before facial deblurring. We report in Table 14