Intelligent Frame Selection as a Privacy-Friendlier Alternative to Face
Recognition
Mattijs Baert, Sam Leroux, Pieter Simoens
IDLab, Department of Information Technology, Ghent University - imec
Technologiepark-Zwijnaarde 126
B-9052 Gent, Belgium
mattijs.baert@ugent.be, sam.leroux@ugent.be, pieter.simoens@ugent.be
Abstract
The widespread deployment of surveillance cameras for facial recognition gives rise to many privacy concerns. This
study proposes a privacy-friendly alternative to large scale facial recognition. While there are multiple techniques to preserve privacy, our work is based on the minimization principle which implies minimizing the amount of collected personal data. Instead of running facial recognition software on
all video data, we propose to automatically extract a highquality snapshot of each detected person without revealing
his or her identity. This snapshot is then encrypted and accessis only granted after legal authorization. We introduce a novel
unsupervised face image quality assessment method which isused to select the high quality snapshots. For this, we train
a variational autoencoder on high quality face images froma publicly available dataset and use the reconstruction probability as a metric to estimate the quality of each face crop.
We experimentally confirm that the reconstruction probability can be used as biometric quality predictor. Unlike mostprevious studies, we do not rely on a manually defined facequality metric as everything is learned from data. Our facequality assessment method outperforms supervised, unsupervised and general image quality assessment methods on thetask of improving face verification performance by rejecting
low quality images. The effectiveness of the whole system isvalidated qualitatively on still images and videos.

1

Introduction

Recent advances in computer vision and machine learning
have dramatically increased the accuracy of face recognition
technologies (Schroff, Kalenichenko, and Philbin 2015;
Masi et al. 2018; Deng et al. 2019b). Face recognition isalready commonly used in commercial products such as
Apple’s FaceID (Apple 2020) or at border checkpoints in
airports where the portrait on a digitized biometric passport
is compared with the holder’s face. Most people have little
to no concerns about these specific applications as they are
limited in scope and have a single well defined goal. As
the technologies mature however it becomes possible todeploy them on a much larger scale. The most well known
example of this is the large scale use of CCTV cameras
equipped with intelligent analysis software. In this way,
facial recognition checkpoints are deployed at areas like gas
Copyright c 2021, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

Figure 1: Overview of our proposed approach. For each subject, we automatically extract one high quality frame (or facecrop) and encrypt it for storage. Access to the images is only
possible after legal authorization in case of an investigation.
stations, shopping centers, and mosque entrances (Larson
2018; Schellevis 2016). From a domestic security pointof view, these technologies are extremely useful. They
can be used to find missing children, identify and track
criminals or locate potential witnesses. There are many
examples where CCTV footage in combination with facialrecognition software has supported and accelerated criminal
investigations. In 2016 for example, the “man in the hat”
responsible for the Brussels terror attacks was identified
thanks to FBI facial recognition software (The Brussels
Times 2016).
The proliferation of facial recognition technology howeveralso raises many valid privacy concerns. A fundamental human rights principle is that surveillance should be necessary
and proportionate. This principle was adopted by the UN
Human Rights Council (HRC) in the “the right to privacy inthe digital age” resolution which states that “States should
ensure that any interference with the right to privacy is
consistent with the principles of legality, necessity andproportionality”(United Nations 2017).
Governments have to balance both aspects of this technology before they implement a certain solution. On the
one hand, there are the obvious advantages of a large scale
blanket surveillance system but this clearly violates the
proportionality principle. On the other hand, a recent study
showed that a majority of the public considers it to be
acceptable for law enforcement to use facial recognition
tools to assess security threats in public spaces as long as
it is within a clearly defined regulatory framework (Smith
2019).

An interesting research direction is the development of
more privacy-friendly alternatives that can still support
law enforcement in criminal investigations. In this work
we present an intelligent frame selection approach thatcan be used as a building block in a more privacy-friendly
alternative to large scale face recognition. The problem of
key frame selection is defined as selecting a single frameout of a video stream that represents the content of thescene (Wolf 1996). In the context of facial recognition, ourgoal is to record a single clear crop of each person visible
in the stream without revealing his or her identity. In sucha way, the minimization strategy of privacy preserving
technologies (Domingo-Ferrer and Blanco-Justicia 2020) is
implemented by collecting the minimal necessary amount of
personal data. According to (Duncan 2007), data minimization is an unavoidable first step to engineer systems in linewith the principles of privacy by design (Cavoukian et al.
2009). Next, to ensure the confidentiality of the collected
data, all images are encrypted (i.e. “hide” strategy) andaccess can only be provided to law enforcement after legal
authorization as shown in figure 1.
Extracting face crops which are suitable for recognition is a difficult problem as surveillance footage is typicallyblurry because of the motion of the subjects. In addition,
it is hard to quantify the quality of a single frame as it
depends on multiple aspects such as the angle, illuminationand head position. Frame selection techniques have beenused before in the context of face recognition to reducethe computational cost or increase the recognition rate
(Wong et al. 2011; Anantharajah et al. 2012; Qi, Liu, andSchuckers 2018; Vignesh, Priya, and Channappayya 2015;
Best-Rowden and Jain 2017; Hernandez-Ortega et al. 2020;
Terhorst et al. 2020). In this work we introduce a novel
deep learning based technique to perform intelligent facerecognition. The key contribution is a new face image
quality assessment (FQA) approach inspired by anomaly
detection. Unlike previous solutions, our system is trainedcompletely unsupervised without the need for labeled face
images and without access to a face recognition system. Our
approach can be used for the same tasks as previous frame
selection techniques but in this work we propose to use it asa building block for a more privacy-friendly alternative tolarge scale face recognition.
This paper is organized as follows. An overview of
the related work is presented in section 2. Next, we propose
a privacy preserving alternative to large scale facial recognition in section 3. Our novel FQA method is introduced in
section 4 and experimentally validated in section 5 and 6.
We conclude in section 7 and give some pointers for future
research.

2

Related work

Deep learning has dominated the field of facial recognitionin the past years. Recent techniques can reach accuracies of
99.63% (Schroff, Kalenichenko, and Philbin 2015) on the
Labeled Faces in The Wild dataset (Huang et al. 2008), themost commonly used benchmark dataset. These data-driven

methods outperform techniques based on engineered features by a large margin (Masi et al. 2018). Face recognition
is typically subdivided into face verification and face identification. Face verification is the task of deciding whether
two pictures show the same person where face identification tries to determine the identity of the person on an image.
A large amount of research is dedicated to extracting
high quality face image representations. These can then be
used to calculate the similarity between two face crops or as
input to a classification model. Different loss functions havebeen designed to get meaningful face image representations.
First attempts used softmax loss (Sun et al. 2014) while
recent approaches focus on euclidean distance-based loss
(Schroff, Kalenichenko, and Philbin 2015), cosine-marginbased loss (Wang et al. 2017) and variations of the softmax
loss (Deng et al. 2019b). Given these high quality featureembeddings, we can directly perform face verification by
calculating the distance between two images (i.e. one-to-one
matching). Face identification requires a database whichcontains a reference image of every identity. To identify aperson on an image, the probe image’s embedding should
be compared with all images in the reference database (i.e.
one-to-many matching).
Face images are a popular biometric since they can becollected in unconstrained environments and without theuser’s active participation. These properties in combination
with the widespread deployment of surveillance cameras
give rise to some severe privacy concerns. As a result,researchers explore techniques to secure the data collectedby surveillance cameras and are developing privacy preserving facial recognition systems. The techniques differ inexactly what privacy-sensitive aspects they protect. Sometechniques avoid deducing soft biometrics (e.g. age, gender,race) from data collected for verification or recognition
purposes (Mirjalili, Raschka, and Ross 2018, 2020). Other
techniques focus on face image de-identification (Newton,
Sweeney, and Malin 2005), which eliminates the possibilityof a facial recognition system to identify the subject whilestill preserving some facial characteristics. A very powerful
technique hides the input image and the facial recognition
result from the server that performs the recognition using
a cryptographic enhanced facial recogniton system (Erkinet al. 2009).
A lot of effort has gone into the development of face
image quality assessment (FQA) metrics. Face image quality is defined as the suitability of an image for consistent,
accurate and reliable recognition (Hernandez-Ortega et al.
2020). FQA methods aim to predict a value which describes
the quality of a probe image. Most previous FQA techniques
focus on improving the performance of a face recognition
system (Wong et al. 2011; Anantharajah et al. 2012; Qi, Liu,and Schuckers 2018; Vignesh, Priya, and Channappayya
2015; Best-Rowden and Jain 2017; Hernandez-Ortega
et al. 2020; Terhorst et al. 2020). An FQA system is thenused as a first step to make sure that we only feed highquality images to the face recognition model, thereforeincreasing the accuracy and reducing the computationalcost. A first family of FQA techniques (Full-reference andreduced-reference FQA) assumes the presence of a highquality sample of the probe image’s subject. These methods
do not work for unknown subjects, which is necessary forour purpose as we will explain in the next section. A second
family of FQA techniques develops hand-crafted features
to assess the quality of a face image (Anantharajah et al.
2012; civil aviation organization ICAO) while more recent
studies apply data driven methods and report a considerable
increase in performance. Different studies (Qi, Liu, andSchuckers 2018; Vignesh, Priya, and Channappayya 2015)
propose a supervised approach where a model is trained
to predict the distance between the feature embeddings of
two images. Since two samples are necessary to perform acomparison, one low quality sample can affect the quality
score of a high quality sample. This is commonly solved
by assuming that an image compliant with the ICAO
guidelines (civil aviation organization ICAO) represents
perfect quality (Hernandez-Ortega et al. 2020). In the workof Hernandez-Ortega et al. a pretrained resnet-50 network(He et al. 2016) is modified by replacing the classification
layer with two regression layers which output the quality
score. Alternatively, it is also possible to use human labeleddata (Best-Rowden and Jain 2017). The most similar to
our work is (Terhorst et al. 2020) which also proposes anunsupervised approach. Here the quality of an image ismeasured as its robustness in the embedding space, whichis calculated by generating embeddings using random
subnetworks of a selected face recognition model.
Compared to previous work, we introduce a novel
completely unsupervised FQA method based on a variational autoencoder. We assume no access to the identities ofthe people and show that it works well for unseen people.
Unlike (Terhorst et al. 2020), no facial recognition modelis used to calculate a quality score. In contrast to previous
work our main goal is not necessarily to improve the facialrecognition accuracy but instead we use it as a building
block to enable a more privacy-friendly alternative to large
scale face recognition, as explained in the next section.

3

Frame selection as an alternative to facerecognition

In this section we introduce a framework based on (Simoens
et al. 2013) that uses intelligent frame selection in thecontext of face recognition to build a more privacy-friendly
alternative to large scale face recognition. Instead ofproactively trying to recognize individuals, we follow thepresumption of innocence principle and do not indiscriminately perform recognition of people as they go about
their daily business. Instead, our system uses key frame
extraction to capture a high quality snapshot of every person
passing by the camera. These snapshots are encrypted andstored locally on the camera or securely transmitted toa cloud back-end for storage. In a normal situation thisdata is then automatically deleted after a well defined time
period as defined in article 17 of the GDPR (“Right to be

Figure 2: Overview of our face quality-aware frame selection system.
forgotten”). In this case, the identity of the people will never
be known and no human operator will have access to theunencrypted pictures. The images can only be decrypted
after legal authorization in case of a criminal investigationor another situation where access to the identities present
in a certain location at a certain time is warranted. The
high quality images can then be used as input for a facerecognition system or to aid the investigation process. Since
only high quality crops are stored, the storage overhead ismuch lower than in a system where the full video is stored.
This system is summarized in Figure 1 using a video fromthe ChokePoint dataset (Wong et al. 2011).
It is important to note that we need to store at least
one crop for each person visible in the video. It is notenough to use a fixed threshold to decide whether a frame
should be stored or not, instead we have to store the best
frame for each individual even if this best frame still
has a relatively low quality compared to images of otherindividuals (for example because the person never perfectly
faces the camera). As a generalization we could also decideto store a short video clip of a few seconds before and after
the best frame has been captured.
An obvious disadvantage of our approach is that it is
not possible to proactively recognize people for example todetect wanted individuals. On the other hand it does support
the criminal investigation after the fact. Our system istherefore complementary and more suited to low risk areas
where a full blown face recognition system would violate
the proportionality principle.

4

Face image quality assessment

The system proposed in the previous section relies on aFace Quality Assessment block to decide which crops tostore. Any FQA method can be used but in this section we
introduce a novel technique based on a variational autoencoder. Compared to other FQA methods, this has the benefit
of being completely unsupervised. We do not assume accessto a face recognition system or to the identities of the people
in the dataset. Our method also generalizes well to otherpeople outside of the original training dataset.
An overview of the proposed system is depicted infigure 2. The first step is to detect all faces in each frame andtrack them across subsequent frames. We use the MTCNN
model (Zhang et al. 2016) to detect faces in a still video
frame. The output is a list of bounding boxes around thedetected faces. To track a subject across a video, we simply
calculate the euclidean distance between the bounding

boxes of subsequent frames. Bounding boxes that are close
to each other are considered to correspond to the same
subject. To evaluate the quality of a face crop, we calculate
the reconstruction probability of a variational autoencoder(VAE) trained on a dataset of high quality images. The
VAE reconstruction probability is commonly used as ananomaly detection metric (An and Cho 2015) to determine
how different an input is to the data seen during training.
By training the VAE on a publicly available dataset of highquality face images, we reformulate the FQA task as ananomaly detection task (i.e. how different is this face cropfrom the high quality face crops seen during training?).
The next paragraph explains the VAE and reconstruction
probability in more details.
A variational autoencoder (VAE) (Kingma and Welling2013) is a probabilistic variant of the standard autoencoder(AE). The encoder and decoder are modeled by probabilistic distributions rather than deterministic functions.
The encoder fφ (x) models the posterior qφ (z|x) of thelatent variable z and a decoder fθ (z) models the likelihoodpθ (x|z) of the data x given the latent variable z. The
prior distribution of the latent variable pθ (z) is chosen asa Gaussian distribution N (0, I). The posterior qφ (z|x)
and likelihood pθ (x|z) are isotropic multivariate normal
distributions N (µz , σz ) and N (µx , σx ) respectively. Figure
3 shows the process of a forward pass of an image x throughthe VAE, the arrows represent a sampling process. To train
a VAE using backpropagation, every operation should
be differentiable which is not the case for the sampling
operations: z ∼ N (µz , σz ) and x̂ ∼ N (µx , σx ). Applying
the re-parameterization trick fixes this problem. A dedicatedrandom variable  ∼ N (0, 1) is sampled such that the
sampling operations can be rewritten as: z ∼ µz +  · σz andx̂ ∼ µx +  · σx . The VAE training objective is written as the
expected log likelihood minus the KL divergence between
the posterior and the prior as described in equation 1.
L(x) = Eqφ (z|x) (pθ (x|z)) − KL(qφ (z|x)|pθ (z))(1)

The first term is the reconstruction term and forces agood reconstruction x̂ of the input data x. The KL term
regularizes the distribution of the latent space by forcing
it to be Gaussian. By training a generative model, like a
VAE, the model learns to approximate the training data
distribution. When a large reconstruction error is observed,
this is an indication that the data is not sampled from thedata distribution the VAE was trained on.
The reconstruction probability is a generalization of
the reconstruction error by taking the variability of thelatent space and the reconstruction into account (An and
Cho 2015). First, an image x is fed to the encoder which
generates the mean vector µz and standard deviation vector
σz . Next, L samples {z 0 , z 1 , ..., z l } are drawn from thelatent distribution N (µz , σz ). All samples z l are fed into
the decoder to get the distribution of the reconstruction of x
which is described by the mean µlx̂ and the standard deviation σx̂l . The reconstruction probability is the probability of

Figure 3: Variational autoencoder with encoder fφ (x) anddecoder fθ (z), each arrow represents a sampling processes.
x averaged over L samples as described by equation 2.
L

RP(x) =

1X
N (x|µlx̂ , σx̂l )
L

(2)

l=1

The reconstruction probability was originally developed
as an anomaly score by (An and Cho 2015). When a VAE
is trained solely on samples of “normal” data, the latent
distribution learns to represent these samples in a low dimensional space. Accordingly samples from “normal” data
result in a high reconstruction probability while anomalies
result in low reconstructing probabilities. We define thebiometric quality of a face image as the reconstruction
probability calculated by a VAE trained on high quality face
images. Correspondingly, a high reconstruction probability
is expected for high quality face images. Note that there is
no explicit definition of face image quality and the quality
score is independent of any face recognition model. The
definition of a high quality face image is derived directlyfrom the training data.
The encoder fφ (x) consists of 5 consecutive blocks ofa convolution layer, batch normalization and a leaky ReLU
activation function with at the end two fully connected
layers. The outputs of the encoder are the parameters
defining qφ (z|x). The decoder fθ (z) consists of 5 blocks ofa transposed convolution layer, batch normalization and aleaky ReLU activation function with again two fully connected layers at the end. The outputs of the decoder are theparameters defining pθ (x|z). To calculate the reconstruction
probability, L is set to 10. The CelebA dataset (Liu et al.
2015) consisting of 202,599 face images serves as training
data. The Adam optimization algorithm (Kingma and Ba
2014) was applied with a learning rate of 0.005 and a batch
size of 144. The VAE was trained for 10 epochs. Each
image is cropped by MTCNN (Zhang et al. 2016), resized
to 64x64 pixels and converted to grayscale.

5

Experimental setup

In this section, we isolate the FQA block for evaluation.
According to the national institute of standards and technology (NIST), the default way to quantitatively evaluate a
FQA system is analyzing the error vs. reject curve (ERC)(Grother and Tabassi 2007; Grother, Ngan, and Hanaoka
2020). As defined in section 2, FQA indicates the suitability of an image for recognition. The ERC measures towhat extent the rejection of low quality samples increases

and computed using the inverse of the empirical cumulative
distribution function of the distances between reference andprobe images M −1 :dt = M −1 (1 − f ).

(6)

Practically, f is chosen to give some reasonable FNMR. As
suggested by (Frontex 2017) a maximum FNMR of 0.05 is
maintained.
Figure 4: Sample images from the VggFace2 dataset ranked
by different quality metrics.

6

Results

VggFace2
the verification performance as measured by the false nonmatch rate (FNMR). The FNMR is the rate at which a biometric matcher miscategorizes two signals from the same
individual as being from different individuals (Schuckers
2010). Face verification consists of calculating a comparison score of two images and comparing this score with somethreshold. The comparison score is defined as the euclidean
distance between the FaceNet (Schroff, Kalenichenko, andPhilbin 2015) embeddings of the two images. To avoid alow quality sample affecting the verification performance ofa high quality sample, one high quality reference image forevery subject is required. For this, an ICAO compliant image
is typically used (Hernandez-Ortega et al. 2020). We used
the BioLab framework (Ferrara et al. 2012) to calculate an
average ICAO compliance score for all images. For every
subject, the image with the highest ICAO score is selected
as a reference image. Note that it is not possible to use the
BioLab framework as face quality assessment method directly because it cannot assess all images accurately and it
is unable to operate in real-time. Now, assume a set of gen(1)(2)
uine image pairs (xi , xi ) (i.e. two images of the same
(1)
(2)
person) of length N. Every image pair (xi , xi ) constitutes a distance di (i.e. comparison score). To determine if
two images match, the distance between the two images iscompared with a threshold dt . Using quality predictor function F (i.e. a FQA method), a quality value qi is calculated(1)
for each image pair. Since xi is always a high quality image from the reference database, the quality qi of image pair(1)(2)
(xi , xi ) can be written as:
(2)

(2)

qi = qi = F (xi ).
(3)
Now consider R as a set of low quality entities composed by
the samples that correspond with a predicted quality value
below some threshold:R(r) = {i : qi < F −1 (r), ∀i < N }.
(4)
−1
F
is the inverse of the empirical cumulative distributionfunction of the predicted quality scores. The parameter r is
the fraction of images to discard, such that F −1 (r) equals
the quality threshold that corresponds with rejecting a fraction r of all images. Then the FNMR can be written as:|di : di ≥ dt , i ∈/ R(r)|FNMR =(5)|di : di ≥ −∞, i ∈/ R(r)|The value of r is manipulated to calculate the FMNR for different fractions of rejected images. The value of dt is fixed

The VggFace2 dataset (Cao et al. 2018) is designed fortraining face recognition models and consists of more than
3.3 million face images of more than 9000 identities. In thefollowing experiments, our approach (VAE) is comparedto FaceQNet (FQN) (Hernandez-Ortega et al. 2020), a
method based on the stochastic embedding robustness(SER) (Terhorst et al. 2020) and a general image quality
assessment (IQA) system (Talebi and Milanfar 2018).
FaceQNet is a fine-tuned resnet-50 network (He et al. 2016)
which is trained on large amounts of face images for a facerecognition task. The classification layer is then replacedby two layers designed for quality regression. The groundtruth quality score is defined as the euclidean distance
between the feature embeddings of the probe image and an
ICAO compliant image of the same subject. The face image
quality assessment method based on stochastic embedding
robustness (SER) calculates a quality score by measuring
the variations of the embeddings generated by random
subnetworks of a resnet-101 (He et al. 2016) model trainedfor facial recognition. The general image quality assessment(IQA) system (Talebi and Milanfar 2018) does not take facefeatures into account and predicts the general image quality.
For all conducted experiments, images were cropped by the
MTCNN face detector (Zhang et al. 2016).
Figure 4 shows five images from the VggFace2 dataset (Caoet al. 2018) ranked by different quality metrics. This allows
a first qualitative evaluation of the five metrics. As presented
on the figure, all metrics agree on assigning the highest
quality to the first image. All FQA metrics assign a low
quality value to the third image because the person looks
down, the general IQA method does not take this aspect
into account and assigns a higher quality value. For thesame reason, the IQA method assigns a low quality value
to the fifth image in contrast to all FQA algorithms. On the
fourth image, our method agrees with the IQA method andselects it as worst quality image as opposed to the other
FQA metrics. We could assume that our method attaches
more importance to blurriness compared to the other FQA
algorithms. It is remarkable that the second image is ranked
as second worst by FQN since the person looks straight into
the camera.
In a second experiment, we evaluate our proposed
FQA algorithm on still face images by analyzing the ERC
plots as explained in section 5. For 103.008 images of 300
subjects from the VggFace2 dataset (Cao et al. 2018), aFigure 5: ERC with an initial FNMR of 0.01.
Figure 7: Consecutive face crops from one tracked identity
in the ChokePoint dataset. The crop with the green bordercorresponds with the highest quality calculated by our FQA
algorithm.

Figure 6: ERC with an initial FNMR of 0.001.

high quality reference image is selected using the ICAO
compliance scores. The ERC plots in figures 5 and 6 display
the FNMR for different fractions of rejected images. The
red line with the label “PERFECT” represents a quality
measure which correlates perfectly with the distance scores.
When an initial FNMR of 0.01 is set, in an ideal scenario,
the FNMR will be zero after rejecting 1% of all images. The
closer an ERC is to the red line, the better the performanceof the used FQA algorithm. For an initial FNMR of 0.01
our approach clearly outperforms FaceQNet, SER and thegeneral image quality assessment program. We hypothesizethat SER would perform better when the same type of
embeddings were used for verification as quality estimation.
In the conducted experiments SER uses ArcFace (Denget al. 2019a) embeddings to estimate face image quality
while the FNMR is calculated using FaceNet embeddings.
For an initial FNMR of 0.001, the difference with the otherapproaches is smaller. It is important to note that our model
is considerably smaller than FaceQNet and the resnet-101
model (He et al. 2016) used by SER. FaceQNet comprises
7 times more parameters than our VAE and resnet-101

Figure 8: Logarithm of the reconstruction probability (i.e.
face quality) for consecutive face crops.

even 14 times more. Additionally, our method is trainedcompletely unsupervised without the need for ground truth
quality values while FaceQNet relies on distance scores as
ground truth quality values. The ground truth generation
process used by FaceQNet also indicates the dependency
on one or more face recognition models. This dependency
is even more prominent for SER since a specific facerecognition network is used for generating the quality
predictions.

ChokePoint
In a third experiment, we focus on the ChokePoint dataset(Wong et al. 2011). This dataset is designed for conducting experiments on face identification and verification underFigure 9: Three frames from the footage of a security camera
in a bicycle parking lot. The corresponding frame crops andquality scores are depicted below each frame.
real-world surveillance conditions. The dataset consists of
48 videos or 64,204 face images. Both crowded and singleidentity videos are included. We now evaluate our system
qualitatively on selecting one high quality face crop of eachidentity in a video stream. Figure 7 shows consecutive facecrops of an example video. The crop outlined in green is theframe that corresponds with the highest quality value calculated by our FQA algorithm. Figure 8 shows how the quality
score changes over time as the subject moves through thescene. We define the quality score as the logarithm of the reconstruction probability. We can see that initialy the quality
score decreases as the person is moving through a darker
area looking down. The shades and the angle of the facemakes these crops less useful for face recognition. As the
person moves closer to the camera, the brightness increases
and the subject becomes clearly visible. This is also reflectedin an increasing quality score. The highest score is obtained
when the person is close to the camera and is looking in thegeneral direction of the camera. As the person turns away,
the score again decreases. This qualitative example showsthat our model is indeed able to assign understandable andmeaningful scores to each frame. We made videos of thisand other examples publicly available 1 .

Bicycle parking
Finally, we also validated our approach on video data fromsecurity cameras in a bicycle parking lot. This to investigate
how well the model generalizes to data collected in the realworld. Figure 9 shows three example frames with its corresponding frame crop and quality score. Even though theseimages are very different from the images the VAE was originaly trained on, we can see that the model generalizes welland is able to assign useful scores to each crop.

7

Conclusion and future work

In this study, a novel face image quality assessment method
is proposed based on a variational autoencoder’s reconstruction probability. This is, by our knowledge, the firsttime a generative model like a VAE is used to tackle theproblem of face image quality assessment. We demonstrate,by quantitative and qualitative results, that our method canbe used as a biometric quality predictor. Unlike other data1
https://drive.google.com/drive/folders/
1GRlFRSxHRfBnfTpI5DG2v2rN3nTCg5Y0?usp=sharing

driven approaches, no facial recognition model is used fortraining and no explicit definition of face quality is given.
Our FQA algorithm is used as a building block in a privacyfriendly alternative to large scale facial recognition. Insteadof identifying all detected faces in a video stream, oursystem saves one high quality face crop without revealing
the person’s identity. This face crop is encrypted and accessis only granted after legal authorization. In such a way,
the system still supports criminal investigations while notviolating the proportionality principle.
In future work, we will further optimize the VAE architecture keeping the constraints on model size andcomputational complexity in mind as the final goal would
be to deploy the model on a stand-alone edge device.
It would be interesting to investigate different hardware
platforms such as FPGAs that allow the model to processdata in real-time with a small energy consumption, making
it possible to embed our system in low cost surveillancecamera’s. Moreover, our method should be evaluated onother datasets and in combination with alternative feature
extractors.

8

Acknowledgments

We thank Klaas Bombeke (MICT UGent) for providing uswith the dataset used in the experiment of Figure 9, andStijn Rammeloo from Barco for the feedback on the initial manuscript. This research was funded by the imec.ICON
SenseCity project and the Flemish Government (AI Research Program).

References
An, J.; and Cho, S. 2015. Variational autoencoder based
anomaly detection using reconstruction probability. Special
Lecture on IE 2(1): 1–18.
Anantharajah, K.; Denman, S.; Sridharan, S.; Fookes, C.;
and Tjondronegoro, D. 2012. Quality based frame selection
for video face recognition. In 2012 6th International Conference on Signal Processing and Communication Systems,
1–5. IEEE.
Apple. 2020. About Face ID advanced technology URL
https://support.apple.com/en-gb/HT208108.
Best-Rowden, L.; and Jain, A. K. 2017. Automatic face image quality prediction. arXiv preprint arXiv:1706.09887 .
Cao, Q.; Shen, L.; Xie, W.; Parkhi, O. M.; and Zisserman,
A. 2018. VGGFace2: A dataset for recognising faces acrosspose and age. In International Conference on Automatic
Face and Gesture Recognition.
Cavoukian, A.; et al. 2009. Privacy by design: The 7 foundational principles. Information and privacy commissionerof Ontario, Canada 5.
civil aviation organization (ICAO), I. 2015. Doc 9303: Machine readable travel documents. Technical report.
Deng, J.; Guo, J.; Niannan, X.; and Zafeiriou, S. 2019a. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR.

Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2019b. Arcface:
Additive angular margin loss for deep face recognition. In
Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 4690–4699.
Domingo-Ferrer, J.; and Blanco-Justicia, A. 2020. PrivacyPreserving Technologies, 279–297. Cham: Springer International Publishing. ISBN 978-3-030-29053-5. doi:10.1007/
978-3-030-29053-5 14. URL https://doi.org/10.1007/9783-030-29053-5 14.
Duncan, G. 2007. Engineering. Privacy by design. Science (New York, N.Y.) 317: 1178–9. doi:10.1126/science.
1143464.
Erkin, Z.; Franz, M.; Guajardo, J.; Katzenbeisser, S.; Lagendijk, I.; and Toft, T. 2009. Privacy-preserving face recognition. In International symposium on privacy enhancing
technologies symposium, 235–253. Springer.
Ferrara, M.; Franco, A.; Maio, D.; and Maltoni, D. 2012.
Face image conformance to iso/icao standards in machinereadable travel documents. IEEE Transactions on Information Forensics and Security 7(4): 1204–1213.
Frontex. 2017. Best practice technical guidelines for automated border control (abc) systems .
Grother, P.; Ngan, M.; and Hanaoka, K. 2020. Ongoing facerecognition vendor test (frvt) part 5: Face image quality assessment. NIST Interagency Report .
Grother, P.; and Tabassi, E. 2007. Performance of biometricquality measures. IEEE transactions on pattern analysis andmachine intelligence 29(4): 531–543.
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of theIEEE conference on computer vision and pattern recognition, 770–778.
Hernandez-Ortega, J.; Galbally, J.; Fierrez, J.; and Beslay,
L. 2020. Biometric Quality: Review and Application
to Face Recognition with FaceQnet.
arXiv preprint
arXiv:2006.03298 .
Huang, G. B.; Mattar, M.; Berg, T.; and Learned-Miller, E.
2008. Labeled faces in the wild: A database forstudying facerecognition in unconstrained environments.
Kingma, D. P.; and Ba, J. 2014. Adam: A method forstochastic optimization. arXiv preprint arXiv:1412.6980 .
Kingma, D. P.; and Welling, M. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 .
Larson, C. 2018. China’s massive investment in artificial
intelligence has an insidious downside. Science 8.
Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV).
Masi, I.; Wu, Y.; Hassner, T.; and Natarajan, P. 2018. Deepface recognition: A survey. In 2018 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), 471–
478. IEEE.

Mirjalili, V.; Raschka, S.; and Ross, A. 2018. Gender privacy: An ensemble of semi adversarial networks for confounding arbitrary gender classifiers. In 2018 IEEE 9th International Conference on Biometrics Theory, Applications
and Systems (BTAS), 1–10. IEEE.
Mirjalili, V.; Raschka, S.; and Ross, A. 2020. PrivacyNet:semi-adversarial networks for multi-attribute face privacy.
arXiv preprint arXiv:2001.00561 .
Newton, E. M.; Sweeney, L.; and Malin, B. 2005. Preserving
privacy by de-identifying face images. IEEE transactions on
Knowledge and Data Engineering 17(2): 232–243.
Qi, X.; Liu, C.; and Schuckers, S. 2018. Boosting face invideo recognition via cnn based key frame extraction. In
2018 International Conference on Biometrics (ICB), 132–
139. IEEE.
Schellevis, J. 2016.
Politie gaat verdachten opsporen met gezichtsherkenning.
NOS URL
https://nos.nl/artikel/2148598-politie-gaat-verdachtenopsporen-met-gezichtsherkenning.html.
Schroff, F.; Kalenichenko, D.; and Philbin, J. 2015. Facenet:
A unified embedding for face recognition and clustering. In
Proceedings of the IEEE conference on computer vision andpattern recognition, 815–823.
Schuckers, M. E. 2010. False Non-Match Rate, 47–96. London: Springer London. ISBN 978-1-84996-202-5. doi:10.
1007/978-1-84996-202-5 3. URL https://doi.org/10.1007/
978-1-84996-202-5 3.
Simoens, P.; Xiao, Y.; Pillai, P.; Chen, Z.; Ha, K.; and Satyanarayanan, M. 2013. Scalable crowd-sourcing of video frommobile devices. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services, 139–152.
Smith, A. 2019. More than half of us adults trust law enforcement to use facial recognition responsibly. Pew Research Center .
Sun, Y.; Chen, Y.; Wang, X.; and Tang, X. 2014. Deep learning face representation by joint identification-verification. In
Advances in neural information processing systems, 1988–
1996.
Talebi, H.; and Milanfar, P. 2018. NIMA: Neural image assessment. IEEE Transactions on Image Processing 27(8):3998–4011.
Terhorst, P.; Kolf, J. N.; Damer, N.; Kirchbuchner, F.; andKuijper, A. 2020. SER-FIQ: Unsupervised estimation offace image quality based on stochastic embedding robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5651–5660.
The Brussels Times. 2016.
The man in the
hat identified thanks to FBI software.
URL
https://www.brusselstimes.com/brussels/37264/the-man-inthe-hat-identified-thanks-to-fbi-software/.
United Nations. 2017. URL https://www.article19.org/data/
files/HRC.34.L.7.Rev1 Privacy in the digital age 1.pdf.

Vignesh, S.; Priya, K. M.; and Channappayya, S. S. 2015.
Face image quality assessment for face selection in surveillance video using convolutional neural networks. In 2015
IEEE Global Conference on Signal and Information Processing (GlobalSIP), 577–581. IEEE.
Wang, F.; Xiang, X.; Cheng, J.; and Yuille, A. L. 2017.
Normface: L2 hypersphere embedding for face verification.
In Proceedings of the 25th ACM international conference on
Multimedia, 1041–1049.
Wolf, W. 1996. Key frame selection by motion analysis. In
1996 IEEE international conference on acoustics, speech,
and signal processing conference proceedings, volume 2,
1228–1231. IEEE.
Wong, Y.; Chen, S.; Mau, S.; Sanderson, C.; and Lovell,
B. C. 2011. Patch-based Probabilistic Image Quality Assessment for Face Selection and Improved Video-based Face
Recognition. In IEEE Biometrics Workshop, Computer Vision and Pattern Recognition (CVPR) Workshops, 81–88.
IEEE.
Zhang, K.; Zhang, Z.; Li, Z.; and Qiao, Y. 2016. Joint facedetection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23(10):1499–1503.