# Learning a Genetic Measure for Kinship Verification Using Facial Images

Authors: Lu Kou; Xiuzhuang Zhou; Min Xu; Yuanyuan Shang
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

Motivated by the key observation that children generally resemble their parents more than other persons with respect to facial appearance, distance metric (similarity) learning has been the dominant choice for state-of-the-art kinship verification via facial images in the wild. Most existing learning-based approaches to kinship verification, however, are focused on learning a genetic similarity measure in a batch learning manner, leading to less scalability for practical applications with ever-growing amount of data. To address this, we propose a new kinship verification approach by learning a sparse similarity measure in an online fashion. Experimental results on the kinship datasets show that our approach is highly competitive to the state-of-the-art alternatives in terms of verification accuracy, yet it is superior in terms of scalability for practical applications.


---

## Body

## 1. Introduction

Due to the fact that rich human characteristics, such as gender, identity, expression, and ethnicity, can be effectively extracted from facial images, a variety of face analysis problems, ranging from face recognition, facial expression recognition, and gender estimation to age estimation, have been extensively studied over the past decades [1]. Kinship verification using facial images, however, is a relatively new and challenging problem in biometrics [2], which is mainly motivated by the phenomenon that children generally look like their parents more than other people due to the kinship relation. Recent evidence in psychology has shown that facial appearance is a reliable cue to measure the genetic similarity between children and their parents [3–9]. In practice, there are some important potential applications of kinship verification via facial images, including missing children search and social media analysis [10, 11]. Figure 1 presents some image examples (with kinship relation) from the dataset KinFaceW [12].Figure 1
Some image examples (with kin relation) from KinFaceW dataset [12]. From top to bottom are the father-son (F-S), father-daughter (F-D), mother-son (M-S), and mother-daughter (M-D) kinship relations, respectively.Over the past few years, a few seminal works on kinship verification using facial images [2, 10, 11, 13–21] have been proposed. Roughly, existing methods for kinship verification are either feature-based [2, 13, 15–17, 19] or learning-based [10, 11, 14, 18, 20]. The former aims for extraction of the discriminative feature from facial images to characterize genetic property in human appearance. Learning-based approaches, however, are focused on learning a genetic measure via training data based on some discriminative learning technologies, such as subspace learning and distance metric learning, to improve the separability of facial images for kinship verification. Despite the promising results by existing learning-based approaches to kinship verification, they aim to learn a distance metric (or transform) in a batch learning way, leading to less efficiency or scalability even for medium-scale applications. Unlike the previous learning-based studies, we aim to learn a sparse similarity measure for kinship verification in an online manner. Our proposed approach not only is able to achieve highly competitive verification accuracy to state-of-the-art kinship verification method but also is superior in terms of scalability, making it more scalable for practical applications with ever-growing amount of data.The rest of this paper is organized as follows. In Section2 we elaborate our approach to kinship verification from facial images. Experiments and evaluations on performance are conducted in Section 3. Finally, we conclude and summarize the paper in Section 4.


## 2. Our Approach

In the past few years, many metric learning methods [23–27] have been proposed in the machine learning community, and they have been successfully applied in various vision tasks, including face recognition [10, 18, 28–34], activity recognition [35], human age estimation [36], gait recognition [37], and person reidentification [30, 38, 39]. The most existing metric learning methods aim at learning a Mahalanobis distance metric that measures the squared distance between a pair of samples x
i and x
j:
(1)
d
M
2
x
i
,
x
j
=
x
i
-
x
j
T
M
x
i
-
x
j
,
where x
i
,
x
j
∈
R
d and M
⪰
0 is a positive semidefinite (PSD) matrix. Existing metric learning methods for kinship verification (e.g., [10]) are often proposed to minimize interclass variation and maximize intraclass variation simultaneously. In this work, we instead attempt to learn a genetic similarity measure explicitly, such that similarity score of a positive image pair is higher than those of the negative pairs. Let D
=
{
(
x
i
,
y
i
)
}
i
=
1
n be the training set consisting of n positive image pairs (with kinship relation), where x
i
,
y
i
∈
R
d denote the facial images of the ith parent and child, respectively. The goal of this problem is to learn a similarity measure function S
(
x
,
y
) that specifies the kinship constraints on anyquadruple input (
x
i
,
y
i
,
x
k
,
y
l
), where (
x
i
,
y
i
)
∈
D, k
≠
i, l
≠
i. Formally, the kinship constraints for a quadruple input can be written by
(2)
S
(
x
i
,
y
i
)
≥
S
(
x
i
,
y
l
)
+
1
S
(
x
i
,
y
i
)
≥
S
(
x
k
,
y
i
)
+
1
,
where (
x
i
,
y
i
)
∈
D, k
≠
i, l
≠
i, and 1 is a margin constant. There exist two triplet constraints for each quadruple input, on which similarity score of the positive pair is higher than that of image pair from different family. This can be schematically illustrated in Figure 2. We consider here a parametric similarity function S
(
x
,
y
) with bilinear form, S
(
x
,
y
)
≜
x
T
A
y, where A
∈
R
d
×
d. Note that here A is not necessarily symmetric or PSD. Given the similarity function and the constraints in (2), the problem of genetic similarity learning can be formulated as the following optimization task:
(3)
min
⁡
A
∑
i
,
k
,
l
‍
L
x
i
,
y
i
,
x
k
,
y
l
;
A
+
λ
r
A
,
where r
A is a sparse regularization term that limits the model complexity and λ
>
0. Different from the widely used OASIS algorithm [22] designed to deal with the triplet constraints, we aim to directly deal with the quadruple constraints for kinship verification. The loss function L for our problem is defined based on the hinge loss l
:
(4)
L
x
i
,
y
i
,
x
k
,
y
l
;
A
=
l
x
i
,
y
i
,
x
k
;
A
+
l
x
i
,
y
i
,
y
l
;
A
,
where l
(
x
,
y
,
z
;
A
)
=
max
⁡
(
0,1
-
S
(
x
,
y
)
+
S
(
x
,
z
)
). The optimization task of (3) corresponds to a batch learning formulation, and it is less efficient and scalable to practical applications even for medium-size training set. To tackle this issue, we instead aim to learn the similarity measure S
(
x
,
y
) parameterized by A in an online manner.Figure 2
Quadruple constraint for kinship verification. There exist two triplet constraints for each quadruple input, on which the similarity score of positive pair (denoted by solid arrow) is higher than those of image pairs from different family (denoted by dotted arrow).At each timet
=
1,2
,
…, a quadruple instance (
x
i
,
y
i
,
x
k
,
y
l
) is received, and the model is sequentially updated by solving the following optimization problem:
(5)
A
t
+
1
=
arg
min
⁡
A
L
x
i
,
y
i
,
x
k
,
y
l
;
A
+
λ
tr
⁡
A
,
where tr
⁡
A is a trace norm that encourages learning a low rank similarity matrix A to limit the model complexity. The above online optimization can be solved by online gradient descent:
(6)
A
t
+
1
=
A
t
-
η
∇
A
L
x
i
,
y
i
,
x
k
,
y
l
;
A
t
-
η
λ
I
,
where η
>
0 is the learning rate, I is the identity matrix, and ∇
A
L is the subgradient of L with respect to A, which can be computed by
(7)
∇
A
L
=
x
i
(
y
l
T
-
2
y
i
T
)
+
x
k
y
i
T
l
(
x
i
,
y
i
,
x
k
;
A
t
)
>
0
,
l
(
x
i
,
y
i
,
y
l
;
A
t
)
>
0
x
i
(
y
l
T
-
y
i
T
)
l
(
x
i
,
y
i
,
x
k
;
A
t
)
>
0
,
l
(
x
i
,
y
i
,
y
l
;
A
t
)
≤
0
(
x
k
-
x
i
)
y
i
T
l
(
x
i
,
y
i
,
x
k
;
A
t
)
≤
0
,
l
(
x
i
,
y
i
,
y
l
;
A
t
)
>
0
0
otherwise.
The proposed online learning algorithm for genetic similarity measure is outlined in  Algorithm 1.Algorithm 1:Online similarity measure learning for kinship verification.
Input:  Training set D
=
{
x
i
;
y
i
}
i
=
1
n
Output:  Similarity matrix A
(1)Initialize A
0
=
I; t
=
1;
(2)
repeat
(3) Construct a quadruple instance  (x
i
,
y
i
,
x
k
,
y
l) from D;
(4) Compute the loss l
(
x
i
;
y
i
;
x
k
;
A
t
-
1
), l
(
x
i
;
y
i
;
y
l
;
A
t
-
1
);
(5) Compute the sub-gradient ∇
A
L  according to (7);
(6) Update the similarity matrix A
t according to (6);
(7) t
=
t
+
1;
(8)
until  convergence


## 3. Experiment and Discussion

We conducted experiments on the kin datasets that are publicly available [10, 12] (KinFaceW-I and KinFace-II) to verify the effectiveness of our proposed approach. For both datasets, there are four kin relations: mother-son (M-S), mother-daughter (M-D), father-son (F-S), and father-daughter (F-D). All of the facial images have been aligned and cropped into 64 × 64 pixels.We performed 5-fold cross validation on the two kin datasets, where each subset was equally divided into 5-fold so that each fold consists of nearly the same number of image pairs with kin relation. The parametersη and λ are empirically set as 1.0 and 10−4, respectively. For the feature representations of the facial image, three descriptors, SIFT [40], learning-based (SPLE) [13], and local binary patterns (LBP) [41] are used. In our experiments, 256 bins are used for LBP, and we followed the parameter setting in [10] for SPLE and used 200 bins to encode a histogram. For the SIFT descriptors, 16 × 16 patches over a grid with space of 8 pixels are extracted from each face image.We have compared our proposed approach with NRML [10] and OASIS [22]. The former is a state-of-the-art learning-based kinship verification approach, and the latter represents a popular algorithm for online similarity learning. Tables 1 and 2 list the mean verification rate of the three approaches with different features on the KinFaceW dataset. As shown in the two tables, our approach outperforms OASIS and is highly competitive to NRML in terms of verification rate. In addition, we can make the observation that SPLE is the best feature representation for our kinship verification problem. This is mainly attributed to the coding scheme of SPLE that is directly learned from the training examples, leading to better verification accuracy than two other hand-crafted feature descriptors.Table 1
Mean verification rate (%) of different approaches on the KinFaceW-I dataset.
Method
Feature
F-S
F-D
M-S
M-D
Mean
LBP
64.7
65.2
59.4
65.4
63.7
NRML [10]
SIFT
70.5
64.0
64.0
60.4
63.8
SPLE
64.1
59.1
63.9
71.0
64.3
LBP
63.1
64.2
57.5
66.1
62.7
OASIS [22]
SIFT
68.3
63.0
63.4
60.2
63.7
SPLE
63.6
60.3
64.0
69.1
64.2
LBP
67.2
64.7
56.9
65.3
63.5
Proposed
SIFT
70.1
66.0
64.1
61.6
65.5
SPLE
66.1
62.2
64.3
70.0
65.7Table 2
Mean verification rate (%) of different approaches on the KinFaceW-II dataset.
Method
Feature
F-S
F-D
M-S
M-D
Mean
LBP
69.0
69.5
69.8
69.0
69.5
NRML [10]
SIFT
68.0
60.9
60.8
61.4
62.8
SPLE
76.8
73.1
76.8
77.0
75.7
LBP
61.6
63.6
68.7
62.6
64.1
OASIS [22]
SIFT
67.2
69.5
65.2
60.1
65.5
SPLE
66.7
67.7
74.7
73.7
70.5
LBP
69.2
68.3
68.9
69.3
68.9
Proposed
SIFT
68.7
70.7
65.7
61.6
66.7
SPLE
74.9
71.0
76.9
76.4
74.8It should be noticed that, compared to NRML, our approach is superior in terms of scalability for practical applications due to the online learning nature in model training. We believe this is very important for the kinship verification on the large-scale dataset with ever-growing amount of data.


## 4. Conclusion

We have presented a new scheme for kinship verification via facial images in wild conditions by explicitly learning a sparse genetic similarity measure in an online manner. Experimental results on kinship datasets demonstrated that our approach is not only highly competitive to the state-of-the-art algorithm in terms of the verification accuracy but also superior in terms of scalability for practical applications with ever-growing amount of data. For future work, we are interested in investigating the fusion of multiple feature representations within the online learning framework to further improve the overall kinship verification performance. Also, it is helpful for face (kin) verification to work out the similarity of facial components (e.g., eyes or mouths). In this setting, robust face landmarks estimation is often needed to parse the face into facial components. How to exploit it in our proposed approach to work on partially occluded faces or faces viewed from the side appears to be another interesting direction of future work.



---

*Source: 472473-2015-01-31.xml*