Progress in Informatics, No.1

Research Paper(2009.1)

Associating Faces and Names in Japanese Photo News Articles
Akio KITAHARAThe University of Electro-Communications
kitaha-a@mm.cs.uec.ac.jp

Keiji YANAIThe University of Electro-Communications
yanai@cs.uec.ac.jp

ABSTRACT
We propose a system which extracts faces and person names from news articles with photos on the Web andassociates them automatically. The system detects face images in news photos with a face detector and extracts
person names from news text with a morphological analyzer. In addition, the bag-of-keypoints representation isapplied to the extracted face images for ﬁltering out non-face images. The system uses the eigenface representation as image features of the extracted faces, and associates them with the extracted names by the modiﬁed
k-means clustering in the eigenface subspace. In the experiment, we obtained the 66% precision rate regarding
association of faces and names.
KEYWORDS
Web photo news, face detection, face recognition, name extraction, associating names and faces

1

Introduction
Many commercial news sites exist on the Web,and they deliver a lot of news articles every day.
Web news sites are one of important news sources
for many people as well as newspapers and TV news
programs. The advantage of Web news is easy tosearch and store, while newspapers and TV news
programs do not provide eﬃcient search methods andneed large physical or HDD space to store. Due tothis useful feature, by accumulating Web news arti-

usable as a name-annotated face image database.
The rest of the paper is organized as follows. In
Section 2, we present related work brieﬂy. Section 3
describes overview of our system. Section 4 explains
how to extract faces and names from news articles
on the Web, and Section 5 describes how to associate faces and names. In Section 6, we describe theexperimental result. In Section 7, we conclude thispaper.
2

Related Workcles, we can use them as a personal news database.
Nowadays, most of Web news articles include pho-

Face recognition processing is needed to extract

tos as well as news texts. Photos can be under-

faces from Web news photos. M. Turk et al. sug-stood intuitively with just a look without reading.

gested the eigenface method which considered pixel

Many of news photos contain person’s faces related

values of a grayscale image to be elements of anto the news articles. However, sometimes it is hard

image feature vector, compressed the vector by theto identify persons in the news photos, especially inprincipal component analysis (PCA), and assumed

case that a news article includes multiple faces in its

the compressed vector as an image feature for facephoto and multiple person names in its text. In thisrecognition [1]. This method has been proved to per-paper, we try to extract faces from news photos andform well for recognizing frontal face images. We

person names from news texts, and associate them

use the eigenface representation as image features ofautomatically. In the experiments, we used Yahoo!

faces. The detail of the eigenface method is describedJapan News as a Web news source. The results are

in Section 4.3.
P. Viola et al. proposed a face detection technique日本語 Web ニュース記事データを用いた顔画像と名前の自動対応付け

to unify weak classiﬁers which used rectangular fea-

the processing ﬂow of the proposed system.

tures of local regions by AdaBoost [2]. This method
is implemented as a face detection module in the4

Extracting Stage

Open Computer Vision Library (OpenCV) [3], since

4.1

Face Detection

it performs much faster than the other methods. In

At ﬁrst, we extract face regions from Web news

our system, we use this AdaBoost-based face detec-

photos. To do this, we use the Adaboost-based face

tor module included in the OpenCV to extract facedetection module included in the Open Computer

regions from Web news photos.

Vision Library (OpenCV). In our system, we han-T. L. Berg et al. proposed associating faces anddle only frontal face images since the OpenCV canperson names in Web news articles. Their methods

extract only frontal face images.

employed a modiﬁed eigenface method which used

4.2

kernel PCA and LDA as an image representation,

Reﬁnement of Face Images with Bag-ofKeypoints

and performed the modiﬁed k-means clustering to

The face detection function in the OpenCV runs

estimate associations between faces and names [4].

very fast, but their output results usually contain

The method T. L. Berg et al. proposed achieved

some irrelevant non-face images. To ﬁlter out them,

about 95% precision at most in terms of association

we apply the bag-of-keypoints representation and SVM [5].

between faces and names extracted from Web news.

The bag-of-keypoints representation got popular re-

The objective of their work is the same as our work.

cently in the research community of object recogni-

The major diﬀerence between theirs and ours is thetion. It is proven that it has excellent ability to rep-news source as the target of each method. While

resent image concepts in the context of visual object

the target of Berg’s method was Web news written

categorization/recognition in spite of its simplicity.

in English, our target is Japanese Web news. For

The basic idea of the bag-of-keypoints represen-

Japanese news articles, extracting person names istation is that an image is represented by a set ofmuch more diﬃcult than for English articles, sincelocal image features. In our system, to sample awords in Japanese texts are not separated with aset of local image points and represent them, we use

space.

the Scale Invariant Feature Transform (SIFT) [6].

3

Overview of the Proposed System
The processing of our system consists of two stages:

an extracting stage and an associating stage.

The resulting distribution of description vectors isthen quantiﬁed by vector quantization against a prespeciﬁed codebook, and the quantiﬁed distribution
vector is used as an image feature vector of the image.

In the extracting stage, the system detects face

The codebook is generated by the k-means cluster-

images for news photos with a face detector and ex-

ing method based on the distribution of SIFT vectors

tracts person names from news text with a Japanese

extracted from many training frontal face images inmorphological analyzer. Afterwards, the bag-of-keypoints advance. As a classiﬁer to classify images associatedrepresentation [5] is applied to the extracted face towith quantiﬁed vectors as face or non-face, we use aﬁlter out irrelevant face images.

Support Vector Machine (SVM) classiﬁer.

In the associating stage, a candidate list is gen-

By applying the bag-of-keypoint-based ﬁltering, theerated at the beginning. The candidate list records

precision of extracted face images were raised fromname candidates for each extracted face image. Next,

about 70% to 90%.

each face is associated with the person name of the

4.3

Eigenfacebest candidate based on the candidate ﬁle. Eigenface

To associate faces and names, we need to iden-

which is eﬀective for frontal face is used as image fea-

tify extracted face images. As a representation oftures, and the modiﬁed k-means method is used to

extracted face images, we adopt the eigenface repre-

estimate associations between faces and names. Fi-

sentation, which regards the PCA-based-compressed

nally, post-processing is carried out to resolve some

vector of a grayscale face image as a feature vector

conﬂicts from the association results. Figure 1 showsof the face image [1]. In advance, eigenface vectors

Progress in Informatics No.1

(2009.1)

Fig. 1 The processing ﬂow of the proposed system.

are estimated from a set of training frontal face im-¶
Extracted person nameages, and the eigenface subspace spanned by some³

• last name + ﬁrst name (e.g. Koizumi Ju-

principal ones out of all the eigenface vectors arenichiro)generated. In the experiments, we select the prin-• last name + position title (e.g. Koizumi

cipal components corresponding to the several high-Prime-Minister )est eigenvalues so that the cumulative percentage ofthe sum of eigenvalues of the selected components• last name + country + position title (e.g.

to the total sum of all the eigenvalues is more thanBlair Great-Britain Prime-Minister)

90%. The PCA compression is carried out by pro• ﬁrst name +・+ last name (e.g. Michel・

jecting the vector of a greasily face image into theeigen subspace. In our system, all the extracted face
images are converted into compressed vectors on the
eigen subspace.
4.4

Extracting Person Names

A Japanese morphological analyzer “ChaSen”[7] isJackson (rule for Western person’s name))µIn case that the 14 following nouns appear justafter family names, it is regarded as position titles.
¶³
position titles

used to extract person names from texts of Web news

prime minister, president, minister, prince,

articles. Based on the output of “ChaSen”, we ex-

princess, emperor, empress, supervisor, head,tract person names with the following four patterns

coach, CEO, governor, representative, secre-with respect to parts of speech.

´tariatµThe names of journalists who wrote the article
texts, and the names of the photographer who took
the photo sometimes are included in the news article
texts. To ignore them, we drop the names extracted
from the last line of article texts. This processing is
eﬀective only for Yahoo! Photo News which is thetarget of our system.

´日本語 Web ニュース記事データを用いた顔画像と名前の自動対応付け

5

Associating Stage¶
modiﬁed k-means clustering (1)5.1

Candidate List

1. Decide k cluster centers

At the associating stage, ﬁrst of all, the candidate

Each cluster center is chosen from eigen-list is generated. At ﬁrst, we ignore all the articles

face vectors of all the face images thatfrom which only faces, only names or none of bothhave the person name associated with theare extracted, since our system associates names andcluster. If a one-to-one correspondence re-faces extracted within the same article. Next, for

garding a name and a face in the candidate

the news articles from which both faces and per-list is found, the vector of the face is as-

son names are extracted, uniﬁcation of person names

signed to the cluster center of the name.

within a article is carried out. Some article texts in-

If not, a center of a name cluster is se-clude several names for the same one person such as

lected randomly from all the vectors of the“Junichiro Koizumi” and “Koizumi Prime-Minister”

faces which have the same name in their

within the same article text. To handle that, wename candidates. The cluster centers areunify names if the family names are regarded to be

assigned in the ascending order of the to-the same. Note that this simple uniﬁcation is valid

tal number of the candidate names so that

only within the same article text. If we do this acrosscenters of all name clusters can be decided.

the articles, all the person names which have the2. Associate faces to name clusters

same family name will be uniﬁed into one person

Each face except one among one-to-one

name. Finally, we generate the candidate list whichcorrespondence pairs is assigned to therecords all the name candidates for each extracted

nearest center of all the person name clus-

face image.

ters in the eigenface subspace. The eigen-

5.2

face representation is used to represent

Modiﬁed k-means Clustering

Each face in the candidate list is associated withface images, and we use the Euclid dis-

one of the candidate person names by the modiﬁedtance as a metric in the eigenface sub-k-means clustering. The value of k is assumed tospace. In case that the distance exceeds

be the number of kinds of person names. In short,the predeﬁned threshold, the face is noteach cluster corresponds to one unique person name.
So that for one-to-one correspondence pairs whichmean that only one name and one face are extracted
from the same article, the name cluster to which the³assigned to any clusters.
µ¶
modiﬁed k-means clustering (2)´³

3. Calculate cluster center

face is assigned is ﬁxed and not changed during the

The center vector of each cluster is calcu-

loop of the k-means clustering. That is the diﬀer-

lated again by averaging all the vectors ofence between the normal k-means and the modiﬁed

the assigned faces.

k-means. The modiﬁed k-means clustering is carriedout according to the following procedure:

4. Judge repeating or ﬁnishing
If all the cluster centers do not change, theprocessing is ﬁnished. Otherwise, return
µ
5.3

to 2.
Post-processing

As a result of the modiﬁed k-means clustering, two
or more faces extracted from the same certain news
might be associated with the same person name. To
resolve such conﬂict, if two or more faces extracted
from the same news are associated with the same
person name, only the nearest face to the center of´Progress in Informatics No.1

(2009.1)

the person name cluster is kept, and all the otherfaces are removed from the cluster. After this processing, the output is regarded as the ﬁnal output offace-name associations.
6

Experimental Setting
We collected news articles with photos and texts
Fig. 3 Relevant faces extracted from news
photos.

from Yahoo! JAPAN Photo News since March 6,2005 to August 31, 2006 for one and half year. In theexperiments, we used them as experimental data.
Being diﬀerent from usual news articles, the maincontents of articles of the Yahoo Photo News arephotos. In addition to a photo, each article has a title
and a short main text which explain the contents ofFig. 4 Irrelevant faces extracted from news
photos.

the photo (Figure 2).

use it for all the other experiments.
To evaluate results, we use the precision rate forall of the experiment results. The precision rate isdeﬁned as follows:
P recision =the number of relevant f ace output images
the number of all f ace output images(1)

As an evaluation measure for these kinds of work,
we have not only the precision rate but also the recall
rate. However, since the number of faces included
in news articles on the Web is enormous, the more
Fig. 2 An example of an article of the Yahoo!
JAPAN Photo News.

As pre-processing, to obtain the eigenface space,
we selected 100 relevant face images by hand fromphoto news articles we fetch from the Web, the more
face images we can get apparently. Therefore, wegive priority to improving the precision rate.
7

Experimental Results

all the extracted face images, normalized them to
50 × 50, and applied PCA in advance.

7.1

Results of Face Extraction

When we apply the modiﬁed k-means, we set a

Figure 3 and Figure 4 show the results of face

threshold regarding the distance between a face im-images extracted from the news photos of Yahoo!

age and the center of a cluster which decides if the

Photo News. The total number of extracted face im-face image is joined into the name cluster, or not. Asages is 38,650. As shown in Figure 4, some output

a distance, we used the Euclidean distance. In theimages were irrelevant images, which included necks

experiment, we set the threshold as the square rootof the persons who wore neckties, waists of the per-of the following four kinds of values: (1) none, (2)sons who put on uniforms, and upper half of circles.

6,000,000, (3)4,000,000, and (4) 2,000,000.

Since it was diﬃcult to investigate whether all the ex-

Regarding face-image-ﬁltering with the bag-of-keypoints tracted images are correct as faces, instead, we chose
100 images randomly and examined them. As a remethod, we made experiments with four kinds of thesize of the codebook: 100, 300, 500, and 800. By thesult, the precision rate of face extraction is estimatedexperiments we select the best size among them andto be 78%.

日本語 Web ニュース記事データを用いた顔画像と名前の自動対応付け

Table 1 Result of ﬁltering non-face.

their precision rate evaluated by random sampling ofcode words size

faces

precision

100 associates for four kinds of evaluation strategies

100

23,12493%

and four kinds of thresholds. In the table, “ALL”

300

26,40595%

and “OVER 3” mean all the association results and500

25,69997%

the association results of names associated with more

800

26,00698%

than three faces, respectively. Both are the results
without bag-of-keypoint-based face ﬁltering. In addition, “BoK-ALL” and “BoK-OVER 3”mean the re-7.2

Results of Face Filteringsults with bag-of-keypoint-based (BoK-based) face

The estimated precision rate 78% was not enoughﬁltering. We made experiments with four kinds offor associating faces and names. Then, we ﬁlter out

thresholds shown in the table, which have been al-irrelevant images from face images extracted by theready explained in Section 6.

AdaBoost-based face detector. The results that non-Table 2 shows that the most strict threshold (4)face images are excluded by the bag-of-keypoints rep-with BoK-based face ﬁltering gave the best associa-

resentation are shown in the Table 1. In case that thetion result among all the results. Although the best

size of codebook was 800, the best precision rate was

precision rate 66% was obtained, the number of asso-

obtained, so that we used this size for all the otherciations was only 223 in the same setting, which was

experiments. Introducing the bag-of-keypoint-based

only about one forty-second of the result of thresholdface ﬁltering raised the precision rate from 78% to(1) without BoK. In case of the threshold (4) with-

98%, which has proved to be very eﬀective.

out BoK, only three face images were associated with

7.3

the person name ranked in the 21st in terms of theResults of name extraction

13,579 kinds of unique person names were extracted

number of associated face images.

in the experiments. In the same way as evaluation

Compared with “ALL”, “OVER 3” tends to betterof extracted faces, we evaluated 100 kinds of namesin the precision rate. This implies that the precision

randomly selected from all the extracted names. As

rates of names with the less faces were relatively de-a result, the estimated precision rate is 78%.

graded.

Some examples of extracting person names correctly and incorrectly are shown as follows.

As described in Section 7.2, the precision rate ofThe

detected faces are improved much by the bag-keypoint-

following three incorrectly-extracted names resulted

based face ﬁltering. However, regarding the results offrom failure of separation of words by the morpho-

association of faces and names, there was no promi-

logical analyzer. In the third case of failure of namenent improvement between “non-BoK” and “BoK”.

extraction, two person’s names were extracted as one

This is because the precision rate of extracted names

person name.
¶Some results of name extraction
Relevant names
Koizumi Junichiro, Matsui Hideki,

To clarify the “trade-oﬀ” relation between the number of associations and the precision rate, we show

President Bush, Bobby Valentine

a graph on the results of BoK-OVER3 in Figure 5.

Irrelevant names

The more strict the threshold got, the less associa-

Sawa Ichiro (correct name:Aisawa Ichido),
Toda Megumi (correct name: Toda Erica),
Koichi・Suzuki (correct name: Saito Koichi,
Suzuki Yoshizumi)µ
7.4

still remains relatively low. Therefore, we need to³ improve the name extraction part in our system.

tions were obtained while the more the precision rateincreased.
Figure 6 shows all the face images associated withthe top ﬁve person names regarding the number of´ associated faces, and Table 3 shows the precision
rates of the results of the ﬁve names. Note that
Results of Association of Faces and Names

Table 2 shows the results of associating faces andalthough “Koizumi Junichiro” and “Prime Ministernames, which include the number of associations andKoizumi” represent the same person, our system can-Progress in Informatics No.1

(2009.1)

was obtained at most.
Thanks to the bag-of-keypoint-based face ﬁltering, we obtained the 98% precision rate in termsof face detection.

On the other hand, the preci-sion rate of name extraction remains 78%. This ispartly because the proposed system prepares only
four kinds of patterns to extract person names fromnews article texts.

Names without titles or fam-

ily names without ﬁrst names were not extracted,
Fig. 5 The precision rates for each threshold.

since the patterns rely on position titles or combination of ﬁrst and family names. The other reason

Table 3 The number of associated face images and the precision rates of the
ﬁve names shown in Figure 6.

of relatively low precision rate regarding name extraction is that a Japanese morphological analyzer“ChaSen” we used in the experiments cannot ex-person namefaces

precision

tract person names except names registered in its

Koizumi Junichiro

98

72.4%

dictionary. Therefore, the system was not able to ex-

Prime Minister Koizumi

55

40.0%

tract uncommon Japanese names and names of non-President Bush

42

64.3%

Japanese persons. Basically, extracting names from

Zico coach∗

36

66.7%

Japanese texts is more diﬃcult than from English

Abe Shinzo
21
100%(*..former head coach of the Japanese national

texts, because words are not separated by a space.

soccer team)ticated name-extraction method to extract person

For future work, we plan to introduce more sophisnames from Web news articles.

not identify them as the same person since it has noknowledge about the Japanese prime minister. However, they can be uniﬁed as the same person by hand

References[1]

easily as post-processing. The precision ratio 100%
was obtained for “Abe Shinzo”, since he never shows[2]

his emotions on his face and his facial expression always stays unchanged.
8[3]Conclusions and Future Work
[4]

In this paper, we proposed the system which extracts faces from news photos and person names innews article texts and associated them. In the ex-[5]periments, 26,006 face images have been extracted
in the precision rate 98% with the AdaBoost-based[6]

face detector and the bag-of-keypoint-based ﬁltering,and 13,579 kinds of unique person names have been
extracted in the precision rate 78% with a Japanesemorphological analyzer “ChaSen” and a name extraction method based on predeﬁned patterns. After extracting faces and names, they are associatedby the modiﬁed k-means clustering in the eigenface
space. In the experiments, the 66% precision rate

[7]

M. Turk and A. Pentland, “Face recognition using
eigenfaces,” Proc. of Computer Vision and Pattern
Recognition, pp. 586–591, 1991.
P. Viola and M. Jones, “Rapid object detection using aboosted cascade of simple features,” Proc. of Computer
Vision and Pattern Recognition, vol. 1, pp. 511–518,
2001.
Open
SourceComputer
VisionLibrary,
http://www.intel.com/technology/comput
ing/opencv/.
T.L. Berg, A.C. Berg, J. Edwards, M. Maire, R. White,
Y.W. Teh, E. Learned-Miller, and DA Forsyth, “Names
and faces in the news,” Proc. of Computer Vision and
Pattern Recognition, pp. 848–854, 2004.
G. Csurka, C. Bray, C. Dance, and L. Fan, “Visual categorization with bags of keypoints,” Workshop on Statistical Learning in Computer Vision,European Conference on Computer Vision, pp. 1–22, 2004.
D.G. Lowe, “Distinctive Image Features from ScaleInvariant Keypoints,” International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004.
ChaSen, http://chasen.naist.jp/hiki/ChaSen/.

日本語 Web ニュース記事データを用いた顔画像と名前の自動対応付け

Table 2 Results of Association.

ALL

OVER 3

BoK-ALL

BoK-OVER 3

threshold2

faces

precision

faces

precision

faces

precision

faces

precision(1) none

9,329

34%

9,108

32%

7,003

42%

6,802

40%

(2) 6,000,000

6,480

42%

6,027

46%

4,947

42%

4,578

47%

(3) 4,000,000

3,853

40%

3,125

44%

2,852

43%

2,282

46%

(4) 2,000,000

1,281

54%

379

62%

960

44%

223

66%

Fig. 6 The top ﬁve person names regarding the number of associated faces.