An Experiment on Generic Image Classification
Using Web Images
Keiji Yanai
Department of Computer Science, The University of Electro-Communications
1-5-1 Chofugaoka, Chofu-shi, Tokyo 182-8585, JAPAN, E-mail: yanai@cs.uec.ac.jp
Abstract. In this paper, we describe an experiment on generic image
classiﬁcation using a large number of images gathered from the Web aslearning images. The processing consists of three steps. In the gathering
stage, a system gathers images related to given class keywords from the
Web automatically. In the learning stage, it extracts image features fromgathered images and associates them with each class. In the classiﬁcation
stage, the system classiﬁes a test image into one of classes corresponding
to the class keywords by using the association between image featuresand classes. In the experiments, we achieved a classiﬁcation rate 44.6%
for generic images by using images gathered from the World-Wide Web
automatically as learning images.

1

Introduction

Due to the recent spread of digital cameras, we can easily obtain digital images of various kinds of real world scenes, so that demand for image recognitionof various kinds of real world images becomes greater. It is, however, hard to
apply conventional image recognition methods to such generic recognition, because most of their applicable targets are restricted. Therefore, at present, it is
impossible to deal with semantics of images of real world scene automatically.
Henceforth, it is desired that automatic attaching keywords to images, classiﬁcation and search in terms of semantic contents of images.
So far, automatic attaching keywords[1] and semantic search[2] for an image
database have been proposed. In these works, since learning images with correct
keywords were required, commercial image collections were used as learning images, for example, Corel Image Library. However, most of images in commercial
image collections are well-arranged images taken by professional photographers,
and many similar images are included in them. They are diﬀerent from imagesof real world scenes taken by the people with digital cameras.
In this paper, we propose utilizing images gathered from WWW (World-WideWeb) as learning images for generic image classiﬁcation instead of commercial
image collections. In other words, this research is Web image mining for genericimage classiﬁcation. We can easily extract keywords related to an image on the
Web (Web image) from the HTML ﬁle linking to it, so that we can regard a
Web image as an image with related keywords. Web images are as diverse asreal world scene, since Web images are taken by a large number of people forvarious kinds of purpose.

test images

WWW
bear
cat
cow
dog
lion

WWW
image
gatheringsystemwords

bear image
cat imagecow imagedog imagelion image

gathered
imagesimage
classification
systemlearning
images

bear
cat
cow
dog
lion

classification
results

Fig. 1. Image classiﬁcation by Web images.

Image classiﬁcation by Web images is performed by combination of an image
gathering system and an image classiﬁcation system (Fig.1). First, we gather
images related to some kinds of words from the Web by utilizing the Image
Collector, which we have proposed in [3]. Next, we extract image features fromgathered images and associate image features with words for image classiﬁcation.
Finally, we classify an image into one of classes corresponding to class keywords
by comparing its image features with ones of images gathered from the Web inadvance.
In this paper, we describe image gathering from the Web, learning and classiﬁcation. Next, we describe experimental results and conclusions.

2

A Method of Image-Gathering

An image-gathering system gathers images from WWW related to the keywords
given by a user. Note that our system is not called an image “search” system but
an image “gathering” system, since our system has the following properties: (1)it does not search for images over the whole WWW directly, (2) it does not makea index of the Web images in advance, and (3) it makes use of search results ofcommercial keyword-based search engines for query keywords. These propertiesare diﬀerent from conventional Web image search systems such as WebSeer[4],WebSEEk[5] and Image Rover[6]. These systems search for images based on thequery keywords, and then a user selects query images from their search results.
These three systems carry out their search in such an interactive manner. Our
system is diﬀerent from those in that our system only needs one-time input of
query keywords due to automatic image selection mechanism described later.
Since an image on WWW is usually embedded in an HTML document that
explains it, the system exploits some existing commercial keyword-based WWW
search engines, and it gathers URLs (Universal Resource Locator) of HTML
documents related to query keywords. In the next step, using those gathered
URLs, the system fetches HTML documents from WWW, analyzes them, andevaluates the intensity of relation between the keywords and images embedded
in HTML documents. If it is judged that images are related to keywords, theimage ﬁles are fetched from WWW. According to the intensity of relation to the
keywords, we divide fetched images into two groups: images in group A having
stronger relation to the keywords, and others in group B. For all gathered images,
image features are computed. We use a color histogram in the Lu∗ v∗ color space
as image features.
In content-based image retrieval (CBIR), a user provides query images or
sketches to the system, because it searches for images based on the similarity ofimage features between query images and images in an image database. In ourimage-gathering system, instead of providing query images or sketches, a user
only needs to provide query keywords to the system. Then, we select images
strongly related to the keywords as group A images, remove noise images fromthem, and regard them as query images only by examining keywords. Removing
noise images is carried out by eliminating images which belong to relatively
small clusters in the result of image-feature-based clustering for group A images.
Images which are not eliminated are regarded as appropriate images to the query
keywords, and we store them as output images. Next, we select images that are
similar to the query images from group B in the same way as CBIR, and addthem to output images. The detail is described in [3].

3A Method of Learning and Classification

We make experiments on image classiﬁcation for images gathered from the Web
by image-feature-based search. First, we extract image features from gathered
images and associate image features with classes represented by keywords in thelearning stage. Next, we classify an image into one of classes corresponding toclass keywords by comparing image features in the classiﬁcation stage.
3.1

Signatures and Earth Mover’s Distance

We exploit two kinds of image features for learning and classiﬁcation: color signature for block segments, and region signature for region segments. A signaturedescribes multi-dimensional discrete distribution, which is represented by a setof vectors and weights. In case of color signatures, a vector and a weight correspond to a mean color vector of each cluster and its ratio of pixels belonging tothat cluster, respectively, where some color clusters are made in advance by clustering color distribution of an image. Since the number of elements of a signature
is variable, it is superior to conventional ﬁxed-size color histograms in terms ofexpressiveness and eﬃciency. In case of region signatures, a set of feature vectorsof regions and their ratio of pixels represents a region signature.
To compute dissimilarity between two signatures, Earth Mover’s Distance(EMD)
has been proposed[7]. Intuitively, given two signatures, one can be seen as a massof earth properly spread in the feature space, the other as a collection of holes inthe same space. Then, the EMD measures the least amount of work needed to
ﬁll the holes with earth. Here, a unit of work corresponds to transporting a unitof earth by a unit of ground distance which is a distance in the feature space.

The EMD is based on the transportation problem and can be solved eﬃciently
by linear optimization algorithms.
Formally, let P = {(p 1 , wp1 ), ..., (p m, wpm )} be the ﬁrst set with m elements,
where pi is the feature vector and wpi is its weight; Q = {(q1 , wq1 ), ..., (q n, wqn )}the second set with n elements; and dij = d(pi , qj ) the ground distance matrix
where dij is the distance between pi and qj . The EMD between sets P and Q
is thenm n
i=1
j=1 fij dij(1)
EMD(P, Q) = m n
i=1
j=1 fij
where F = [fij ], with fij ≥ 0 the ﬂow between pi and qj , is the optimal admissible ﬂow from P to Q. In addition, an easy-to-compute lower bound for the
EMD between signatures with equal total weights is the distance between theircenters of mass.
3.2

Color Signatures

To obtain color signatures, ﬁrst, we normalize the size of learning images into
240 × 180, and divide them into 16 and 9 block regions as shown in Fig. 2. We
make a color signature for each of these 25 block regions. Next, we select somedominant colors by clustering color vectors of each pixel into color clusters by
the k-means method. In the experiments, the number of color clusters is 15 orless, and it is decided in order not to make a cluster whose weight is less than
0.005. We make a color signature for each block with elements consisting of amean color vector of each cluster and its ratio of pixels belonging to that cluster.
A mean color vector is represented by the Lu∗ v∗ color space which is designed inorder that Euclid distance between two points in this space matches the humancolor sense, so that we use Euclid distance as ground distance.
In the classiﬁcation stage, ﬁrst, we extract color signatures from each blockin an image to be classiﬁed (a test image) in the same way as the learning stage
after normalizing its size. We obtain 25 sets of signatures for one test image.
Next, we search all blocks of learning images of each class for the block withthe minimum distance (dissimilarity) to each block of the test image. Here, thedistance is computed by the EMD. In the next step, we sum up the minimum
distances between the test image and learning images of each class for 25 allblocks. This search and computation is carried out for all the classes. We compare
the total distances among all the classes, and we classify the test image into theclass whose total distance is the smallest. In the actual implementation, we used
lower bound of the EMD to reduce a frequency of computation of the EMD.
3.3

Region Signatures

To obtain region signatures, we carry out region segmentation for images insteadof dividing images into block segments after normalizing their size (Fig.3). Many
methods of region segmentation have been proposed so far. Here, we employ asimple segmentation method based on k-means clustering used in [8]. First, we60x45block region

original
image

image
240x180color signature( (p1,w1 ), ,(pn,wn) )x 25 sets selecting
x 25 sets
dominant color(up to 15 colors)

Fig. 2. Color signatures for color segments.

region signature
original
image

image
240x180( (f1 ,w1 ), ,(fn,wn) )regionsegmentation

Fig. 3. Region signatures for region segments.

divide a learning image into 4 × 4 small blocks, and for each block we compute amean color vector in the Lu∗ v∗ color space and a texture feature vector, which
consists of square means of HL elements, LH elements and HH elements obtainedby Daubechies-4 wavelet transform to each 4 × 4 block. Both vectors are threedimension, so that a six-dimension feature vector is obtained for each block.
Next, we cluster all blocks in a learning image into some regions by the k-means
method in the similar way as computing color signatures. In the experiments,
the number of color clusters is 15 or less, and it is decided in order not to makea cluster whose weight is less than 0.005. Then, we compute a mean 6-dimension
feature vector for each region. In addition, for making a region signatures we
extract three more features about shape of a region. We use normalized inertia
of order 1 to 3 as three features to describe shape of a region. Finally, we makea region signature with elements consisting of a nine-dimensional feature vector
for each region and its ratio of pixels belonging to that region.
In the classiﬁcation stage, we employ the k-nearest neighbor (k-NN) method
to classify a test image into a class. The value of k is decided as 5 by preliminary
experiments. We used Euclid distance as ground distance to compute the EMD.
In our method of image classiﬁcation, image features of not only a target
object but also non-target objects such as background included in the image
are used as a clue of classiﬁcation, since non-target objects usually have strong
relation to a target object. For example, a cow usually exists with grass ﬁeld
and/or fence in farm, and a lion usually exists in Savannah or zoo. Although
the number of combination of a target object and non-target objects is large,
we think that we can deal with this largeness by gathering a large amount ofimage from the Web and using them as learning images. Here, we do not set up”reject”, and then all test images are classiﬁed into any class.

Table 1. Nine experiments.
num.of num.of precision test images

no. classes images
1 10
4548
2 10
3102
3 10
500
4 10
4548
5 10
3102
6 20
5694
7 20
3485
8 20
5694
9 20
3485

(%) num.

68.2
100†
100‡
68.2
100
61.2
100†
61.2
100†

src.

20 WWW
20 WWW
10 Corel
50 Corel
50 Corel20 WWW
20 WWW
20 special
20 special†selection of correct images by hand.
‡Corel Image as a learning set．

4

Table 2. Results of image-gathering (left) andclassiﬁcation (right) in experiment no.1
class
bear
cat
dog

num. pre. rec. pre. F
419 56.4 21.0 31.1 25.1
354 62.0 28.0 60.9 38.4
570 75.7 40.0 23.3 29.4
elephant
506 65.5 25.0 23.1 24.0
tropical fish 275 89.9 22.0 74.6 34.0
lion
504 77.0 45.0 25.2 32.3
penguin
576 57.0 33.5 29.0 31.1
sheep
347 64.0 13.0 34.2 18.8
tiger
405 68.7 24.0 32.2 27.5
whale
592 72.4 66.5 39.0 49.2
total/avg. 4582 68.2 31.8 37.3 34.3
by region sig.

29.4 30.3 29.8

Experimental Results

We made nine experiments from no.1 to no.9 shown in Table 1.
In the experiment no.1, we gathered images from the Web for 10 kinds ofwords related to animals shown in Table 2. In the image-gathering processing,about ten thousands URLs were fetched from six major text search engines,
Google, InfoSeek, Excite, Lycos, InfoNavi and Goo Japan. The total number ofgathered image was 4582, and the precision by subjective evaluation was 68.2%,
which is deﬁned to be NOK /(NOK + NN G ), where NOK , NN G are the numberof relevant images and the number of irrelevant images to their keywords. In theleft side of Table 2, we show the number and the precision of gathered images.
In the image classiﬁcation experiment, we regard each of the 10 words as one
class. In the right side of Table 2, we show the classiﬁcation result evaluated by
10-fold cross-validation. In this section, tables describe results by color signatures
mainly, and results by region signatures are shown only in the bottom line ofeach table. In the table, the recall is deﬁned to be MOK /Mtest , the precision isdeﬁned to be MOK /(MOK + MN G ) and F-measure is the harmonic mean of therecall and the precision, where MOK , MN G , Mtest are the number of correctlyclassiﬁed images, the number of incorrectly classiﬁed images, and the number oftest images for each class, respectively. All values are represented in percentage.
In the experiment no.1, we obtained 31.0 as the F-measure value.
In the experiment no.2, we select only correct images for each class fromgathered images by hand, and the classiﬁcation experiment was carried out using
them. The result is shown in Table 3. Compared to no.1, the F-measure increased.
Especially, the result of “whale” was good, since most of “whale” images on the
Web were images of “whale watching” scene.
In the experiment no.3, we made a classiﬁcation experiment not for Web
images but for the 500 images of 10 classes picked up from Corel Image Gallery.
The classiﬁcation result evaluated by 10-fold cross-validation is shown in TableTable 3. Results of image-gathering and classiﬁcation in experiment no. 2, 3, 4, 5
exp. no.2
exp. no.3
exp. no.4
exp. no.5
rec. pre. F rec. pre. F rec. pre. F rec. pre. F
17.1 46.2 25.0 36.0 62.1 45.6 8.0 15.4 10.5 4.0 40.0 7.3
34.3 78.7 47.8 61.2 85.7 71.4 4.1 33.3 7.3 6.1 42.9 10.7
58.6 21.5 31.4 24.0 75.0 36.4 24.0 14.8 18.3 58.0 21.3 31.2
elephant
25.0 32.1 28.1 68.0 69.4 68.7 34.0 34.7 34.3 16.0 25.8 19.8
tropical fish
35.7 62.5 45.5 58.0 93.5 71.6 22.0 61.1 32.4 30.0 46.9 36.6
lion
47.9 35.1 40.5 82.0 77.4 79.6 30.0 19.5 23.6 36.0 27.3 31.0
penguin
47.9 27.3 34.8 50.0 42.4 45.9 26.0 19.7 22.4 48.0 25.5 33.3
sheep
17.1 36.4 23.3 80.0 46.0 58.4 8.0 23.5 11.9 4.0 18.2 6.6
tiger
10.7 60.0 18.2 72.0 69.2 70.6 4.0 7.4 5.2 10.0 45.5 16.4
whale
75.0 55.6 63.8 94.0 53.4 68.1 86.0 32.6 47.3 86.0 40.6 55.1
avg. by color 36.9 45.5 40.8 62.5 67.4 64.9 24.6 26.2 25.4 29.8 33.4 31.5
class
bear
cat
dog

avg. by region 35.4 37.2 36.2 67.1 69.2 68.1 23.2 20.7 21.9 26.0 22.8 24.3

3. Since Corel Image Gallery includes many similar images to each other, a highF-measure value, 68.1, was obtained by region signatures.
In the experiment no.4 and no.5, we used the gathered images in the experiment no.1 and no.2 as learning images and the Corel images as test images. The results are shown in Table 3. In no.4 and no.5, we obtained 25.4 and31.5 as F-measure, respectively. Since “dog”, “tropical ﬁsh”, “lion”, “penguin”
and “whale” have some typical patterns and both of the gathered images andthe Corel images include the images with the typical patterns, their F-measureachieved high values. On the other hand, since “bear”, “cat”, “elephant”, “sheep”
and “tiger” had no typical patterns, their F-measures were relatively low.
In the experiment no.6 and no.7, we made an experiment for 20 words (Table
4) which includes many diﬀerent kinds of words in the same way as the experiment no.1 and no.2. Compared to the expected F-measure, 5.0, in case of therandom classiﬁcation, we obtained much better F-measure, 42.3 and 46.7 shownin Table 5. These results are superior to the result of the experiment no.1 andno.2 for only 10 classes, because all classes used in no.1 and no.2 are related toanimals and their learning images include many similar images even between diﬀerent classes. In case of “apple”, “Kinkaku Temple” and “noodle”, their result
were about 60.0, since their scene have some typical patterns and many of theirimages were applicable to them. On the other hand, for “house” we obtained
only a very low F-measure value, since “house” images had much variation. Fromthese results, diﬃculty of classiﬁcation depends on the properties of the class.
It is hard to collect such various kinds of images as images used in the experiment no.6 and no.7 by means of commercial image databases, and it has cometo be possible only by image-gathering from the World-Wide Web.
In the experiment no.8 and no.9, we used the gathered images in the experiment no.6 and no.7 as learning images and a special test image set as test images.
We make a special test image set by selecting various kinds of 50 typical images

Table 4. 20 class keywords
20: apple, bear, bike, lake, car, cat, entrance ceremony, house, Ichiro, Ferris wheel, lion,
Moai, Kinkaku Temple, note PC, bullet train, park, penguin, noodle, wedding, Mt.Yari
Table 5. Results of experiment no.6, 7, 8, 9
exp. no.6
exp. no.7
exp. no.8
exp. no.9
classrec. pre. F rec. pre. F rec. pre. F rec. pre. F
avg. by color 34.9 53.6 42.3 35.7 67.8 46.7 39.8 50.7 44.6 38.5 60.4 47.0
avg. by region 34.3 37.7 35.9 37.0 45.5 40.8 40.1 43.1 41.5 42.1 47.9 44.8

for each class from Corel Image Gallery and Web images by hand. The classiﬁcation results are shown in Table 5. In no.8 and no.9, we obtained 44.6 and 47.0
as F-measure, respectively. These results are comparable to conventional works
of generic image recognition. However, unlike them, we provide learning images
not by hand, but by gathering images from the World-Wide Web automatically.

5

Conclusions

In this paper, we described experiments on generic image classiﬁcation using
images gathered from the World-Wide Web. While the main targets of the conventional works on knowledge retrieval from the Web are numeric data and text
data, we have proposed knowledge retrieval of image data from the Web in thispaper. For future works, we plan to make much improvement in classiﬁcation
methods and extraction of image features to obtain more improved classiﬁcation
rate.

References
1. Barnard, K., Forsyth, D.A.: Learning the semantics of words and pictures. In: Proc.
of IEEE International Conference on Computer Vision. Volume II. (2001) 408–415
2. Belongie, S., Carson, C., Greenspan, H., Malik, J.: Recognition of images in large
databases using a learning framework. Technical Report 07-939, UC Berkeley CS
Tech Report (1997)
3. Yanai, K.: Image collector: An image-gathering system from the World-Wide Web
employing keyword-based search engines. In: Proc. of IEEE International Conference of Multimedia and Expo. (2001) 704–707
4. Framkel, C., Swain, M.J., Athitsos, V.: WebSeer: An image search engine for theWorld Wide Web. Technical Report TR-96-14, University of Chicago (1996)
5. Smith, J.R., Chang, S.F.: Visually searching the Web for content. IEEE Multimedia
4 (1997) 12–20
6. Sclaroﬀ, S., LaCascia, M., Sethi, S., Taycher, L.: Unifying textual and visual cues
for content-based image retrieval on the World Wide Web. Computer Vision andImage Understanding 75 (1999) 86–98
7. Rubner, Y., Tomasi, C., Guibas, L.J.: The earth mover’s distance as a metric forimage retrieval. International Journal of Computer Vision 40 (2000) 99–121
8. Wang, J.Z., Li, J., Wiederhold, G.: SIMPLIcity: semantics-sensitive integrated
matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence 23 (2001) 947–963