Web Image Mining: Can We Gather Visual Knowledge
for Image Recognition from the Web ?
Keiji Yanai

yanai@cs.uec.ac.jp
Department of Computer Science, The University of Electro-Communications

Abstract
Because of the wide spread of digital imaging devices andthe World Wide Web, we can easily obtain digital imagesof various kinds of real world scenes. Currently, however,classiﬁcation/recognition of generic real world images is far
from practical due to a diversity of real world scenes.
To deal with such diversity, we have proposed gathering real world images from the World-Wide Web and using
them as training images for image classiﬁcation. We callthis research project ”Web Image Mining”. Web images
are as diverse as real world scene, since Web images aretaken by a large number of people for various kinds of purpose. It is expected that diverse training images enable us
to classify/recognition diverse real world images. In this paper, we describe our ongoing project, “Web Image Mining
for Generic Image Recognition”.

1

Introduction

Due to the recent spread of digital imaging devices suchas digital cameras, we can easily obtain digital images ofvarious kinds of real world scenes. Therefore, the demand
for generic image recognition/classiﬁcation of various kindsof real world images becomes greater.
So far, automatic attaching keywords [1, 4, 6, 12] andsemantic search [2] for an image database have been proposed. In these works, since training images with correct
keywords were required, commercial image collections wereused as training images, for example, Corel Image Library.
However, most of images in commercial image collections
are well-arranged images taken by professional photographers, and many similar images are included in them. They
are diﬀerent from images of real world scenes taken by thepeople with commodity digital cameras.
All of the existing works quoted above focused on
the classiﬁcation methods. However, we consider thatthe important elements for generic image classiﬁcation/recognition are not only classiﬁcation methods but
also training images. Even if the classiﬁcation method has
high ability, the system cannot work for various images sufﬁciently in case that a diversity of learning images is notenough. A diversity of images real world scenes is extremely
high in general, so that we have to gather as diverse images
as possible to make system more practical. It is almost impossible to gather such various images of various kinds ofreal world scenes exhaustively by hand.
Then, we have proposed gathering visual knowledge forgeneric image classiﬁcation/recognition of real world scenes

from the World Wide Web[14]. We call this project “WebImage Mining for Generic Image Recognition”. To say it
concretely, our system utilizes images gathered automatically from the World Wide Web as training images forgeneric image classiﬁcation instead of commercial image
collections. We can easily extract keywords related to an
image on the Web (Web image) from the HTML ﬁle linking
to it, so that we can regard a Web image as an image withrelated keywords. Web images are as diverse as real world
scenes, since Web images are taken by a large number ofpeople for various kinds of purpose. It is expected that diverse training images enable the system to classify diverse
real world images.
Assuming that we gather all the images in the wholeworld, a newly generated image must be very similar tosome of the gathered images. This expectation is basedon the heuristic that most of the scenes we see in our everyday life are similar to the scenes we have ever seen. In
addition, people usually take pictures not at random but
intentionally. They tend to place their targets on the center of pictures. In most case, the targets are parts of thereal world scene such as objects and views. Although adiversity of scenes in the real world is huge, a diversity ofimages, which are taken as photos by the people, is lessthan that. Therefore, we expect that diverse training images must enable the system to classify diverse real worldimages.
In addition, the main targets of the conventional works
on Web mining are numeric data and text data. However,there are a large number of multimedia data such as images,
movies and sounds on the Web. The number of images on
the Web especially is rapidly increasing recently because of
the recent rapid spread of digital cameras. We think thatuse of multimedia data on the Web, namely visual knowledge on the Web, is promising and important for resolving
real world image recognition/classiﬁcation.
The processing in the system we are developing in our“Web Image Mining” project consists of three steps. In thegathering stage, the system gathers images related to givenclass keywords from the Web automatically. In the learning stage, it extracts image features from gathered images
and associates them with each class. In the classiﬁcation
stage, the system classiﬁes an unknown image into one of
the classes corresponding to the class keywords by using
the association between image features and classes. The
system is constructed by integrating three modules, which
are an image-gathering module, an image-learning module,
and an image classiﬁcation module (Figure 1).
In this paper, we describe methods of image-gatheringinput image

World Wide Web"bear"
"cat"
"cow"
"dog"
"lion"

bear image
cat imagecow imagedog imagelion image

imagegathering
moduleclass
keywords(unknown image)gathered
imagesimagelearning
modulelearning
images(known image)

image
feature
space
class

image
knowledge

image
classification
module

bear
cat
cow
dog
lion

classification
results

Figure 1: Web Image Mining system, which is constructed as an integrated system of an image-gathering module, animage-learning module and an image classiﬁcation module.
from the World Wide Web and classiﬁcation of unknown
images using images gathered from the Web. They areslightly modiﬁed compared to [14]. The main diﬀerence
is in the image-gathering module. We propose a method
to gather more images with higher accuracy for the image
gathering stage. Finally, we describe experimental results
and conclusions.

2

Image Gathering

The image-gathering module gathers images from the Web
related to class keywords. Note that we do not call thismodule the image “search” module but the image “gathering” module, since its objective is not to search for a few
highly relevant images but to gather a large number of relevant images.
At present, some commercial image search engines on the
Web such as Google Image Search, Ditto and AltaVista Image Search are available. Their preciseness of search results
is, however, not good since they employs only keywordbased search. Then, some integrated search engines employing both keyword-based search and content-based image retrieval have been proposed. WebSeer [5], WebSEEk[11] and Image Rover [10] have been reported so far. Thesesystems search for images based on the query keywords, andthen a user selects query images from search results. After
this selection by the user, the systems search for images
that are similar to the query images based on image features. These three systems carry out their search in an interactive manner. Therefore, they are not suitable for “WebImage Mining”, which requires gathering a large number ofimages.
Since an image on the Web is usually embedded in an
HTML document that explains it, ﬁrst the module exploits
some existing commercial text-based Web search engines
and gathers URLs (Universal Resource Locator) of HTML
documents related to the class keywords. In the next step,using those gathered URLs, the module fetches HTML documents from the Web, analyzes them. If it is judged thatimages are related to keywords, the image ﬁles are fetchedfrom the Web. Then, we remove irrelevant images fromthem based on their image features, and regard them as
output images.
Removing irrelevant images is carried out by eliminating images which belong to relatively small clusters in the
result of image-feature-based clustering. Images which are
not eliminated are regarded as appropriate images to theclass keywords, and we store them as output images. Ourpreference of larger clusters to smaller ones is based on thefollowing heuristic observation: an image that has many
similar images is usually more suitable to an image represented by keywords than one that has only a few similar
images.
The processing of the image-gathering module consists ofcollection and selection stages.
In the collection stage, the system obtains URLs using
some commercial web search engines, and by using those
URLs, it gathers images from the web. The detail of thealgorithm is as follows.
1. A user provides the system with two kinds of query keywords. One is a main keyword that best represents animage, and the other is an optional subsidiary keyword.
For example, when we gather “apple” images, we use“apple” as a main keyword and “fruit” as a subsidiary
keyword. Subsidiary keywords help to restrict the kindof gathered images. In this case, it prevents “apple computer” images from being gathered.
2. The system sends the main and subsidiary keywords as
queries to the commercial search engines and obtains the
URLs of the HTML documents related to the keywords.
3. It fetches the HTML documents indicated by the URLs.
4. It analyzes the HTML documents, and extracts the
URLs of images embedded in the HTML documentswith image-embedding tags (“IMG SRC” and “A HREF”).
The system fetches ﬁles whose images satisfy one of thefollowing conditions.
Condition:• If the image is embedded by the “SRC IMG” tag, the“ALT” ﬁeld of the “SRC IMG” includes the keywords.
• If the image is linked by the “A HREF” tag directly,
the words between the “A HREF” and the “/A” include the keywords.
• The name of the image ﬁle includes the keywords.
The reason why we set up the above conditions is thatour preliminary experiments turned out that an ALT
ﬁeld, link words and a ﬁle name had high tendency toinclude keywords related to the image. In this paper
we do not use the other clues such as a TITLE tag andfrequency of keywords, because these clues have only
low tendency to include keywords related to the image.
Although the number of gathered images in this method
is not large, we can compensate it with the re-gathering
processing described later.
In the selection stage, the system remove irrelevant images from them based on their image features.
1. The system ﬁrst makes image feature vectors for all
the collected images. We use signatures and the EarthMover Distance(EMD)[8] as image features and dissimilarity. Here, we compute one signature from one image.
Next, all the distances (dissimilarity) between two images are calculated based on the EMD.
2. Based on the distance between images, they are groupedby the hierarchical cluster analysis method. Our system
uses the farthest neighbor method (FN). In the beginning, each cluster has only one image, and the system repeats merging clusters until all distances between them
are more than a certain threshold.
3. It throws away small clusters that have fewer images
than a certain threshold value, regarding them as being
irrelevant. All the images in the remaining clusters are
regarded as output images.
In the image-gathering module, we can gather more images as we obtain more URLs of HTML documents. However, for one set of query keywords, the number of URLs
obtained from Web search engines was limited because commercial search engines restrict the maximum number of
URLs returned for one query. Thus, we introduce the query
expansion method [9] for generating automatically new setsof query keywords for search engines.
The system extracts the top several words (only nouns,
adjectives, and verbs) with high frequency except for initial query keywords from all HTML ﬁles with embedded
output images of the initial image gathering, and regards
them as subsidiary query keywords. It generates several
sets of query keywords by adding each subsidiary words
to the main keyword, and then obtains a large number of
URLs for the query keyword sets. For carrying out thesecond image gathering, using obtained URLs, the system
goes through the collection and selection stages again. This“query expansion and re-gathering” enables the number ofimages gathered from the Web to increase greatly.

3

Image Classification

First, in the learning stage, the image-learning module extracts image features from images gathered by the gathering

module and associates image features with the classes represented by the class keywords. Next, in the classiﬁcation
stage, we classify an unknown image into one of the classes
corresponding to the class keywords by comparing image
features.
We use Earth Mover Distance(EMD)[8] as image featuresand dissimilarity, and image-feature-based search which isa k-nearest neighbor variant as a classiﬁcation method. It
is the same as [14].
In our method of image classiﬁcation, image features of
not only a target object but also non-target objects suchas background included in the image are used together as aclue of classiﬁcation, since non-target objects usually havestrong relation to a target object. For example, a cow usually exists with grass ﬁeld and/or fence in farm, and a lion
usually exists in Savannah or zoo. Although the numberof combination of a target object and non-target objects
is large, we think that we can deal with this largeness bygathering a large amount of image from the Web and using
them as training images. Here, we do not set up ”reject”,
and then all test images are classiﬁed into any class.

We exploit two kinds of image features for learning andclassiﬁcation: color signature for block segments, and region
signature for region segments. A signature describes multidimensional discrete distribution, which is represented bya set of vectors and weights.
In case of color signatures, a vector and a weight correspond to a mean color vector of each cluster and its ratioof pixels belonging to that cluster, respectively, where somecolor clusters are made in advance by clustering color distribution of an image. To obtain color signatures, ﬁrst,
we normalize the size of training images into 240 × 180,and divide them into 25 block regions. Next, we make acolor signature for each of these 25 block regions by clustering color vectors of each pixel into color clusters by the
k-means method.
In case of region signatures, a set of feature vectors ofregions and their ratio of pixels represents a region signature. To obtain region signatures, we carry out region segmentation for images instead of dividing images into block
segments after normalizing their size. Here, we employ asimple segmentation method based on the k-means clustering used in [13] and a sophisticated color segmentation
method, JSEG [3].
To compute dissimilarity between two signatures, EarthMover’s Distance (EMD) has been proposed [8]. The EMD
are found to be the most excellent distance on the averageamong distances commonly used in content-based image
retrieval, as indicated by the prior work of Y.Rubner et al.
[7].
In the classiﬁcation stage, in case of the color signature,
we sum up the minimum distances between an unknown
input image and training images of each class for 25 allblocks, and classify it into the class whose total distance isthe smallest. In case of the region signature, we employ the
k-nearest neighbor (k-NN) method to classify an unknown
input image into one of the class. The value of k is decided
as 5 by the preliminary experiments.

Table 3: Results of the experiment no. 4
exp. no.4
methodrec. pre.
F
avg. by color
28.6 53.0 37.1
avg. by region (1)24.5 27.9 26.1
avg. by region (2)21.6 24.6 23.0

Table 1: Four experiments.
no.
1
2
3
4

# of

# of

precisionclasses

images(%)#

source

20
20
20
50

3790
11205
34043
17379

77.8
80.8
80.1
—‡

50
50
50
LO

W+C†

test images

W+C†
W+C†
Web

†Web images + Corel images ‡not examined
CV: cross-validation, LO: leave-one-out

4

Experimental Results

We made four experiments from no.1 to no.4 as shown in
Table 1. The experiment no.1, 2 and 3 are 20-class classiﬁcation experiments, and the experiment no.4 is a 50-classclassiﬁcation experiment. In the experiment no.1, 2 and3, we made the experiments using three diﬀerent training
image sets gathered from the Web independently. We used
the query expansion technique for the experiments no.2 andno.3. In each experiments, we extracted the top 5 and 15
words from the initially-gathered HTML ﬁles.
In the experiment no.1, we gathered images from the Web
for 20 kinds of class keywords shown in Table 2. By theimage-gathering module about ten thousands URLs were
fetched from three commercial text search engines, Google,
InfoSeek, Goo Japan. The total number of gathered image
was 3790, and the precision by subjective evaluation was
77.8%, which is deﬁned to be NOK /(NOK + NN G ), where
NOK , NN G are the number of relevant images and the number of irrelevant images to their keywords. In the second
and third columns of Table 2, we show the number of URLs
of images gathered from the Web and their precision.
In the columns after the fourth of Table 2, we show
the classiﬁcation result using the gathered images from the
Web as training images. Note that the precision of training images is not 100% unlike the conventional works on
image classiﬁcation. In the experiments no.1, 2 and 3,
we used a special hand-made test image set for evaluation. We make a special test image set by selecting various kinds of 50 typical images for each class from Corel
Image Gallery and Web images by hand. The table describes only results by color signatures in each class, sincemost of results by color signatures are superior to resultsby region signatures using k-means and JSEG. In the table, “region (1)” and “region (2)” mean region signatureusing the k-means clustering and region signature using the
JSEG region segmentation method. In the tables, the recall is deﬁned to be MOK /Mtest , the precision is deﬁned tobe MOK /(MOK + MN G ) and F-measure is the harmonicmean of the recall and the precision, where MOK , MN G ,and Mtest are the number of correctly classiﬁed images, thenumber of incorrectly classiﬁed images, and the number oftest images for each class, respectively. All values are represented in percentage terms. In the experiment no.1, we
obtained 37.3 as the F-measure value by color signatures.
In Table 2, we also show the results of the experimentno.2 and 3. Their diﬀerences from no.1 are only the number of training images. These results shows the F-measurerose as the number of training images increased. The re-sults of “apple”, “house”, “car” and “Mt.Yari” especiallywere improved a lot. On the other hand, the classiﬁcation
rate of “Ichiro” remained low, since “Ichiro” images hadmuch variation and no typical pattern. About 500 training
images were not enough to classify them. These results indicate that the diﬃculty to classify images depends on thenature of the class greatly.
In the experiment no.4, we made a classiﬁcation experiment for 50 class keywords, which were selected from words
related to nature, artifacts and scene. We obtained 28.6,
53.0 and 37.1 as the recall, the precision and the F-measure,respectively, by color signatures (Table 3). This results arecomparable to the results of the experiment of 20 classes.
This indicates that the diﬃculty of classiﬁcation depends
on the dispersion of image features of each class in the image feature space, not simply on the number of classes. It
is hard to collect such various kinds of images as images
used in the experiment no.4 by means of commercial image databases, and it has come to be possible by imagegathering from the World Wide Web.

5

Conclusions

In this paper, we described our ongoing project, “WebImage Mining for Generic Image Recognition”. This
project aims at generic image classiﬁcation using images
automatically-gathered from the Web as training images
instead of hand-made image collections.
Although classiﬁcation rate obtained in the experiments
for generic real world images is not high and not suﬃcient for practical use, the experimental results suggest thatgeneric image classiﬁcation using visual knowledge on theWorld Wide Web is one of the promising ways for resolving
real world image recognition/classiﬁcation.
There are many issues to be solved for making thisproject more practical. How many classes does a genericclassiﬁcation system have to treat for practical use? How
many training images are required for each class? Whatshould we deﬁne as a “class”? Should we remove as many irrelevant images as possible from training image sets? Whatis the classiﬁcation way to treat with training sets including
some irrelevant samples well? Because of such issues, evaluation is the biggest problem for generic image classiﬁcation/recognition. Experimental results sometimes dependon learning sets and training sets more greatly than classiﬁcation algorithms. Therefore, we need comprehensive
standard benchmark test sets like TREC.
Lately, Corel images are a de facto standard set for theevaluation of generic image classiﬁcation systems. As we
indicated in this paper, however, commercial photos suchas Corel images are well-arranged, and most of ones in theclassapple
bear
mountain bike
Lake Biwa a)carcat
entrance b)
house
Ichiro c)
Ferris wheel
Kinkaku Temple d)
lion
Moai
note-size PC
Shinkansen train e)park
penguinnoodle f )wedding
Mt.Yari g)
total avg. by color total
avg. by region (1)avg. by region (2)Table 2: Results of the experiment no. 1, 2 and 3.
exp. no.1
exp. no.2
num. pre. rec. pre. F
num. pre. rec. pre. F
86 (86) 9.4 100.0 17.2
614 (86) 47.1 93.0 62.5
261 (47) 13.7 13.7 13.7
729 (76) 23.5 22.6 23.1
82 (62) 1.5 100.0 3.0
575 (78) 6.2 66.7 11.3
272 (38) 32.4 30.7 31.5
679 (49) 22.5 47.1 30.5
41 (76) 0.0 0.0 0.0
694 (92) 38.9 26.9 31.8
267 (82) 23.5 21.4 22.4
611 (92) 15.7 28.6 20.3
440 (96) 89.7 24.1 38.0
648 (94) 46.6 49.1 47.8
33 (79) 1.6 100.0 3.2
306 (82) 0.0 0.0 0.0
75 (80) 0.0 0.0 0.0
392 (68) 3.5 66.7 6.7
81 (89) 9.0 77.8 16.1
395 (74) 19.2 83.3 31.2
222 (89) 73.8 68.2 70.9352 (86) 23.0 100.0 37.3
70 (86) 23.5 75.0 35.8
481 (75) 23.5 41.4 30.0
104 (78) 29.4 93.8 44.8
329 (93) 49.0 89.3 63.3
83 (55) 10.8 77.8 18.9
499 (58) 15.4 83.3 26.0
98 (66) 11.1 75.0 19.4
548 (77) 16.7 60.0 26.1
328 (83) 62.9 28.9 39.6
762 (89) 95.2 17.0 28.8
195 (91) 13.0 43.8 20.0
441 (78) 1.9 50.0 3.6
365 (90) 78.6 46.6 58.5
786 (88) 40.0 48.3 43.8
125 (88) 7.0 40.0 11.9
645 (89) 31.6 37.5 34.3
562 (94) 94.5 13.0 22.9
719 (93) 92.7 15.1 26.0
3790 (78) 29.3 51.5 37.3 11205 (81) 30.6 51.3 38.3
29.7 34.0 31.7
33.1 36.2 34.6
28.6 31.9 30.2
30.8 32.0 31.4

exp. no.3
num. pre. rec. pre. F
2561 (83) 55.3 67.1 60.6
2533 (73) 29.4 17.4 21.9
2386 (88) 23.1 19.0 20.8
2191 (62) 67.6 30.8 42.3
1836 (92) 61.1 30.0 40.2
2532 (84) 49.0 19.1 27.5
1934 (92) 70.7 46.1 55.8
2575 (77) 19.7 25.0 22.0
481 (61) 1.8 100.0 3.4
731 (83) 12.8 100.0 22.7
700 (75) 36.1 95.7 52.4
1738 (92) 27.5 93.3 42.4
549 (87) 43.1 91.7 58.7
852 (59) 23.1 71.4 34.9760 (65) 11.1 85.7 19.7
2193 (88) 58.1 26.1 36.0
1791 (82) 14.8 88.9 25.4
2212 (96) 42.9 62.5 50.82062 (78) 28.1 37.2 32.0
1426 (85) 72.7 38.8 50.634043 (80) 37.4 57.3 45.2
35.3 40.1 37.6
31.2 31.5 31.4a) the biggest lake in Japan b) a school entrance ceremony c) the name of a famous baseball player d) a famous temple in Japane) Japanese bullet train f) Chinese noodle g) a famous mountain in Japansame category are similar to each other unlike Web images which are diverse real world images. We think that ageneric image classiﬁcation system should be able to treat
diverse real world images like Web images. For this, the system requires a large amount of visual knowledge for many
classes. Since the World Wide Web has them, visual knowledge on the Web will enables us to realize a generic image
recognition system.

References[1] K. Barnard and D. A. Forsyth. Learning the semantics ofwords and pictures. In Proc. of IEEE International Conference on Computer Vision, volume II, pages 408–415, 2001.
[2] S. Belongie, C. Carson, H. Greenspan, and J. Malik. Recognition of images in large databases using a learning framework. Technical Report 07-939, UC Berkeley CS Tech Report, 1997.
[3] Y. Deng and B. S. Manjunath. Unsupervised segmentation of color-texture regions in images and video. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 23(8):800–810, 2001.
[4] P. Duygulu, K. Barnard, J. F. G. de Freitas, and D. A.
Forsyth. Object recognition as machine translation: Learning a lexicons for a ﬁxed image vocabulary. In Proc. ofEuropean Conference on Computer Vision, 2002.
[5] C. Framkel, M. J. Swain, and V. Athitsos. WebSeer: An
image search engine for the World Wide Web. Technical
Report TR-96-14, University of Chicago, 1996.

[6] Y. Mori, H. Takahashi, and R. Oka. Image-to-word transformation based on dividing and vector quantizing images
with words. In Proc. of First International Workshop on
Multimedia Intelligent Storage and Retrieval Management,
1999.
[7] Y. Rubner, J. Puzicha, C. Tomasi, and J. M. Buhmann.
Empirical evaluation of dissimilarity measures for color
and texture. Computer Vision and Image Understanding,84(1):25–43, 2001.
[8] Y. Rubner, C. Tomasi, and L. J. Guibas. The earth mover’s
distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99–121, 2000.
[9] G. Salton and C. Buckley. Improving retrieval performanceby relevance feedback. Journal of the American Society for
Information Science, 41(4):288–297, 1990.
[10] S. Sclaroﬀ, M. LaCascia, S. Sethi, and L. Taycher. Unifying
textual and visual cues for content-based image retrieval on
the World Wide Web. Computer Vision and Image Understanding, 75(1/2):86–98, 1999.
[11] J. R. Smith and S. F. Chang. Visually searching the Web
for content. IEEE Multimedia, 4(3):12–20, 1997.
[12] J. Z. Wang and J. Li. Learning-based linguistic indexing of
pictures with 2-D MHMMs. In Proc. of ACM International
Conference Multimedia, pages 436–445, 2002.
[13] J. Z. Wang, J. Li, and G. Wiederhold.
SIMPLIcity: semantics-sensitive integrated matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(9):947–963, 2001.
[14] K. Yanai. Generic image classiﬁcation using visual knowledge on the web. In Proc. of ACM International Conference
Multimedia 2003, 2003.