Pacific Association for Computational Linguistics

Classification of Images Using Their Neighboring Sentences†Kazutaka SHIMADA†

Tetsuro ITO†Tsutomu ENDO††

Department of Computer Science and Intelligent Systems
Oita University
700 Dannoharu Oita, 870-1192 Japan
E-mail:{shimada, ito}@csis.oita-u.ac.jp††

Department of Artificial Intelligence
Faculty of Computer Science and Systems Engineering
Kyushu Institute of Technology
680-4 Kawazu, Iizuka, Fukuoka 820-8502, JAPANE-mail: endo@pluto.ai.kyutech.ac.jp
Abstract
There are many images on the WWW. Traditional image analysis, however, can not estimate the contents of images because they lack information such as colors, resolutions and so on. On the other hand a page on
the WWW consists of sentences and images, and the images are closely
related to their neighboring sentences. In this paper, we propose a method for classification of images using surface expressions in their neighboring sentences. First, the method extracts images and relational sentences from a page on the WWW. It then classifies the contents of images
using weighted keywords. We evaluated the performance for three keyword sets, which were constructed by handwork, a TF*IDF method and aBayes theorem based method. The highest accuracy attained to 99.5% by
the Bayes theorem based method for the training data, and 79.0% by theTF*IDF method for the test data. The keywords by the TF*IDF method
and the Bayes theorem based method outperformed those by the handwork for both the training and test data. They were suited to this classification.
Keywords Surface expression, Neighboring sentences, Classification, WWW

1. Introduction
There is a vast amount of information on theWWW. Information on the WWW is not only
text but also images and tables. Traditionalimage analysis, however, can not identify thecontents of images because they lack information such as colors, resolutions and so on. On
the other hand a page on the WWW consistsof sentences and images, and the images areclosely related to their neighboring sentences.
Their neighboring sentences are appropriate

for classifying images of the WWW because
classification using them can handle low-resolution images. There are several approaches to
identify the relation between text and images[1][2][3][4][11][13]. However, [1],[11] and [13]only align a person's name and picture. Our
proposed method can classify images of corporate advertising about computer systems.
[2]-[4] require some information such as colors and text in the images. To use informationfrom images on the WWW is unsuitable because of the low-resolution.

- 250 (C) 2001 The Pacific Association for Computational Linguistics

Pacific Association for Computational Linguistics
We are developing a multi-specifications
summarization system using extracted important data from specifications[5][6] and integrating the summary and images for multimedia summarization. The outline of the system is shown in Figure 1. A multimedia summarization system is required to identify andclassify the content of images to integrate
sentences and images. Since we deal with images on the WWW, an image analysis method
not fully dependent on the quality of images
is necessary. A home page is exemplified in
Figure 2. We proposed a method for identification and classification of images using theirneighboring sentences in [7]. The keywords
for classification were constructed by handwork, but they were subjective. Experimental
results were insufficient. In this paper, weevaluate the performance for three keyword
sets, which are constructed by handwork, aTF*IDF method and a Bayes theorem based
method.

2. Image and Neighboring Sentences
Extraction
We deal with pages of corporate advertising
about computer systems. Our system extracts
images and their neighboring sentences fromthe HTML source.

Figure 2. A home pageSpecifications

HTMLImportant Data Extraction

Text and Image Extraction

Sentence Generation

Image Classification

Integration
Multimedia
Summarization

Figure 1. Outline of our system- 251 (C) 2001 The Pacific Association for Computational Linguistics

Pacific Association for Computational Linguistics
2.1 Image ExtractionThere are two kinds of images in a page ofcorporate advertising about computer systems.
Type1: Letter images, button images and soon.
Type2: Images about the products.
These image types are exemplified in Figure
3. We deal with Type2 images for classification. Our system extracts Type2 images using
the size of an image. Conditions for extraction
are constructed from 337 images, and aregiven as follows:
1. The size of an image is more than
55 * 55(pixels).
2. The ratio of width to height is less than
3.8.
If the height of bordering images is the same,we handle them as one image because a large
image is often divided into several small ones
for the WWW.

2.2 Neighboring Sentences Extraction
The system extracts neighboring sentences ofimages. Their neighboring sentences are extracted using the tags of HTML as follows:
1. Extract <IMG> tag from a HTMLsource.
2. Extract sentences around the tag.
3. If "ALT" tag exists in <IMG>,
examine the contents of "ALT", andextract sentences which include it.
4. Estimate the layout of the extractedsentences from tags.
There are three tags about the layout:<TABLE>, <BR> and <HR>. The orderof priority of the tags is as follows:"<TABLE>" > "<BR>" > "<HR>".
Extract the sentences which are thenearest to <IMG> tag.
The extraction process is exemplified in
Figure 4. "Text1" and "Text3" are the neighboring sentences of "Image1" and "Image2",
respectively.

Type1

Type2
Figure 3. Image types

Text1<TABLE><TR><TD>Text1</TD><TD><IMG src="Image1"></TD>
</TR>
<TR>
<TD colspan="2">Text2</TD></TR></TABLE>

Image1
Text2(a)
Text3

<HR>
Text3<BR><IMG src="Image2"><HR>
Text4<BR>

Image2
Text4(b)Figure 4. Layout and HTML- 252 (C) 2001 The Pacific Association for Computational Linguistics

Pacific Association for Computational Linguistics
tem classifies each image into the class showing the maximum of the total score.

3. Keyword Extraction and Weighting
We define keywords and weights to classify
images. There are three kind of keywords,
which are constructed by handwork, a TF*IDF
method and a Bayes theorem based method.
The system classifies images into 12 classes.
The class names and their image examples areshown in Table 1. These class names denotethe topics of "Sentence Generation" in Figure1.
3.1 Handwork
We extract the keywords by handwork fromthe training data. The keywords extracted byhandwork are exemplified in Figure 5. Thekeywords possess the weight 1 or 2. The system searches the keywords in the 1st sentence.
If a keyword exists in this sentence, its scoreis multiplied by 2. If there are the tags, <B>
and/or <FONT>, the score is multiplied by n.
The value of n is shown in Table 2. The sys-

3.2 TF*IDF Method
The TF*IDF weight of a term in one document
is the term frequency(tf) divided by its document frequency(df). A document consists ofneighboring sentences of an image. Keywords
are extracted automatically using surface expressions. The process of the keyword extraction is as follows:
1. Eliminate hiragana expressions.
2. Eliminate 2-bytes symbols(ex. ◆ ,□ ).
3. Eliminate a word with 1 character.
4. Divide a word into several ones bysymbols(ex. CD-R/RW → CD-R, CD-RW).
5. Eliminate a word including numerals.
(exceptions: IEEE1394, MP3 and so on)
6. Eliminate stopwords.
Figure 6 is an example of the keyword extraction.
The weight is obtained by:(画像 |グラフィック)(を)?(扱う |あつかう |処理 |処理する)?ボード(Image|Graphic) (processing) ? board"?" denotes a word which produces once or nothing, "|" denotes "or"Figure 5. Keywords extracted by handwork
Table 2. The weight of tags

Table 1. Class names
Class nameContent of image

Tagn

Basic Spec.

CD-ROM, DVD-ROM<FONT color=...>

1.5

Extension

USB, PCI<B >1.5Image
Processing

IEEE1394,
Painting soft

<FONT size=+...>

1.75

Display

TFT, CRT display<FONT size=−...>

0.75

Operation

Keyboard, Mouse

2.0Size

Comparison of size

<FONT size=+...>
and<FONT color=...> or <B>

Communication

Communicationequipment

Sound

Speaker

Information

Service

Other models

Relational models

Soft

Attending soft

OthersOthers... USB を 2 つ装備している .....
(... equipped with USB ports[2] ...)↓
Keywords: USB, 装備(USB, equipped)
Figure 6. Keyword extraction- 253 (C) 2001 The Pacific Association for Computational Linguistics

Pacific Association for Computational Linguistics
4.1 Image Extraction
We used 337 images(the training data) forconstructing the conditions to extract images,and 450 images(the test data). The experimental results are shown in Table 3.
In our system, a recall rate is important because the image extraction is thepre-processing for the image classification.
The conditions to extract images produced thehigh recall rates for both the training and testdata.

1. Calculate the weight of each document.
Ndw(dk, i)=w×tfi×log
dfi
dw(dk, i): The weight of termi
in a documentk
w: The weight for tfi
N: The number of documents
2. Calculate the weight of each class.
dw(dk, i)×cw(Cj, i)=i∈Cj

df(Cj, i)dficw(Cj, i):The weight of termi in Classj
df(Cj, i): The number of documents
including termi in Classj
3.3 Bayes Theorem Based Method
The Bayes theorem based method is a probabilistic one. The probability that termi belongs
to class Ck is given by:
Pr(Ck|i) =Pr(Ck)×Pr(i|Ck)r

Pr(Cj)×Pr(i|Cj)
j=1

r: the number of classes(r=1, ... ,12).
We handle the probabilities as the weights ofthe keywords.

4. Evaluation
In this section, first, we evaluated the proposed image extraction method using 787 images. Next, we used three keyword
sets(Handwork, TF*IDF method, Bayes theorem based method) and evaluated their performance.

4.2 Image Classification
We used 325 images possessing their neighboring sentences. The images and their neighboring sentences are exemplified in Figure 7.
First, we used the three keywords sets, whichwere constructed by handwork, the TF*IDF
method and the Bayes theorem based method,and evaluated their performance. The numberof keywords which were constructed byhandwork was 184. The number of keywords
for the TF*IDF method and the Bayes theorem
based method was 1111. The training dataconsists of 182 documents. The experimental
results are shown in Table 4. The symbol "w"
in Table 4 denotes the weight of each keyword in the 1st sentence. For the TF*IDF
method, we normalized the weights because
the frequency of each class is different.
Table 3. Results of image extraction
Training data
Recall98.0%
Precision
91.5%

Test data
97.2%
94.0%

●背面 (the back)1. PC カード (PCMCIA support)
2. USB
3. シリアルポート (Serial ports)

Class name: Extension◆イージー・アクセスボタン（Easy Access Button）
使いやすいインターネット対応ボタンやメール対応ボタンを装備しています。(PC1 is equipped with some buttons
for the internet, e-mail and so on)

Class name: Operation
Figure 7. Examples of images and their neighboring sentences- 254 (C) 2001 The Pacific Association for Computational Linguistics

Pacific Association for Computational Linguistics
Table 4. Results for the training data
w=1
89.0%
90.1%
90.7%
97.3%

Standard
TF*IDF
Normalized1
Normalized2
Bayes theorem
Handwork

w=2
90.1%
90.7%
92.9%
98.9%

w=5
91.2%
91.8%
92.3%
99.5%
78.0%

w=10
90.1%
89.5%
91.8%
99.5%

w=20
87.9%
90.1%
90.1%
99.5%

w=10
68.5%
69.9%
73.4%
74.8%

w=20
66.4%
67.8%
70.6%
72.7%

Table 5. Results for the test dataStandard
Normalized1
Normalized2
Bayes theorem
Handwork

TF*IDF

Normalized1=1−dfc

w=1
70.6%
70.6%
78.3%
74.1%

Normalized2=log

N

w=2
72.0%
73.4%
79.0%
74.8%

Ndfc

N: The number of documents
dfc: The frequency of a class
The Bayes theorem based method produced
the best performance. The TF*IDF method obtained the high accuracy. The accuracy ofTF*IDF method was lower than that of theBayes theorem based method because theTF*IDF method overtrained the weight ofeach keyword.
Next, we evaluated their performance by
the test data. The test data consists of 143
documents. The experimental results areshown in Table 5. The TF*IDF method was effective in case of insufficiency of the keywords in documents because each weight of
the TF*IDF method was larger than that of theBayes theorem based method. We calculated
the weights of the TF*IDF method and theBayes theorem based method using the testdata. As a result, the accuracy of the TF*IDF
method(Normalized2 and w=2) and the Bayes
theorem based method(w=5) were 89.5% and97.2%, respectively.
We evaluated the number of keywords. We
reduced the number of the keywords by athreshold using the TF*IDF value. The number of the keywords was 659. As a result, theaccuracy using the keywords was 84.6%(thetraining data). For the test data, it was 63.4%.

w=5
68.5%
70.6%
76.2%
75.5%
66.4%

The reduction of the number of keywords
using the TF*IDF value was not effective, especially for the test data.
On the whole, the TF*IDF method and theBayes theorem based method calculated thevalid weights of the keywords and outperformed the handwork for both the training
and test data. They were suited to this classification. The reason why the results by thehandwork is not good is that the weight ofeach keyword is 1 or 2 only. It is, however,difficult to determine the continuous weightby handwork. The Bayes theorem based
method performed better than the TF*IDF
method if there were sufficient keywords. Thesystem requires increasing the number ofkeywords for the high accuracy, especially for
the Bayes theorem based method.

5. Conclusions
In this paper, we propose a method for classification of images using their neighboring
sentences and evaluated the performance forthree keyword sets, which were constructedby handwork, a TF*IDF method and a Bayes
theorem based method. In particular, the keywords by the TF*IDF method and the Bayes
theorem based method outperformed those by
the handwork. We showed the effectivenessof image classification using neighboring sentences of images by way of the experiments.
Our system can classify low-resolution imagesby using their neighboring sentences. Future- 255 (C) 2001 The Pacific Association for Computational Linguistics

Pacific Association for Computational Linguistics
work will include improvement of an algorithm to extract the keywords for the TF*IDF
method and the Bayes theorem based method.

References[1] K. Yamada, K. Sugiyama and H.
Nakagawa. Learning relations between linguistic expressions and pictures in newspaper,
Technical report of IEICE, NLC97-63,
pp.65-70, 1998(in Japanese).
[2] Y. Watanabe, Y. Okada, K. Kaneji and Y.
Sakamoto. Retrieving related TV news reports
and newspaper articles, IEEE Intelligent
SYSTEMS, Vol.14, No.5, pp.40-44, 1999.
[3] Y. Watanabe and M. Nagao. Diagram understanding for pictorial book of flora using
integration of pattern information and naturallanguage information, JSAI, Vol.11, No.6,
pp.888-895, 1996(in Japanese).
[4] Y. Watanabe and M. Nagao. Image analysis using natural language information extracted from explanation text, Vol.13, No.1,
pp.66-74, 1998(in Japanese).
[5] K. Shimada and T. Endo. Extracting important data from specifications, IPSJ SIG,NL133-15, pp.107-113, 1999(in Japanese).
[6] K. Shimada and T. Endo. Sentence generation from table structure of extracted important data, Technical report of IEICE, TL99-29,
pp.25-31, 1999(in Japanese).
[7] K. Shimada, T. Ito and T. Endo. Image
identification and classification using surface
expressions, IPSJ SIG, NL139-8, pp.55-60,
2000(in Japanese).
[8] W T. Chuang and J. Yang. Extracting sentence segments for text summarization: A machine learning approach, SIGIR 2000, pp.
152-159, 2000.
[9] M. Pazzani and D. Billsus. Learning andrevising user profiles: The identification of interesting web sites, Machine Learning 27,
pp.313-331, 1997.
[10] J. Favela and V.Meza. Image-retrieval
agent: integrating image content and text,
IEEE Intelligent SYSTEMS, Vol.14, No.5,
pp36-39, 1999.
[11] R. Houghton. Named Faces: Putting

names to faces, IEEE Intelligent SYSTEMS,
Vol.14, No.5, pp.45-50, 1999.
[12] R. Srihari. Computational models for integrating linguistic and visual information: asurvey, Artificial Intelligence Review 8,
pp.349-369, 1995.
[13] R. Srihari. Use of captions and other collateral text in understanding photographs, Artificial Intelligence Review 8, pp.349-369, 1995.

- 256 (C) 2001 The Pacific Association for Computational Linguistics