Proceedings of the Third NTCIR Workshop

Evaluation of Web Retrieval Methods Using Anchor TextKenji TATEISHI, Hideki KAWAI, Susumu AKAMINE, Katsushi MATSUDAand Toshikazu FUKUSHIMAInternet Systems Research Laboratories, NEC Corporation
8916-47, Takayama-Cho, Ikoma, Nara 630-0101, Japan{tateishi,kawai,akamine,mat,fuku}@hml.cl.nec.co.jp

AbstractＧ

In this paper, we evaluate two types of anchor
texts: a page anchor and a site anchor. Since theＨＩＡ

Insidethe siteTop PageAnchor texts of page Eanchor text tends to summarize information referredＢahead, it can be expected that the terms appearing
there haveOutsidethe siteＣ

important meaning in information
Ｄretrieval. We introduce a retrieval method to giveＥＦPage anchor= {Anchor（B,E),Anchor(C,E)}
Site anchor= {Anchor(G,A),Anchor(H,A), Anchor(I,A)}

high priority to the terms in the anchor text. In theexperiment, we compared the proposed method with

Figure 1. An example of the page anchor

the base line which indexed only page documents.

and the site anchor

The result indicated that both methods had almost thesame accuracy, and that there were many queries in

2. Anchor text

which the accuracy much differed between twomethods. It can be expected that the improvement on

We used the following two types of

the queries in which the proposed method was

information retrieval.

inferior to base line will be achieved by deletinganchor texts for(1) An anchor text which summarizes content of aoverlapped anchor texts toward the same page.

web page (hereafter, page anchor)(2) An anchor text which summarizes content of a

1. Introduction

web site (hereafter, site anchor)
First of all, the page anchor is the text in a link to a

The text in a link is called anchor text. Since thegiven web page. For instance, in Figure 1, pageanchor text tends to summarize information referredanchors of page “E” are equal to {Anchor (B, E),

ahead, it can be expected that the terms appearingAnchor(C, E)}. Here, Anchor(x,y) shows an anchor

there have an important role in information retrieval.

text in a link of page x to page y. This page anchor

We participated in this NTCIR-WEB task to clarifyhas the same definition as a so-called usual anchor

the effect of indexing them. We introduced not only

text.

conventionally used page anchors, but also site

Next, the site anchor is the text in a link to the top

anchors, and used the retrieval method to give highpage of a given web site. For instance, {Anchor(G,A),

priority to the terms in anchor texts. Hereafter, weAnchor(H,A),Anchor(I,A)} correspond to the site

discuss the two types of anchor texts, the retrieval

anchors of page “E” in Figure 1. Since the web site ismethod and the evaluation result.

usually constructed under the assumption of visitorsbrowsing each page via the top page, the expressions
shared on the entire web site tend to be omitted. For
instance, suppose a web site where gourmet© 2003 National Institute of Informatics

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

3.2 Ranking method
AncA AND B AND C

We use a ranking method that gives high priority tothe anchor text, which processes according to theAnchor
A AND B

following.
Step 1: Decrease web pages by using the ANDAnchor

function made from all query terms. Each query

Cterm can exist either in the web page document orin the anchor texts of the page.
Figure 2. An image of the retrieval model

Step 2: To the web pages narrowed in Step 1, give

(For three query terms)

high priority to web pages whose anchor texts
contain the terms.

information on an entire country is treated. There,

Step 3: Delete one query term, and repeat from

“Gourmet” and “Restaurant retrieval” are often

Step 1.

shown as anchor texts of the main page while

Thus, it is intuitively the pyramidal ranking shown inminimal information is attached to the anchor texts ofFigure 2, and the web pages whose anchor texts

the internal pages such as “Kansai”, “Chinesecontain the same kind of query terms can attain acooking” and “Nara”. Therefore, not only pagehigh position in the ranking.
In step 1,anchors but also the site anchor have to be taken inAND

we reduced web pages beforehand withorder for the anchor text to summarize the entire

thefunction

becausethepreliminary

content of the web page.

experiment informed us it was better than the simple
model using only the step 2 process.

3. Retrieval method

In step 2, we calculate the sum of tf * idf over all
query terms for the anchor texts only, and simply

We explain the retrieval method used to evaluate
two types of anchor texts in Section 2.

multiply tf by idf without any normalization of that
length. The tf means the frequency of a certain termin the anchor texts. When a web page has multiple

3.1 Decision of the top page for the siteanchor

anchor texts, we connected them into one and counts
tf.
In step 3, when TopicPart is a Title, the rightmost

The top page of the web site where a certain page

query term is deleted because it was provided

E belongs means a page that represents the whole

beforehand that the query term on the right hand side

web site or a certain area of it. It must fulfil theis more important for the NTCIR-Web task. On thefollowing three conditions.

other hand, when TopicPart is a Description, we(1) The page has the same domain as E.

delete a query term whose idf is minimal.

(2) The page is hierarchically above E.
(3) The number of external (from different

4. Evaluationdomain) links is more than n.
According to these requirements, we assume that a4.1 Comparison systems

top page has many external links. From here on we
set it as 2.

We compared the following three kinds of retrievalsystem.
(1) The base line system that indexes only web
page documents (hereafter, Baseline)Proceedings of the Third NTCIR Workshop

Table 1. Experimental results (upper table: TopicPart = Title，， below table: TopicPart =Description)
Title
BaselinePage Anchor
Page+Site Anchor
DescriptionBaseline
Page Anchor
Page+Site Anchor

10G
R-Pre
Prec@10 Prec@20
0.22980.20210.18620.24130.20430.18150.23910.20430.1815

Ave Pre0.13980.14200.1408100G
R-Pre
Prec@10 Prec@20
0.32130.31060.20070.32340.30320.19220.31490.28620.1859Ave Pre0.12460.12970.128510G
R-Pre
Prec@10 Prec@20
0.26600.20430.18350.25110.1856
0.15610.25110.1856
0.1561

Ave Pre0.14690.13710.1371100G
R-Pre
Prec@10 Prec@20
0.31490.29470.20400.26520.25540.1716
0.25870.24780.1713

Ave Pre0.12290.11890.1178(2) The retrieval system in Section 3 that uses the4.2 Experimental result

page anchor as the anchor text (hereafter, PageAnchor).
(3) The retrieval system in Section 3 that uses theThe experiment results are shown in Table 1 (uppertable:Title,below table:Description).

Whenpage anchor and the site anchor as the anchor

TopicPart was the Title, both Page Anchor andtext (hereafter, Site Anchor)

Baseline have almost the same performance, although

We implemented a base line system which used aPage Anchor slightly exceeded Baseline in some ofranking method of Okapi[2]. Okapi is a retrieval

the evaluation scales. The accuracy of Site Anchor

method based on the probabilistic model, and a

was inferior to that of Page Anchor in all thesystem using this method has given successful results

evaluation scales. On the other hand, when TopicPart

in past TRECs. Note that since Baseline was not

was the Description, neither Page Anchor nor Site

submitted to the evaluation, it was not included in theAnchor performed as well as Baseline. Therefore,pooling web pages. The experiment was conductedfrom this result, we could not observewith both a 10G and a 100G index The evaluationin the method, which gave high priority to the terms,

scales are Prec@10, Prec@20, R-Precision andwhich appear in the anchor texts. Past TRECs

Average Precision.

also reached similar conclusions[3], so this result

From all these systems, we selected the character

improvement
havesupports them.

basis indexing style and used “Chasen”[1] formorphological analysis when TopicPart was the4.3 Discussion

Description. The morphemes we took as the query
terms were noun, verbal noun, and adjectival noun.

Table 2 shows the comparison of Baseline andAmong these morphemes, those mutually adjacent

Page Anchor results for each query when TopicPart iswere connected and treated as one term. We

the Title and the evaluation scale is Prec@10. Thereconfirmed by a preliminary experiment that higher

were many queries in which the accuracy muchaccuracy could be obtained when the mutuallydiffered between two systems. There were 17 of 47

adjacent morphemes were connected rather than not.

queries queries in which Page Anchor was inferior toMoreover, we excluded some noun terms as stop

Baseline.

words. For example, 情 報 (information), 説 明

When we put attention to this case, there were two(explanation), 文書(document), and 関心(concern),reasons. The one was because the query terms , firstetc.

of all, hardly hit in the anchor texts and Page Anchor
could not make use of its feature. 4 of 17 quries were

The Third NTCIR Workshop, Sep.2001 - Oct. 2002

{各種資格試験,各種資格試験,各種資格試験

function.

各種資格試験シリーズ,各種資格試験,各種資格試験,各種資格試験,各種資格試験,各種資格

5. Conclusion

試験,各種資格試験,各種資格試験}
In this paper, we evaluated two types of anchor texts:
Figure 3. An example of overlapped page

page anchor and site anchor, for verifying the effectanchors “資格試験
資格試験 (Qualifying exam)” whenof indexing the anchor text. We also introduced athe query is “資格試験
資格試験,情報処理資格試験 情報処理,IT情報処理 (Qualifying

retrieval method to give high priority to the terms inexam, information processing, IT)”the anchor text. . In the experiment, we compared theadopted to this case. This result can be found fromthe line of “Anchor Hit” in Table 2. “Anchor Hit”
means the number of web pages within the top 10
which more than one query terms apper in the anchor
texts. Thus, for example, “サルサ,学ぶ,方法 (Salsa,Learn, methods)” and ”速読法,効果 (Speed reading
method, effects)” had few web pages that hit query
terms in the anchor texts.
Next, the another case was because there weremany overlapped anchor texts towad the same web
page and then Page Anchor gave higher score to theweb page than should have given. These strongly
correspond to the ones used for navigation in thesame web site. Figure 3 illustrates an example ofoverlapped page anchors “ 資 格 試 験 (Qualifyingexam)” when the query is “資格試験,情報処理,IT(Qualifying exam, information processing, IT)”. This
problem can be solved by deleting the overlapping
anchor texts toward the same web page from thesame web site.

only page documents. The result indicated that bothmethods had almost the same accuracy, and that there
were many queries in which the accuracy muchdiffered between two methods. It can be expected thatthe improvement on the queries in which theproposed method was inferior to base line will beachieved by deleting overlapped anchor texts toward
the same page.

In future works, we plan to solve

this problem and to make a fairer evaluationcompared with the baseline system. In addition, wewill develop a new type of the anchor text which
includes site path anchor texts from the top page to acertainpage(forexample,Anchor(A,C)

andAnchor(C,E) in Figure 1) in addition to page anchorsand site anchors.

References[1] Y. Matsumoto, A. Kitauchi, T. Yamashita, Y. Hirano, O.
Imaichi, T. Imamura, Japanese morphological analysis

However, we don’t know exactly whether theaforementioned problem represents all

problems ofaccuracy decline, though it is also an important
problem The proposed system and Baseline differs inthat it decreases web pages with the AND function in
addition to using anchor text. The proposed system
didn’t performed as well as Baseline when TopicPart
was the Description (See Table 2) was thatmorphological analysis caused the system to retain
many unnecessary query terms,

proposed method with the base line which indexedthen the AND

function decreased web pages too much. Therefore,
we plan to evaluate the proposed system again by
preparing another base line system which is equal tothe proposed system at the point of theANDsystem“Chasen”,

Availablefrom

http://cactus.aist-nara.ac.jp/lab/nlt/chasen/distribution.html[2] S.E.Robertson, S.Walker. Okapi/Keenbow at TREC-8,
The Eighth Text Retrieval Conference, 1999.
[3] David Hawking, Overview of the TREC-9 Web Track,
The Ninth Text Retrieval Conference, 2000.

Proceedings of the Third NTCIR Workshop

Table 2. The comparison of Page Anchor and Baseline at Prec@10 by each query
Anchor
KeywordHit
80 サルサ,学ぶ,方法 (Salsa, learn, methods)
20
2 速読法,効果 (Speed reading method, effects)24
10 テーピング,方法 (Taping, method)31
10 資格試験,情報処理,ＩＴ (Qualifying exam, information processing, IT)
38
10 加速器,医療,治療 (Accelerator, medical treatment, treatment)
58
0 信越本線,碓氷峠,方法 (Shinetsu main line, Usui Pass, method)
10
10 オーロラ,条件,観測 (Aurora, conditions, observation)34
10 キューブリック,映画,感想 (Kubrick, film, impression)49
10 ポリフェノール,種類,効果 (Polyphenol, type, effect)14
10 夢,将来,努力 (Dreams, future, effort)
18
10 ロープワーク,結び方 (Rope work, knots)
19
10 梅,名所,東京 (Plum tree, place of interes)36
1 パイプオルガン,コンサートホール,住所 (Pipe organ, concert hall, address)
41
10 印象派,モネ,美術館 (Impressionist, Monet, art museum)42
10 イースター,復活祭,キリスト (Easter, Christ)48
10 アントシアニン,ブルーベリー,視力 (Anthocyanin, blueberry, eyesight)
57
10 亀,寿命 (Turtle, lifespan)
13
10 京都,寺,神社 (Kyoto, temple, shrine)
16
10 ゲノム,創薬,動向 (Genome, drug design, trend)23
10 絶滅,哺乳類,危機 (Extinction, mammals, crisis)
28
10 著作権,デジタルコンテンツ,ネットワーク (Copyright, digital content, network)29
10 スピーカー,評価,比較 (Speaker, evaluation, comparison)
39
10 宮崎駿,アニメーション,映画 (Miyazaki Hayao, animation, film)
53
10 自動車,将来像,日本 (Automobile, future image, Japan)
56
1 変分法,入門 (Calculus of variations, introduction)
Ｔｏｍｍｙｆｅｂｒｕａｒｙ,川瀬智子,ＴｈｅＢｒｉｌｌｉａｎｔＧｒｅｅｎ (Tommy February, Kawase
61
0 Tomoko, The Brilliant Green)
62
7 柴犬,日本犬,特徴 (Shiba inu, Japanese dog, characteristic)
12
10 正月,雑煮,地方 (New Years, ozoni soup, locality)
22
10 株式投資,インターネット,入門 (Stock investment, internet, introduction)27
3 宮部みゆき,書評,レビュー (Miyabe Miyuki, book review, review)30
2 アカデミー賞,受賞者,歴代 (Academy Award, recipient, successive generation)
32
2 憲法第九条,解釈,意見 (Article 9 of the Constitution, interpret, opinion)
33
9 石川県,特産品,お土産 (Ishikawa Prefecture, local product, souvenir)35
9 三国志,ゲーム,題材 (Sanguozhi (The Three Kingdom), game, theme)46
9 天然酵母パン,店,場所 (Natural yeast bread, shop, location)
59
0 Ｎゲージ,ＮＯゲージ,意味 (N gauge, HO gauge, meaning)63
10 グレートバリアリーフ,オーストラリア,旅行 (Great Barrier Reef, Australia, travel)11
10 遣唐使,習慣,文化 (Japanese envoy to Tang Dynasty China, customs, culture)17
10 野球,ベースボール,比較 ("Yakyu" (Japanese baseball), American baseball,
43
10 シフォンケーキ,作り方,菓子 (Chiffon cake, directions, cake)
アロマセラピー,アロマオイル,アロマキャンドル (Aromatherapy, aroma oil, aroma
447 candle)
47
3 カプサイシン,とうがらし,効能 (Capsaicin, capsicum, effect)
52
10 湖,水質,透明度 (Lake, water quality, clarity)
15
10 オゾン層,オゾンホール,人体 (Ozone, ozone hole, human body)40
7 本上まなみ,主演作品,女優 (Honjo Manami, works starred in, actress)
60
0 世界樹,北欧神話,名前 (Yggdrasil the world tree, Norse mythology, name)37
10 バイク,ツーリング,レポート (Motorbike, touring, report)

Query ID

PageBaseline
DiffAnchor
0.6
0 -0.6
0.50.1 -0.4
0.7
0.3 -0.4
0.4
0.1 -0.3
0.6
0.3 -0.3
0.4
0.1 -0.3
0.7
0.5 -0.2
0.4
0.2 -0.2
0.5
0.3 -0.2
0.1
0 -0.10.8
0.7 -0.1
0.2
0.1 -0.1
0.2
0.1 -0.1
0.5
0.4 -0.1
1
0.9 -0.1
0.8
0.7 -0.1
0.1
0 -0.1
0.5
0.5
00.1
0.1
00.3
0.3
00.5
0.5
00
000.1
0.1
000
0
0
0
00.1
0.5
0.8
00.1
00.6
0.2
0.4
0.2
00.1
0.2
00.6

0.1
0.5
0.9
0.1
0.2
0.1
0.7
0.3
0.5
0.3
0.1
0.2
0.4
0.2
0.8

00
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.2
0.2
0.2

00.4
0.2
0.3
00.4
00.2
0.6
0.4
0.6
0.3
0.8
0.6

0.2
0.2
0.2
0.3
0.3
0.4
0.6