Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

Overview of the NTCIR-5 WEB Query Term Expansion Subtask
Masaharu Yoshioka
Graduate School of Information Science and Technology, Hokkaido University
N14 W9, Kita-ku, Sapporo-shi, Hokkaido, JAPAN
National Institute of Informatics
yoshioka@ist.hokudai.ac.jp

Abstract
The query term expansion subtask was conducted
to establish an evaluation framework for information
retrieval (IR) systems that focus on the effectiveness ofquery term expansion techniques. However, the quality of query term expansions are affected by several
factors (e.g., IR system using expanded query, qualityof initial query, etc.), so it is difficult to evaluate thistechnique.
In this subtask, I assume the topic difficulty for thequery term expansion technique is caused by the mismatch between different information-need expressions(query terms and relevant documents). To take into
account this topic difficulty, I propose feature quantities to characterize this difficulty and propose a newframework to evaluate each query expansion system.

1 Introduction
It is very difficult for many users of Information Retrieval (IR) utilities to select appropriate query terms torepresent the information need. Because query terms
are often imprecise and inappropriate, the documents
selected may contain only some of the query words
and be irrelevant to the user’s needs.
To reduce the mismatch between query terms andinformation need, many IR systems use query term expansion techniques to find better query terms. However, the effectiveness of this technique depends on
the quality of query terms in the initial query anddocuments used for query term expansion. CronenTownsend et al. [3] used a query clarity score basedon a language model to decide if the query terms contain relevant information for the query term expansion;
this approach was shown to be effective.
The Reliable Information Access (RIA) Workshop[5] conducted a failure analysis [1] for a set of topics
using seven different popular IR systems and proposed
a topic categorization based on the types of failures
they encountered. They also conducted a relevance
feedback experiment using a different IR systems [7].

This study, however, did not examine the relationshipbetween topic difficulty based on the mismatch and theeffect of relevance feedback.
In this subtask, I aim to establish an evaluation
framework for IR systems that focuses on query term
expansion. However the quality of query term expansion is affected by several factors, for example, whenI evaluate a query term expansion technique using results of information retrieval, these results may be affected by the characteristics and quality of the IR system. In other cases, when a user carefully selects
the query terms, the query term expansion is not performed well.
To evaluate the appropriateness of this evaluation
framework and several IR systems with query term
expansion techniques, each participant conducted retrieval experiments that used the survey type topics of
the NTCIR-4 web test collection [4].
The remainder of this paper is divided into five sections. Section 2 proposes various statistical features
for defining the topic difficulty and the effectivenessof the query expansion term. I also explain the collection of such information. In Section 3, I briefly review the NTCIR-4 web test collection and analyze thecharacteristics of topics in the test collection using theinformation defined in Section 2. Section 4 presents
guideline for the retrieval experiment and for submission of the result. Section 5 analyzes the experimental
results and Section 6 gives the conclusions of the paper.

2 Statistical Features for Evaluation of
the Query Term Expansion Technique
Buckley et al. [2] hypothesized a possible reason
why query expansion improves the query performanceas follows.
1. one or two good alternative words to original
query terms (synonyms)
2. one or two good related words

Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

3. a large number of related words that establish thatsome aspect of the topic is present (context)

The ratio between the size of relevant documents that satisfy the Boolean query ((2)) and thesize of relevant documents ((2)+(3)).

4. specific examples of general query terms

 The ratio between the size of relevant docu-

5. better weighting to original query terms
The first four reasons relate to the query term expansion. I can evaluate reasons 1, 2, and 4 using a thesaurus. However, since Voorhees [6] confirmed simple
automatic query term expansion based on a thesaurus
did not improve query performance, it may be inappropriate to use a thesaurus for this evaluation.
Therefore, I propose to use mismatch between different information-need expressions (query terms andrelevant documents) for this evaluation.

2.1 Feature Quantities for Characterizing
Mismatch between Initial Query and Relevant Documents
When a user carefully selects good query terms,
query term expansion is not well performed. Therefore, it is crucial for this subtask to evaluate the qualityof the initial query based on the mismatch between theinitial query and the relevant documents.
When the query is represented with a Boolean operator this mismatch is characterized as a mismatch between the documents that satisfy this query and the relevant documents. When the initial query is good, documents that satisfy the Boolean query (Boolean satisfied documents) and relevant documents are equivalent((1) and (3) in Figure 1 are an empty set).
Documents
Relevantthat satisfy
Documents
the Boolean queryments that satisfy the Boolean query ((2)) and thesize of the Boolean satisfied documents ((1)+(2)).

2.2 Feature Quantities for Evaluating Effectiveness of the Query Term
Feature quantities proposed in 2.1 can be also used
to evaluate a query term when the Boolean query isconstructed using only this term.
In addition to these features, I propose the following
three criteria for selecting feature quantities.
1. Appropriateness of the alternative term for each
initial query term.
2. Appropriateness of the context definition term forthe query.
3. Appropriateness of the term that characterizes therelevant documents.
A good alternative term should exist for relevant
documents that do not contain the initial query term.
Therefore, the number of documents that have a query
expansion term and do not have an initial query term
is useful for evaluation.
A good term for context definition is a distinct termthat exists in relevant documents. Therefore, the number of documents that have a query expansion termin the relevant documents, the Boolean satisfied documents, and total documents is useful for evaluation.
The following feature quantities are defined foreach query expansion term.

   The number of relevant documents thathave a query expansion term.

   The number of Boolean satisfied docuFigure 1. Mismatch between Initial Queryand Relevant Documents
However, because it is difficult to construct goodqueries, (1) and (3) are not an empty set in almost every query. The size of (1) and (3) characterizes thequality of the query from the viewpoint of query expansion. For example, when there are many documents in (1), the initial query is too general and requires new query terms that define the context of thequery. Conversely, when (3) has many documents theinitial query is too strict and it is necessary to determine alternative words to relax the query.
Since the number of documents in (1) and (3) are
affected by the number of relevant documents, I use
following two feature quantities for this evaluation.

ments that have a query expansion term.

  The number of Boolean satisfied relevant documents that have a query expansion term.

   The number of documents that have aquery expansion term in the document database.

The following feature quantities are defined foreach set of an initial query term and query expansion
term.



  The number of relevant documents thathave a query expansion term and do not have an
initial query term.

   The number of relevant documents thathave a query expansion term and an initial queryterm.

Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan   The number of Boolean satisfied documents that have a query expansion term and an
initial query term.

   The number of documents that have aquery expansion term and an initial query termin the document database.

I use feature quantities that are based on mutual information content for evaluating the distinctiveness ofeach term [9]. These quantities are the mutual information content between relevant documents and theterm .  is the probability of the term  being in
the document database and   is the probability
for the relevant documents.

    ¾ 

When term  exists explicitly in the relevant documents,   increases.

3 Analysis of The NTCIR-4 Survey TypeTopics
3.1 The NTCIR-4 Web Test Collection
The NTCIR-4 Web test collection [4] is a set of
100 gigabytes of html document data and 80 topics
for retrieval experiments. 35 topics out of 80 are forthe survey retrieval topics and the other 45 are for thetarget retrieval topics. The survey retrieval topics aredesigned for finding most relevant documents and thetarget retrieval topics are for finding just one, or only
a few, relevant documents of the highly ranked documents. Since the target retrieval topics may miss relevant candidate documents, only the survey retrieval
topics were used for this query term expansion subtask.
Figure 2 shows a sample topic in this test collection.
TITLE includes 1–3 terms with Boolean expressions. The attribute “CASE” in TITLE, ALT0,ALT1, ALT2, ALT3 means:(a) All the terms are related to one another by the OR
operator.
(b) All the terms are related to one another by the
AND operator.
(c) Only two terms can be related using the OR operator; the rest are specified by the attribute “RELAT.”
For the sample topic described in Figure 2, I canformulate the Boolean query ( オフサイド (offside) and(サッカー (soccer) or ルール (rule))) from TITLE and(オフサイド (offside) and サッカー (soccer) and ルー
ル (rule)) from ALT3.

TOPIC NUM0001/NUM
TITLE CASE=“c” RELAT=“2-3” オフサ イド ,サッカー, ルール /TITLEDESC サッカーのオフサイド というルールについて説明されている文書を探したい /DESC
NARRBACK サッカ ー で オ フ サ イ ドと はど うい うル ール な のか を 知 りた い 。
/BACKTERM オ フ サ イ ド は オ フェン
ス 側の 反 則で あ る 。オ フ サ イド が 適 用 され る状 況に は い く つか の パ タ ーン が む 爐 蝓 Ё サッ
カ ーの ル ール の 中で 最 もわ か りに く い も の であ る 。/TERMRELE 適 合 文 書 は オ フ
サ イド が 適 用 され る 状 況を 説 明し て い るも の/RELE/NARR
ALT0 CASE=“b” オフサイド /ALT0
ALT1 CASE=“b” オ フ サ イド , 選 手, 位 置
/ALT1
ALT2 CASE=“b” オ フ サ イド , サッカ ー/ALT2
ALT3 CASE=“b” サッカー, オフサイド , ルール
/ALT3
USER 大学 2 年, 男性, 検索歴 4 年, 熟練度 3, 精
通度 5/USER/TOPIC(a) An original sample topic
TOPIC NUM0001/NUM
TITLE CASE=“c” RELAT=“2-3” offside, soccer,
rule /TITLEDESC I want to find documents that explain theoffside rule in soccer. /DESC
NARR BACK I want to know about theoffside rule in soccer. /BACK TERM Offside
is a foul committed by a member of the offense side.
There are several patterns for situations in which theoffside rule can be applied, and it is the most difficult
soccer rule to understand. /TERM RELE
Relevant documents must explain situations where
the offside rule applies/RELE /NARR
ALT0 CASE=“b” offside /ALT0
ALT1 CASE=“b” offside, player, position
/ALT1
ALT2 CASE=“b” offside, soccer /ALT2
ALT3 CASE=“b” soccer, offside, rule /ALT3USER 2nd year undergraduate student, male, 4years of search experience, skill level 3, familiarity
level 5 /USER/TOPIC(b) An English translation of the sample topic

Figure 2. A sample topic from the NTCIR4 Web test collection [4]Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

3.2 Feature Quantities of Topics in NTCIR-4
Web Survey Retrieval Topics
A graph in Figure 3 shows a characteristic of the
topics in the Survey Retrieval Topics. The X axis of
the graph is and the Y axis is  . The
size of each circle indicates the size of the relevant
documents.
These statistical values were calculated using anorganizer reference IR system named The Appropriate Boolean Query Reformulation for Information Retrieval (ABRIR) [8]. Because such values may differ
according to the method of extracting index keywords
from the documents 1
From this graph, all initial queries were not sufficiently appropriate to distinguish all relevant documents from the other documents. For the topics thathave higher
 and lower  , such as topics 1, 4, 6, 55, and 98, the term for context definition
may be good query expansion terms. For the topics
that have lower
 and higher  , such as
65, 76, and 82, alternative terms may be good query
expansion terms. The topics that have lower
and lower , such as 45, 62, 63, 80, and 84,
may require various types of query expansion terms.

4 Guideline for the Retrieval Experiments
We used the NTCIR-4 web test collection data for
the formal run. We used the survey type topics only
(topic numbers 0001, 0003, 0004, 0006, 0019, 0021,
0022, 0023, 0028, 0029, 0034, 0044, 0045, 0055,
0058, 0061, 0062, 0063, 0065, 0068, 0070, 0071,
0073, 0074, 0076, 0080, 0082, 0084, 0086, 0088,
0091, 0095, 0097, 0098, and 0099).
The quality of query term expansions are affectedby several factors (e.g., The IR system that uses the
expanded query, the quality of the initial query, etc.)
so it is difficult to evaluate this technique by itself.
Therefore, we conducted several retrieval experiments to reduce the effect of different elements.
The effect of the query term expansion and thenumber of terms used for the query term expansion.
– No query term expansion vs. query term expansion.
– Query term expansion with a limited number of terms (10) vs. query term expansion
with no limitation.

The effect of documents that are used for query
term expansion

1 ABRIR in NTCIR-4 [8] uses a noun plus two adjacent nouns
as index words. Verbs were also used as index words in this experiment.

– Pseudo-relevant documents vs.
lected relevant documentsuser se-The effectiveness of expanded query terms– Statistical analysis of expanded query terms
and relevant documents.
– Correlation between statistical analysis andretrieval performance.

Comparison between ideal query expansion
terms– Generation of ideal query expansion term
candidates and retrieval results using all relevant document information defined in thetest collection

The following are the specifications of the retrieval
experiment formally run by the participants.
Initial Query Terms We used the TITLE(tt) field forall experiments.
Type of relevance feedback We use two different
methods to select the feedback documents.
feedback-type

Automatic selection (e.g., use the top-N
ranked documents) (auto):Used all relevant documents as selected as
relevant documents (relevant-A: use “S(H)”and “A” as relevant documents, relevant-B:
use “S(H)”, “A” and “B” as relevant documents):

Simulation of user selection using relevant
document information and system output(e.g., initial retrieval results. (user)– You could use relevant document information, however, you had to explain
the interface of your IR system andmake assumptions about the users’ behavior as a scenario. The assumptions
should be consistent for all topics; youmust not have changed this assumption
manually for each topic.
– You could use grades (“S(H)”, “A”,
“B” and “C”) for relevant document
information. For example, you could
use “B” relevant documents when you
could not find “S(H)” or “A” in a document list. You could also use thisgrade as a user’s confidence of relevance for calculating the weight of the
document’s importance.
– You could use other related information on WWW documents, such as
document length, URL pattern, language, etc.

Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

Figure 3. Characteristics of the Topics– You could also use the top-N pseudorelevant documents.
– To restrict amount of work required ofa user, I introduced the following restriction for scenario definition.


A user could read 20 documents
at most for relevant judgments andselect at most five documents asrelevant.
If your system required a user tocheck more than 20 documents,
you could use more documents.
However, the precise reason wasnoted in the scenario.
If your system required a user toselect more than five relevant documents (e.g., selection of document cluster with more than five
documents), you could use more
documents. However, the precise
was noted reason in the scenario.

Each participant submitted sets of retrieval results,
expansion term candidates lists, and document lists
that were used for query term expansion.

5 Analysis of the Retrieval Experiments
Results
5.1 Summary of Participants
Four participants, listed below in alphabetical orderof affiliation, and one organizer reference system submitted their completed run results.


Justsystem CorporationNational Institute of Informatics



National Institute of Informatics, the Universityof Tokyo, and KYA group

NTT Cyber Solutions Laboratories; NTT Corporation

Several query term expansion techniques and information retrieval models were used in these runs forwhich results were submitted.
5.1.1

JSWEB

Experimented with relevant document vectors thatwere generated based on the existence of the keyword in the relevant documents. They also proposed
combining relevant document vectors (one from theuser selected relevant documents and the other fromthe pseudo-relevant documents). The retrieval method
was based on a vector space IR model.

Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

5.1.2

NCSSI

Experimented with a clustering technique for the
initial retrieval results and a named entity recognition technique for selecting query expansion terms
from the appropriate cluster (user selected cluster or
pseudo-relevant cluster). They used an organizer reference model, ABRIR, based on a probabilistic model
as an IR system.
5.1.3

R2D2

Experimented with Robertson’s Selection Value(RSV) for selecting query expansion terms using
pseudo relevant documents. The retrieval method wasbased on the modified Okapi. They also used link information for scoring the retrieved documents.
5.1.4

ZKN

Experimented with Larvenko’s relevance model for selecting query expansion terms using pseudo relevant
documents. The retrieval method was based on the inference network and language model.
5.1.5

ABRIR: Organizer Reference System

Experimented with mutual information between terms
and relevant documents for selecting query expansion
terms. The retrieval method was based on the Okapi.

5.2 Additional Relevance Judgment
Since there were several submission results whose
top-ranked documents are not included in the judged
document list2 , it is unfair to evaluate the system results using the relevant document list in the test collection.
Due to time limitations the relevance judgment forthis subtask is not the same as that of the NTCIR-4
Web[4]3 .
5.2.1

Pooling

Because the quality of the document list used for query
term expansion may affect the quality of the query
term expansion, these document lists were used forpooling in addition to the submitted results. In addition to evaluating the Boolean query quality, experimental results using ABRIR with original Boolean
query were included in the pool.
I took the top 50 ranked documents from each of
the submitted results and the top 10 ranked documentsfrom the document list used for the query term expansion.
2 Documents that were checked by the assessor. A non-checked

document may be relevant.
3 I plan to rectify this mismatch before final data release.

The documents in the judged document list were removed from this pool. Pooled documents were ranked
using the same method as in [4].
5.2.2

Relevance Assessment

In relevance assessment a page-unit document model 4
was used as the document model. However, the relevant document list of NTCIR-4 consists of a one-click
distance document model 5 .
The assessor judged the “Multi-Grade Relevance”
of the individual documents as : highly relevant, fairly
relevant, partially relevant or irrelevant. This is same
as for NTCIR-4.

5.3 Summary of Evaluation Results
Table 1 shows the overall evaluation of the submitted runs. In most of the runs, the query term expansion technique improved the retrieval performance on
average, but there was no run that improved the query
performance for all topics.
Table 2, 3, 4, 5 show maximum, minimum, and average value for average precision, R-precision, and relevance retrieved for each topic. In this table, the highest performance run IDs for each team was selected
and used to calculate these values.
Table 6 shows the number of Run IDs where query
performance improved by using query term expansion
techniques. There is no direct correlation between theeffectiveness of the query term expansion techniqueand the mismatch between the initial query and the relevant documents showed in Figure 3.
It is interesting that there are several topics whose
number of Run IDs for a user is lower than that for automatic (e.g., 0021, 0058, 0076, 0082). Those topics
have higher values compared with other topics and it means good query expansion terms for these
topics are terms that can be used as alternative terms
for the initial query terms.
This result shows that nonrelevant documents may
be useful for finding alternative terms for initial query
terms.

6 Conclusion
In this paper, I propose a new framework to evaluate
query term expansion techniques using mismatch between different information-need expressions (query
terms and relevant documents). Although there isno direct correlation between the effectiveness of thequery term expansion technique and the mismatch between the initial query and the relevant documents, I
4 An assessor judged the relevance of a page only on the basis of
the entire information given on it
5 Assessors judged the relevance of a page by using out-linked
page information in addition to the information given on itProceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

Table 1. Evaluation Results for Average
Run ID

type offeedback

JSWEB-auto-01
JSWEB-auto-02
JSWEB-auto-03
NCSSI-auto-01
NCSSI-auto-02
R2D2-auto-01
R2D2-auto-02
ZKN-auto-01
ZKN-auto-02
ABRIR-auto
JSWEB-relevant-B-02
JSWEB-relevant-B-03
NCSSI-user-01
NCSSI-user-02
ABRIR-user

automatic
automatic
automatic
automatic
automatic
automatic
automatic
automatic
automatic
automatic
user
user
user
user
user

No. of topics where
performace improve
AP RP
RR
2
1
0
3
2
0
17 15
11
22 17
23
21 12
17
19 15
21
19 19
21
25 17
13
23 18
16
28 20
25
7
5
5
18 18
13
27 18
17
28 15
16
32 18
20

10 query terms
expansion (average)
AP
RP
RR
0.011 0.0236 212
0.0197 0.0344 516
0.0714 0.1094 1101
0.1708 0.2107 2432
0.1536 0.1962 2322
0.1747 0.2239 2257
0.181 0.2236 2257
0.1523 0.2011 2139
0.1537 0.1968 2153
0.2198 0.2506 2591
0.0235 0.049
755
0.0976 0.1466 1453
0.2434 0.2705 2508
0.2196 0.2487 2415
0.2569 0.2834 2689

No query term
expansion (average)
AP
RP
RR
0.0743 0.0992 1512
0.0743 0.0992 1512
0.0743 0.0992 1512
0.1511 0.1991 2256
0.1511 0.1991 2256
0.162 0.2066 2155
0.162 0.2066 2155
0.139 0.1824 2137
0.139 0.1824 2137
0.169 0.2085 2422
0.0743 0.0992 1512
0.0743 0.0992 1512
0.173 0.2258 2353
0.173 0.2258 2353
0.1801 0.2268 2469

AP: Average Precision, RP: R-Precision, RR: Relevant Retrieved
confirmed that this mismatch is one of the factors that
affects the performance of the technique.
For the future work further analysis is necessary toestablish a framework to evaluate the query term expansion technique in isolation.

Workshop Meeting, 2004. http://research.nii.ac.jp/ntcirws4/NTCIR4-WN/WEB/NTCIR4WN-OV-WEB-AEguchiK.pdf.
[5] D. Harman and C. Buckley. Sigir 2004 workshop:
Ria and ”where can ir go from here?”. SIGIR Forum,38(2):45–49, 2004.
[6] E. Voorhees. Query expansion using lexical-semantic
relations. In Proceedings of the 17th Annual InternaAcknowledgments
tional ACM SIGIR Conference on Research and Development in Information Retrieval, pages 61–69, 1994.
This research was partially supported by a Grant-in[7] R. H. Warren and T. Liu. A review of relevance feedback experiments at the 2003 reliable information acAid for Scientific Research on Priority (2), 15017202
cess (ria) workshop. In SIGIR ’04: Proceedings of thefrom the Ministry of Education, Culture, Sports, Sci27th annual international conference on Research andence, and Technology, Japan.
development in information retrieval, pages 570–571,
New York, NY, USA, 2004. ACM Press.
[8] M. Yoshioka and M. Haraguchi. Study on the combiReferences
nation of probabilistic and boolean ir models for www
documents retrieval. In Working Notes of the Fourth NT[1] C. Buckley. Why current ir engines fail. In SIGIRCIR Workshop Meeting, Supplement Volume, pages 9–’04: Proceedings of the 27th annual international con16, 2004. http://research.nii.ac.jp/ntcir-ws4/NTCIR4ference on Research and development in information
WN/WEB/NTCIR4WN-WEB-YoshiokaM.pdf.
retrieval, pages 584–585, New York, NY, USA, 2004.
[9] M. Yoshioka and M. H. W. Oniki. An appropriate
ACM Press.
boolean query reformulation interface for information[2] C. Buckley and D. Harman. Reliable information
retrieval based on adaptive generalization. In Internaaccess final workshop report.
Technical report,
tional Workshop on Challenges in Web Information ReNortheast Regional Research Center, MITRE, 2004.
trieval and Integration, pages 145–150, 2005.
http://nrrc.mitre.org/NRRC/Docs Data/RIA 2003/ria final.pdf.
[3] S. Cronen-Townsend, Y. Zhou, and W. B. Croft. Predicting query performance. In Proceedings of the 25th Annual International ACM SIGIR Conference on Researchand Development in Information Retrieval, pages 299–
306, 2002.
[4] K. Eguchi, K. Oyama, A. Aizawa, and H. Ishikawa.
Overview of the informational retrieval task at
ntcir-4 web. In Working Notes of the Fourth NTCIRProceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

Table 2. Evaluation Results for Each Topic (Automatic, 10 query terms expansion)
Topic
0001
0003
0004
0006
0019
0021
0022
0023
0028
0029
0034
0044
0045
0055
0058
0061
0062
0063
0065
0068
0070
0071
0073
0074
0076
0080
0082
0084
0086
0088
0091
0095
0097
0098
0099

Average Precision
Max
Min
Average0.319 0.0038 0.17126
0.2887 0.1851 0.24148
0.3559 0.082 0.23542
0.4366 0.1583
0.364
0.127 0.0226 0.05954
0.6828 0.0356 0.35538
0.3524 0.1905 0.25428
0.027 0.0105 0.01808
0.153 0.0461 0.08648
0.037 0.0117 0.02108
0.308
00.16002
0.067 0.0016 0.03704
0.2545 0.0454 0.15978
0.6531 0.0336 0.36368
0.5257 0.1165 0.40794
0.034500.02080.3341 0.0001 0.22234
0.0818 0.0005 0.04574
0.4412 0.0004 0.1106
0.0736 0.011 0.04548
0.3342 0.079
0.22750.0572 0.0017 0.02072
0.231 0.0681 0.15914
0.2114 0.0862 0.14536
0.533
0.024 0.33062
0.0949 0.0006 0.0465
0.5247 0.0041 0.32178
0.029800.00786
0.0173 0.0054 0.01396
0.4652 0.2809 0.40524
0.2317 0.0585 0.10846
0.1992 0.0083 0.11598
0.302 0.0017
0.195
0.193 0.0256 0.10018
0.2224 0.0214 0.12946

Max
0.25
0.35
0.5
0.53040.17390.69
0.4381
0.06670.23810.12780.38240.14080.31820.619
0.56620.14290.40620.16670.45990.1
0.36360.06060.19150.25930.58040.129
0.55870.02630.08110.48570.41670.25
0.34210.25
0.26

R-Precision
Min
Average00.16668
0.2
0.28
0.1667 0.26668
0.2686 0.47506
00.078260.06
0.4
0.3093 0.35464
00.040020.0476 0.14284
0.0451 0.07968
00.21766
0.0282 0.0986
00.20910.119 0.40476
0.2469 0.4887
00.0571600.2812600.112120.0084 0.15864
00.04
0.1212 0.29088
00.03030.1489 0.17872
0.0741 0.17778
0.0769 0.36572
00.051620.0508 0.39144
00.00526
0.027 0.05408
0.3143 0.42286
0.08330.2
00.125
0.0263 0.23684
0.05
0.13
0
0.172

Relevant Retrieved
Max Min Average
12
8
10.4
20
14
17.8
6
5
5.2
470 250
422.2
18
14
16.8
97
6
72.4
150
88
113.6
12
7
9.8
33
11
23.4
38
30
34.2
27
0
19.6
27
3
19.8
19
14
16.8
41
14
34.2
474 204
400.4
6
0
4
121
2
93.2
29
6
20.4
165
9
66.6
10
2
8.2
55
35
49.4
14
7
11.2
43
30
40.4
26
14
21.6
169
50
131.6
27
4
17
429
46
301.4
21
0
8
20
5
12.6
33
20
28.8
10
6
8.6
12
2
8.2
27
3
18.8
17
12
14.6
43
21
32.4

Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

Table 3. Evaluation Results for Each Topic (Automatic, no expansion)
Topic
0001
0003
0004
0006
0019
0021
0022
0023
0028
0029
0034
0044
0045
0055
0058
0061
0062
0063
0065
0068
0070
0071
0073
0074
0076
0080
0082
0084
0086
0088
0091
0095
0097
0098
0099

Average Precision
Max
Min
Average
0.2716 0.068 0.18818
0.2543 0.2122 0.22402
0.2165 0.057 0.12428
0.4926 0.3903 0.42308
0.1317 0.0185 0.07324
0.4097 0.0303 0.29346
0.1836 0.1586 0.17044
0.0221 0.0089 0.0154
0.0806 0.0468 0.0662
0.0316 0.0061
0.02
0.232900.12838
0.07930.02
0.052240.1397 0.0125 0.0804
0.5145 0.0131 0.32742
0.5185 0.1744 0.3953
0.0601 0.0002 0.03346
0.2882 0.0323 0.2045
0.0908 0.0056 0.04444
0.1051 0.0076 0.04048
0.0845 0.0116
0.042
0.258 0.0815 0.16518
0.0621 0.0002 0.03178
0.197 0.0927 0.13876
0.1854 0.0507 0.12034
0.5197 0.2806 0.39016
0.0668 0.0011 0.03662
0.5132 0.0289 0.31354
0.029900.00840.0183 0.0008 0.01256
0.3919 0.1247 0.31798
0.0494 0.0191 0.0399
0.1973 0.0024 0.10554
0.134 0.0269 0.08002
0.0841 0.0001 0.04762
0.1699 0.037 0.11188

Max
0.16670.35
0.16670.56010.17390.47
0.3247
0.06670.119
0.10530.32350.15490.18180.47620.57180.14290.34380.16670.189900.31820.09090.17020.22220.57690.129
0.56670.02630.05410.45710.16670.25
0.21050.1
0.24

R-Precision
Min
Average
0.0833 0.15002
0.2
0.27
0
0.133360.5142 0.53336
00.11302
0.05
0.34
0.2784 0.2928
00.01334
0.0714 0.0952
0.0376 0.0782
00.188240.0282 0.10706
0.0455 0.12728
0.0714 0.32858
0.2706 0.47252
00.08574
0.0703 0.2578
0.0303 0.09396
0.0506 0.10634
000.1212 0.20304
00.06060.0851 0.14466
0.1111 0.17776
0.3147 0.43148
00.058080.1302 0.40448
00.0052600.037860.1714 0.35428
0.0833 0.09998
00.125
0.0263 0.14736
0
0.07
0.04
0.164

Relevant Retrieved
Max Min Average
12
10
11.6
20
15
17.8
6
5
5.2
506 454
470.2
18
12
15.8
96
5
71.2
118
85
95.4
12
8
9.8
24
11
20.4
36
26
30.2
24
0
18
30
4
22.2
17
5
13.2
41
11
33.2
468 203
383.6
6
1
4
119
52
100.6
27
8
18.8
103
24
54
10
2
8
52
36
46.2
13
2
9.6
43
36
41.6
26
14
22
165
97
131.2
27
5
16
419 102
305.6
21
0
8.4
15
4
10.4
31
20
25.6
9
6
7.8
12
3
7.6
20
3
16.2
14
1
11.2
43
21
33.8

Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

Table 4. Evaluation Results for Each Topic (User, 10 query terms expansion)
Topic
0001
0003
0004
0006
0019
0021
0022
0023
0028
0029
0034
0044
0045
0055
0058
0061
0062
0063
0065
0068
0070
0071
0073
0074
0076
0080
0082
0084
0086
0088
0091
0095
0097
0098
0099

Average Precision
Max
Min
Average
0.6276 0.0918 0.2384
0.2717 0.2606 0.16034
0.3375 0.2096 0.15288
0.485 0.3127 0.24788
0.191 0.0116 0.07552
0.3876 0.1365 0.1524
0.38
0.2791 0.20108
0.0862 0.0097 0.02388
0.1621 0.1293 0.0888
0.2685 0.1732 0.1324
0.298500.112960.0722 0.0001 0.02736
0.4089 0.0019 0.12644
0.6531 0.1224 0.26578
0.5257 0.1077 0.20132
0.086700.02424
0.3418 0.0192 0.1306
0.1747 0.0005 0.04462
0.4412 0.0404 0.1343
0.0782 0.0629 0.04294
0.36
0.1325 0.15258
0.0745 0.0001 0.02288
0.2213 0.094
0.0967
0.2669 0.1167 0.1247
0.4197 0.0632 0.15806
0.220500.0646
0.5391 0.0033 0.2107
0.042500.01376
0.038 0.0112 0.01322
0.4905 0.3421 0.2495
0.2217 0.1101 0.0898
0.2368 0.0001 0.09394
0.348 0.1998 0.16464
0.3767 0.0011 0.1143
0.2462 0.0403 0.10556

Max
0.5
0.35
0.33330.57220.26090.41
0.433
0.06670.26190.33830.44120.11270.40910.619
0.56620.14290.35160.21210.45990.1
0.42420.12120.21280.29630.43360.29030.581
0.0526
0.10810.45710.41670.25
0.36840.3
0.3

R-Precision
Min
Average
0.1667 0.21668
0.2
0.18
0.1667 0.13334
0.417 0.30338
0.0435 0.11306
0.17
0.18
0.3866 0.24948
00.02668
0.1667 0.12858
0.2556 0.17592
00.14706
00.0366200.14546
0.2619 0.28094
0.2273 0.25578
00.057160.0781 0.15156
00.07272
0.1392 0.1713
00.02
0.1667 0.19394
00.042420.1702 0.1149
0.1481 0.14814
0.1783 0.19232
00.077420.0429 0.2378
00.01578
0.0541 0.04866
0.3714 0.25712
0.0833 0.11666
0
0.1
0.2632 0.18948
0
0.12
0.1
0.136

Relevant Retrieved
Max Min Average
12
11
7
20
15
10.8
6
5
3.2
494 363
265.2
21
11
10.4
90
48
43.2
150 130
83.4
12
7
6
35
33
20.2
111
81
56.4
32
0
10.8
26
1
10
20
7
9
41
28
21.6
474 192
214.6
5
0
1.8
120
35
54.6
44
5
15.4
165
75
69.8
10
9
5.8
55
45
30.2
18
1
6.4
43
38
24.8
26
23
14.4
160
88
74.8
29
1
11.6
433
45
179.2
21
0
7.2
20
5
8
33
26
17.2
10
8
5.4
12
1
4.4
33
26
17.8
20
5
8.4
44
33
23.8

Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

Table 5. Evaluation Results for Each Topic (User, no expansion)
Topic
0001
0003
0004
0006
0019
0021
0022
0023
0028
0029
0034
0044
0045
0055
0058
0061
0062
0063
0065
0068
0070
0071
0073
0074
0076
0080
0082
0084
0086
0088
0091
0095
0097
0098
0099

Average Precision
Max
Min
Average
0.2691 0.068
0.11920.2231 0.2091 0.12854
0.2165 0.0782 0.07508
0.4926 0.3897 0.2546
0.1381 0.0185 0.05894
0.3554 0.2412
0.19
0.1803 0.1619
0.101
0.0214 0.0132 0.01044
0.0683 0.0577 0.0372
0.0163 0.0061 0.00768
0.243900.09620.06810.02
0.029640.1148 0.0125 0.04744
0.5145 0.0131 0.20562
0.5185 0.1744 0.22012
0.055 0.0002 0.01826
0.29
0.0323 0.1224
0.0502 0.0056 0.02072
0.1051 0.0083 0.03858
0.0599 0.0343 0.02924
0.2743 0.0815 0.10142
0.0658 0.0002 0.02608
0.135 0.0927 0.07204
0.1963 0.0507 0.08138
0.4557 0.2806 0.2204
0.0859 0.0011 0.03166
0.5296 0.0289 0.21374
0.033300.00820.0175 0.0008 0.00694
0.3808 0.1247 0.17612
0.0419 0.0191 0.01958
0.2036 0.0024 0.08072
0.0636 0.0269 0.03072
0.0444 0.0001 0.01708
0.1616 0.037
0.0716Max
0.16670.25
0.16670.56010.26090.38
0.3247
0.0667
0.09520.06770.35290.11270.18180.47620.57180.14290.35160.15150.189900.25760.09090.14890.22220.53150.129
0.573
0.0526
0.05410.45710.08330.25
0.15790.1
0.24

R-Precision
Min
Average
0.0833 0.08334
0.2
0.14
0.1667 0.10002
0.5061 0.31472
00.10436
0.33
0.218
0.2938 0.1835
00.02668
0.0714 0.0476
0.0376 0.0346
00.135280.0282 0.05072
0.0455 0.08182
0.0714 0.19524
0.2706 0.26862
00.057160.0703 0.15314
0.0303 0.0606
0.0591 0.08018
000.1364 0.1091
00.036360.0851 0.07658
0.1111 0.1037
0.3147 0.24826
00.045160.1302 0.25366
00.01052
00.021640.1714 0.21142
0.0833 0.04998
00.1
0.0263 0.06842
0
0.04
0.04
0.1

Relevant Retrieved
Max Min Average
12
12
7.2
19
16
10.8
6
5
3.2
506 451
282
21
12
10.8
88
81
51.4
118
93
60.8
11
8
6
24
22
13.6
30
27
17
24
0
9.4
31
4
12.8
17
5
7.6
40
11
18.2
468 203
217.2
4
1
1.6
119
52
58
23
8
10.8
103
26
42.2
10
9
5.8
52
36
27.2
14
2
5.8
43
36
24.4
26
20
13.8
161
97
78.4
28
5
12
430 102
190
18
0
6.6
15
4
6.8
29
20
15.2
9
7
5
9
3
3.6
20
3
8.6
14
1
5.8
42
34
23

Proceedings of NTCIR-5 Workshop Meeting, December 6-9, 2005, Tokyo, Japan

Table 6. Effectiveness of Query Term
Expansion Technique for Each Topic

Topic
0001
0003
0004
0006
0019
0021
0022
0023
0028
0029
0034
0044
0045
0055
0058
0061
0062
0063
0065
0068
0070
0071
0073
0074
0076
0080
0082
0084
0086
0088
0091
0095
0097
0098
0099

No. of Run IDs
where performace improveautomaticuser
AP RP RR AP RP RR
6
3
2
4
3
0
4
3
2
4
2
1
9
7
0
3
1
0
5
3
5
3
2
3
2
1
1
2
1
0
5
4
5
1
1
1
8
7
6
5
4
5
4
3
1
2
0
1
5
4
4
5
5
5
4
3
8
5
5
5
7
5
6
3
1
1
1
1
1
2
0
0
7
6
8
3
3
3
4
5
4
4
4
2
7
5
7
1
0
1
0
0
3
2
0
1
6
6
6
2
1
1
3
4
6
2
1
2
5
5
4
4
4
4
5
2
1
3
1
0
5
6
7
4
3
4
2
0
5
1
1
2
6
7
0
4
4
1
5
4
1
4
4
1
2
4
7
0
0
0
5
2
3
3
1
2
6
4
6
2
1
1
0
1
0
4
1
1
3
2
8
2
2
2
8
6
5
4
2
2
10
2
0
5
1
2
5
1
3
3
0
3
7
6
4
5
5
5
6
4
6
4
3
4
5
3
0
3
3
2

AP: Average Precision, RP: R-Precision,
RR: Relevant Retrieved